I'm BDavis (WMF) on wiki, bd808 on irc & GitLab, and BryanDavis on Gerrit & Wikitech.
I've got a thing for 🦄s. Don't judge.
I work for or provide services to the Wikimedia Foundation, but this is my only Phabricator account. Edits, statements, or other contributions made from this account are my own, and may not reflect the views of the Foundation.
[02:04] wm-bb (~wm-bridge@wikimedia/bot/wm-bridgebot) left IRC (Quit: Bouncer quit) [02:04] wm-bb (~wm-bridge@wikimedia/bot/wm-bridgebot) joined the channel [02:04] ChanServ sets mode +v wm-bb [02:05] < bd808> I just cycled wm-bb connection. Double messages to telegram now or ? [02:06] < wm-bb> <bd808> omg! single message
Such doc updates! Much wow! https://wikitech.wikimedia.org/w/index.php?title=Tool%3ABridgebot&diff=2172216&oldid=2169061
This work actually got done as part of T357729: wikibugs having a hard time staying connected to libera.chat IRC network. https://gitlab.wikimedia.org/toolforge-repos/wikibugs2-znc is now being used by Bridgebot as well.
The main problem in the deploy was that there was a cut-and-paste error in the bridgebot.toml config file. Once that was spotted and fixed things came up as hoped.
The code and config are ready to try switching everything over. I don't want to do this in my evening however due to the possibility of exciting new failure modes cropping up after running for a little while.
I have seen one crash on startup in testing but it was not repeatable. It looks like it was triggered by something the irc client saw in scrollback when attaching:
[0005] DEBUG irc: (/layers/heroku_go/go_deps/cache/gitlab.wikimedia.org /toolforge-repos/bridgebot-matterbridge@v0.0.0-20240424042617-38c64944bf1d/bridge/irc/handlers.go:117: github.com/42wim/matterbridge/bridge/irc.(*Birc).handleJoinPart) handle girc.Event{Source:(*girc.Source)(0xc0001f5fb0), Tags:girc.Tags{"time":"2024-04-24T23:19:17.667Z"}, Timestamp:time.Date(2024, time.April, 24, 23, 19, 17, 667000000, time.Local), Command:"JOIN", Params:[]string{"#wikimedia-cloud"}, Sensitive:false, Echo:false} panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xb36c66]
Adding tiny shell wrappers for the Procfile to call seems to work around the issue.
$ ssh login.toolforge.org $ become bridgebot $ webservice buildservice shell --mount all -m 2G -c 1 $ /layers/heroku_go/go_target/bin/bridgebot -conf /app/etc/testing.toml [0000] INFO router: (/layers/heroku_go/go_deps/cache/gitlab.wikimedia.org/toolforge-repos/bridgebot-matterbridge@v0.0.0-20240424042617-38c64944bf1d/gateway/router.go:66: github.com/42wim/matterbridge/gateway.(*Router).Start) Parsing gateway testing-irc-telegram [0000] INFO router: (/layers/heroku_go/go_deps/cache/gitlab.wikimedia.org/toolforge-repos/bridgebot-matterbridge@v0.0.0-20240424042617-38c64944bf1d/gateway/router.go:75: github.com/42wim/matterbridge/gateway.(*Router).Start) Starting bridge: irc.testing ...
[21:07] < wm-bb> Does it work now? [21:07] < bd808> omg, it did work!
We are a few versions behind on https://github.com/heroku/buildpacks-go, but I don't see anything in the commits or CHANGELOG that looks directly Procfile related. The latest tagged release also may not be compatible with pack (https://github.com/heroku/buildpacks-go/commit/111bb19806bb838c457ef1778a30487dc50f1cb0).
I love the idea of attempting to create a more equitable resource distribution for networking in Toolforge. I'm not sure yet however how this would actually work as hoped in practice unless there was deep integration with the Kubernetes scheduler.
Adding a buildpack like https://github.com/kr/heroku-buildpack-inline to our stack might be an interesting way to implement this feature request. That solution would not readily support the use cases that use an existing buildpack, but it would provide tool maintainers with maximum flexibility with minimal overhead for building their own tools.
Sorry for the mess @taavi. Thank you for noticing the problem and reporting it.
import logging logging.basicConfig(level=logging.DEBUG) logging.captureWarnings(True)
I recently learned that matterbridge uses the https://github.com/spf13/viper library and its AutomaticEnv feature when processing the config file. This allows envvars like MATTERBRIDGE_IRC_LIBERA_BRIDGEBOT_PASSWORD to be used to set secret values at runtime. This is hoped to make conversion to a custom container image a bit simpler by removing the need for a custom interpolation system for the config file.
https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/29 looks to have fixed the unintended regression.
I can assert that webhook inputs were received by the running tool for both https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/10 and https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/37
$ sudo ls -lh /home/rluts/2fa-reset-request.txt -rw------- 1 rluts wikidev 42 Apr 22 11:57 /home/rluts/2fa-reset-request.txt $ sudo cat /home/rluts/2fa-reset-request.txt https://phabricator.wikimedia.org/T363092
$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki 'RLuts' Two-factor authentication disabled for RLuts.
The root problem here seems to be:
[step-build] 2024-04-19T08:31:50.877313171Z Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable)
For currently unexplained reasons, the DNS resolver used by the builder process resolved archive.ubuntu.com to an IPv6 address. The network in Cloud VPS & Toolforge is IPv4 only, so this in turn led to nothing working.
There aren't any open tasks in the search, so somehow this came to fruition.
Build service probably makes this investigation and implementation obsolete in that we are actively directing folks away from the shared containers and towards building their own.
GitLab EE has mechanisms for jobs to authenticate to a Hashicorp Vault service and fetch secrets. It is not completely clear from reading https://docs.gitlab.com/ee/ci/secrets/ if there is really any support for this in CE or not.
One concrete use case for this feature would be installing toolforge-*-cli packages in a buildservice managed container. This is an idea that @Anomie and I have discussed as a potential partial solution for T356377: [toolforge] simplify calling the different toolforge apis from within the containers / T321919: Figure out and document how to call the Kubernetes API as your tool user from inside a pod.
Play around with it via curl --no-buffer -H 'Accept: text/event-stream' 'https://gitlab-webhooks.toolforge.org/sse/'
I was able to create R3385 bd808-testing-T362909 via the Phabricator UI. Is it only the Conduit API that is failing? Are the failures intermittent?
This is basically a missing integration with the system that @Andrew built for T170355: Figure out process for deleting an unused tool (wikitech docs). There is also nothing currently deleting or otherwise marking GitLab projects and their associated Diffusion mirrors as abandoned or archived when the tool that created them is deleted.
The tool in question is https://toolsadmin.wikimedia.org/tools/id/api-docs. Things are definitely goofy here with there being no toolinfo record. There are several related log events in logstash, and they seem to start with this error that seems to echo the name validation failure.
Traceback (most recent call last): File "/srv/app/striker/tools/views/tool.py", line 83, in create tool = utils.create_tool(form.cleaned_data["name"], req.user) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/app/striker/tools/utils.py", line 86, in create_tool sudoers.save() File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/django/db/models/base.py", line 743, in save self.save_base(using=using, force_insert=force_insert, File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/django/db/models/base.py", line 780, in save_base updated = self._save_table( ^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldapdb/models/base.py", line 120, in _save_table connection.add_s(new_dn, new_values) File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldapdb/backends/ldap/base.py", line 317, in add_s return cursor.connection.add_s(dn, modlist) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 236, in add_s return self.add_ext_s(dn,modlist,None,None) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 1012, in add_ext_s return self._apply_method_s(SimpleLDAPObject.add_ext_s,*args,**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 972, in _apply_method_s return func(self,*args,**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 222, in add_ext_s resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 543, in result3 resp_type, resp_data, resp_msgid, decoded_resp_ctrls, retoid, retval = self.result4( ^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 553, in result4 ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 128, in _ldap_call result = func(*args,**kwargs) ^^^^^^^^^^^^^^^^^^^^ ldap.ALREADY_EXISTS: {'msgtype': 105, 'msgid': 17, 'result': 68, 'desc': 'Already exists', 'ctrls': []}
Current lag is very likely caused by T352010: Gradually drop old pagelinks columns as shown on https://wikitech.wikimedia.org/wiki/Map_of_database_maintenance which is linked to by https://replag.toolforge.org/ prominently.
As explained briefly in the irc snippet, I have a use case where Tool-bridgebot uses a 3rd party golang component (matterbridge) which expects to read configuration from a static file. I want to move this tool from a legacy hand built deployment to a modern stack of a toolforge jobs managed deployment using a toolforge build generated image. To do so I need to replace the current chmod 0600 config file in the tool's $HOME with configuration that uses toolforge envvars for transmitting secrets into the running container.
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework does this work now.
I was doing some other cleanup of pending GitLab accounts and I think I rejected the https://gitlab.wikimedia.org/safan41 account. Logging in to GitLab again will restore the account to pending status.
I was doing some other cleanup of pending GitLab accounts and I think I may have rejected the https://gitlab.wikimedia.org/sethabathaba account. Attempting to login again will restore the account to pending status.
Rendering of T1152#9715566 via https://web.libera.chat/. Note that this client does not render the strikethrough style, but it also does not show any strange formatting characters:
another closed task comment
Rendering of T1152#9715558 in my Textual client with some additional styling triggered by my irc nick being part of the the message:
test comment on closed task
I'm going to switch the formatting from grey foreground to strikethrough text style. The strikethrough will only be visible to some IRC clients, but is expected to be more intuitive as it matches the Phabricator styling for closed tasks. HexChat, IRCCloud, The Lounge, & Textual are explicitly listed at https://defs.ircdocs.horse/info/formatting as supporting the strikethrough style.
I really need to work on this again. The bot breaks a lot and @LucasWerkmeister is amazing for noticing and restarting it. This really just needs me to take a little time to understand the golang build chain used in the upstream project so I can easily iterate on a fix like the one @Platonides proposed years ago in T305487#7915998.
The open and assigned task nag script has complained that I haven't closed this task, so whatever it is closed now. Maybe someday I'll write up the reasons I added this feature to blubber. :shrug:
Getting comment, pipeline, and job events out of GitLab would be a nice addition to the data that is currently collected by gitlab-webhooks using the system hooks integration. Unfortunately these are not as easy for us to manage centrally as they must be configured at the project level in GitLab CE. Group level integration for these events is only possible with a paid subscription.
Quick mockup using jq of data likely to be used in building messages:
$ grep '"object_kind": "merge_request"' events.log | grep "bd808" | jq '{"action": .object_attributes.action, "user": .user.username, "message": .object_attributes.title, "repo": .object_attributes.target.path_with_namespace, "branch": .object_attributes.target_branch, "url": .object_attributes.url, "owner": .object_attributes.author_id, "state": .object_attributes.state }' { "action": "open", "user": "bd808", "message": "Relicense under GPL-3.0-or-later", "repo": "toolforge-repos/wikibugs2", "branch": "main", "url": "https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_request s/22", "owner": 14, "state": "opened" } { "action": "approved", "user": "bd808", "message": "Relicense under GPL-3.0-or-later", "repo": "toolforge-repos/wikibugs2", "branch": "main", "url": "https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/22", "owner": 14, "state": "opened" } { "action": "merge", "user": "bd808", "message": "Relicense under GPL-3.0-or-later", "repo": "toolforge-repos/wikibugs2", "branch": "main", "url": "https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/22", "owner": 14, "state": "merged" }
The 2014-04-08 scap triggered outage that "earned" me the shirt (and my Chaos Monkey badge):
In 2016 I started making a sticker version because 1) easier to carry places, 2) no worries about not having the correct size, 3) cheaper.
So bd808 made one: http://oi61.tinypic.com/2ryj5kz.jpg
tinypic.com has been lost to the sands of time. That photo was:
The photo of the just arrived shirts that I posted to twitter on 2014-09-15:
test
tools.wikibugs-testing@tools-bastion-12:~$ toolforge build start --ref work/bd808/expect-phorge-api-errors https://gitlab.wikimedia.org/toolforge-repos/wikibugs2 Waiting for the logs... if the build just started this might take a minute [place-tools] 2024-04-11T17:49:59.348045941Z 2024/04/11 17:49:59 Copied /ko-app/entrypoint to /tekton/bin/entrypoint [place-tools] failed to create fsnotify watcher: too many open files [step-init] 2024-04-11T17:49:59.913177243Z 2024/04/11 17:49:59 Setup /step directories [step-init] failed to create fsnotify watcher: too many open files [place-scripts] 2024-04-11T17:50:01.125871795Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-0-vmtp5 [place-scripts] 2024-04-11T17:50:01.150840971Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-2-wldbl [place-scripts] 2024-04-11T17:50:01.187348966Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-3-l5vh7 [place-scripts] 2024-04-11T17:50:01.214228628Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-9-s6s4t [place-scripts] 2024-04-11T17:50:01.237079271Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-10-6th52 [place-scripts] 2024-04-11T17:50:01.270148540Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-11-zcrvx [place-scripts] 2024-04-11T17:50:01.304439329Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-12-4gvkt [place-scripts] failed to create fsnotify watcher: too many open files [step-clone] failed to create fsnotify watcher: too many open files [step-copy-builder-to-tmp] 2024-04-11T17:50:02.407982038Z 2024/04/11 17:50:02 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-copy-builder-to-tmp] failed to create fsnotify watcher: too many open files [step-prepare] failed to create fsnotify watcher: too many open files [step-inject-buildpacks] failed to create fsnotify watcher: too many open files [step-copy-stack-toml] 2024-04-11T17:50:03.342859273Z 2024/04/11 17:50:03 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-copy-stack-toml] failed to create fsnotify watcher: too many open files [step-detect] failed to create fsnotify watcher: too many open files [step-analyze] 2024-04-11T17:50:04.254967516Z 2024/04/11 17:50:04 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-analyze] failed to create fsnotify watcher: too many open files [step-restore] 2024-04-11T17:50:04.688674016Z 2024/04/11 17:50:04 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-restore] failed to create fsnotify watcher: too many open files [step-build] failed to create fsnotify watcher: too many open files [step-fix-nested-procfile-launcher] failed to create fsnotify watcher: too many open files [step-fix-permissions] failed to create fsnotify watcher: too many open files [step-export] 2024-04-11T17:50:06.064284001Z 2024/04/11 17:50:06 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-export] failed to create fsnotify watcher: too many open files [step-results] failed to create fsnotify watcher: too many open files
Retry worked.
I thought that was part of {T316419}, but apparently I was mistaken. +1 from me.