20:52 taavi: added `denisse` as a member
- 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in T308278
- 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in T303147
- 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
- 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
- 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # T307620
- 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # T307620
- 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180
- 22:01 brennen: gitlab-runners: re-registering all shared runners
- 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster
- 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
- 15:50 ori: On deployment-cache-{text,upload}06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service (T310957)
- 14:07 ori: restarted acme-chief on deployment-acme-chief03
- 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
- 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack
- 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
- 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
- 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]
- 22:39 brennen: phabricator: tagged release/2022-06-15/1 (T310742)
- 16:31 hashar: integration-agent-docker-1035: docker image prune
- 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster
- 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
- 17:18 brennen: starting 1.39.0-wmf.16 (T308069) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
- 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
- 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production
- 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
- 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore
- 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for T309932
- 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
- 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for T295578
- 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for T295578 sanity check
- 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
- 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration
14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for T302711 - 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
- 11:26 hashar: Restarting Jenkins on contint2001
- 11:19 hashar: Restarting Jenkins on releases1002
- 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
- 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
- 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
- 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster
- 11:47 jelto: apply gitlab-settings to gitlab1004 - T307142
- 11:46 jelto: apply gitlab-settings to gitlab1003 - T307142
19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything T309437 - 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
- 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
- 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
- 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
- 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` (T309419)
- 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
- 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` (T309415)
- 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep
- 18:33 dancy: Updated Jenkins beta-* job configs
- 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of 2c7b5825
- 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch
- 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] T309230
- 15:46 dancy: Restarted apache2 on gerrit1001
- 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts T299797
- 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
- 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
- 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
- 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - T296982
- 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - T296982
- 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - T296982
- 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
- 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
- 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar` # T307339#7937268
- 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # T252071
23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for T308558 - 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
- 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
- 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request
- 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom T299797
- 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test T307137
- 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day and `docker system prune --force` on Sunday | https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
- 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
- 00:46 brennen: gitlab: disabling container registries on all existing projects (T307537)
- 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
- 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster
12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # T304057 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # T252071 - 15:40 brennen: train 1.39.0-wmf.9 (T305215): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
- 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for T306068
- 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for T306897
- 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
- 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
- 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204
16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service
- 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
- 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for T306329
- 16:48 brennen: phabricator: adding self to acl*blog-admins
- 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per T306054; it's apparently unused, so this shouldn't break anything.
- 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # T304461, output is at P24204
- 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki={enwiki,cswiki} --delete # T304461
- 20:03 thcipriani: rebooting phabricator
- 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for T304596
- 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for T304948
- 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*` T304932T302477
- 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771
13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting T304736 02:37 Reedy: beta-update-databases-eqiad is back to @hourly
- 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
- 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
- 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
- 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
- 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord
- 20:00 hashar: reloading Zuul for Id844e1 # T299320
- 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for T304652
- 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for T303899
- 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck
- 17:35 brennen: restarting phabricator for T304540, brief downtime expected
- 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster
- 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
- 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # T300203
- 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
- 13:22 hashar: Rolling back Quibble jobs from 1.4.4 T304147
- 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)
- 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
- 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
- 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
- 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
- 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
- 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
- 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
- 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for T267890
- 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for T267890
- 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for T267890
- 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 | T300340
- 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079
04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki
20:31 brennen: requiring 2fa for all users under /repos
10:53 zabe: restarted apache on deployment-mediawiki11 # T302699 - 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for T302908
- 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
- 15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
- 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04
- 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
- 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
- 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin
- 20:05 zabe: apply T302658 on deployment-prep centralauth databases
- 13:24 zabe: apply T302660 on deployment-prep centralauth databases
- 13:19 zabe: apply T302659 on deployment-prep centralauth databases
- 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
- 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
- 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
- 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
- 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
- 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
- 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
- 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1
- 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
- 20:10 dancy: Updating scap in beta
- 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
- 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # T252071
- 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # T252071
- 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # T284774
- 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # T112147
- 13:28 urbanecm: deployment-prep: Create database for incubatorwiki (T210492)
- 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # T302226
- 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549T225730
- 07:27 hashar: Refreshing all Jenkins jobs
10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo
- 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
- 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for T293705
- 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for T293705
- 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
- 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T301995
- 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
- 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
- 11:12 hashar: Bringing deployment-deploy03 back
- 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome
- 18:20 hashar: Tag Quibble 1.4.1 @ d4bd2801de # T300301
- 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
- 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
- 10:09 hashar: Reloading Zuul for I997fee
- 09:59 hashar: Reloading Zuul for I2ffa01
- 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
- 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
- 16:24 hashar: Restarting CI Jenkins for plugins updates
- 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
- 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
- 15:26 hashar: Reloading Zuul for If80b4b
- 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
- 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
- 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
- 14:26 jnuche: Jenkins upgrade complete T301361
- 13:54 jnuche: Jenkins contint instances are going to be restarted soon
18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)
15:22 taavi: deleted shutoff deployment-mx02
- 17:34 taavi: remove scap from deployment-kafka-main/jumbo
- 16:23 taavi: hard reboot misbehaving deployment-echostore01
- 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03
- 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
- 14:53 hashar: Building Docker images for Quibble 1.4.0 (prepared by kostajh)
- 13:51 kostajh: Tag Quibble 1.4.0 @ 4231bc2 # T300340 T291549 T225730
- 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
- 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta
- 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to 9e3ea96 for T222216
- 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # T284774
- 21:45 taavi: running recountCategories.php on all beta wikis per T299823#7652496
- 14:27 hashar: taking heapdump of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`
- 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # T299431
- 19:34 Amir1: Reloading Zuul to deploy 757464
- 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # T300214
- 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300236
- 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300214
- 00:24 thcipriani: restarting jenkins
- 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - T252071
- 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - T234259
- 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for T252071
- 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for T252071
- 16:45 hashar: integration: creating integration-agent-docker-1023 based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # T290783
- 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
- 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for T232948
- 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for T232948
- 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for T299995
- 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948 . Docker role instances now always have a 24G partition for Docker
- 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # T299996
- 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # T299996
- 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
- 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996
- 09:22 Reedy: unblocked beta again
- 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/
- 21:44 Reedy: unstick beta ci jobs
- 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
- 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
- 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker} # T299933
- 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker}
- 13:40 taavi: apply T299827 on deployment-prep centralauth database
- 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts
- 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
- 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`
- 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
- 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
- 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
- 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
- 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
- 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743 # T292729
- 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents | https://gerrit.wikimedia.org/r/755713
- 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
- 12:10 hashar: contint2001 : docker container prune && docker image prune
- 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
- 12:04 hashar: contint1001 `docker image prune`
- 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org
- 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
- 18:01 taavi: added ryankemper as a member of the deployment-prep project
- 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # T299389
- 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389
- 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
- 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0
- 16:28 hashar: Building Quibble 1.3.0 Docker images
- 16:16 hashar: Tagged Quibble 1.3.0 @ 2b2c7f9a45 # T297480 T226869 T294931
- 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro
- 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
- 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # T290783
- 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings T265615 T299211
- 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta
- 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
- 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
- 16:27 dancy: testing scap prep auto on deployment-deploy03
- 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
- 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
- 10:52 hashar: Restarting Jenkins CI for plugins update
- 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # T298691
- 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins
09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings
- 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
- 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for T222291
- 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
- 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002
Last edited on 24 June 2022, at 20:52
Wikitech
Content is available under CC BY-SA 3.0 unless otherwise noted.