Page MenuHomePhabricator
Create Task
Maniphest T245757

Upgrade MediaWiki clusters to Debian Buster (debian 10)
Closed, ResolvedPublic
Actions

Description

Our 4 MediaWiki clusters, application, api, jobrunners/videoscalers, parsoid need to be migrated to Debian Buster.

progress: see https://docs.google.com/spreadsheets/d/1Ris18-joRFfd3OHjGJIraVUk-bpmIRORsPoms9D7BcM/edit?usp=sharing

Provisional plan for the migration:

  • Upgrade all current stretch servers to ICU 63 T264991
  • Rebuild all our php-7.2 packages for Debian Buster (buster-wikimedia)
    • php7.2-cli
    • php7.2-common
    • php7.2-curl
    • php7.2-dba
    • php7.2-fpm
    • php7.2-gd
    • php7.2-gmp
    • php7.2-mysql
    • php7.2-opcache
    • php7.2-phpdbg
    • php7.2-readline
    • php7.2-xml
  • Build missing packages for Buster
    • ploticus
    • prometheus-nutcracker-exporter
    • prometheus-php-fpm-exporter
  • Fix puppet code to support Buster
    • ttf-alee replaced with fonts-alee
    • ttf-wqy-zenhei replaced with fonts-wqy-zenhei
    • code to add PHP72 component on buster
  • Reimage mwdebug1001 to buster OR introduce mwdebug1003, so not to mess with development testing
    • first iteration done with testvm1001, decom'ed again
    • mwdebug1003 to be introduced early December (T267248)
    • add PHP72 APT component on mwdebug1003
    • Reimage parse2001 to buster (parsoid)
    • Reimage mw2243 to buster (jobrunner)
  • Reimage mw1265 to Buster (weight=5)

Q3

Details

SubjectRepoBranchLines +/-
switch mwmaint.discovery.wmnet from codfw to eqiadoperations/dnsmaster+2 -2
DHCP: switch mwmaint2002 from stretch to buster installeroperations/puppetproduction+0 -1
DHCP: switch mw1307 to use buster installeroperations/puppetproduction+0 -1
install_server: switch parsoid servers to busteroperations/puppetproduction+0 -43
tcpircbot: allow deploy1002/2002, do not allow deploy1001/2001operations/puppetproduction+3 -4
mcrouter: move mcrouter proxy for C6 from mw1320 to mw1321operations/puppetproduction+1 -1
mcrouter: move mcrouter proxy for B6 from mw1287 to mw1288operations/puppetproduction+1 -1
mcrouter: move mcrouter proxy for codfw C3 to mw2337operations/puppetproduction+1 -1
mcrouter: move mcrouter proxy for A7 from mw1270 to mw1271operations/puppetproduction+1 -1
mcrouter: move mcrouter proxy for D6 from mw1367 to mw1368operations/puppetproduction+1 -1
mcrouter: move mcrouter proxy for codfw B3 to mw2258operations/puppetproduction+1 -1
DHCP: switch remaining eqiad appservers to use buster installeroperations/puppetproduction+0 -24
DHCP: switch all eqiad appservers to use buster installeroperations/puppetproduction+0 -121
DHCP: switch all codfw appservers from stretch to buster installeroperations/puppetproduction+0 -150
DHCP: switch mw1266,mw1267,mw1276,mw1277 to use buster installeroperations/puppetproduction+0 -4
mw maintenance: Install php-readline from component/php72operations/puppetproduction+9 -2
Install php-readline from component/php72operations/puppetproduction+7 -1
install_server: Reimage mw1265 to buster.operations/puppetproduction+0 -2
DHCP: switch mw2243 to buster installeroperations/puppetproduction+0 -1
profile::mediawiki::videoscaler: Add Support for Busteroperations/puppetproduction+2 -0
Rebuild for Buster: - Use golang-github-prometheus-client-golang-dev and add adapt-to-prometheus090.patch - debhelper 12 - Disable dh_dwzoperations/debs/prometheus-php-fpm-exportermaster+14 -2
Also apply the PHP 7.2 component for Busteroperations/puppetproduction+21 -28
mediawiki::php: allow opting-in to use the PHP72 component on busteroperations/puppetproduction+5 -1
Avoid transitional package ttf-wqy-zenhei in favour of fonts-wqy-zenheioperations/puppetproduction+2 -2
mediawiki: Use python-pil on busteroperations/puppetproduction+8 -1
mediawiki::php: fix hardcoded stretch dist name for PHP 72 packagesoperations/puppetproduction+2 -1
site: introduce mwdebug1003 as debug server on busteroperations/puppetproduction+5 -0
decom testvm1001.eqiad.wmnetoperations/puppetproduction+0 -8
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT248925 Make MediaWiki release tarball compatible with PHP 8.0
 ResolvedJdforrester-WMFT300463 Make PHP 8.0 voting on MW master
 ResolvedJdforrester-WMFT313563 Bump lcobucci/jwt & league/uri-components for php8
 ResolvedJdforrester-WMFT313564 Bump onoi/message-reporter in vendor.git to 1.4.2 for php 8 support
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 Resolved• toanT243590 Fix WikibaseDataModel CI for php 7.4
 ResolvedLucas_Werkmeister_WMDET316923 Restore skipped test in ReferenceListTest.php
 ResolvedJoeT318918 Undeploy patch to use old PHP serialization in PHP 7.4
 ResolvedJdforrester-WMFT264168 Drop PHP 7.2 support from Wikibase master branch, once Wikimedia production is on 7.4
 ResolvedLadsgroupT270740 Drop hacky support for doctrine/dbal class renaming
 InvalidNoneT303505 [S] Remove Deprecated methods "serialize" and "unserialize" after php production upgrade to PHP 7.4
 ResolvedReedyT251043 Cleanup css-sanitizer when it only requires PHP >= 7.4
 OpenNoneT166010 The Great Namespaceization Effort
 Resolved• tstarlingT277618 var_dump() on various objects writes gigabytes of data and takes minutes to run
 ResolvedJdforrester-WMFT261872 Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedPRODUCTION ERRORLegoktmT293568 PHP Notice: Undefined offset in wikimedia/remex-html when rendering rest.php error page
 Resolved• tstarlingT297667 mysqli/mysqlnd memory leak
 ResolvedJoeT271736 Migrate WMF production from PHP 7.2 to PHP 7.4
 ResolvedKrinkleT248191 Can't reopen table in wikidb-unittest_ (from SpecialPageFatalTest)
 ResolvedhasharT278203 Migrate all CI jobs from stretch to buster or later and drop stretch testing support
 ResolvedJdforrester-WMFT252432 Drop MediaWiki testing in stretch and instead test only in buster
 ResolvedDzahnT245757 Upgrade MediaWiki clusters to Debian Buster (debian 10)
 ResolvedhasharT252434 Test MW code in buster as well as stretch
 ResolvedJdforrester-WMFT250514 Create buster-based images for quibble
 ResolvedLadsgroupT279068 Wikibase data-client bridge selenium failure on buster but not stretch
 ResolvedMoritzMuehlenhoffT250515 Please provide our special component/php72 in buster-wikimedia
 ResolvedjijikiT264991 Upgrade the MediaWiki servers to ICU 63
 ResolvedMoritzMuehlenhoffT253377 WMF deployed EasyTimeline extension depends on Ploticus package which is not available in Debian Buster (but available again in Debian Bullseye)
 ResolvedTrizek-WMFT267145 CommRel support for ICU 63 upgrade
 ResolvedDzahnT267248 create mwdebug1003 - ganeti VM with buster and appserver role
 Resolved• GillesT268188 Release WikimediaDebug for mwdebug1003 addition
 ResolvedDzahnT267607 upgrade mwmaint servers to buster
ResolvedRequestPapaulT275928 decommission mwmaint2001.codfw.wmnet
 ResolvedjijikiT268524 Upgrade Parsoid servers to buster
DeclinedNoneT245888 Rename wtp* servers to parse* (Parsoid PHP servers)
 ResolvedDzahnT268248 upgrade scandium to buster
 ResolvedDzahnT265963 Replace production deployment servers and update them to Buster
Unknown Object (Task)
 ResolvedPapaulT266363 (Need By: TBD) rack/setup/install deploy2002
 Resolved• CmjohnsonT265653 (Need By: TBD) setup/install deploy1002
 ResolvedRequest• CmjohnsonT275831 decommission deploy1001.eqiad.wmnet
 ResolvedRequestPapaulT275832 decommission deploy2001.codfw.wmnet
 ResolvedAndrewT269004 Upgrade labweb servers to buster
 ResolvedArielGlennT269377 Upgrade snapshot hosts to Buster
 ResolvedDzahnT270517 Investigate opcache hit rate on Buster appserver
 ResolvedLegoktmT273312 Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade
 ResolvedPapaulT273803 mw2220 - broken IPMI / mgmt
ResolvedDzahnT274023 Convert mwdebug VMs to debian buster
 Declined• GillesT274026 Add 3 new canaries to WikimediaDebug
 ResolvedVolansT274689 sre.hosts.decomission -> generate_dns_snippets - > Cumin execution failed
 ResolvedDzahnT274403 mw1379 - down after reboot attempt and DRAC can't powercycle
 ResolvedLegoktmT275752 Jobrunner timeouts on cross-DC file uploads because of HTTP/2

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Feb 19 2021, 11:48 PM

Completed auto-reimage of hosts:

['mw1317.eqiad.wmnet']

and were ALL successful.

ArielGlenn closed subtask T269377: Upgrade snapshot hosts to Buster as Resolved.Feb 21 2021, 10:40 AM
ops-monitoring-bot added a comment.Feb 22 2021, 6:51 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1316.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102221850_dzahn_17022_mw1316_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 22 2021, 6:55 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1315.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102221855_dzahn_21586_mw1315_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 22 2021, 7:47 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1349.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102221947_dzahn_7004_mw1349_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 22 2021, 8:03 PM

Completed auto-reimage of hosts:

['mw1316.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 22 2021, 8:07 PM

Completed auto-reimage of hosts:

['mw1315.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 22 2021, 8:16 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1314.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102222015_dzahn_4645_mw1314_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 22 2021, 8:20 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1312.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102222019_dzahn_8398_mw1312_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 22 2021, 8:25 PM

Completed auto-reimage of hosts:

['mw1349.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 22 2021, 8:29 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1279.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102222029_dzahn_17792_mw1279_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 22 2021, 9:30 PM

Completed auto-reimage of hosts:

['mw1314.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 22 2021, 9:33 PM

Completed auto-reimage of hosts:

['mw1312.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 22 2021, 9:42 PM

Completed auto-reimage of hosts:

['mw1279.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 22 2021, 10:21 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1286.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102222221_dzahn_30877_mw1286_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 22 2021, 10:24 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1410.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102222223_dzahn_620_mw1410_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 22 2021, 10:25 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1412.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102222224_dzahn_1715_mw1412_eqiad_wmnet.log.

Dzahn updated the task description. (Show Details)Feb 22 2021, 10:36 PM
jijiki added a comment.Feb 22 2021, 10:52 PM

@MoritzMuehlenhoff do you think it makes sense to keep 1 api and 1 app in stretch a bit longer as to keep comparing performance? IIRC there might be some upcoming perf optimisations on the mediawiki side.

ops-monitoring-bot added a comment.Feb 22 2021, 11:00 PM

Completed auto-reimage of hosts:

['mw1410.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 22 2021, 11:01 PM

Completed auto-reimage of hosts:

['mw1412.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 22 2021, 11:34 PM

Completed auto-reimage of hosts:

['mw1286.eqiad.wmnet']

and were ALL successful.

Urbanecm mentioned this in T275752: Jobrunner timeouts on cross-DC file uploads because of HTTP/2.Mar 1 2021, 9:54 AM
Legoktm added a subtask: T275752: Jobrunner timeouts on cross-DC file uploads because of HTTP/2.Mar 1 2021, 4:12 PM
Dzahn closed subtask T265963: Replace production deployment servers and update them to Buster as Resolved.Mar 2 2021, 1:12 AM
Dzahn added a subtask: T275831: decommission deploy1001.eqiad.wmnet.Mar 2 2021, 11:43 PM
gerritbot added a comment.Mar 2 2021, 11:45 PM

Change 635108 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] tcpircbot: allow deploy1002/2002, do not allow deploy1001/2001

https://gerrit.wikimedia.org/r/635108

gerritbot added a project: Patch-For-Review.Mar 2 2021, 11:45 PM
gerritbot added a comment.Mar 2 2021, 11:49 PM

Change 635108 merged by Dzahn:
[operations/puppet@production] tcpircbot: allow deploy1002/2002, do not allow deploy1001/2001

https://gerrit.wikimedia.org/r/635108

Dzahn updated the task description. (Show Details)Mar 5 2021, 8:58 PM
• Cmjohnson closed subtask T275831: decommission deploy1001.eqiad.wmnet as Resolved.Mar 11 2021, 3:32 PM
Dzahn closed subtask T268248: upgrade scandium to buster as Resolved.Mar 19 2021, 9:14 PM
Dzahn changed the status of subtask T267607: upgrade mwmaint servers to buster from Open to Stalled.Mar 19 2021, 9:17 PM
taavi mentioned this in T278664: Upgrade deployment-prep MediaWiki clusters to Buster.Mar 29 2021, 5:34 AM
hashar closed subtask T250515: Please provide our special component/php72 in buster-wikimedia as Resolved.Mar 29 2021, 12:38 PM
ops-monitoring-bot added a comment.Mar 29 2021, 1:05 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['parse2001.codfw.wmnet', 'parse2002.codfw.wmnet', 'parse2003.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202103291305_jiji_19021.log.

gerritbot added a comment.Mar 29 2021, 1:11 PM

Change 675506 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] install_server: switch parsoid servers to buster

https://gerrit.wikimedia.org/r/675506

gerritbot added a comment.Mar 29 2021, 1:19 PM

Change 675506 merged by Effie Mouzeli:
[operations/puppet@production] install_server: switch parsoid servers to buster

https://gerrit.wikimedia.org/r/675506

jijiki added a comment.Mar 29 2021, 3:26 PM

I have reimaged parse2001 as a test, and it appears that puppet is unable to run successfully because:

Error: Execution of '/usr/bin/scap deploy-local --repo parsoid/deploy -D log_json:False' returned 70: 15:19:26 Fetch from: http://deploy1001.eqiad.wmnet/parsoid/deploy/.git
15:19:26 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 347, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 147, in main
    getattr(self, stage)()
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 291, in fetch
    git.fetch(self.context.cache_dir, git_remote)
  File "/usr/lib/python2.7/dist-packages/scap/git.py", line 374, in fetch
    git.clone(*cmd)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 1428, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 775, in __init__
    self.wait()
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 793, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 816, in handle_command_exit_code
    raise exc
ErrorReturnCode_128:

  RAN: /usr/bin/git clone --jobs 46 http://deploy1001.eqiad.wmnet/parsoid/deploy/.git /srv/deployment/parsoid/deploy-cache/cache

  STDOUT:


  STDERR:
Cloning into '/srv/deployment/parsoid/deploy-cache/cache'...
fatal: unable to access 'http://deploy1001.eqiad.wmnet/parsoid/deploy/.git/': Could not resolve host: deploy1001.eqiad.wmnet

15:19:26 deploy-local failed: <ErrorReturnCode_128>

  RAN: /usr/bin/git clone --jobs 46 http://deploy1001.eqiad.wmnet/parsoid/deploy/.git /srv/deployment/parsoid/deploy-cache/cache

  STDOUT:


  STDERR:
Cloning into '/srv/deployment/parsoid/deploy-cache/cache'...
fatal: unable to access 'http://deploy1001.eqiad.wmnet/parsoid/deploy/.git/': Could not resolve host: deploy1001.eqiad.wmnet


Error: /Stage[main]/Parsoid/Service::Node[parsoid]/Scap::Target[parsoid/deploy]/Package[parsoid/deploy]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/scap deploy-local --repo parsoid/deploy -D log_json:False' returned 70: 15:19:26 Fetch from: http://deploy1001.eqiad.wmnet/parsoid/deploy/.git
15:19:26 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 347, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 147, in main
    getattr(self, stage)()
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 291, in fetch
    git.fetch(self.context.cache_dir, git_remote)
  File "/usr/lib/python2.7/dist-packages/scap/git.py", line 374, in fetch
    git.clone(*cmd)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 1428, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 775, in __init__
    self.wait()
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 793, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 816, in handle_command_exit_code
    raise exc
ErrorReturnCode_128:

  RAN: /usr/bin/git clone --jobs 46 http://deploy1001.eqiad.wmnet/parsoid/deploy/.git /srv/deployment/parsoid/deploy-cache/cache

  STDOUT:


  STDERR:
Cloning into '/srv/deployment/parsoid/deploy-cache/cache'...
fatal: unable to access 'http://deploy1001.eqiad.wmnet/parsoid/deploy/.git/': Could not resolve host: deploy1001.eqiad.wmnet

15:19:26 deploy-local failed: <ErrorReturnCode_128>

  RAN: /usr/bin/git clone --jobs 46 http://deploy1001.eqiad.wmnet/parsoid/deploy/.git /srv/deployment/parsoid/deploy-cache/cache

  STDOUT:


  STDERR:
Cloning into '/srv/deployment/parsoid/deploy-cache/cache'...
fatal: unable to access 'http://deploy1001.eqiad.wmnet/parsoid/deploy/.git/': Could not resolve host: deploy1001.eqiad.wmnet


Notice: /Stage[main]/Parsoid/Service::Node[parsoid]/Base::Service_unit[parsoid]/File[/lib/systemd/system/parsoid.service]: Dependency Package[parsoid/deploy] has failures: true
Warning: /Stage[main]/Parsoid/Service::Node[parsoid]/Base::Service_unit[parsoid]/File[/lib/systemd/system/parsoid.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Parsoid/Service::Node[parsoid]/Base::Service_unit[parsoid]/Exec[systemd reload for parsoid]: Skipping because of failed dependencies
Warning: /Stage[main]/Parsoid/Service::Node[parsoid]/Base::Service_unit[parsoid]/Service[parsoid]: Skipping because of failed dependencies
Notice: Applied catalog in 22.96 seconds
jijiki mentioned this in T268524: Upgrade Parsoid servers to buster .Mar 30 2021, 4:50 PM
Dzahn added a comment.Mar 30 2021, 6:56 PM
In T245757#6953720, @jijiki wrote:

I have reimaged parse2001 as a test, and it appears that puppet is unable to run successfully because:

Error: Execution of '/usr/bin/scap deploy-local --repo parsoid/deploy -D log_json:False' returned 70: 15:19:26 Fetch from: http://deploy1001.eqiad.wmnet/parsoid/deploy/.git

@jijiki This is where the deploy1001 appears in:

deployment/parsoid/deploy-cache/.config:git_server: deploy1001.eqiad.wmnet

editing that file should fix it.

Other options from the past appear to include: "run scap with --refresh-config, delete cached .config file".

For more background also see T197470 , T197470#4414254 , T162814, T196663#4265139 afaict

ops-monitoring-bot added a comment.Apr 8 2021, 1:20 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['parse2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104081320_jiji_21421.log.

ops-monitoring-bot added a comment.Apr 8 2021, 2:00 PM

Completed auto-reimage of hosts:

['parse2001.codfw.wmnet']

and were ALL successful.

jijiki closed subtask T268524: Upgrade Parsoid servers to buster as Resolved.Apr 15 2021, 3:42 PM
Jdforrester-WMF closed subtask T252434: Test MW code in buster as well as stretch as Resolved.Apr 16 2021, 4:56 PM
Dzahn updated the task description. (Show Details)Apr 16 2021, 11:34 PM
gerritbot added a comment.Apr 16 2021, 11:36 PM

Change 680483 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] DHCP: switch mw1307 to use buster installer

https://gerrit.wikimedia.org/r/680483

ops-monitoring-bot added a comment.Apr 16 2021, 11:39 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1402.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202104162338_dzahn_27978_mw1402_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Apr 16 2021, 11:39 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1403.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202104162338_dzahn_28020_mw1403_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Apr 16 2021, 11:40 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1307.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202104162340_dzahn_28210_mw1307_eqiad_wmnet.log.

gerritbot added a comment.Apr 16 2021, 11:41 PM

Change 680483 merged by Dzahn:

[operations/puppet@production] DHCP: switch mw1307 to use buster installer

https://gerrit.wikimedia.org/r/680483

Dzahn updated the task description. (Show Details)Apr 16 2021, 11:52 PM

Remaining 3 special cases kept on stretch now reimaged to buster as well.

Decom'ed mwdebug1003 VM.

Everything here is completely donenow... except mwmaint1002. Which will happen during the DC switchover.

ops-monitoring-bot added a comment.Apr 17 2021, 12:16 AM

Completed auto-reimage of hosts:

['mw1403.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 17 2021, 12:16 AM

Completed auto-reimage of hosts:

['mw1402.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 17 2021, 12:49 AM

Completed auto-reimage of hosts:

['mw1307.eqiad.wmnet']

and were ALL successful.

thcipriani removed a project: Release-Engineering-Team (Deployment services).Apr 20 2021, 1:10 AM
thcipriani edited projects, added Release-Engineering-Team (Radar); removed Release-Engineering-Team-TODO.Apr 20 2021, 3:33 AM
thcipriani moved this task from Limbo to Watching/External on the Release-Engineering-Team (Radar) board.Apr 20 2021, 3:34 AM
Addshore subscribed.May 12 2021, 2:45 PM
jijiki removed a project: User-jijiki.May 19 2021, 10:54 AM
Izno added a subtask: T279059: Remove parsoidJS leftovers from production.Jun 1 2021, 11:11 PM
Izno added a subtask: T275826: L10n cache files building up on backup deploy hosts.
Izno added a subtask: T257317: scap deploy --init on deployment server fails on first puppet run.
Izno added a parent task: T248191: Can't reopen table in wikidb-unittest_ (from SpecialPageFatalTest).Jun 1 2021, 11:15 PM
Izno added a parent task: T278203: Migrate all CI jobs from stretch to buster or later and drop stretch testing support.
Izno removed a parent task: T252432: Drop MediaWiki testing in stretch and instead test only in buster.Jun 1 2021, 11:17 PM
Jdforrester-WMF added a parent task: T252432: Drop MediaWiki testing in stretch and instead test only in buster.Jun 3 2021, 12:12 AM
hashar removed a subtask: T257317: scap deploy --init on deployment server fails on first puppet run.Jun 7 2021, 2:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 7 2021, 2:10 PM
Dzahn changed the status of subtask T267607: upgrade mwmaint servers to buster from Stalled to Open.Jul 13 2021, 11:11 AM
Dzahn claimed this task.Jul 14 2021, 2:56 PM

mwmaint1002 done

Michael subscribed.Aug 2 2021, 8:50 AM
Dzahn changed the task status from Open to Stalled.Aug 5 2021, 1:37 PM

this is only open due to a single remaining server, the mwmaint servers in codfw. this will be upgraded after we switch DCs back on September 13th

Dzahn changed the status of subtask T267607: upgrade mwmaint servers to buster from Open to Stalled.Aug 10 2021, 11:12 AM
gerritbot added a comment.Sep 15 2021, 5:16 PM

Change 721358 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] DHCP: switch mwmaint2002 from stretch to buster installer

https://gerrit.wikimedia.org/r/721358

gerritbot added a project: Patch-For-Review.Sep 15 2021, 5:16 PM
gerritbot added a comment.Sep 16 2021, 1:57 PM

Change 721546 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] switch mwmaint.discovery.wmnet from codfw to eqiad

https://gerrit.wikimedia.org/r/721546

gerritbot added a comment.Sep 16 2021, 2:09 PM

Change 721546 merged by Dzahn:

[operations/dns@master] switch mwmaint.discovery.wmnet from codfw to eqiad

https://gerrit.wikimedia.org/r/721546

gerritbot added a comment.Sep 16 2021, 2:15 PM

Change 721358 merged by Dzahn:

[operations/puppet@production] DHCP: switch mwmaint2002 from stretch to buster installer

https://gerrit.wikimedia.org/r/721358

Stashbot mentioned this in T267607: upgrade mwmaint servers to buster .Sep 16 2021, 2:35 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-16T14:35:07Z] <mutante> reimaging mwmaint2002 to buster (T267607, T245757)

Dzahn changed the task status from Stalled to In Progress.Sep 16 2021, 2:38 PM
Dzahn changed the status of subtask T267607: upgrade mwmaint servers to buster from Stalled to In Progress.
Dzahn closed subtask T267607: upgrade mwmaint servers to buster as Resolved.Sep 16 2021, 3:19 PM
Dzahn updated the task description. (Show Details)
Dzahn closed this task as Resolved.Sep 16 2021, 3:22 PM

https://noc.wikimedia.org (mwmaint.discovery.wmnet) has been switched from codfw to eqiad.

mwmaint2002 has been upgraded to buster. monitoring all green.

This was the last open check box and completes the task.

fgiunchedi reopened this task as Open.Sep 24 2021, 8:33 AM
fgiunchedi subscribed.

I don't think this is resolved, see T275752 for jobrunner on buster slowness in upload

Legoktm closed this task as Resolved.Nov 4 2021, 7:10 PM
Legoktm closed subtask T275752: Jobrunner timeouts on cross-DC file uploads because of HTTP/2 as Resolved.Nov 8 2021, 6:20 PM
Jdforrester-WMF removed a subtask: T273334: Re-imaged mw app servers can end up with missing l10n cache for old versions of MW needed for rollback.Nov 30 2021, 8:20 PM
Jdforrester-WMF removed a subtask: T275826: L10n cache files building up on backup deploy hosts.
Jdforrester-WMF removed a subtask: T279059: Remove parsoidJS leftovers from production.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL