Page MenuHomePhabricator
Create Task
Maniphest T283235

Upgrade s5 to Debian Buster and MariaDB 10.4
Closed, ResolvedPublic
Actions

Description

Steps to upgrade:

Please read the doc about procedure for more details.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Marostegui claimed this task.May 20 2021, 11:12 AM
• Marostegui triaged this task as Medium priority.

This should be stalled for now and only to be done once we are happy with s6's performance/stability in around 3-4 weeks or so.

• Marostegui changed the task status from Open to Stalled.May 20 2021, 11:12 AM
jcrespo updated the task description. (Show Details)May 20 2021, 11:18 AM
• Marostegui updated the task description. (Show Details)May 20 2021, 11:18 AM
gerritbot added a comment.May 20 2021, 11:22 AM

Change 693142 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Switchover codfw s5 backups from db2099 to db2101 (buster)

https://gerrit.wikimedia.org/r/693142

gerritbot added a project: Patch-For-Review.May 20 2021, 11:22 AM
jcrespo updated the task description. (Show Details)May 20 2021, 11:23 AM
• Marostegui added a comment.May 26 2021, 10:40 AM

All s5 codfw buster hosts upgraded to 10.4.19

• Marostegui added a comment.Jun 3 2021, 9:07 AM

I am going to start working on this next week

• Marostegui changed the task status from Stalled to Open.Jun 4 2021, 6:07 AM
• Marostegui moved this task from Blocked to Ready on the DBA board.
• Marostegui added a comment.EditedJun 4 2021, 6:50 AM

Starting to upgrade eqiad hosts 10.4 hosts to the latest 10.4.19 uploaded

• Marostegui added a comment.Jun 4 2021, 7:08 AM

@jcrespo let me know if you want to upgrade db1150 to 10.4.19 or you want me to do it.

gerritbot added a comment.Jun 4 2021, 7:15 AM

Change 698151 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db2113 to Buster and 10.4

https://gerrit.wikimedia.org/r/698151

gerritbot added a comment.Jun 4 2021, 7:16 AM

Change 698151 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db2113 to Buster and 10.4

https://gerrit.wikimedia.org/r/698151

jcrespo added a comment.Jun 4 2021, 7:32 AM
In T283235#7133534, @Marostegui wrote:

@jcrespo let me know if you want to upgrade db1150 to 10.4.19 or you want me to do it.

I can do it quickly.

• Marostegui added a comment.Jun 4 2021, 7:33 AM

Cool thanks!

Stashbot added a comment.Jun 4 2021, 7:38 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T07:38:20Z] <jynus> stop and upgrade db1150 T283235

jcrespo added a comment.Jun 4 2021, 7:52 AM
In T283235#7133581, @Marostegui wrote:

Cool thanks!

MariaDB read only s4 Version 10.4.19-MariaDB, Uptime 188s, read_only: True, event_scheduler: True, 215.45 QPS, connection latency: 0.003659s, query latency: 0.000495s 	
MariaDB read only s5 Version 10.4.19-MariaDB, Uptime 154s, read_only: True, event_scheduler: True, 108.23 QPS, connection latency: 0.003970s, query latency: 0.000481s
gerritbot added a comment.Jun 4 2021, 8:09 AM

Change 698157 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Switchover eqiad s5 backups from db1145 to db1150 (buster)

https://gerrit.wikimedia.org/r/698157

jcrespo updated the task description. (Show Details)Jun 4 2021, 8:10 AM
Stashbot added a comment.Jun 4 2021, 8:33 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T08:33:39Z] <marostegui> Upgrade db1110 T283235

• Marostegui added a parent task: T266486: Schema change to turn user_last_timestamp.user_newtalk to binary(14).Jun 4 2021, 8:59 AM
• Marostegui added a parent task: T268392: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14).
• Marostegui added a parent task: T273360: Schema change for dropping default of img_timestamp and making it binary(14).
• Marostegui moved this task from Ready to In progress on the DBA board.Jun 4 2021, 9:45 AM
gerritbot added a comment.Jun 4 2021, 12:24 PM

Change 698185 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1019: Depool clouddb1015

https://gerrit.wikimedia.org/r/698185

gerritbot added a comment.Jun 4 2021, 12:25 PM

Change 698185 merged by Marostegui:

[operations/puppet@production] dbproxy1019: Depool clouddb1015

https://gerrit.wikimedia.org/r/698185

Stashbot added a comment.Jun 4 2021, 12:27 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T12:27:57Z] <marostegui> Upgrade mysql on clouddb1015 T283235

• Marostegui added a comment.Jun 4 2021, 12:30 PM

I am starting to upgrade all eqiad hosts, including clouddb* replicas

Stashbot added a comment.Jun 4 2021, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T12:46:02Z] <marostegui> Upgrade mysql on clouddb1016 T283235

gerritbot added a comment.Jun 7 2021, 5:08 AM

Change 698370 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2113: Disable notifications

https://gerrit.wikimedia.org/r/698370

gerritbot added a comment.Jun 7 2021, 5:09 AM

Change 698370 merged by Marostegui:

[operations/puppet@production] db2113: Disable notifications

https://gerrit.wikimedia.org/r/698370

ops-monitoring-bot added a comment.Jun 7 2021, 5:12 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2113.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106070512_marostegui_17172.log.

gerritbot added a comment.Jun 7 2021, 5:36 AM

Change 698371 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1018: Depool clouddb1020

https://gerrit.wikimedia.org/r/698371

gerritbot added a comment.Jun 7 2021, 5:36 AM

Change 698371 merged by Marostegui:

[operations/puppet@production] dbproxy1018: Depool clouddb1020

https://gerrit.wikimedia.org/r/698371

ops-monitoring-bot added a comment.Jun 7 2021, 5:38 AM

Completed auto-reimage of hosts:

['db2113.codfw.wmnet']

and were ALL successful.

• Marostegui updated the task description. (Show Details)Jun 7 2021, 5:38 AM
• Marostegui added a comment.Jun 7 2021, 5:42 AM

Candidate master in codfw done (db2113) - checking its tables before proceeding with the master.
@jcrespo this can probably be pushed: https://gerrit.wikimedia.org/r/693142 anytime you like.

Stashbot added a comment.Jun 7 2021, 6:05 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-07T06:05:55Z] <marostegui> Upgrade mysql on dbstore1003 T283235

LSobanski subscribed.Jun 7 2021, 12:01 PM
• Marostegui added a comment.Jun 7 2021, 2:19 PM

db2113 tables checked, all clean.

gerritbot added a comment.Jun 7 2021, 5:40 PM

Change 693142 merged by Jcrespo:

[operations/puppet@production] dbbackups: Switchover codfw s5 backups from db2099 to db2101 (buster)

https://gerrit.wikimedia.org/r/693142

gerritbot added a comment.Jun 8 2021, 4:58 AM

Change 698658 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2123: Disable notifications

https://gerrit.wikimedia.org/r/698658

gerritbot added a comment.Jun 8 2021, 4:59 AM

Change 698658 merged by Marostegui:

[operations/puppet@production] db2123: Disable notifications

https://gerrit.wikimedia.org/r/698658

ops-monitoring-bot added a comment.Jun 8 2021, 4:59 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2123.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106080459_marostegui_32562.log.

gerritbot added a comment.Jun 8 2021, 5:09 AM

Change 698659 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db2123 to buster

https://gerrit.wikimedia.org/r/698659

gerritbot added a comment.Jun 8 2021, 5:17 AM

Change 698659 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db2123 to buster

https://gerrit.wikimedia.org/r/698659

• Marostegui updated the task description. (Show Details)Jun 8 2021, 5:23 AM
ops-monitoring-bot added a comment.Jun 8 2021, 5:26 AM

Completed auto-reimage of hosts:

['db2123.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Jun 8 2021, 5:27 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2123.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106080527_marostegui_5778.log.

ops-monitoring-bot added a comment.Jun 8 2021, 5:56 AM

Completed auto-reimage of hosts:

['db2123.codfw.wmnet']

and were ALL successful.

• Marostegui added a comment.Jun 8 2021, 6:13 AM

db2123 (master on codfw) reimaged to Buster. Checking its tables now.

• Marostegui updated the task description. (Show Details)Jun 8 2021, 6:13 AM
• Marostegui added a comment.Jun 8 2021, 6:15 AM

sanitarium host (db1154) upgraded

• Marostegui updated the task description. (Show Details)Jun 8 2021, 6:29 AM
gerritbot added a comment.Jun 8 2021, 6:55 AM

Change 698715 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db1130 to Buster

https://gerrit.wikimedia.org/r/698715

gerritbot added a comment.Jun 8 2021, 6:57 AM

Change 698715 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db1130 to Buster

https://gerrit.wikimedia.org/r/698715

• Marostegui added a comment.Jun 8 2021, 2:23 PM

db2123 finished checking its tables - all clean

gerritbot added a comment.Jun 9 2021, 4:42 AM

Change 698907 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2123: Enable notifications

https://gerrit.wikimedia.org/r/698907

gerritbot added a comment.Jun 9 2021, 4:43 AM

Change 698907 merged by Marostegui:

[operations/puppet@production] db2123: Enable notifications

https://gerrit.wikimedia.org/r/698907

Stashbot added a comment.Jun 9 2021, 10:04 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-09T10:04:23Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1130 T283235', diff saved to https://phabricator.wikimedia.org/P16337 and previous config saved to /var/cache/conftool/dbconfig/20210609-100423-marostegui.json

gerritbot added a comment.Jun 9 2021, 10:05 AM

Change 698955 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1130: Disable notifications

https://gerrit.wikimedia.org/r/698955

gerritbot added a comment.Jun 9 2021, 10:06 AM

Change 698955 merged by Marostegui:

[operations/puppet@production] db1130: Disable notifications

https://gerrit.wikimedia.org/r/698955

ops-monitoring-bot added a comment.Jun 9 2021, 10:07 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1130.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106091007_marostegui_1368.log.

ops-monitoring-bot added a comment.Jun 9 2021, 10:28 AM

Completed auto-reimage of hosts:

['db1130.eqiad.wmnet']

and were ALL successful.

• Marostegui added a comment.Jun 9 2021, 10:30 AM

db1130 (candidate master) reimaged - checking tables now.

• Marostegui updated the task description. (Show Details)Jun 9 2021, 10:30 AM
• Marostegui added a comment.Jun 10 2021, 4:55 AM
In T283235#7145224, @Marostegui wrote:

db1130 (candidate master) reimaged - checking tables now.

The check came back clean.

gerritbot added a comment.Jun 10 2021, 5:10 AM

Change 698157 merged by Marostegui:

[operations/puppet@production] dbbackups: Switchover eqiad s5 backups from db1145 to db1150 (buster)

https://gerrit.wikimedia.org/r/698157

• Marostegui added a comment.EditedJun 10 2021, 5:11 AM

Merged the above patch to start generating eqiad backups from the buster host.

• Marostegui updated the task description. (Show Details)Jun 10 2021, 5:11 AM
• Marostegui changed the task status from Open to Stalled.Jun 11 2021, 4:45 AM

Waiting for the switchover to happen to be able to continue the rest of steps

gerritbot added a comment.Jun 22 2021, 5:11 AM

Change 700725 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Remove s5 (stretch) from backup sources

https://gerrit.wikimedia.org/r/700725

• Marostegui updated the task description. (Show Details)Jun 22 2021, 5:13 AM
gerritbot added a comment.Jun 22 2021, 5:13 AM

Change 700746 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db1100 to Buster.

https://gerrit.wikimedia.org/r/700746

• Marostegui changed the task status from Stalled to Open.Jun 22 2021, 5:13 AM
gerritbot added a comment.Jun 22 2021, 5:14 AM

Change 700746 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db1100 to Buster.

https://gerrit.wikimedia.org/r/700746

• Marostegui updated the task description. (Show Details)Jun 22 2021, 5:56 AM
• Marostegui closed subtask T284529: Switchover s5 from db1100 to db1130 as Resolved.Jun 22 2021, 6:01 AM
ops-monitoring-bot added a comment.Jun 22 2021, 6:03 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1100.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106220603_marostegui_5451.log.

ops-monitoring-bot added a comment.Jun 22 2021, 6:24 AM

Completed auto-reimage of hosts:

['db1100.eqiad.wmnet']

and were ALL successful.

• Marostegui added a comment.Jun 22 2021, 6:25 AM

Old master upgraded to Buster and 10.4 - running mysqlcheck now.

• Marostegui added a comment.Jun 22 2021, 2:31 PM

db1100 had all the tables checked and clean. Started replication

• Marostegui added a comment.Jun 23 2021, 4:49 AM

db1100 has caught up - going to start repooling

gerritbot added a comment.Jun 23 2021, 4:49 AM

Change 701013 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1100: Enable notifications

https://gerrit.wikimedia.org/r/701013

gerritbot added a comment.Jun 23 2021, 4:51 AM

Change 701013 merged by Marostegui:

[operations/puppet@production] db1100: Enable notifications

https://gerrit.wikimedia.org/r/701013

• Marostegui updated the task description. (Show Details)Jun 23 2021, 5:14 AM
• Marostegui added a comment.Jun 23 2021, 10:49 AM

This is all done, pending the backup clean up which has been scheduled for Monday by Jaime

gerritbot added a comment.Jun 28 2021, 8:16 AM

Change 700725 merged by Jcrespo:

[operations/puppet@production] dbbackups: Remove s5 (stretch) from backup sources

https://gerrit.wikimedia.org/r/700725

Stashbot added a comment.Jun 28 2021, 8:19 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-28T08:19:01Z] <jynus> stop and remove db1145:s5 db2099:s5 T283235

jcrespo updated the task description. (Show Details)Jun 28 2021, 8:32 AM

Cleanup of s5 old backup sources should be done, and if everything went ok, instances should be no longer on icinga, tendril, grafana, but please double check!

• Marostegui closed this task as Resolved.Jun 28 2021, 8:36 AM

Thanks! All good (also checked zarcillo)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL