Page MenuHomePhabricator
Create Task
Maniphest T310484

Reboot dbproxy for kernel upgrades
Closed, ResolvedPublic
Actions

Description

  • dbproxy1012.eqiad.wmnet
  • dbproxy1013.eqiad.wmnet
  • dbproxy1014.eqiad.wmnet
  • dbproxy1015.eqiad.wmnet
  • dbproxy1016.eqiad.wmnet
  • dbproxy1017.eqiad.wmnet
  • dbproxy1020.eqiad.wmnet
  • dbproxy1021.eqiad.wmnet
  • dbproxy2001.codfw.wmnet
  • dbproxy2002.codfw.wmnet
  • dbproxy2003.codfw.wmnet
  • dbproxy2004.codfw.wmnet

Details

SubjectRepoBranchLines +/-
wmnet: Failover m3 and m5 master.operations/dnsmaster+2 -2
wmnet: Failover m1 and m2operations/dnsmaster+2 -2
dbproxy2*: Disable notificationsoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 Resolved• MarosteguiT310484 Reboot dbproxy for kernel upgrades

Event Timeline

• Marostegui triaged this task as High priority.Jun 13 2022, 10:37 AM
• Marostegui created this task.
• Marostegui moved this task from Triage to Ready on the DBA board.
gerritbot added a comment.Jun 13 2022, 10:41 AM

Change 805095 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy2*: Disable notifications

https://gerrit.wikimedia.org/r/805095

gerritbot added a project: Patch-For-Review.Jun 13 2022, 10:41 AM
gerritbot added a comment.Jun 13 2022, 10:44 AM

Change 805095 merged by Marostegui:

[operations/puppet@production] dbproxy2*: Disable notifications

https://gerrit.wikimedia.org/r/805095

• Marostegui updated the task description. (Show Details)Jun 13 2022, 11:10 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 13 2022, 11:30 AM
• Marostegui added a comment.Jun 13 2022, 11:47 AM

Active proxies:

# for i in m1 m2 m3 m5; do host $i-master | grep alias ;done
m1-master.eqiad.wmnet is an alias for dbproxy1012.eqiad.wmnet.
m2-master.eqiad.wmnet is an alias for dbproxy1015.eqiad.wmnet.
m3-master.eqiad.wmnet is an alias for dbproxy1016.eqiad.wmnet.
m5-master.eqiad.wmnet is an alias for dbproxy1021.eqiad.wmnet.
• Marostegui updated the task description. (Show Details)Jun 13 2022, 11:54 AM
• Marostegui updated the task description. (Show Details)
gerritbot added a comment.Jun 13 2022, 11:58 AM

Change 805114 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Failover m1 and m2

https://gerrit.wikimedia.org/r/805114

gerritbot added a project: Patch-For-Review.Jun 13 2022, 11:58 AM
gerritbot added a comment.Jun 13 2022, 2:42 PM

Change 805114 merged by Marostegui:

[operations/dns@master] wmnet: Failover m1 and m2

https://gerrit.wikimedia.org/r/805114

Stashbot added a comment.Jun 13 2022, 2:42 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-13T14:42:53Z] <marostegui> Failover m1 and m2 to a different proxy T310484

• Marostegui added a comment.Jun 13 2022, 2:43 PM

m1 and m2 failed over.

Maintenance_bot removed a project: Patch-For-Review.Jun 13 2022, 3:30 PM
Stashbot added a comment.Jun 14 2022, 6:27 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-14T06:27:35Z] <marostegui> Reboot dbproxy1012 and dbproxy1015 T310484

• Marostegui updated the task description. (Show Details)Jun 14 2022, 6:29 AM
gerritbot added a comment.Jun 14 2022, 6:32 AM

Change 805271 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Failover m3 and m5 master.

https://gerrit.wikimedia.org/r/805271

gerritbot added a project: Patch-For-Review.Jun 14 2022, 6:32 AM
gerritbot added a comment.Jun 14 2022, 7:13 AM

Change 805271 merged by Marostegui:

[operations/dns@master] wmnet: Failover m3 and m5 master.

https://gerrit.wikimedia.org/r/805271

• Marostegui added a comment.Jun 14 2022, 7:13 AM

m3 and m5 dbproxies failed over, waiting a few hours for connections to move gracefully to the other proxies

Maintenance_bot removed a project: Patch-For-Review.Jun 14 2022, 7:30 AM
Stashbot added a comment.Jun 15 2022, 5:03 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-15T05:03:17Z] <marostegui> Reboot dbproxy1016 and dbproxy1021 T310484

• Marostegui closed this task as Resolved.Jun 15 2022, 5:06 AM
• Marostegui updated the task description. (Show Details)

All done

===== NODE GROUP =====
(12) dbproxy[2001-2004].codfw.wmnet,dbproxy[1012-1017,1020-1021].eqiad.wmnet
----- OUTPUT of 'sudo uname -v' -----
#1 SMP Debian 5.10.120-1 (2022-06-09)

dbproxy1018 and dbproxy1019 are owned by WMCS

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL