Page MenuHomePhabricator

June 2021 Datacenter switchover
Closed, ResolvedPublic

Description

This is the meta task for the June 2021 Datacenter switchover (eqiad -> codfw).

Schedule:

Services: Monday, June 28th, 2021 14:00 UTC
Traffic: Monday, June 28th, 2021 15:00 UTC
MediaWiki: Tuesday, June 29th, 2021 14:00 UTC

Switching back: TBD, but at least 1 month later

See also: https://wikitech.wikimedia.org/wiki/Switch_Datacenter - section Schedule

Related Objects

StatusSubtypeAssignedTask
Resolved Marostegui
DeclinedNone
Resolved Marostegui
ResolvedJclark-ctr
Resolved Marostegui
Resolved Marostegui
ResolvedRequestwiki_willy
ResolvedLegoktm
Resolvedsgrabarczuk
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
ResolvedAndrew
Resolved Marostegui
ResolvedAndrew
DeclinedAndrew
ResolvedAndrew
ResolvedAndrew
Resolved Ladsgroup
DuplicateNone
ResolvedBstorm
Resolved Marostegui
ResolvedBTullis
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
ResolvedKormat
Resolved Marostegui
ResolvedTrizek-WMF
ResolvedKormat
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
Resolvedsgrabarczuk
Resolved Marostegui
Resolvedsgrabarczuk
Resolved Cmjohnson
Resolved Marostegui
Resolved Marostegui
Resolvedsgrabarczuk
ResolvedRequest Cmjohnson
Resolved Marostegui
ResolvedRequestwiki_willy
ResolvedRequest Cmjohnson
ResolvedRequest Cmjohnson
ResolvedRequest Cmjohnson
ResolvedRequest Cmjohnson
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
ResolvedKormat
ResolvedKormat
ResolvedTrizek-WMF
Resolved Marostegui
Resolved Marostegui
Resolvedsgrabarczuk
Resolved Marostegui
ResolvedKormat
Resolved Marostegui
Resolved Marostegui
ResolvedKormat
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui
Resolved Marostegui

Event Timeline

Dzahn triaged this task as Medium priority.May 3 2021, 7:04 PM
Aklapper added a parent task: Restricted Task.May 17 2021, 9:43 AM

Change 701610 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] admin_state: depool eqiad for datacenter switchover (June 2021)

https://gerrit.wikimedia.org/r/701610

Change 701610 merged by Ssingh:

[operations/dns@master] admin_state: depool eqiad for datacenter switchover (June 2021)

https://gerrit.wikimedia.org/r/701610

Mentioned in SAL (#wikimedia-operations) [2021-06-28T18:40:34Z] <ebernhardson@deploy1002> Synchronized wmf-config/: T281515: Prepare Cirrus more_like for dc switchover (duration: 01m 02s)

I did a successful run through of the live-test mode just now, where we "switch" from codfw -> eqiad. The only issue I ran into is T285519#7182377, which I live-hacked a fix for.

Change 702128 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Change masters cnames

https://gerrit.wikimedia.org/r/702128

Change 702128 merged by Marostegui:

[operations/dns@master] wmnet: Change masters cnames

https://gerrit.wikimedia.org/r/702128

The switchover is mostly complete now, we were read only from 2021-06-29 14:21:26.671853 to 2021-06-29 14:23:23.504447, or 1m57s.

The raw notes of the issues we encountered are at https://etherpad.wikimedia.org/p/2021-switchdc-notes, later today I'll distill those into actionable Phabricator tasks and write up a report for how it went and what should be improved.

Legoktm claimed this task.

A recap blog post was published a few days ago: https://techblog.wikimedia.org/2021/07/23/june-2021-data-center-switchover/

T287539: September 2021 Datacenter switchover (codfw -> eqiad) tracks switching back to eqiad in September, closing this as resolved accordingly.