News/Wiki Replica c1 and c3 shutdown
< News
Tracked in Phabricator
Task T142807RESOLVED
The labsdb1001.eqiad.wmnet (aka c1.labsdb) and labsdb1003.eqiad.wmnet (aka c3.labsdb) servers were shutdown and permanently removed from service. This page contains historical information about the plan that was followed prior to their shutdown.
TL;DR

What is changing?
Monday 2017-10-30, 14:30 UTC
Done
Tuesday 07 2017-11-07, 14:30 UTC
Not done
Wednesday 2017-12-13
Done
Thursday 2017-12-14
Done
Wednesday 2018-01-17
Why are we doing this?
There are two clusters of physical servers which provide the Wiki Replicas to Toolforge and other Cloud VPS users. The older of the two clusters consists of the physical hosts labsdb1001.eqiad.wmnet and labsdb1003.eqiad.wmnet​. These hosts are also known by the aliases c1.labsdb and c3.labsdb in documentation and application code. These two hosts are among the oldest hardware still operating in our production server farm. The hardware is out of warranty and old enough that we are concerned that it could fail catastrophically at any time.
As announced on 2017-09-25, a new cluster of physical servers is ready to replace the older hardware. This change comes along with a breaking feature change however. These new servers will not allow users to create their own databases/tables co-located with the replicated content. This was a feature of the older database servers that some tools used to improve performance by making intermediate tables that could then be JOINed to other tables to produce certain results.
We looked for solutions that would allow us to replicate user created data across the three servers, but we could not come up with a solution that would guarantee success. The user created tables on the current servers are not backed up or replicated and have always carried the disclaimer that these tables may disappear at any time. With the improvements in our ability to fail over and rebalance traffic under load, it is more likely on the new cluster that these tables would randomly appear and disappear from the point of view of a given user. This kind of disruption will break tools if we allow it. It seems a safer solution for everyone to disallow the former functionality.
User created databases and tables are still supported on the tools.db.svc.eqiad.wmflabs server (also known as tools.labsdb). If you are using tables co-located on the current c1.labsdb or c3.labsdb hosts we are recommending that your tool/scripts be updated to instead keep all user managed data on tools.db.svc.eqiad.wmflabs and perform any joining of replica data and user created data in application space rather than with cross-database joins.
Last edited on 7 July 2020, at 11:12
Wikitech
Content is available under CC BY-SA 3.0 unless otherwise noted.
Privacy policy
Terms of Use
Desktop
 Home Random Log in  Settings  Donate  About Wikitech  Disclaimers
WatchEdit