Switch Datacenter/DeploymentServer
This page describes the procedure to switch over the master deployment server from one host to another.
Procedure
- Disable puppet on deployment servers:
sudo cumin 'R:class = role::deployment_server' 'disable-puppet "Switchover of the deployment server"'
- Merge a DNS change that points all the deployment CNAME to the new master hosts, see this change.
- Change deployment_server and scap::deployment_server variables in hiera, see this change.
- Run puppet on the new master
- Run puppet on all the other servers
- Using
grep block_deployments /etc/scap.cfg, verify that:- Deployments on the old master are blocked
- Deployments on the new master are not blocked
- Since keyholder configuration might have changed but not been reloaded on the spare deployment server, restart the keyholder service on the new master and test it
deployXXXX:~ # sudo systemctl restart keyholder-proxy.service deployXXXX:~ #SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -i /etc/keyholder.d/deploy_jenkins -l deploy-jenkins releases1002.eqiad.wmnet
- Temporary workaround for task T197470: Run the following on all deployment servers after replacing deployment server url accordingly
deployXXXX:~# find /srv/deployment -name DEPLOY_HEAD | xargs sed -i "s/git_server: deploy2002.codfw.wmnet/git_server: deploy1002.eqiad.wmnet/"
- Test scap deployment
cd /srv/mediawiki-staging; scap sync-world "check the deployment server after switchover". This will also test helmfile deployments. - Test a scap3 deployment
- email ops@ about the switch of main deployment server