Phabricator/Deployment

Deployment windows

Except for emergencies, Phabricator deployments are generally handled during mw:Phabricator/Maintenance windows. As of 2024, this is done during Collaboration Services office hours. Other windows may be scheduled, depending on deployer availability.

GitLab deployment repository

Phabricator deployments are handled via scap, using configuration in the phabricator/deployment repo.

For minor changes you intend to deploy immediately, you can merge changes locally to the appropriate repository, bump any affected submodules, and push directly to the wmf/stable branch.

~/code/wmf/phabricator/deployment $ git pull --recurse-submodules
# Assuming an update to the phabricator submodule:
~/code/wmf/phabricator/deployment $ git submodule update
~/code/wmf/phabricator/deployment $ git add phabricator
~/code/wmf/phabricator/deployment $ git commit
~/code/wmf/phabricator/deployment $ git push

For larger sets of changes, you should create a Merge Request in GitLab which updates the submodules (example).

~/code/wmf/phabricator/deployment $ git pull --recurse-submodules
~/code/wmf/phabricator/deployment $ git checkout -b work/2023-11-09-release
# Assuming an update to the phabricator submodule:
~/code/wmf/phabricator/deployment $ git submodule update
~/code/wmf/phabricator/deployment $ git add phabricator
~/code/wmf/phabricator/deployment $ git commit
~/code/wmf/phabricator/deployment $ git push --set-upstream origin work/2023-11-09-release

Then follow the link to create a merge request.

When you're ready to deploy, merge this Merge Request via the GitLab UI ("Merge" button) to bump submodules.

Notes:

As of October 2023, the AVA repository and the Translations repository are still in Gerrit instead of GitLab (cf. phab:T321380)
When deploying CSS and JS changes, dependency and packaging maps are generated by bin/celerity map, which updates resources/celerity/map.php
When removing or adding source code files, running Arcanist's /bin/arc liberate updates the library metadata.

Downtime host alerts

If you can run cookbooks:

sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' phab1004
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phabricator.wikimedia.org
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phab.wmfusercontent.org

If you can't run cookbooks:

Downtime Icinga
- Go to https://icinga.wikimedia.org/icinga/ and search for "phab" in the sidebar.
- Select the following hosts:
  - phab.wmfusercontent.org
  - phab1004 (production in eqiad)
  - phab2002 (fallback in codfw)
- In the "Select command" dropdown, select "Schedule Downtime for Checked Host(s)"
Silence Alerts
- Click the bell on https://alerts.wikimedia.org/
- Click "New Silence"
- Under "Label" select "Instance"; toggle "regex" to be enabled; in the text field add a new silence for each of the following:
  - ^(phab1004)(:[0-9]+)?$
  - ^(phab\.wmfusercontent\.org)(:[0-9]+)?$
  - ^(phabricator\.wikimedia\.org)(:[0-9]+)?$
- Ensure you've put something in the comment field at the bottom, "Phab deployment T123456" or whatever you're doing.
- In the "No Alerts matched" dialog, click the "Submit" button
- If the resulting dashboard URL is a 404, check/correct the domain name

Deploy to fallback host

ssh deploy2002.codfw.wmnet
cd /srv/deployment/phabricator/deployment/
git pull
git submodule update
scap deploy -l phab2002.codfw.wmnet 'test deploy to phab2002 for Txxxxxx' (replace with task ID; also logs to #wikimedia-operations ^connect)

Deploy to production host

ssh deploy1002.eqiad.wmnet
cd /srv/deployment/phabricator/deployment/
git pull
git submodule update
scap deploy -l phab1004.eqiad.wmnet 'deploy to phab1004 for Txxxxxx' (replace with task ID; also logs to #wikimedia-operations ^connect)
In parallel, check for errors on the Logstash dashboard; or via ssh phab1004.eqiad.wmnet followed by tail -F /var/log/apache2/phabricator_error.log
In parallel, check for the next minutes for any explosions on IRC #wikimedia-operations ^connect
Optionally,
- systemctl status phd to make sure things are running
- tail /var/log/phd/daemon.log to check for daemon problems (noisy because of repo syncing errors)

Clean up deployed Phabricator tasks in Phabricator

Go into phab project and rename "next" milestone to the date of the deploy
- i.e. "2023-08-23"
- Remove the additional hashtag "#phab-next" from the milestone
- Paste generated markup with stuff from release (past example)
- archive this old milestone
- create a new "next" milestone
  - Set description to "A future deployment of Phabricator"
- Tag and resolve open tasks that got deployed and need no further work