GitLab/Upgrade

From Wikitech
GitLab Backup/Upgrade
Use this link to create a security update task
When the timeline of the upgrade is known, add a maintenance window to Deployments

WMF GitLab is installed via Debian package named gitlab-ce. This is called an Omnibus setup. So to upgrade GitLab, this package is upgraded to a newer version. The upgrade process is described here. For more detail, please see the GitLab upgrade documentation and the Omnibus specific upgrade documentation.

WMF upgrade path

The upgrade of GitLab should be applied in the following order:

  1. upgrade gitlab-prod-1002 machine in WMCS/horizon, project devtools (using the manual upgrade process)
  2. upgrade GitLab replica (currently gitlab1003 and gitlab2002, please check)
  3. upgrade production GitLab (currently gitlab1004, please check)
  4. upgrade all gitlab-runners (Shared, Trusted and Cloud)

Recommended upgrade cadence:

  • security patches (issued monthly or ad-hoc):
    • "Critical" and "high" security patches which affect our installation (CE): as soon as possible
    • "Low", "medium" and "high" which only affect disabled features: next maintenance windows (usually Friday)
  • minor upgrades (issued monthly): only if needed for upstream features or bugfixes
  • major upgrades (issued annually): once per year, after release has had time to be tested by early adopters and the previous major version is out of support.

We follow GitLabs security patches when upgrading GitLab. That means we upgrade GitLab when there is a new security release. Upgrade to non-security releases will only be done if needed for certain features or bugfixes or in preparation for another security update.

Upgrade GitLab Debian package

  • Update modules/aptrepo/files/updates in operations/puppet/ similar to 774905. Set hook commands --eq to desired version
  • Get +1 for aptrepo change, merge and puppet-merge
  • run puppet on apt1002
  • Run sudo -i reprepro --component thirdparty/gitlab-bullseye checkupdate bullseye-wikimedia on apt1002 to check if new version is found
  • Run sudo -i reprepro --component thirdparty/gitlab-bullseye update bullseye-wikimedia on apt1002 to update package
  • repeat for thirdparty/gitlab-runner if needed
  • read Reprepro#Updating external repositories for more information

Prerequisites

  • A valid GitLab personal access API token for every instance you want to upgrade (scope the token for api and admin_mode)
  • Make yourself familiar with GitLab release and maintenance policy
  • Pick the closest supported version unless another version is explicitly requested. Check if it's a major, minor or patch upgrade.
  • Read release notes of all versions between current and selected one
  • Determine the proper upgrade path using the upgrade path calculator. Make sure an upgrade from the current to the new version is possible. Never upgrade over two major versions in a single step
  • Make sure the package with the new version is present in debmonitor
  • Determine if any manual migrations are required in your upgrade (docs). This is run as part of the cookbook, or commands are listed in the manual upgrade section below.
  • Make sure no other maintenance or train is running (Deployments or jouncebot: nowandnext in IRC).

Automated Upgrade (sre.gitlab.upgrade cookbook)

The upgrade process for GitLab hosts is automated by the sre.gitlab.upgrade cookbook. Login to cumin host and open a persistent windows (tmux or screen).

sudo cookbook sre.gitlab.upgrade -h

    Usage example:
        cookbook sre.gitlab.upgrade --host gitlab1004 --version 15.4.4-ce.0 -r 'some reason' -t T12345

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Short hostname of the gitlab host to upgrade, not FQDN (default: None)
  --version VERSION     Version of new GitLab Debian package in Debian versioning schema (default: None)
  -r REASON, --reason REASON
                        The reason for the downtime. The current username and originating host are automatically added. (default: None)
  -s, --skip-replica-backups
                        Skips creating a backup on replica hosts (default: False)
  -t TASK_ID, --task-id TASK_ID
                        An optional task ID to refer in the downtime message (i.e. T12345). (default: None)

You need a valid admin API token to run the cookbook. Make sure you are admin (see paste). Go to personal access API token and create a new, temporary personal access token with scope api and admin_mode. You have to paste the access token to a prompt in the cookbook.

When the cookbook passed, proceed with upgrading the GitLab Runners (see below).

Note:

Manual Upgrade

The following steps describe the manual upgrade process. This process is used on the test instance in WMCS currently and can be used for manual troubleshooting. If you don't have a good reason, use the cookbook sre.gitlab.upgrade mentioned above.

Preparations

  • make both full GitLab data and full configuration data backups before upgrading:
sudo /usr/bin/gitlab-backup create CRON=1 STRATEGY=copy GZIP_RSYNCABLE=yes SKIP=builds,artifacts,registry GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1
sudo /usr/bin/gitlab-ctl backup-etc
  • On the test instance, clean the old apt packages to make sure enough disk space is available
sudo apt clean
  • preload to-be-installed GitLab CE packages before upgrading:
sudo apt-get update && sudo apt-get install gitlab-ce=14.0.10-ce.0 --download-only
  • if you have GitLab Runners connected to your GitLab Server, it is recommended to pause all runners and wait until all jobs are finished before starting the upgrade. Use the script in gitlab-settings to pause all active runners:
./runners active | tee active.txt
./runners pause < active.txt
  • Check if any background migrations are running:
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

Upgrading GitLab

  • Downtime GitLab host in Icinga if long migration is expected (see release notes, not needed in most cases):
sudo cookbook sre.hosts.downtime -r "upgrade <gitlab_host> to new version https://phabricator.wikmiedia.org/<ID>" -M 15 <gitlab_host>
  • Run required manual configuration steps if needed (see release notes, not needed in most cases)
  • Install new GitLab version:
sudo apt-get install gitlab-ce=14.0.10-ce.0
  • Wait for GitLab to finish all migrations and restarts

GitLab Runner

This section describes how to upgrade GitLab Runners.

Upgrading GitLab Shared Runners

Shared runners are currently in the gitlab-runners project in WMCS. This Runners have unattendedUpgrades enabled. So after updating the gitlab-runner package on apt host, the Shared Runners are updated automatically the next day. For time-critical updates proceed with the steps below: 

sudo cumin 'O{project:devtools name:^gitlab-runner.*}' 'apt-get update && apt-get -y install gitlab-runner' # for Test Runners
sudo cumin 'O{project:gitlab-runners name:^runner.*}' 'apt-get update && apt-get -y install gitlab-runner' # for Shared Runners
  • Runners will restart automatically. If needed, restart runners (login to cloudcumin hosts):
sudo cumin 'O{project:gitlab-runners name:^runner.*}' '/usr/bin/gitlab-runner restart'

Upgrading GitLab Trusted Runners

Trusted Runners live in codfw and eqiad (gitlab-runner100* and gitlab-runner200*). 

sudo cumin 'P{O:gitlab_runner}' 'apt-get update && apt-get -y install gitlab-runner'
  • Runners will restart automatically. If needed, restart runners:
sudo /usr/bin/gitlab-runner restart

Upgrading GitLab Cloud Runners

Cloud Runners are managed by gitlab-cloud-runner/ project in GitLab. To upgrade the version, create a merge request and update the image.tag field for the gitlab-runner values file.

Review and merge the change. CI is configured to update the gitlab-runner helm deployment automatically.

Steps after upgrading

  • Run required manual steps/migrations (see release notes. not needed in most cases)
  • Check that all background migrations are fully finished and background migration queue is empty
  • Make sure that GitLab is up and running after upgrade; please give it several minutes to calm down
  • Check monitoring systems of GitLab (especially Icinga alerts)
  • Run basic smoke tests (make sure that web UI works, authentication works, ssh cloning works)
  • Only manual: Re-enable paused runners in the GitLab admin interface at https://gitlab.wikimedia.org/admin/runners . Use the script in gitlab-settings to unpause all active runners:
    • ./runners unpause < active.txt
  • Revoke the API tokens when no longer needed
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=GitLab/Upgrade&oldid=2165020"