Page MenuHomePhabricator
Create Task
Maniphest T280382

WDQS hosts low on /srv disk space
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Context

Following the data-transfer of the most recent wikidata.jnl, we've hit low enough disk space to trigger the warning threshold.

DISK WARNING - free space: /srv 45621 MB (4% inode=99%)

While Blazegraph's need for free space for compaction specifically is quite low compared to other datastores, the raw amount of space left gives us an unacceptably low amount of headroom for our journal file(s) to keep expanding.

We should take short-term action to address the lack of available disk space. We can double our existing space by migrating from raid10 to raid0. This will cost us redundancy, but it's an acceptable tradeoff in the short term. Medium-term, our newer instances will have more storage and in particular will have at least 4 expansion slots free each if we use the same spec we used for WCQS.

Acceptance criteria

  • Migrated to raid0
    • Switch partman recipe to raid0
    • Re-image each server
  • Do a combover of all the current servers, verifying which hosts this issue applies to (currently looks like it might be every server except potentially wdqs101[1-3])

Current Status

[EQIAD PUBLIC]
wdqs1004.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1006.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1006.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1007.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1012.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1013.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)

[EQIAD INTERNAL]
wdqs1003.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1008.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1011.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)

[CODFW PUBLIC]
wdqs1006.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1006.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs2003.codfw.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs2004.codfw.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs2007.codfw.wmnet => (SUCCESS, NEW_JOURNAL)

[CODFW INTERNAL]
wdqs1006.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs2006.codfw.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1006.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)

[TEST]
wdqs1006.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)
wdqs1010.eqiad.wmnet => (SUCCESS, NEW_JOURNAL)

Details

SubjectRepoBranchLines +/-
wdqs: hack issue blocking reimage on some hostsoperations/puppetproduction+8 -0
install_server: add new installer to test raid0 configuration:operations/puppetproduction+41 -0
wdqs: add missing raid0 dependencyoperations/puppetproduction+1 -1
wdqs: switch to raid0 for more spaceoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Jun 1 2021, 10:35 AM

Script wmf-auto-reimage was launched by jbond on cumin2001.codfw.wmnet for hosts:

wdqs2003.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106011035_jbond_31463_wdqs2003_codfw_wmnet.log.

ops-monitoring-bot added a comment.Jun 1 2021, 10:53 AM

Completed auto-reimage of hosts:

['wdqs2003.codfw.wmnet']

Of which those FAILED:

['wdqs2003.codfw.wmnet']
ops-monitoring-bot added a comment.Jun 1 2021, 10:59 AM

Script wmf-auto-reimage was launched by jbond on cumin2002.codfw.wmnet for hosts:

wdqs2003.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106011059_jbond_2734214_wdqs2003_codfw_wmnet.log.

jbond added a comment.EditedJun 1 2021, 11:05 AM
In T280382#7125504, @jbond wrote:

from iLo the error i see is

PXELINUX 6.03 lwIP 20150819 Copyright (C) 1994-2014 H. Peter Anvin et al

Failed to load ldlinux.c32
Boot failed: press a key to retry, or wait for reset...

When buster-raid0-installer was created the corresponding installer directory in volatile on the puppetmaster was not (see first line of WMF-specific instructions for this below). Adding this directory allowed the installation to complete with out issues. Note '/srv' was reformatted, im not sure if this was intended

In T280382#7125544, @ops-monitoring-bot wrote:

Completed auto-reimage of hosts:

['wdqs2003.codfw.wmnet']

Of which those FAILED:

['wdqs2003.codfw.wmnet']

This one actually succeeded but i ran the script on cumin2001 instead of cumin2002 which meant the final puppet parts failed, re-running to hopefully get a clean install

jbond added a project: User-jbond.Jun 1 2021, 11:07 AM
jbond moved this task from Unsorted 💣 to Active 🚁 on the User-jbond board.
ops-monitoring-bot added a comment.Jun 1 2021, 11:28 AM

Completed auto-reimage of hosts:

['wdqs2003.codfw.wmnet']

and were ALL successful.

jbond added a comment.Jun 1 2021, 11:38 AM
In T280382#7125667, @ops-monitoring-bot wrote:

Completed auto-reimage of hosts:

['wdqs2003.codfw.wmnet']

and were ALL successful.

Things look good to me from a basic level

$ df -h /srv                                                                                                                                          [11:37:56]
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        2.9T  481M  2.8T   1% /srv
jbond moved this task from Active 🚁 to Watching 👀 on the User-jbond board.Jun 1 2021, 11:39 AM
MoritzMuehlenhoff subscribed.Jun 2 2021, 2:12 PM
In T280382#7114272, @RKemper wrote:

I'm mentioning this because if we just default to layout 2 for raid0 reimages in general, IF a re-image is not full i.e. it leaves /srv/ or some other partition, AND the data was originally written by a pre-3.14 kernel, then data loss could/would occur. This is probably more of a theoretical concern; last I talked to jbond he seemed to think that it would be okay to just default to layout 2 for all raid0 reimages.

Nice detective work! I think it's entirely safe to make layout 2 the global default:

  • Debian jessie was released in 2015 (where we started to use Debian) and already featured a 3.16 kernel. Prior to that Ubuntu was used and all jessie/Ubuntu machines are now decommisioned since they are way beyond our hardware lifecycle.
  • The partman infrastructure to actually be able to retain a partition is fairly new, it was implemented by Stevie maybe a year ago, so we don't have any cases of old Ubuntu systems which were reimaged to Debian while retaining /srv either.
gerritbot added a comment.Jun 2 2021, 6:17 PM

Change 689525 abandoned by Ryan Kemper:

[operations/puppet@production] wdqs: hack issue blocking reimage on some hosts

Reason:

see https://gerrit.wikimedia.org/r/c/operations/puppet/ /697832 for the fix

https://gerrit.wikimedia.org/r/689525

Maintenance_bot removed a project: Patch-For-Review.Jun 2 2021, 7:10 PM
ops-monitoring-bot added a comment.Jun 2 2021, 9:07 PM

Script wmf-auto-reimage was launched by ryankemper on cumin1001.eqiad.wmnet for hosts:

wdqs1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106022107_ryankemper_28365_wdqs1004_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Jun 2 2021, 9:10 PM

Script wmf-auto-reimage was launched by ryankemper on cumin2002.codfw.wmnet for hosts:

wdqs2007.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106022110_ryankemper_2977652_wdqs2007_codfw_wmnet.log.

Stashbot added a comment.Jun 2 2021, 9:10 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T21:10:30Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 --new wdqs1004.eqiad.wmnet on ryankemper@cumin1001 tmux session reimage

Stashbot added a comment.Jun 2 2021, 9:10 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T21:10:36Z] <ryankemper> T280382 T281437 sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2007.codfw.wmnet on ryankemper@cumin2002 tmux session wdqs_reimage

ops-monitoring-bot added a comment.Jun 2 2021, 9:20 PM

Completed auto-reimage of hosts:

['wdqs1004.eqiad.wmnet']

Of which those FAILED:

['wdqs1004.eqiad.wmnet']
ops-monitoring-bot added a comment.Jun 2 2021, 9:23 PM

Script wmf-auto-reimage was launched by ryankemper on cumin1001.eqiad.wmnet for hosts:

wdqs1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106022123_ryankemper_9191_wdqs1004_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Jun 2 2021, 9:43 PM

Completed auto-reimage of hosts:

['wdqs2007.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Jun 2 2021, 9:49 PM

Completed auto-reimage of hosts:

['wdqs1004.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 2 2021, 9:56 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T21:56:01Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1004.eqiad.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 2 2021, 9:59 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T21:59:58Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2007.codfw.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin2002 tmux session wdqs_reimage

ops-monitoring-bot added a comment.Jun 2 2021, 10:10 PM

Script wmf-auto-reimage was launched by ryankemper on cumin1001.eqiad.wmnet for hosts:

wdqs1003.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106022210_ryankemper_20767_wdqs1003_eqiad_wmnet.log.

Stashbot added a comment.Jun 2 2021, 10:11 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T22:11:00Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 wdqs1003.eqiad.wmnet on ryankemper@cumin1001 tmux session wdqs_reimage_2

ops-monitoring-bot added a comment.Jun 2 2021, 10:11 PM

Completed auto-reimage of hosts:

['wdqs1003.eqiad.wmnet']

Of which those FAILED:

['wdqs1003.eqiad.wmnet']
Stashbot added a comment.Jun 2 2021, 10:11 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T22:11:29Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 --new wdqs1003.eqiad.wmnet on ryankemper@cumin1001 tmux session wdqs_reimage_2

ops-monitoring-bot added a comment.Jun 2 2021, 10:11 PM

Script wmf-auto-reimage was launched by ryankemper on cumin1001.eqiad.wmnet for hosts:

wdqs1003.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106022211_ryankemper_21294_wdqs1003_eqiad_wmnet.log.

Stashbot added a comment.Jun 2 2021, 10:30 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T22:30:31Z] <ryankemper> T280382 Cleaned up no-longer-needed files removed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/697832 => ryankemper@cumin1001:~$ sudo -E cumin -b 6 'P{install*}' 'sudo rm -fv /srv/tftpboot/buster-raid0-installer/pxelinux.cfg'

Stashbot added a comment.Jun 2 2021, 10:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T22:34:23Z] <ryankemper> T280382 Cleaned up no-longer-needed files removed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/697832 => ryankemper@cumin1001:~$ sudo -E cumin -b 2 'P{apt*}' 'sudo rm -rfv /srv/tftpboot/buster-raid0-installer/pxelinux.cfg'

ops-monitoring-bot added a comment.Jun 2 2021, 10:38 PM

Completed auto-reimage of hosts:

['wdqs1003.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Jun 2 2021, 10:39 PM

Script wmf-auto-reimage was launched by ryankemper on cumin2002.codfw.wmnet for hosts:

wdqs2003.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106022239_ryankemper_2989124_wdqs2003_codfw_wmnet.log.

Stashbot added a comment.Jun 2 2021, 10:39 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T22:39:55Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2003.codfw.wmnet on ryankemper@cumin2002 tmux session wdqs_reimage

ops-monitoring-bot added a comment.Jun 2 2021, 11:07 PM

Completed auto-reimage of hosts:

['wdqs2003.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 2 2021, 11:26 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T23:26:00Z] <ryankemper> T280382 wdqs2007.codfw.wmnet has been re-imaged and had the appropriate wikidata/categories journal files transferred. df -h shows disk space is no longer an issue following the switch to raid0: /dev/mapper/vg0-srv 2.7T 998G 1.6T 39% /srv

Stashbot added a comment.Jun 2 2021, 11:28 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T23:28:30Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2007.codfw.wmnet --dest wdqs2003.codfw.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 2 2021, 11:47 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T23:47:26Z] <ryankemper> T280382 wdqs1004.eqiad.wmnet has been re-imaged and had the appropriate wikidata/categories journal files transferred. df -h shows disk space is no longer an issue following the switch to raid0: /dev/md2 2.9T 998G 1.8T 36% /srv

Stashbot added a comment.Jun 2 2021, 11:56 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T23:56:42Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1004.eqiad.wmnet --dest wdqs1003.eqiad.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 2 2021, 11:57 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-02T23:57:23Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2007.codfw.wmnet --dest wdqs2003.codfw.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 12:18 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T00:18:29Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1007.eqiad.wmnet --dest wdqs1003.eqiad.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 12:35 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T00:35:14Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1007.eqiad.wmnet --dest wdqs1003.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 1:47 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T01:47:12Z] <ryankemper> T280382 wdqs2003.codfw.wmnet has been re-imaged and had the appropriate wikidata/categories journal files transferred. df -h shows disk space is no longer an issue following the switch to raid0: /dev/md2 2.9T 998G 1.8T 36% /srv

ops-monitoring-bot added a comment.Jun 3 2021, 1:51 AM

Script wmf-auto-reimage was launched by ryankemper on cumin2002.codfw.wmnet for hosts:

wdqs2006.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106030150_ryankemper_3011426_wdqs2006_codfw_wmnet.log.

Stashbot added a comment.Jun 3 2021, 1:51 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T01:51:15Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 wdqs2006.codfw.wmnet on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 2:05 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T02:05:22Z] <ryankemper> T280382 wdqs1003.eqiad.wmnet has been re-imaged and had the appropriate wikidata/categories journal files transferred. df -h shows disk space is no longer an issue following the switch to raid0: /dev/md2 2.9T 998G 1.8T 36% /srv

ops-monitoring-bot added a comment.Jun 3 2021, 2:07 AM

Script wmf-auto-reimage was launched by ryankemper on cumin1001.eqiad.wmnet for hosts:

wdqs1008.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106030207_ryankemper_3293_wdqs1008_eqiad_wmnet.log.

Stashbot added a comment.Jun 3 2021, 2:07 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T02:07:40Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 wdqs1008.eqiad.wmnet on ryankemper@cumin1001 tmux session wdqs_reimage

ops-monitoring-bot added a comment.Jun 3 2021, 2:19 AM

Completed auto-reimage of hosts:

['wdqs2006.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Jun 3 2021, 2:32 AM

Completed auto-reimage of hosts:

['wdqs1008.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 3 2021, 4:29 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T04:29:43Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1005.eqiad.wmnet --dest wdqs1008.eqiad.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 4:29 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T04:29:59Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2004.codfw.wmnet --dest wdqs2006.codfw.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 4:36 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T04:36:34Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2004.codfw.wmnet --dest wdqs2006.codfw.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 4:37 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T04:37:07Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1005.eqiad.wmnet --dest wdqs1008.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 6:23 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T06:23:48Z] <ryankemper> T280382 wdqs1008.eqiad.wmnet has been re-imaged and had the appropriate wikidata/categories journal files transferred. df -h shows disk space is no longer an issue following the switch to raid0: /dev/md2 2.6T 998G 1.5T 40% /srv

Stashbot added a comment.Jun 3 2021, 6:23 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T06:23:54Z] <ryankemper> T280382 wdqs2006.codfw.wmnet has been re-imaged and had the appropriate wikidata/categories journal files transferred. df -h shows disk space is no longer an issue following the switch to raid0: /dev/md2 2.6T 998G 1.5T 40% /srv

Stashbot added a comment.Jun 3 2021, 6:46 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T18:46:18Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 wdqs1005.eqiad.wmnet on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 6:46 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T18:46:24Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 wdqs2005.codfw.wmnet on ryankemper@cumin2002 tmux session wdqs_reimage

ops-monitoring-bot added a comment.Jun 3 2021, 6:47 PM

Script wmf-auto-reimage was launched by ryankemper on cumin1001.eqiad.wmnet for hosts:

wdqs1005.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106031846_ryankemper_5874_wdqs1005_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Jun 3 2021, 6:47 PM

Script wmf-auto-reimage was launched by ryankemper on cumin2002.codfw.wmnet for hosts:

wdqs2005.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106031846_ryankemper_3168105_wdqs2005_codfw_wmnet.log.

RKemper updated the task description. (Show Details)Jun 3 2021, 6:47 PM
ops-monitoring-bot added a comment.Jun 3 2021, 7:12 PM

Completed auto-reimage of hosts:

['wdqs1005.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Jun 3 2021, 7:15 PM

Completed auto-reimage of hosts:

['wdqs2005.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 3 2021, 7:27 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T19:27:39Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1013.eqiad.wmnet --dest wdqs1005.eqiad.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 7:28 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T19:28:00Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2001.codfw.wmnet --dest wdqs2005.codfw.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 8:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T20:34:09Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2001.codfw.wmnet --dest wdqs2005.codfw.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 8:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T20:34:32Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1013.eqiad.wmnet --dest wdqs1005.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 10:35 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T22:35:15Z] <ryankemper> T280382 wdqs2005.codfw.wmnet has been re-imaged and had the appropriate wikidata/categories journal files transferred. df -h shows disk space is no longer an issue following the switch to raid0: /dev/md2 2.6T 998G 1.5T 40% /srv

Stashbot added a comment.Jun 3 2021, 10:36 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T22:36:43Z] <ryankemper> T280382 Cancelled transfer to wdqs1005; the source host wdqs1013 has a wikidata.jnl that is 80% too big; will transfer from different node -> wdqs1005 and then fix the journal on wdqs1013 after

Stashbot added a comment.Jun 3 2021, 10:39 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T22:39:41Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1008.eqiad.wmnet --dest wdqs1005.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 3 2021, 10:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T22:41:13Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 wdqs2001.codfw.wmnet on ryankemper@cumin2002 tmux session wdqs_reimage

ops-monitoring-bot added a comment.Jun 3 2021, 11:11 PM

Script wmf-auto-reimage was launched by ryankemper on cumin2002.codfw.wmnet for hosts:

wdqs2001.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106032241_ryankemper_3197307_wdqs2001_codfw_wmnet.log.

ops-monitoring-bot added a comment.Jun 3 2021, 11:42 PM

Completed auto-reimage of hosts:

['wdqs2001.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 4 2021, 12:05 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T00:05:55Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1008.eqiad.wmnet --dest wdqs1005.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin1001 tmux session wdqs_reimage

Stashbot added a comment.Jun 4 2021, 12:07 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T00:07:20Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2007.codfw.wmnet --dest wdqs2001.codfw.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 4 2021, 2:09 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T02:09:34Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2007.codfw.wmnet --dest wdqs2001.codfw.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 4 2021, 2:30 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T02:30:33Z] <ryankemper> T280382 wdqs1005.eqiad.wmnet has been re-imaged and had the appropriate wikidata/categories journal files transferred. df -h shows disk space is no longer an issue following the switch to raid0: /dev/md2 2.9T 998G 1.8T 36% /srv

Stashbot added a comment.Jun 4 2021, 4:22 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T04:22:22Z] <ryankemper> T280382 wdqs2001.codfw.wmnet has been re-imaged and had the appropriate wikidata/categories journal files transferred. df -h shows disk space is no longer an issue following the switch to raid0: /dev/md2 2.9T 998G 1.8T 36% /srv

ops-monitoring-bot added a comment.Jun 4 2021, 4:25 AM

Script wmf-auto-reimage was launched by ryankemper on cumin2002.codfw.wmnet for hosts:

wdqs2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106040424_ryankemper_3238923_wdqs2002_codfw_wmnet.log.

Stashbot added a comment.Jun 4 2021, 4:25 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T04:25:40Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 wdqs2002.codfw.wmnet on ryankemper@cumin2002 tmux session wdqs_reimage

ops-monitoring-bot added a comment.Jun 4 2021, 4:54 AM

Completed auto-reimage of hosts:

['wdqs2002.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 4 2021, 5:17 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T05:17:06Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2001.codfw.wmnet --dest wdqs2002.codfw.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories on ryankemper@cumin2002 tmux session wdqs_reimage

Stashbot added a comment.Jun 4 2021, 5:24 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-04T05:24:27Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs2001.codfw.wmnet --dest wdqs2002.codfw.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph on ryankemper@cumin2002 tmux session wdqs_reimage

Gehel added a comment.Jun 7 2021, 1:09 PM

Current status: all nodes except wdqs1009 have been reimaged and now have sufficient space. wdqs1009 is our test server to expose skolemnized journal to users and do internal testing on the new streaming updater. We will only reimage it once this experiment is completed (once T264006 is done).

Stashbot added a comment.Jun 8 2021, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T19:18:37Z] <ryankemper> T280382 sudo systemctl stop wdqs-updater wdqs-blazegraph on wdqs1010 in preparation for transfer

Stashbot added a comment.Jun 8 2021, 7:20 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T19:20:44Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1009.eqiad.wmnet --dest wdqs1010.eqiad.wmnet --reason "transferring skolemized wikidata.jnl so we can reimage wdqs1009" --blazegraph_instance blazegraph --without-lvs on ryankemper@cumin1001 tmux session wdqs_1009

Stashbot added a comment.Jun 8 2021, 7:36 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T19:36:22Z] <ryankemper> T280382 Cancelling the data-transfer run to restart it; realized that the cookbook will start up the wdqs-updater again so will locally hack the cookbook on cumin1001 to prevent that

Stashbot added a comment.Jun 8 2021, 9:27 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T21:27:56Z] <ryankemper> T280382 Disabled puppet on wdqs1010 out of abundance of caution; will re-enable after wdqs1009 is reimaged and xfer back is complete

ops-monitoring-bot added a comment.Jun 8 2021, 9:28 PM

Script wmf-auto-reimage was launched by ryankemper on cumin1001.eqiad.wmnet for hosts:

wdqs1009.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106082128_ryankemper_20112_wdqs1009_eqiad_wmnet.log.

Stashbot added a comment.Jun 8 2021, 9:29 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T21:29:11Z] <ryankemper> T280382 sudo -i wmf-auto-reimage-host -p T280382 wdqs1009.eqiad.wmnet on ryankemper@cumin1001 tmux session wdqs_1009

ops-monitoring-bot added a comment.Jun 8 2021, 9:54 PM

Completed auto-reimage of hosts:

['wdqs1009.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 9 2021, 12:00 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-09T00:00:57Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1010.eqiad.wmnet --dest wdqs1009.eqiad.wmnet --reason "transferring skolemized wikidata.jnl so we can reimage wdqs1009" --blazegraph_instance blazegraph --without-lvs on ryankemper@cumin1001 tmux session wdqs_1009

Stashbot added a comment.Jun 9 2021, 2:39 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-09T02:39:19Z] <ryankemper> T280382 Re-enabled puppet on wdqs1010

Stashbot added a comment.Jun 9 2021, 2:49 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-09T02:49:24Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1010.eqiad.wmnet --dest wdqs1009.eqiad.wmnet --reason "xfer categories following reimage" --blazegraph_instance categories --without-lvs on ryankemper@cumin1001 tmux session wdqs_1009

RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Jun 14 2021, 3:30 PM
RKemper updated the task description. (Show Details)Jun 14 2021, 3:34 PM
Gehel closed this task as Resolved.Jun 21 2021, 11:41 AM
Stashbot added a comment.Jul 21 2021, 10:37 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-21T22:37:58Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1010.eqiad.wmnet --reason "transferring fresh categories journal to resolve categories update lag unknown alert status" --blazegraph_instance categories --without-lvs on ryankemper@cumin1001 tmux session wdqs

Stashbot added a comment.Jul 21 2021, 10:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-21T22:41:54Z] <ryankemper> T280382 sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1009.eqiad.wmnet --reason "transferring fresh categories journal to resolve categories update lag unknown alert status" --blazegraph_instance categories --without-lvs on ryankemper@cumin1001 tmux session wdqs

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL