completion traffic is now served from codfw which has proper indices, lowering prio

Removing assignee that was automatically set by Phabricator when the task got marked as resolved.

scap does the docker pull on any of the k8s worker as defined by the kubernetes-workers group and parse1002 is n that group:

deploy1002$ grep -R parse1002 /etc/dsh/group
/etc/dsh/group/kubernetes-workers:parse1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2024-04-26T09:54:21Z] <dcausse@deploy1002> Finished scap: Backport for [[gerrit:1024478|cirrus: Shift autocomplete traffic to codfw (T363516)]] (duration: 17m 57s)

Mentioned in SAL (#wikimedia-operations) [2024-04-26T09:41:30Z] <dcausse@deploy1002> dcausse and ebernhardson: Backport for [[gerrit:1024478|cirrus: Shift autocomplete traffic to codfw (T363516)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-04-26T09:36:24Z] <dcausse@deploy1002> Started scap: Backport for [[gerrit:1024478|cirrus: Shift autocomplete traffic to codfw (T363516)]]

Change #1024478 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Shift autocomplete traffic to codfw

https://gerrit.wikimedia.org/r/1024478

This is still happening, raising to UBN

Change #1024478 restored by DCausse:

[operations/mediawiki-config@master] cirrus: Shift autocomplete traffic to codfw

https://gerrit.wikimedia.org/r/1024478

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2003.codfw.wmnet with OS bullseye completed:

• kubestagemaster2003 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via gnt-instance
  • Host up (Debian installer)
  • Add puppet_version metadata to Debian installer
  • Set boot media to disk
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404260748_jayme_2095943_kubestagemaster2003.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2003.codfw.wmnet with OS bullseye

Change #1024543 merged by JMeybohm:

[operations/puppet@production] kubestagemaster2003: Add as insetup::serviceops

https://gerrit.wikimedia.org/r/1024543

Change #1024543 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubestagemaster2003: Add as insetup::serviceops

https://gerrit.wikimedia.org/r/1024543

Never mind, they just said it's fixed :)

There's someone reporting that they're still not seeing the expected results for some queries, although I can't reproduce: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#c-2804:F14:8092:9F01:3468:323E:5807:DBA8-20240425214000-Matma_Rex-20240425212500

Change #1024516 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] magru: update edgeuno transit IP

https://gerrit.wikimedia.org/r/1024516

Change #1024478 abandoned by Ebernhardson:

[operations/mediawiki-config@master] cirrus: Shift autocomplete traffic to codfw

Reason:

rebuild only took 45 minutes, decided not to shuffle while it was in progress

https://gerrit.wikimedia.org/r/1024478

Decided against shuffling traffic, rebuild is almost compete already for enwiki. I can see in the logs where the enwiki eqiad build jumped from 44% to complete, but no reason why. nothing in logstash for that period either. I've created T363521 to put something in place to prevent this in the future.

Change #1024481 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elasticsearch: Configure alerts for short-lived certs

https://gerrit.wikimedia.org/r/1024481

Mentioned in SAL (#wikimedia-operations) [2024-04-25T19:33:12Z] <ebernhardson> T363516 started manual rebuild of enwiki titlesuggest indices in eqiad

Change #1024478 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Shift autocomplete traffic to codfw

https://gerrit.wikimedia.org/r/1024478

hmm, i can confirm this is happening. The completion index is built new every day in each datacenter. Usually they are the same, but somehow the eqiad index is about half the size of the codfw index (6.7g vs 14.5g). Auto complete is fairly high traffic, we should probably shift the autocomplete traffic to codfw until it can be fixed which probably requires a rebuild and a couple hours.

@YLiou_WMF here's the task -- please sign L3

Change #1023917 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] prometheus: Ensure TLS certificates are provided by CFSSL

https://gerrit.wikimedia.org/r/1023917

Thanks! I added Lina to WMF-NDA in Phabricator for access to private tickets.

@Dzahn Done!

@KFrancis Could you add Lina to the 'NDA and MOU' spreadsheet please? That way Brett can see the email address and we don't run into sync issues like with T358578. Thanks

Change #1024449 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] admin: add Linda Farid to LDAP_only (nda)

https://gerrit.wikimedia.org/r/1024449

Change #1024447 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] deployment_server: stop including redis::client::python

https://gerrit.wikimedia.org/r/1024447

Change #1024336 merged by Dzahn:

[operations/puppet@production] releases: Enable profile::auto_restarts::service for docker/containerd

https://gerrit.wikimedia.org/r/1024336

Hi, @Lina_Farid_WMDE, thanks for signing that. Could you share your email address so I can get a patch in?

Change #1023954 merged by Dzahn:

[operations/puppet@production] redis: use python3-redis to support bullseye

https://gerrit.wikimedia.org/r/1023954

In T363399#9745051, @MoritzMuehlenhoff wrote:

Will parsoidtest1001 be installed with Bullseye? scandium is currently running buster, but all the mediawiki manifests are compatible with bullseye (cloudweb already runs it), and so is the component/php74.

Updating the host ownership in the Puppet role should also be part of this task.

Change #1019131 merged by JHathaway:

[operations/puppet@production] Postfix profile

https://gerrit.wikimedia.org/r/1019131

In any case, at this point I 'd prefer to do an in-place upgrade rather than a reimage, given how unreliable a reimage is and how impactful it can be for stateful services.

In T361087#9745154, @jcrespo wrote:

In T361087#9744977, @cmooney wrote:

In T361087#9744384, @jcrespo wrote:

Booting failed (PXE):

PXELINUX 6.03 lwIP 20150819 Copyright (C) 1994-2014 H. Peter Anvin et al


Debian 12 (bookworm) amd64 (Wikimedia edition)

                                              boot: 
Loading debian-installer/amd64/linux... ok
Loading debian-installer/amd64/initrd.gz...
Boot failed: press a key to retry, or wait for reset...

Hmm. Not sure if we've seen this problem before. DHCP clearly worked as did the debian image download, but Linux failed to load for some reason.

@jcrespo the only difference was selecting bullseye rather than bookworm on the second attempt?

Yes. Check with @MoritzMuehlenhoff he did something to fix something, but not sure what, or if it applies here.