In T362729#9721040, @ssingh wrote:

Thanks for the task @RobH! As in the previous runs, please feel free to leave these for Traffic:

Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
Run the sre.hosts.reimage cookbook

Since if you do insetup::traffic, we will have to reimage to cache::text or upload anyway so it doesn't make sense to repeat the reimages, unless you think is a particular reason. Thanks!

@Jhancock.wm: I put a typo in the top, it should be (3) dimms per host not 2, not changing it but updating in this comment so you can acknowledge and update the task description to ensure everyone's on the same page.

In T361229#9669564, @Jhancock.wm wrote:

I have located and set aside the parts to be installed.

I am available every week day between 1300 UTC and 1700 UTC. Please let me know what time/day in that works best.

Reassigning from myself over to @Fabfur for reimaging at Traffic's leisure.

ESAMS remote hands began hands on work at 11:10 CET and it is now ongoing.

Remote work task is via CS1553796, remote hands has confirmed receipt of the SSDs and work to take place on March 27th @ 11AM CET.

I no longer track contracts, those are handled via Coupa which has end date tracking. This old request is now invalid.

CS1553796 created. Will update one they confirm the window.

We would like remote hands to fetch shipmnet DEL0158639 which contains (8) 6.5TB NVMe PCIe SSDs from Dell NL to Wikimedia.

Proposted Work Window: 2023-03-27 @ 1100 CET

Once fetched, please unbox, photograph the contents and packing slip, and stage them for installation in our servers.

We would like to schedule the actual installation to take place om 2023-03-27 @ 1100 CET. The installation of these PCIe cards by Remote hands should take anywhere from 1-3 hours.

We would like remote hands to unbox and install (1) PCIe SSD NVMe card into (8) of our hosts which only contain (1) PCIe NVMe SSD and will upgrade them to (2) PCIe NVMe SSDs per host.

This will be repeated for a total of eight hosts in our racks as follows:
hostname/serial Rack:U-space

cp3066/6QGW8X3 BW27:U2
cp3067/3QGW8X3 BY27:U2
cp3068/5QGW8X3 BW27:U3
cp3069/2QGW8X3 BY27:U3
cp3070/1QGW8X3 BW27:U4
cp3071/7QGW8X3 BY27:U4
cp3072/4QGW8X3 BW27:U5
cp3073/JPGW8X3 BY27:U5

We would prefer the cadence of the work to be as follows:

• Unplug cables from listed host that has been powered down.
• Note serial of the PCIe NVMe SSD and install it into the host.
• Push host back into rack rails and re-attach all cables.
• Power on host and update us remotely so they can begin testing of the new hardware as remote hands works on the next one.

During the work window I'll be online remotely from the USA, so any updates can be sent via the ticket or via text message to +1.727.255.4597 or via email or google hangout to rhalsell@wikimedia.org.

Once the first host has the PCIe Card installed and cables re-attached, please update us so we can begin remote testing to ensure there are no issues while remote hands continues to install PCIe SSDs into the rest of the 8 total hosts.

Would remote hands please review the above directions for clarity or questions and also confirm the work window of 2023-03-27 @ 1100 CET.

Thank you in advance,

Remote hands won't have any ability to power down a host other than by pressing the front power button. It would reduce potential complexity if we power down all the hosts for them to work on to prevent confusion. That way they know if it is powered off and matches the list, they can work on it.

In T355353#9642542, @jcrespo wrote:

I will take care, as I discussed previously with John, but to avoid future mistakes, @RobH is there a way to transmit the desired recipe clearer? I wrote: "Partman recipe and/or desired Raid Level: db.cfg" And I can see that as confusing because of the hostname, so trying to make your team's workflow easier, in any way I can- any tip?

This installation is blocked until patchsets to allow installation are complete. I've removed the assignment from @Jhancock.wm to @jcrespo but once that is complete please assign it back to them.

This installation is blocked until patchsets to allow installation are complete. I've removed the assignment from @VRiley-WMF to @jcrespo but once that is complete please assign it back to them.

My patchset had mistakes, and @jcrespo has advised he is workong on these patchsets. As such, I've abandoned my patchset.

Chatted with @ssingh as I had neglected some items we had discussed previously:

It appears when these were racked they had the incorrect model selected. There was already an R650 in Netbox (that could have been used) but I've also appended in https://netbox.wikimedia.org/dcim/device-types/288/

Awesome, post-offsite please coordinate with @fgiunchedi on when to install these. As the systems are hot swap, it shouldn't cause an issue, but I'd clear a window just in case.

Indeed, my bad! Fixed the incorrect entires and cleared 25 entries off the report.

Is there a file where this mapping is maintained and if so, I can update in the future when I add new device models to netbox?

In T358594#9580737, @Volans wrote:

Cleanup completed, leaving the task open for DCOps to prevent this from happening.

I'm told there is a question on 'can we pull these raid controllers to use elsewhere' and the answer is 'no, or the host you remove it from has no controller.'

Moritz asked me about this, and I have some background. So orders placed in January 2023 via the dell portal for standard configs also included a number of hosts with raid which should not have had raid.

Both disconnects are currently pending with the vendors. EQ's has a ticket submitted directly where CyrusOne is via our account reps. Updates to both will take place on their individual tasks.

Only two sub-tasks open, T350621 and T342239 which are both being taken care of on their own tasks. As such a master tracking task is no longer required and is being resolved.

Please review this racking task for the GPU test host we're ordering for codfw and provide the needed details for network, confirm hostname, etc...

@Andrew: I've assigned this task to you for you to populate the racking details, additionally please add the servers to the site.pp file with the insetup role and their partition info.

When I login to cr2-codfw, I cannot see the serial of the line card in quetion:

As the disconnects will reference contract IDs and disconnect fees, each site's disconnect has bene put to is own S4 space subtask.

@ayounsi: I never saw a blocker come on on this, so we're good to go ahead and disconnect the cross connections at each site for this correct?