Page MenuHomePhabricator
Create Task
Maniphest T236437

(Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of 36 new mw systems ordered via T228214.

Racking Proposal: Due to the recent order of additional mw hosts on upcoming task T233639, the racking for this needs to be re-evaluated by @Joe and team to ensure they get the best placement for this batch of 36 and the other batch of 23 systems incoming. This section will be updated by them.

Individual server setup checklist (this checklist should be duplicated for each system):

  • - receive in system on procurement task T228214.
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementation steps by service owners -> https://wikitech.wikimedia.org/wiki/Application_servers#Adding_a_new_server_into_production
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedDzahnT236437 (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Joe added a comment.Nov 5 2019, 6:21 AM

Sorry I'm lagging behind on this. Point is, I shouldn't be the only person to answer this question but most of my team is out this week. I'll try to reassign this to someone with time to respond.

jijiki added a project: serviceops.Nov 20 2019, 3:30 PM
wiki_willy added a comment.Nov 21 2019, 10:59 PM

@Joe et al - just following up on this, since we've had these servers onsite for a while now. Also, once you have any specific racking instructions for us, please keep in mind that we'll need some time to complete all the tasks associated with getting these hosts set up, so it should be reflected in your requested "need by date" as well.

Thanks,
Willy

wiki_willy renamed this task from rack/setup/install mw13[49-84].eqiad.wmnet to (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet.Nov 22 2019, 8:57 PM
jijiki subscribed.Nov 22 2019, 10:07 PM

@wiki_willy I will provide racking instructions on Monday for you, sorry we have delayed you this much.

wiki_willy added a comment.Nov 22 2019, 10:08 PM

Thanks @jijiki , much appreciated

jijiki added a comment.Nov 25 2019, 9:21 PM

Currently:

RackA 5A6A 7B 6B 7C 6D 4D 5
mw servers6617216306 (decom)30 (decom)

We will decommission 36 servers from D 4 and D 5.

I recommend we add:

  • D 1: 10 servers
  • D 2: 6 servers
  • D 6: 10 servers
  • D 8: 10 servers

Regarding the 23 upcoming ones, we will split them between rows A, B, C. Please let me know if that rack setup makes sense. Thank you!

jijiki added a comment.Dec 5 2019, 4:46 PM

@Jclark-ctr Do we have an estimate when we are able to have those servers racked?

wiki_willy added a comment.Dec 5 2019, 5:24 PM

Hi @jijiki - we received these on Sept 25, but just recently received the racking instructions from your team on Nov 25, so there might be some shuffling around we need to do, before getting these racked to meet the recommendations. When do you need these handed over to you by? And we can try to meet that need by date. Thanks, Willy

jijiki added a comment.Dec 5 2019, 5:29 PM

@wiki_willy given we are responsible for this delay, we would like to check with your team and tell us what is the earliest we can do, and we can in turn can plan around that. Sorry for the delays on our end. I would like to avoid requesting a date that is unrealistic, given that xmas is round the corner. Thank you!

wiki_willy added a comment.Dec 5 2019, 5:30 PM

Thanks for confirming @jijiki - I'll let @Jclark-ctr provide an ETA on them, when he gets in a bit later today. Thanks, Willy

Jclark-ctr added a comment.Dec 5 2019, 5:42 PM

@jijiki If no surprises i could have them Racked by Dec 20th

jijiki added a comment.Dec 5 2019, 5:43 PM

That would be lovely, thank you!

wiki_willy renamed this task from (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet to (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.Dec 5 2019, 5:46 PM
Papaul subscribed.Dec 6 2019, 1:18 AM

@Jclark-ctr Please see below for available mgmt IP's that you can use for those servers. Once you have the asset tags please update the table with the information and we can enter the the name, ip and asset tag in the DNS file. Let me know if you have any questions. Thanks

mgmt IP's

HostIPAsset tag
mw134910.65.1.24
mw135010.65.1.25
mw135110.65.2.6
mw135210.65.2.7
mw135310.65.2.10
mw135410.65.2.14
mw135510.65.2.17
mw135610.65.2.18
mw135710.65.2.19
mw135810.65.2.21
mw135910.65.2.26
mw136010.65.2.33
mw136110.65.2.34
mw136210.65.2.35
mw136310.65.2.42
mw136410.65.2.69
mw136510.65.2.70
mw136610.65.2.109
mw136710.65.2.119
mw136810.65.2.120
mw136910.65.2.121
mw137010.65.2.129
mw137110.65.2.130
mw137210.65.2.131
mw137310.65.2.132
mw137410.65.2.133
mw137510.65.2.134
mw137610.65.2.135
mw137710.65.3.15
mw137810.65.3.25
mw137910.65.3.27
mw138010.65.3.28
mw138110.65.3.30
mw138210.65.3.31
mw138310.65.3.32
mw138410.65.3.33
Jclark-ctr added a comment.Dec 17 2019, 10:24 PM

@jijiki

I recommend we use these racks we are limited in space . In row D and would not have the enough sfp to rj45 to accommodate the putting these in 10g racks
D 1: 15 servers
D 3: 3 servers
D 6: 15 servers
D 8: 3 servers

jijiki added a comment.Dec 18 2019, 10:59 AM

@Jclark-ctr if you feel this will work better, we are happy. Either way, this racking is still better than the original one (30 servers in D 5). Thank you!

Jclark-ctr added a comment.Dec 20 2019, 11:59 PM

@jijiki small delay had a few tickets become urgent. Being a contractor i will be in week after christmas and will wrap them up at beginning of new year.

jijiki added a comment.Jan 9 2020, 12:31 PM

Hey @Jclark-ctr ! Do we have an update about when we will have those servers ready? Thank you!

jijiki added a subscriber: • Cmjohnson.EditedJan 22 2020, 4:21 PM

@Jclark-ctr @Cmjohnson @wiki_willy please let us know the status of racking these servers and update the task accordingly, thank you!

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Jan 23 2020, 8:47 AM
Jclark-ctr updated the task description. (Show Details)

Host racked bios, ip , and password set. Needs dns

server ip asset tag rack switch port
mw1349 10.65.1.24 WMF5291 D1 42
mw1350 10.65.1.25 WMF5292 D1 41
mw1351 10.65.2.6 WMF5293 D1 40
mw1352 10.65.2.7 WMF5294 D1 39
mw1353 10.65.2.10 WMF5295 D1 38
mw1354 10.65.2.14 WMF5296 D1 37
mw1355 10.65.2.17 WMF5297 D1 36
mw1356 10.65.2.18 WMF5298 D1 35
mw1357 10.65.2.19 WMF5299 D1 33
mw1358 10.65.2.21 WMF5300 D1 34
mw1359 10.65.2.26 WMF5301 D1 29
mw1360 10.65.2.33 WMF5302 D1 30
mw1361 10.65.2.34 WMF5303 D1 31
mw1362 10.65.2.35 WMF5304 D1 32
mw1363 10.65.2.42 WMF5305 D3 40
mw1364 10.65.2.69 WMF5306 D3 41
mw1365 10.65.2.70 WMF5307 D3 42
mw1366 10.65.2.109 WMF5308 D6 27
mw1367 10.65.2.119 WMF5309 D6 26
mw1368 10.65.2.120 WMF5310 D6 25
mw1369 10.65.2.121 WMF5311 D6 24
mw1370 10.65.2.129 WMF5312 D6 23
mw1371 10.65.2.130 WMF5313 D6 22
mw1372 10.65.2.131 WMF5314 D6 21
mw1373 10.65.2.132 WMF5315 D6 20
mw1374 10.65.2.133 WMF5316 D6 19
mw1375 10.65.2.134 WMF5317 D6 18
mw1376 10.65.2.135 WMF5318 D6 17
mw1377 10.65.3.15 WMF5319 D6 16
mw1378 10.65.3.25 WMF5320 D6 15
mw1379 10.65.3.27 WMF5321 D6 14
mw1380 10.65.3.28 WMF5322 D6 13
mw1381 10.65.3.30 WMF5323 D6 12
mw1382 10.65.3.31 WMF5324 D6 11
mw1383 10.65.3.32 WMF5325 D8 25
mw1384 10.65.3.33 WMF5326 D8 26

Joe raised the priority of this task from Medium to High.Feb 4 2020, 7:53 AM

Can we please expedite this?

We really need these servers to join rotation.

• Cmjohnson added a comment.Feb 4 2020, 5:30 PM

@Joe working on these next...will have to you in next day or 2.

wiki_willy mentioned this in T241849: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet.Feb 5 2020, 5:08 PM
mark subscribed.Feb 12 2020, 4:47 PM

@wiki_willy With Chris having been ill the past few days, what's a realistic new ETA for this?

wiki_willy added a comment.Feb 12 2020, 5:08 PM

@mark - just chatted with Chris and he's working on them now (he's back today after being sick), so ETA is end of day. Thanks, Willy

RobH unsubscribed.Feb 12 2020, 5:32 PM
gerritbot added a comment.Feb 12 2020, 7:11 PM

Change 571782 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns for mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/571782

gerritbot added a project: Patch-For-Review.Feb 12 2020, 7:11 PM
gerritbot added a comment.Feb 12 2020, 7:23 PM

Change 571782 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns for mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/571782

gerritbot added a comment.Feb 12 2020, 7:51 PM

Change 571785 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/571785

gerritbot added a comment.Feb 12 2020, 7:54 PM

Change 571785 merged by Cmjohnson:
[operations/dns@master] Adding production dns mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/571785

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2020, 8:10 PM
gerritbot added a comment.Feb 12 2020, 8:44 PM

Change 571797 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding macs mw servers. Missing mw1351,79-81 for bad passwd

https://gerrit.wikimedia.org/r/571797

gerritbot added a project: Patch-For-Review.Feb 12 2020, 8:44 PM
• Cmjohnson updated the task description. (Show Details)Feb 12 2020, 8:45 PM

@Jclark-ctr when you're at data center later today could you look into these please.

mw1351 and mw1381 have the wrong password set
mw1379 and mw1380 are not getting a connection, most likely cable related.

gerritbot added a comment.Feb 12 2020, 8:55 PM

Change 571797 merged by Cmjohnson:
[operations/puppet@production] Adding macs mw servers. Missing mw1351,79-81 for bad passwd

https://gerrit.wikimedia.org/r/571797

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2020, 9:10 PM
wiki_willy added a comment.Feb 12 2020, 11:40 PM

Latest update: there were some settings that need to be redone, so @Cmjohnson will get these fixed tomorrow, with Thursday as the new ETA. Apologies for the long delay. Thanks, Willy

gerritbot added a comment.Feb 13 2020, 4:57 PM

Change 572031 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac to dhcpd for mw1351

https://gerrit.wikimedia.org/r/572031

gerritbot added a project: Patch-For-Review.Feb 13 2020, 4:57 PM
gerritbot added a comment.Feb 13 2020, 5:00 PM

Change 572031 merged by Cmjohnson:
[operations/puppet@production] Adding mac to dhcpd for mw1351

https://gerrit.wikimedia.org/r/572031

Maintenance_bot removed a project: Patch-For-Review.Feb 13 2020, 5:10 PM
gerritbot added a comment.Feb 13 2020, 6:53 PM

Change 572066 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac/dhcpd file entry for mw1379[80-81]

https://gerrit.wikimedia.org/r/572066

gerritbot added a project: Patch-For-Review.Feb 13 2020, 6:53 PM

Change 572066 merged by Cmjohnson:
[operations/puppet@production] Adding mac/dhcpd file entry for mw1379[80-81]

https://gerrit.wikimedia.org/r/572066

Maintenance_bot removed a project: Patch-For-Review.Feb 13 2020, 7:10 PM
• Cmjohnson reassigned this task from • Cmjohnson to Joe.Feb 13 2020, 7:47 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson removed a project: ops-eqiad.

@Joe These are ready for you now, initial puppet run has been completed. I am removing from ops-eqiad tag. If you need any additional on-site work please add tag back and ping me. assigning to you

Joe removed Joe as the assignee of this task.Feb 18 2020, 7:00 AM
Joe added subscribers: RLazarus, Dzahn.

@RLazarus and @Dzahn can you pick up the task of putting these into rotation?

They will eventually replace mw1221-mw1258, and I think we can go 1:1 in terms of replacing Appservers and Apis. Let's keep an eye on maintaining (or reinstating!) the per-row balance of both clusters.

gerritbot added a comment.Feb 18 2020, 9:01 PM

Change 572975 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add new eqiad mw appservers with spare role

https://gerrit.wikimedia.org/r/572975

gerritbot added a project: Patch-For-Review.Feb 18 2020, 9:01 PM
gerritbot added a comment.Feb 18 2020, 9:20 PM

Change 572975 merged by Dzahn:
[operations/puppet@production] add new eqiad mw appservers with spare role

https://gerrit.wikimedia.org/r/572975

gerritbot added a comment.Feb 18 2020, 10:00 PM

Change 572992 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] Site: Assign mw1349 as mediawiki::appserver

https://gerrit.wikimedia.org/r/572992

gerritbot added a comment.Feb 18 2020, 10:02 PM

Change 572992 merged by RLazarus:
[operations/puppet@production] site: Assign mw1349 as mediawiki::appserver

https://gerrit.wikimedia.org/r/572992

Maintenance_bot removed a project: Patch-For-Review.Feb 18 2020, 10:11 PM
gerritbot added a comment.Feb 18 2020, 10:37 PM

Change 573002 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[labs/private@master] Add fake keys for mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/573002

gerritbot added a project: Patch-For-Review.Feb 18 2020, 10:37 PM
gerritbot added a comment.Feb 18 2020, 11:02 PM

Change 573002 merged by RLazarus:
[labs/private@master] Add fake keys for mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/573002

RLazarus mentioned this in rLPRI38addf7a0d35: Add fake keys for mw13[49-84].eqiad.wmnet.Feb 18 2020, 11:04 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 18 2020, 11:10 PM
gerritbot added a comment.Feb 19 2020, 12:15 AM

Change 573019 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] assign mw1350 through mw1355 as MediaWiki appservers

https://gerrit.wikimedia.org/r/573019

gerritbot added a project: Patch-For-Review.Feb 19 2020, 12:15 AM
gerritbot added a comment.Feb 19 2020, 12:35 AM

Change 573019 merged by Dzahn:
[operations/puppet@production] assign mw1350 through mw1355 as MediaWiki appservers

https://gerrit.wikimedia.org/r/573019

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 1:11 AM
Dzahn added a comment.Feb 19 2020, 1:30 AM

The first 7: mw1349 through mw1355 have been added as regular appservers and are pooled now. But just with weight 10. We will change weights and add more (API) appservers tomorrow.

jijiki added a comment.Feb 19 2020, 10:17 AM

Weights of mw1349-mw1355 were switched to 30

gerritbot added a comment.Feb 19 2020, 6:49 PM

Change 573352 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] site: Assign mw13[56-62] as mediawiki::appserver::api.

https://gerrit.wikimedia.org/r/573352

gerritbot added a project: Patch-For-Review.Feb 19 2020, 6:49 PM
gerritbot added a comment.Feb 19 2020, 6:51 PM

Change 573352 merged by RLazarus:
[operations/puppet@production] site: Assign mw13[56-62] as mediawiki::appserver::api.

https://gerrit.wikimedia.org/r/573352

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 7:10 PM
Dzahn added a comment.Feb 19 2020, 9:04 PM

etherpad with role layout and status: https://etherpad.wikimedia.org/p/T236437

ops-monitoring-bot added a comment.Feb 19 2020, 9:31 PM

Icinga downtime for 1:00:00 set by rzl@cumin1001 on 11 host(s) and their services with reason: new installs

mw[1364-1373,1384].eqiad.wmnet
ops-monitoring-bot added a comment.Feb 19 2020, 9:54 PM

Icinga downtime for 1:00:00 set by rzl@cumin1001 on 11 host(s) and their services with reason: new installs

mw[1364-1373,1384].eqiad.wmnet
gerritbot added a comment.Feb 19 2020, 10:38 PM

Change 573402 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftooldata: add more new eqiad MW API appservers

https://gerrit.wikimedia.org/r/573402

gerritbot added a project: Patch-For-Review.Feb 19 2020, 10:39 PM
gerritbot added a comment.Feb 19 2020, 10:45 PM

Change 573402 merged by Dzahn:
[operations/puppet@production] site/conftooldata: add more new eqiad MW API appservers

https://gerrit.wikimedia.org/r/573402

ops-monitoring-bot added a comment.Feb 19 2020, 10:49 PM

Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 11 host(s) and their services with reason: new_install

mw[1363,1374-1383].eqiad.wmnet
Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 11:11 PM
ops-monitoring-bot added a comment.Feb 19 2020, 11:13 PM

Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 11 host(s) and their services with reason: new_install

mw[1363,1374-1383].eqiad.wmnet
Dzahn claimed this task.Feb 20 2020, 12:04 AM
Dzahn updated the task description. (Show Details)Feb 20 2020, 12:13 AM

All hosts have roles, weight 30, have been pooled and changed to status "Active" in Netbox.

{"mw1349.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1349.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1350.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1350.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1351.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1351.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1352.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1352.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1353.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1353.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1354.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1354.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1355.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1355.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1356.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1356.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1357.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1357.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1358.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1358.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1359.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1359.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1360.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1360.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1361.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1361.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1362.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1362.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1363.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1363.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1364.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1364.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1365.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1365.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1366.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1366.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1367.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1367.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1368.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1368.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1369.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1369.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1370.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1370.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1371.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1371.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1372.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1372.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1373.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1373.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1374.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1374.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1375.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1375.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1376.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1376.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1377.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1377.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1378.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1378.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1379.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1379.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1380.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1380.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1381.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1381.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1382.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1382.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1383.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1383.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1384.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1384.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
Dzahn closed this task as Resolved.Feb 20 2020, 12:15 AM
elukey subscribed.Feb 23 2020, 4:53 PM

As FYI I just powercycled mw1372 since it was frozen (no ssh, no mgmt serial console usable, etc..).

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL