Page MenuHomePhabricator
Create Task
Maniphest T236394

rack/setup/install bast3004
Closed, ResolvedPublic
Actions

Description

This task will track the racking/setup/installation of bast3004 ordered on task T230620.

Racking Proposal: Setup via the google sheet rack layout.

bast3004:

  • - receive in system on procurement task T230620
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

SubjectRepoBranchLines +/-
add bast3004 to esams prometheus nodesoperations/puppetproduction+2 -0
DHCP: switch esams DHCP server from bast3002 to bast3004operations/puppetproduction+3 -3
install_server: set 'next-server' for bast3004operations/puppetproduction+2 -0
bastionhost: fix migration class fqdn comparison, rename varsoperations/puppetproduction+15 -11
bastionhost: recreate migration class, convert to profileoperations/puppetproduction+40 -0
smokeping: replace bast3002 with bast3004 as targetoperations/puppetproduction+4 -4
hieradata/common: add bast3004 to bastion hosts for firewallsoperations/puppetproduction+2 -0
Add dhcp macaddrs for esams rack 16 hostsoperations/puppetproduction+40 -0
esams: mgmt dns for rack 16operations/dnsmaster+17 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolvedwiki_willyT235805 ESAMS Refresh/Rebuild (October 2019)
Unknown Object (Task)
 ResolvedMoritzMuehlenhoffT236216 rack/setup/install ganeti300[123]
 ResolvedPapaulT236217 rack/setup/install dns300[12]
 ResolvedPapaulT236294 rack/setup/install lvs300[567]
 ResolvedDzahnT236394 rack/setup/install bast3004
 ResolvedRequestPapaulT236329 decommission bast3002

Event Timeline

RobH triaged this task as Medium priority.Oct 24 2019, 3:22 PM
RobH created this task.
Restricted Application added a project: SRE. · View Herald TranscriptOct 24 2019, 3:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
RobH added a parent task: Unknown Object (Task).Oct 24 2019, 3:22 PM
gerritbot added a comment.Oct 24 2019, 3:35 PM

Change 545880 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] esams: mgmt dns for rack 16

https://gerrit.wikimedia.org/r/545880

gerritbot added a project: Patch-For-Review.Oct 24 2019, 3:35 PM

Change 545880 merged by BBlack:
[operations/dns@master] esams: mgmt dns for rack 16

https://gerrit.wikimedia.org/r/545880

RobH unsubscribed.Oct 24 2019, 3:58 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 4:11 PM
gerritbot added a comment.Oct 24 2019, 4:45 PM

Change 545893 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Add dhcp macaddrs for esams rack 16 hosts

https://gerrit.wikimedia.org/r/545893

gerritbot added a project: Patch-For-Review.Oct 24 2019, 4:45 PM
gerritbot added a comment.Oct 24 2019, 4:48 PM

Change 545893 merged by BBlack:
[operations/puppet@production] Add dhcp macaddrs for esams rack 16 hosts

https://gerrit.wikimedia.org/r/545893

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 5:10 PM
Dzahn subscribed.Oct 24 2019, 5:35 PM

merging in duplicate ticket T236409 where i started OS install

Dzahn merged a task: T236409: setup bast3004.Oct 24 2019, 5:35 PM
Dzahn updated the task description. (Show Details)

confirmed mgmt and production DNS exists, mgtm password is set, IPMI over LAN working..

started OS install

Dzahn added a comment.Oct 24 2019, 5:58 PM
[bast3004:~] $ gen_fingerprints
 +---------+---------+-----------------------------------------------------+
 | Cipher  | Algo    | Fingerprint                                         |
 +---------+---------+-----------------------------------------------------+
 | RSA     | SHA-256 | SHA256:/0S1drRdtp6jpPBDq0bb45228GwTdouVTOGNKYDGZ8Y  |
 +---------+---------+-----------------------------------------------------+
 | ECDSA   | SHA-256 | SHA256:vp48DKQHJ0x/cjPZ7MsXe+ChoMZdkVLS0m0xwg832dw  |
 +---------+---------+-----------------------------------------------------+
 | ED25519 | SHA-256 | SHA256:O7bKwLBunrULntnBP7G/z2Stcvk4mjJ3EgNNxqCb8+E  |
 +---------+---------+-----------------------------------------------------+

 +---[RSA 2048]----+ +---[ECDSA 256]---+ +--[ED25519 256]--+
 |       . o       | |         +..o= . | |      .o         |
 |        + E    . | |    .   o *.*.o E| |     .  +        |
 |       . + .  oo*| |   o .   B * .   | |    .  +         |
 |            ..o**| |    + = B = .    | |     o. .        |
 |        S   ..=o+| |     * +S= .     | |  . + ..S        |
 |         o o +.*.| |    . o.. o +    | |   = +..o. .     |
 |        ..=.+.=+.| |     o =.+ + =   | |  o * Eo+oo..    |
 |         ooBB=o..| |      +.+o+ + .  | | o.B *=.+B=o     |
 |        ..o+*Bo  | |     . .=. . .   | | oB +.+B*O=o.    |
 +----[SHA256]-----+ +----[SHA256]-----+ +----[SHA256]-----+
Dzahn updated the task description. (Show Details)Oct 24 2019, 6:00 PM
Dzahn added a subtask: T236329: decommission bast3002.Oct 24 2019, 6:22 PM
Dzahn updated the task description. (Show Details)Oct 24 2019, 6:55 PM
gerritbot added a comment.Oct 24 2019, 7:02 PM

Change 545921 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: replace bast3002 with bast3004 as target

https://gerrit.wikimedia.org/r/545921

gerritbot added a project: Patch-For-Review.Oct 24 2019, 7:02 PM
gerritbot added a comment.Oct 24 2019, 7:06 PM

Change 545922 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] hieradata/common: add bast3004 to bastion hosts for firewalls

https://gerrit.wikimedia.org/r/545922

gerritbot added a comment.Oct 24 2019, 7:31 PM

Change 545922 merged by Dzahn:
[operations/puppet@production] hieradata/common: add bast3004 to bastion hosts for firewalls

https://gerrit.wikimedia.org/r/545922

Dzahn added a parent task: T235805: ESAMS Refresh/Rebuild (October 2019).Oct 24 2019, 7:59 PM
Dzahn added a comment.Oct 24 2019, 8:11 PM

^ made active bastion host with the global firewall change above

created wikitech pages

https://wikitech.wikimedia.org/wiki/Bast3004
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/bast3004.wikimedia.org

Can be used as a jump host now.

Setting to "Active" in netbox.

gerritbot added a comment.Oct 24 2019, 8:17 PM

Change 545921 merged by Dzahn:
[operations/puppet@production] smokeping: replace bast3002 with bast3004 as target

https://gerrit.wikimedia.org/r/545921

Papaul updated the task description. (Show Details)Oct 24 2019, 8:22 PM
Dzahn claimed this task.Oct 24 2019, 8:24 PM
Dzahn added a subscriber: Papaul.

service "bastion host" is ready but service "tftp" still needs to be migrated. taking it.

gerritbot added a comment.Oct 24 2019, 8:46 PM

Change 545935 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] Revert "delete unused bastionhost::migration class"

https://gerrit.wikimedia.org/r/545935

gerritbot added a comment.Oct 24 2019, 10:47 PM

Change 545935 merged by Dzahn:
[operations/puppet@production] bastionhost: recreate migration class, convert to profile

https://gerrit.wikimedia.org/r/545935

gerritbot added a comment.Oct 24 2019, 11:00 PM

Change 545971 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] bastionhost: fix migration class fqdn comparison, rename vars

https://gerrit.wikimedia.org/r/545971

gerritbot added a comment.Oct 24 2019, 11:03 PM

Change 545973 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP: replace bast3002 with bast3004 as next-server

https://gerrit.wikimedia.org/r/545973

gerritbot added a comment.Oct 24 2019, 11:16 PM

Change 545971 merged by Dzahn:
[operations/puppet@production] bastionhost: fix migration class fqdn comparison, rename vars

https://gerrit.wikimedia.org/r/545971

Stashbot added a comment.Oct 24 2019, 11:46 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T23:46:41Z] <mutante> bast3002 - rsyncing /home, /srv/tfptboot and /srv/prometheus to /srv/bast3002/ on bast3004 (T236394 T236329)

Stashbot mentioned this in T236329: decommission bast3002.Oct 24 2019, 11:46 PM
Dzahn added a comment.Oct 25 2019, 12:44 AM
  • home dirs copied to individual $user_bast1002.tar.gz files in each user home (where the user exists on both old and new server) so users have their old files if they need them and can just delete the file if they don't
  • prometheus data saved in /srv/bast3002/prometheus/ (36G) and local prometheus data /srv/prometheus is 1.2G and unchanged
  • tftpdata (installers) rsynced into the actual /srv/tftpboot so both servers are identical there
gerritbot added a comment.Oct 25 2019, 3:21 AM

Change 546001 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] switch esams prometheus node from bast3002 to bast3004

https://gerrit.wikimedia.org/r/546001

gerritbot added a comment.Oct 25 2019, 3:52 AM

Change 546003 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: set 'next-server' for bast3004

https://gerrit.wikimedia.org/r/546003

gerritbot added a comment.Oct 25 2019, 3:55 AM

Change 546003 merged by Dzahn:
[operations/puppet@production] install_server: set 'next-server' for bast3004

https://gerrit.wikimedia.org/r/546003

gerritbot added a comment.Oct 25 2019, 4:12 AM

Change 545973 merged by Dzahn:
[operations/puppet@production] DHCP: switch esams DHCP server from bast3002 to bast3004

https://gerrit.wikimedia.org/r/545973

gerritbot added a comment.Oct 25 2019, 8:35 AM

Change 546001 merged by Filippo Giunchedi:
[operations/puppet@production] add bast3004 to esams prometheus nodes

https://gerrit.wikimedia.org/r/546001

Maintenance_bot removed a project: Patch-For-Review.Oct 25 2019, 9:10 AM
fgiunchedi subscribed.Oct 25 2019, 9:12 AM

Prometheus data sync'd again from bast3002 and copied in place, DNS flipped, Prometheus is live on this host now and not active on bast3002 anymore

Dzahn changed the status of subtask T236329: decommission bast3002 from Stalled to Open.Oct 25 2019, 3:14 PM
Dzahn closed this task as Resolved.Oct 25 2019, 3:20 PM
Dzahn updated the task description. (Show Details)

Thank you, @fgiunchedi!

It's set to active in Netbox and i tested an install to confirm DHCP/tftpboot is working after that was switched too.

Resolving.

Papaul closed subtask T236329: decommission bast3002 as Resolved.Oct 28 2019, 9:32 AM
• ema mentioned this in T236686: bast3004 can't reach mgmt networks .Oct 28 2019, 1:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL