Log In
T257324
Consolidate edge bastion server into ganeti
Closed, ResolvedPublic
Assigned To
MoritzMuehlenhoff
Authored By
BBlack
Jul 7 2020, 2:42 PM
Tags
SRE (Backlog)
Traffic (Done)
Patch-For-Review
Subscribers
Aklapper
BBlack
MoritzMuehlenhoff
Description
Things to look into here and various notes:
Security - are we ok with ssh bastions inside ganeti alongside other public service instances?
Installer issues - DHCP stuff will be ok?
Public networking for edge ganetis is configured?
Details
ProjectSubject
operations/puppetDisable bast3004/bast4002/bast5001 as bastions
operations/puppetUpdate bastions in smokeping config
operations/puppetMake bast4003/bast5002 bastion hosts
operations/puppetAdd bast4003/bast5002
operations/puppetAdd bast3005
Customize query in gerrit
Related Objects
Task Graph
Mentions
StatusAssignedTask
OpenNoneT257323 Consolidate misc servers at edge sites
ResolvedMoritzMuehlenhoffT257324 Consolidate edge bastion server into ganeti
BBlack triaged this task as Medium priority.
Jul 7 2020, 2:42 PM
BBlack created this task.
BBlack updated the task description. (Show Details)
Jul 7 2020, 2:47 PM
ema moved this task from Triage to General on the Traffic board.
Jul 8 2020, 8:43 AM
MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.
Sep 24 2020, 1:24 PM
Security - are we ok with ssh bastions inside ganeti alongside other public service instances?
Sounds fine to me. As long as we have two baremetal bastions in eqiad/codfw which can access all the internal nodes in all edges, I don't see any issue with running the bastions on Ganeti.
Installer issues - DHCP stuff will be ok?
That's sorted, like other instances being installed in Ganeti.
Public networking for edge ganetis is configured?
We have already have edge Ganeti instances with a public IP (e.g. install3001.wikimedia.org), I think the setup is stil WIP; but should be working in general.
BBlack moved this task from General to Epic Wishlist on the Traffic board.
Sep 29 2020, 8:28 PM
gerritbot added a comment.
Jan 11 2021, 4:06 PM
Change 655450 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add bast3005
https://gerrit.wikimedia.org/r/655450
gerritbot added a project: Patch-For-Review.
Jan 11 2021, 4:07 PM
gerritbot added a comment.
Jan 12 2021, 8:28 AM
Change 655450 merged by Muehlenhoff:
[operations/puppet@production] Add bast3005
https://gerrit.wikimedia.org/r/655450
Maintenance_bot removed a project: Patch-For-Review.
Jan 12 2021, 9:10 AM
gerritbot added a comment.
Jan 14 2021, 2:43 PM
Change 656172 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add bast4003/bast5002
https://gerrit.wikimedia.org/r/656172
gerritbot added a project: Patch-For-Review.
Jan 14 2021, 2:43 PM
gerritbot added a comment.
Jan 14 2021, 3:34 PM
Change 656172 merged by Muehlenhoff:
[operations/puppet@production] Add bast4003/bast5002
https://gerrit.wikimedia.org/r/656172
Maintenance_bot removed a project: Patch-For-Review.
Jan 14 2021, 4:10 PM
Stashbot added a comment.
Jan 15 2021, 8:45 AM
Mentioned in SAL (#wikimedia-operations) [2021-01-15T08:45:31Z] <moritzm> installing bast4003 T257324
Stashbot added a comment.
Jan 15 2021, 9:07 AM
Mentioned in SAL (#wikimedia-operations) [2021-01-15T09:07:58Z] <moritzm> installing bast5002 T257324
gerritbot added a comment.
Jan 15 2021, 10:30 AM
Change 656380 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Make bast4003/bast5002 bastion hosts
https://gerrit.wikimedia.org/r/656380
gerritbot added a project: Patch-For-Review.
Jan 15 2021, 10:30 AM
gerritbot added a comment.
Jan 15 2021, 11:59 AM
Change 656380 merged by Muehlenhoff:
[operations/puppet@production] Make bast4003/bast5002 bastion hosts
https://gerrit.wikimedia.org/r/656380
MoritzMuehlenhoff added a comment.
Jan 15 2021, 3:25 PM
I've created new bastions in Ganeti (bast3005, bast4003, bast5002), which are working fine. I'll send out an announcement to the ops list next week and eventually we can free up the former baremetal servers currently serving as bastions (and reduce our setup for the forthcoming second EU data centre).
I also had a look at the current hardware used for bastions:
We don't really need additional capacity in the cache site Ganeti clusters, so my proposal would be to decom the current hardware bastions and keep the server as spares. They could serve as drop in replacements for the DNS/LVS/Ganeti servers in case of something like a mainboard failure (and we wouldn't even need remote hands).
BBlack added a comment.
Jan 15 2021, 4:12 PM
We actually do have some upcoming projects which might necessitate more Ganeti capacity. In general the plan is to move all the non-ganeti DNS boxes into ganeti as well if possible, and to spin up DoH instances in ganeti everywhere as well (which may turn out to need multiple instances and have real scaling issues). But we don't need more capacity there *now* just yet, and so long as they're kept powered up as online spares, we can always deal with the decision to move them into the cluster at a later time.
MoritzMuehlenhoff added a comment.
Jan 15 2021, 4:17 PM
In T257324#6751667, @BBlack wrote:
We actually do have some upcoming projects which might necessitate more Ganeti capacity. In general the plan is to move all the non-ganeti DNS boxes into ganeti as well if possible, and to spin up DoH instances in ganeti everywhere as well (which may turn out to need multiple instances and have real scaling issues). But we don't need more capacity there *now* just yet, and so long as they're kept powered up as online spares, we can always deal with the decision to move them into the cluster at a later time.
Sounds good. We can also easily integrate those into Ganeti later (with reduced weight in ulsfo/eqsin compared to the other Ganeti nodes)
gerritbot added a comment.
Jan 18 2021, 2:20 PM
Change 656894 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Disable bast3004/bast4002/bast5001 as bastions
https://gerrit.wikimedia.org/r/656894
gerritbot added a comment.
Jan 18 2021, 2:29 PM
Change 656895 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Update bastions in smokeping config
https://gerrit.wikimedia.org/r/656895
gerritbot added a comment.
Jan 18 2021, 2:37 PM
Change 656895 merged by Muehlenhoff:
[operations/puppet@production] Update bastions in smokeping config
https://gerrit.wikimedia.org/r/656895
gerritbot added a comment.
Jan 20 2021, 1:15 PM
Change 656894 merged by Muehlenhoff:
[operations/puppet@production] Disable bast3004/bast4002/bast5001 as bastions
https://gerrit.wikimedia.org/r/656894
MoritzMuehlenhoff closed this task as Resolved.
Jan 21 2021, 2:09 PM
MoritzMuehlenhoff claimed this task.
This is done.
Dzahn mentioned this in T273336: decommission bast4002.
Jan 29 2021, 10:53 PM
RobH mentioned this in Unknown Object (Task).
Mar 16 2021, 5:44 PM
MoritzMuehlenhoff mentioned this in T243057: Move Prometheus off eqsin/ulsfo/esams bastions.
Jun 3 2021, 12:28 PM
ssingh mentioned this in T288579: decommission bast4002.wikimedia.org.
Aug 10 2021, 8:42 PM
BBlack moved this task from Epic Wishlist to Done on the Traffic board.
Fri, Oct 8, 5:44 PM
Log In to Comment
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL