Complete build out of Ceph cluster and attach "diskless" cloudvirts
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	May 22 2020, 12:57 AM

Description

Complete the pilot project of hypervisors with Ceph instance storage:

Rack OSD nodes (DCOps)
Provision nodes as OSDs
Join nodes to existing cluster
Assign proper OpenStack roles to cloudvirt10[31-39] to use ceph storage
Drain and decommission cloudvirt100[1-9]
Profit!!

Details

	Subject	Repo	Branch	Lines +/-
	wmcs eqiad1: change ceph nodes to use the 'eqiad1-compute' ceph pool	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	taavi	T211393 openstack-browser and horizon: Security group and floating IP quota information being pulled from Nova instead of Neutron for eqiad1-r
Resolved	Andrew	T211777 Can't get quota information from Neutron API
Resolved	Andrew	T261137 upgrade cloud-vps openstack to Openstack version 'Victoria'
Resolved	dcaro	T261136 upgrade cloud-vps openstack to Openstack version 'Ussuri'
Resolved	Andrew	T261138 Upgrade Horizon to latest OpenStack release
Resolved	Andrew	T261135 upgrade cloud-vps openstack to Openstack version 'Train'
Resolved	Andrew	T261134 upgrade cloud-vps openstack to Openstack version 'Stein'
Resolved	Andrew	T259399 Upgrade cloudvirts to Debian Buster
		Unknown Object (Task)
Resolved	dcaro	T216195 Move cloudvirt hosts to 10Gb ethernet
Resolved	Andrew	T194334 [Epic] Modern Cloud VPS storage layer
Resolved	Andrew	T261132 Move all cloud-vps VMs to Ceph
Resolved	Andrew	T253365 Complete build out of Ceph cluster and attach "diskless" cloudvirts
Resolved	• Cmjohnson	T251627 (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet
Resolved	Jclark-ctr	T276922 cloudvirt1038: PCIe error
Resolved	Andrew	T258619 Decide on network setup for WMCS ceph cluster
Resolved	Andrew	T258826 Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan
Resolved	Andrew	T258968 Migrate existing ceph workload off of cloudcephosd100[1-3] and rebuild with new vlan config
Resolved	Andrew	T259057 Move cloudcephosd100[1-3] to new vlans
Resolved	Andrew	T259192 Investigate/prototype ceph backup options
Resolved	dcaro	T260692 Ceph VM image backups
Resolved	aborrero	T260941 Practice restoring ceph backups
Declined	dcaro	T267195 CloudVPS: improve VM backups to make them redundant and discoverable
Resolved	dcaro	T271094 [dumps] Review backup strategy
Open	None	T273720 [ceph][rbd] Periodically cleanup dangling snapshots
Open	None	T273723 [backups] Periodically cleanup non-handled backups
		Unknown Object (Task)
Resolved	• Cmjohnson	T251619 (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org
Resolved	Jclark-ctr	T251632 (Need By: 2020-06-12) rack/setup/install WMCS 10G switches
Resolved	ayounsi	T258764 Error: 'cloudsw1-c8-eqiad.mgmt.eqiad.wmnet' is not a valid parent for host 'cloudcephosd1004'
Resolved	aborrero	T266749 cloudcephosd1006: down, possible memory error

Event Timeline

bd808 created this task.May 22 2020, 12:57 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 22 2020, 12:57 AM

bd808 triaged this task as High priority.May 22 2020, 12:59 AM

bd808 added subtasks: T251619: (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org, T251627: (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet, T251632: (Need By: 2020-06-12) rack/setup/install WMCS 10G switches.

bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

bd808 renamed this task from Complete build out of Ceph cluster to Complete build out of Ceph cluster and attach "diskless" cloudvirts.May 22 2020, 1:06 AM

bd808 updated the task description. (Show Details)

Andrew closed subtask T258619: Decide on network setup for WMCS ceph cluster as Resolved.Jul 23 2020, 5:59 PM

Andrew closed subtask T258826: Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan as Resolved.Jul 27 2020, 6:16 PM

Andrew reopened subtask T258826: Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan as Open.Jul 28 2020, 12:45 AM

Andrew closed subtask T258826: Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan as Resolved.Jul 28 2020, 6:39 PM

Change 616897 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs eqiad1: change ceph nodes to use the 'eqiad1-compute' ceph pool

https://gerrit.wikimedia.org/r/616897

Change 616897 merged by Andrew Bogott:
[operations/puppet@production] wmcs eqiad1: change ceph nodes to use the 'eqiad1-compute' ceph pool

https://gerrit.wikimedia.org/r/616897

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2020, 8:10 PM

Andrew closed subtask T259057: Move cloudcephosd100[1-3] to new vlans as Resolved.Jul 28 2020, 8:56 PM

• Cmjohnson closed subtask T251619: (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org as Resolved.Jul 29 2020, 3:25 PM

Andrew closed subtask T258968: Migrate existing ceph workload off of cloudcephosd100[1-3] and rebuild with new vlan config as Resolved.Jul 30 2020, 3:04 PM

Andrew closed subtask T251627: (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet as Resolved.Aug 3 2020, 5:49 PM

Andrew closed subtask T259192: Investigate/prototype ceph backup options as Resolved.Aug 19 2020, 4:26 AM

hashar mentioned this in T260916: Move CI instances to use ceph in WMCS.Aug 20 2020, 3:37 PM

Andrew added a parent task: T261132: Move all cloud-vps VMs to Ceph.Aug 24 2020, 3:05 PM

The original description here included draining cloudvirt1001-1009 but I've added that under T261132. All the new hardware referred to in this task is now up and running and host VMs.

Andrew reopened subtask T259057: Move cloudcephosd100[1-3] to new vlans as Open.Aug 31 2020, 8:32 PM

Andrew closed subtask T259057: Move cloudcephosd100[1-3] to new vlans as Resolved.Sep 1 2020, 1:37 PM

Andrew closed subtask T260692: Ceph VM image backups as Resolved.Sep 7 2020, 4:56 PM

• Cmjohnson closed subtask T251632: (Need By: 2020-06-12) rack/setup/install WMCS 10G switches as Resolved.Oct 1 2020, 6:27 PM

Andrew reopened subtask T260692: Ceph VM image backups as Open.Oct 8 2020, 4:44 PM

dcaro closed subtask T260692: Ceph VM image backups as Resolved.Jul 28 2021, 1:48 PM