Request creation of wmdeanalytics VPS project
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	GoranSMilovanovic
	Feb 9 2022, 5:30 PM

Description

Project Name: wmdeanalytics

Wikitech Usernames of requestors: GoranSMilovanovic

Purpose: WMDE is running the wmde-dashboards CloudVPS project and hosts its Data Science and Analytics solution for Wikidata, Wiktionary, and the WMDE New Editors team from the instances created there. We plan to switch to a new CloudVPS project (wmdeanalytics, requested in this ticket) and then abandon the old one (wmde-dashboards). There are two primary reasons for this migration: (1) we are running our instances on deprecated versions of Debian in wmde-dashboards, and (2) we want to provide training for the members of our team on how to deploy and maintain from scratch the Wikidata Analytics repo because of upcoming changes in our analytics.

Brief description: There are needs for a higher expected disk quota (we are using the /srv instance disk expansion method on one of our instances. We need to host some quite demanding analytical systems. As of the installations, all that we will need will be Docker and Docker Composer once the setup of the instances is finished. In all respects, the new CloudVPS project that we are asking for should have exactly the same properties as the existing wmde-dashboards CloudVPS project.

How soon you are hoping this can be fulfilled: As soon as possible.

Related Objects
Search...

Status	Assigned	Task
Resolved	rook	T301380 Request creation of wmdeanalytics VPS project
Resolved	Andrew	T305829 horizon (or nova) doesn't correctly report duplicated VM names
Resolved	Andrew	T305831 Cloud VPS: evaluate if VM name global uniqueness enforcement can be dropped

Event Timeline

GoranSMilovanovic created this task.Feb 9 2022, 5:30 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 9 2022, 5:30 PM

Request looks good to me. +1

project created

openstack project create --description 'wmdeanalytics' wmdeanalytics --domain default
openstack role add --project wmdeanalytics --user goransm projectadmin
openstack role add --project wmdeanalytics --user goransm user
openstack quota set --cores 24 wmdeanalytics
openstack quota set --ram 49152 wmdeanalytics

rook closed this task as Resolved.Feb 9 2022, 7:02 PM

GoranSMilovanovic added a comment.Feb 11 2022, 3:38 PM

This comment was removed by GoranSMilovanovic.

Hey @aborrero @mdipietro thank you!

One question: is it possible to add more users to the newly created wmdeanalytics project? On the old wmde-dashboards project I did not have sufficient rights to add new users from Horizon.

@Tobi_WMDE_SW and @Manuel should let us know of the Wikitech/LDAP user names of the WMDE developers who will be introduced to Wikidata Analytics, and that developers should also have access to wmdeanalytics.

Can do

root@cloudcontrol1003:~# openstack role add --project wmdeanalytics --user manuel-wmde projectadmin
root@cloudcontrol1003:~# openstack role add --project wmdeanalytics --user manuel-wmde user
root@cloudcontrol1003:~# openstack role add --project wmdeanalytics --user tgritschacher user
root@cloudcontrol1003:~# openstack role add --project wmdeanalytics --user tgritschacher projectadmin
root@cloudcontrol1003:~# openstack role assignment list --project wmdeanalytics
+----------------------------------+---------------+-------+---------------+--------+--------+-----------+
| Role                             | User          | Group | Project       | Domain | System | Inherited |
+----------------------------------+---------------+-------+---------------+--------+--------+-----------+
| 4d8cad783d6342efa8414d7d36fbc034 | goransm       |       | wmdeanalytics |        |        | False     |
| f473273fac7146b3bdbf22e5d4504f95 | goransm       |       | wmdeanalytics |        |        | False     |
| 4d8cad783d6342efa8414d7d36fbc034 | manuel-wmde   |       | wmdeanalytics |        |        | False     |
| f473273fac7146b3bdbf22e5d4504f95 | manuel-wmde   |       | wmdeanalytics |        |        | False     |
| 4d8cad783d6342efa8414d7d36fbc034 | tgritschacher |       | wmdeanalytics |        |        | False     |
| f473273fac7146b3bdbf22e5d4504f95 | tgritschacher |       | wmdeanalytics |        |        | False     |
+----------------------------------+---------------+-------+---------------+--------+--------+-----------+

done

rook closed this task as Resolved.Feb 11 2022, 3:55 PM

Hi @rook @Orteipidm

Could you please add the following LDAP users to wmdeanalytics:

karapayneWMDE
addshore
itamar

Addshore subscribed.Feb 24 2022, 5:45 PM

rook claimed this task.Feb 24 2022, 5:50 PM

For reference that is:
https://ldap.toolforge.org/user/kpayne
https://ldap.toolforge.org/user/itamar
https://ldap.toolforge.org/user/addshore

C'est fait!

root@cloudcontrol1003:~# for i in kpayne itamar addshore ; do openstack role add --project wmdeanalytics --user $i projectadmin ; openstack role add --project wmdeanalytics --user $i user ; done
root@cloudcontrol1003:~# openstack role assignment list --project wmdeanalytics
+----------------------------------+---------------+-------+---------------+--------+--------+-----------+
| Role                             | User          | Group | Project       | Domain | System | Inherited |
+----------------------------------+---------------+-------+---------------+--------+--------+-----------+
| 4d8cad783d6342efa8414d7d36fbc034 | addshore      |       | wmdeanalytics |        |        | False     |
| f473273fac7146b3bdbf22e5d4504f95 | addshore      |       | wmdeanalytics |        |        | False     |
| 4d8cad783d6342efa8414d7d36fbc034 | goransm       |       | wmdeanalytics |        |        | False     |
| f473273fac7146b3bdbf22e5d4504f95 | goransm       |       | wmdeanalytics |        |        | False     |
| 4d8cad783d6342efa8414d7d36fbc034 | itamar        |       | wmdeanalytics |        |        | False     |
| f473273fac7146b3bdbf22e5d4504f95 | itamar        |       | wmdeanalytics |        |        | False     |
| 4d8cad783d6342efa8414d7d36fbc034 | kpayne        |       | wmdeanalytics |        |        | False     |
| f473273fac7146b3bdbf22e5d4504f95 | kpayne        |       | wmdeanalytics |        |        | False     |
| 4d8cad783d6342efa8414d7d36fbc034 | manuel-wmde   |       | wmdeanalytics |        |        | False     |
| f473273fac7146b3bdbf22e5d4504f95 | manuel-wmde   |       | wmdeanalytics |        |        | False     |
| 4d8cad783d6342efa8414d7d36fbc034 | tgritschacher |       | wmdeanalytics |        |        | False     |
| f473273fac7146b3bdbf22e5d4504f95 | tgritschacher |       | wmdeanalytics |        |        | False     |
+----------------------------------+---------------+-------+---------------+--------+--------+-----------+

rook closed this task as Resolved.Feb 24 2022, 5:56 PM

@rook @Orteipidm @ItamarWMDE @Addshore

In the old CloudVPS project wmde-dashboards I was using an g2.cores8.ram16.disk160 instance.

That instance is not even available in the new CloudVPS project wmdeanalytics.

I have spawned a g3.cores8.ram16.disk20 instead; however, the Task status is Scheduling for 31 minutes already.

Please advise.

I tried launching another, seemed to startup. Try removing the one stuck in scheduling and rebuilding it. I'll remove my test VM

One hour, eight minutes right now @rook

I have deleted the old instance and tried to spawn a new one as suggested.

@rook We would really need to find out where is the problem with spinning up virtual instances in this project, and rather quickly (sorry; I am leaving my position as a Data Scientist for Wikidata and the WMDE team needs to take over my responsibilities, and we are already late in that respect, so timing is crucial here).

@Tobi_WMDE_SW @Manuel @ItamarWMDE

The plan for this - wmdeanalytics - CloudVPS project is to have:

one g3.cores16.ram36.disk20 instance (16 cores, ~36Gb RAM) instance for Wikidata Analytics
one g3.cores4.ram8.disk20 instance (4 cores, 8Gb RAM) instance for Wiktionary Cognate, and
one g3.cores2.ram4.disk20 instance (2 cores, 4Gb RAM) instance for the WMDE New Editors team.

The g3.cores16.ram36.disk20 instance for Wikidata Analytics will have to be setup so as to use additional disk space.

@GoranSMilovanovic as requested in IRC, can you please post a screenshot of the settings used to create that instance? It might give us a lead as to what could cause this problem.

@ItamarWMDE the screenshot is already shared ☝️

Hey, thanks to @Majavah who did some internal inspection, we believe the problem is that there is already a virtual machine with the same name somewhere in Cloud VPS.

We enforce VM names to be unique across all the service. The bug I see here is that there wasn't a proper error reporting from the system to you, to let you know how to proceed. We will track this problem on a separated ticket.

aborrero mentioned this in T305829: horizon (or nova) doesn't correctly report duplicated VM names.Apr 11 2022, 10:58 AM

Thank you @aborrero.

@ItamarWMDE @Tobi_WMDE_SW @Manuel

I hope you aware of the fact that changing the instance name in CloudVPS implies the change in its URL, e.g.

current URL for Wikidata Analytics is https://wikidata-analytics.wmcloud.org/
because the name of the CloudVPS instance in the wmde-dashboards project is wikidata-analytics.

@Manuel This is especially relevant for you since you have placed an (understandable, rational) demand to always keep the old URLs alive.
@ItamarWMDE Do you think there is something that we can do about this?

ItamarWMDE added a project: User-ItamarWMDE.Apr 11 2022, 12:14 PM

In T301380#7844583, @GoranSMilovanovic wrote:

Thank you @aborrero.

@ItamarWMDE @Tobi_WMDE_SW @Manuel

I hope you aware of the fact that changing the instance name in CloudVPS implies the change in its URL, e.g.

current URL for Wikidata Analytics is https://wikidata-analytics.wmcloud.org/

because the name of the CloudVPS instance in the wmde-dashboards project is wikidata-analytics.

@Manuel This is especially relevant for you since you have placed an (understandable, rational) demand to always keep the old URLs alive.
@ItamarWMDE Do you think there is something that we can do about this?

This is not correct. I think you are confusing proxy URL/name with virtual machine name.

The URL you mention is a front proxy URL that can point to whatever backend virtual machine you choose.

In other words, you can have today:

https://wikidata-analytics.wmcloud.org (proxy address) --> wikidata-analytics.wmde-dashboards.eqiad1.wikimedia.cloud (virtual machine)

and tomorrow:

https://wikidata-analytics.wmcloud.org (proxy address) --> other-vm.other-project.eqiad1.wikimedia.cloud (virtual machine)

If you can't do this, then this is a different bug.

@aborrero Thank you.

What we really need is:

any_cloudVPS_instance --> https:://whatever_we_want.wmcloud.org

and I am not sure if that is possible or not, but I am sure @ItamarWMDE will help me sort it out.

Happy to help, I think this can be achieved through the DNS Records → Web Proxies section in Horizon. Let's try this together after spinning up the instance under a different name.

ItamarWMDE moved this task from Radar to Radar on the User-ItamarWMDE board.Apr 11 2022, 3:11 PM

aborrero closed this task as Resolved.Apr 11 2022, 4:00 PM

GoranSMilovanovic mentioned this in T251792: Wiktionary Cognate Dashboard not updated.Apr 12 2022, 1:05 PM

Andrew closed subtask T305831: Cloud VPS: evaluate if VM name global uniqueness enforcement can be dropped as Resolved.Aug 2 2022, 7:08 PM

Andrew closed subtask T305829: horizon (or nova) doesn't correctly report duplicated VM names as Resolved.Dec 2 2022, 12:38 AM

Request creation of wmdeanalytics VPS projectClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Request creation of wmdeanalytics VPS project
Closed, ResolvedPublic
Actions

Related Objects
Search...