Log In
T204506
cloudvps: maps project trusty deprecation
Closed, ResolvedPublic
Assigned To
Andrew
Authored By
Krenair
Sep 17 2018, 12:31 PM
Tags
Maps (All map-related tasks)
Cloud-VPS (Ubuntu Trusty Deprecation) (VPS Projects)
User-TheDJ (Inbox)
cloud-services-team (Kanban) (Inbox)
Subscribers
aborrero
Aklapper
akosiaris
Andrew
aude
Awjrichards
bd808
View All 32 Subscribers
Tokens
Description
Ubuntu Trusty is no longer available in Cloud VPS since Nov 2017 for new instances. However, the EOL of Trusty is approaching in 2019 and we need to move to Debian Stretch before that date.
All instances in the maps project needs to upgrade as soon as possible.
The list of affected VMs is:
maps-tiles2.maps.eqiad.wmflabs
maps-tiles3.maps.eqiad.wmflabs
maps-warper2.maps.eqiad.wmflabs
maps-wma1.maps.eqiad.wmflabs
Listed administrator are:
More info in openstack browser: https://tools.wmflabs.org/openstack-browser/project/maps
TODO:
figure out the current configuration for the servers
create new Debian 9 (Stretch) VMs for each and configure puppet client
Created maps-tiles1 instance and a maps-puppetmaster instance to experiment with a puppet config
maps-tiles1 is mapped to maps.wmflabs.org (previously unused)
setup puppet
create new manifests for new versions of packages and configurations in puppet and deploy
transfer any data that needs to be transfered for proper tile server operation (likely mostly via nfs ?)
test that all functionality is transferred and that everything works ok
shut down old servers, turn on new ones
delete old servers
Details
Project
Subject

operations/puppet
nfs: add another VM to the Maps nfs mount
Customize query in gerrit
Related Objects
Task Graph
Mentions
Duplicates
StatusAssignedTask
ResolvedAndrewT204506 cloudvps: maps project trusty deprecation
ResolvedChippyyT211149 Ubuntu to Debian migration for maps-warper2 instance of the wikimaps warper
ResolvedTheDJT212596 Allow tiles1.wmflabs.org (or possibly entire eqiad-r?) access to osm postgres
Resolvedbd808T217992 Unable to delete "tiles.wmflabs.org" proxy entry via horizon
There are a very large number of changes, so older changes are hidden. Show Older Changes
TheDJ added a comment.
Dec 12 2018, 1:09 PM
I have updated https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Technology_stack with some information about the current tiles servers as far as I could deconstruct from looking at the instances. If anyone has any further information, that would be appreciated.
TheDJ added a comment.
Edited · Dec 12 2018, 1:36 PM
It seems overpass-wiki instance is not in use. Was once made by Jotpe in 2015. I've sent out an email via his wikimedia wiki account. I suggest we delete it if there is no response before the 18th.
TheDJ added a comment.
Dec 12 2018, 1:39 PM
And i've sent out an email on maps-l and wikitech-l
TheDJ added a project: User-TheDJ.
Dec 12 2018, 3:28 PM
Krenair mentioned this in T212008: upgrade Cloud VPS servers hosting tiles.wmflabs.org, wma.wmflabs.org, overpass-wiki.
Dec 14 2018, 8:17 PM
Krenair merged a task: T212008: upgrade Cloud VPS servers hosting tiles.wmflabs.org, wma.wmflabs.org, overpass-wiki.
Dec 14 2018, 8:19 PM
Krenair updated the task description. (Show Details)
Krenair added subscribers: Sasheto, sbassett, Zache and 5 others.
Sasheto added a comment.
Dec 14 2018, 8:29 PM
I'm willing to attempt the OS conversion on the VM's but at this point I don't have shell access to the tools project ( apparently infrastructure access is not needed - sorry I'm new... ) and I also need admin access to the maps project. I don't know how to go about getting this but participating in this discussion was suggested as a means, so I'm just leaving this comment in here for now.
dschwen added a comment.
Dec 14 2018, 8:41 PM
I'm working on redoing the maps-wma1 instance as maps-wma. This involves a region change and as a consequecnce it seems the /mnt/nfs/labstore1003-maps directory, which contains my home directory on the old instance is empty on the new instance. Same goes for the project directory. Will I have to copy everything over? Why is there no home dir?
Andrew added a comment.
Dec 14 2018, 8:52 PM
In T204506#4824815, @dschwen wrote:
Why is there no home dir?
For murky historical reasons, mounting NFS on a new VM in that project requires a puppet patch. I'll make one for maps-wma now. Are there others that need the same?
gerritbot added a subscriber: gerritbot.
Dec 14 2018, 8:53 PM
Change 479764 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nfs: add another VM to the Maps nfs mount
https://gerrit.wikimedia.org/r/479764
gerritbot added a project: Patch-For-Review.
Dec 14 2018, 8:53 PM
Change 479764 merged by Andrew Bogott:
[operations/puppet@production] nfs: add another VM to the Maps nfs mount
https://gerrit.wikimedia.org/r/479764
Andrew added a comment.
Dec 14 2018, 8:56 PM
@dschwen, if you reboot maps-wma your nfs mounts should be more like what you'd expect now.
zhuyifei1999 removed a subscriber: zhuyifei1999.
Dec 15 2018, 8:22 AM
Chippyy added a comment.
Dec 17 2018, 6:55 PM
maps-warper2 has been migrated to maps-warper3 and web proxy (warper.wmflabs.org) switched too, with everything seeming to work okay, but I'd like at least a day before we turn off the old instance just in case...
TheDJ added a comment.
Dec 17 2018, 7:59 PM
@Chippyy can you please document the setup somewhere for the future ? That would be super helpful.
TheDJ added a comment.
Dec 17 2018, 8:08 PM
@Sasheto hi, I haven't forgotten about your request... Please don't take this the wrong way, but lately there has been quite a LOT of malicious activity and your account is brand new.. And I can't find any history of your activity with the projects... At this time that makes me uncomfortable to give you access to this set of servers as one of them is actually rather critical and could allow a person to do a lot of harm to wikimedians. I'm considering how we should approach this.
Sasheto added a comment.
Dec 17 2018, 8:18 PM
@TheDJ Thanks for letting me know and no, I'm not taking it personally. You don't find any history because I'm super new and don't have any contributions yet. Let me know if you have something in mind that I could do to gain the trust of this community prior to doing more critical tasks.
Chippyy added a comment.
Dec 17 2018, 10:19 PM
In T204506#4828954, @TheDJ wrote:
@Chippyy can you please document the setup somewhere for the future ? That would be super helpful.
created T212166 to track this.
TheDJ updated the task description. (Show Details)
Dec 18 2018, 3:21 PM
TheDJ added a comment.
Edited · Dec 19 2018, 2:24 PM
Current progress:
Chippyy added a comment.
Dec 19 2018, 7:40 PM
I have shut down the maps-warper2 instance
(I'd like to keep it around for a week or so before deletion though, just in case)
aborrero added a comment.
Dec 21 2018, 11:38 AM
Hi! Thanks for your work on this.
FYI since the deadline already passed, we agreed on shutting down remaining Trusty instances on 2019-01-18. More info at https://wikitech.wikimedia.org/wiki/News/Trusty_deprecation#Cloud_VPS_projects
It would be great if you folks can have the migration done by then.
Chippyy closed subtask T211149: Ubuntu to Debian migration for maps-warper2 instance of the wikimaps warper as Resolved.
Jan 8 2019, 5:01 PM
TheDJ closed subtask T212596: Allow tiles1.wmflabs.org (or possibly entire eqiad-r?) access to osm postgres as Resolved.
Edited · Jan 14 2019, 10:28 PM
maps-tiles1 now has access to the pgsql server.
When running the renderer I see SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1; ERROR: permission denied for relation planet_osm_polygon but the old tiles servers seem to have that problem as well.
Now i need a way to verify if the new renderer is creating proper tiles. I hope to be able to get to that this weekend.
Krenair added a comment.
Jan 18 2019, 10:44 AM
maps-tiles2, maps-tiles3 and maps-wma1 should be shutoff today... It looks like maps-wma1 is still critical, what about the other two? Should this be extended given there is work actively being done on it?
aborrero added a comment.
Jan 22 2019, 4:38 PM
Please, someone provide some estimation of time required for the maps-wma1 instance. https://tools.wmflabs.org/openstack-browser/server/maps-wma1.maps.eqiad.wmflabs
Dalba removed a subscriber: Dalba.
Jan 22 2019, 4:42 PM
Krenair added a comment.
Jan 29 2019, 5:36 PM
It's now over a week since more information was requested.
TheDJ added a comment.
Edited · Jan 30 2019, 12:37 PM
I've had 4 hours since Christmas that i was able to spend on this ticket, all 4 were spent on that postgres issue (the patch for which was open for 3 weeks btw... just sayin)
I still haven't had time to verify if tiles1 is actually working now. (I suspect it's not actually.)
TheDJ added a comment.
Jan 30 2019, 12:46 PM
For maps-wma1, there is a new maps-wma but i'm not sure if @dschwen has worked on it since T204506#4824815
bd808 mentioned this in T215560: Can't read from OSM replicas when connecting from Stretch bastion.
Feb 8 2019, 6:59 AM
TheDJ added a comment.
Edited · Feb 11 2019, 11:01 PM
Spend a couple of hours. I now have tile rendering working again on tiles3 of the old instance as well as on tiles1 of the new server.
The new server is generating into a separate directory for now, as I wanted to confirm operation. Will work on consolidating the configurations in the next few days and attempt to verify operation of the new server. Then i should be able to let go of the old tiles instances.
http://tiles.wmflabs.org/osm/slippymap.html now shows tiles from the new instance. I'll clean that every day, as i don't want us to double the /data/project usage.
FYI, pretty sure that tile updates were not running for many many months, as @Bstorm suspected in T215560: Can't read from OSM replicas when connecting from Stretch bastion
dschwen added a comment.
Feb 11 2019, 11:05 PM
For maps-wma1, there is a new maps-wma but i'm not sure if @dschwen has worked on it since T204506#4824815
Yeah, I have not. The home directory has not magically appeared. Do I have to rebuild the VM? I'm confused.
TheDJ added a comment.
Feb 11 2019, 11:06 PM
@dschwen the instance needs to be rebooted for it to appear.
TheDJ added a subscriber: Bstorm.
Feb 11 2019, 11:07 PM
TheDJ removed a subscriber: Bstorm.
dschwen added a comment.
Feb 11 2019, 11:08 PM
Ok, will do that later. No access from work.
TheDJ added a comment.
Feb 11 2019, 11:12 PM
@dschwen I triggered a reboot on it and can confirm the homedirs are mounted on that instance.
TheDJ added a comment.
Edited · Feb 13 2019, 10:20 AM
Hmm, osm tile rendering seems extremely slow...
/usr/bin/render_speedtest Rendering client Starting 1 rendering threads Initial startup costs Rendered 1 tiles in 0.00 seconds (62500.00 tiles/s) Zoom(0) Now rendering 1 tiles Rendered 1 tiles in 656.16 seconds (0.00 tiles/s) Zoom(1) Now rendering 2 tiles Rendered 2 tiles in 921.96 seconds (0.00 tiles/s)
This is on both old and new tile servers.
Anyone know if that is normal ? 15 minutes for 2 tiles at zoom level 1 ?
MaxSem added a comment.
Feb 13 2019, 8:28 PM
11 minutes for z0 is good (Kartotherian does this in ~13).
TheDJ added a comment.
Feb 14 2019, 11:49 AM
@MaxSem thanks for confirming !
TheDJ removed a project: Patch-For-Review.
Feb 14 2019, 11:50 AM
TheDJ added a comment.
Feb 26 2019, 9:43 PM
Hmm, think I just figured out that none of the cached tiles have been expired since at least early 2015... learning so much...
TheDJ added a comment.
Feb 27 2019, 9:50 PM
I have now setup munin and have a grip on how the clusterization of renderd works. Next step is to make tiles1 the new master for renderd and http traffic ! Hope to get to that tomorrow evening.
Bstorm mentioned this in T209527: Set up scratch and maps NFS services on cloudstore1008/9.
Mar 5 2019, 11:58 PM
TheDJ added a comment.
Mar 10 2019, 8:47 PM
Currently blocked on T217992: Unable to delete "tiles.wmflabs.org" proxy entry via horizon
TheDJ added a subtask: T217992: Unable to delete "tiles.wmflabs.org" proxy entry via horizon.
Mar 10 2019, 8:47 PM
bd808 closed subtask T217992: Unable to delete "tiles.wmflabs.org" proxy entry via horizon as Resolved.
Mar 12 2019, 5:42 AM
TheDJ added a comment.
Mar 12 2019, 8:12 AM
The tiles traffic and tiles rendering is now primarily on the new server. I intend to verify this for a few days before shutting down the old servers completely.
Gehel awarded a token.
Mar 12 2019, 2:08 PM
Krenair awarded a token.
Mar 12 2019, 2:26 PM
Andrew awarded a token.
Mar 12 2019, 2:58 PM
TheDJ added a comment.
Edited · Mar 16 2019, 4:14 PM
After T218145: maps: take back root owned files/dirs from root_squash protected nfs old tiles can suddenly be regenerated again, causing an explosion of tile render events, in turn causing most render jobs to be dropped. I have temporarily disabled serving of hikebike, osm-no-labels and osm-bw, to allow the server to slowly catch up a bit on rendering old tiles.
TIL, that if you append /status to a tile, you can find out when it was last generated, and adding /dirty causes it to be put on the render queue.
Still seem to have some issues in other spots.
Also still need to take care of adding config files to redirect renderd logging to /var/log/renderd.log and a logrotate for that as well.
Krenair added a subscriber: Don-vip.
Mar 17 2019, 7:26 PM
A user on one of the subtasks wrote:
In T217992#5030855, @Don-vip wrote:
Hello,
JOSM maintainer here.
Since a few days we're unable to access two imagery layers hosted on tiles.wmflabs.org: '​https://tiles.wmflabs.org/osm-no-labels/{zoom}/{x}/{y}.png' and '​https://tiles.wmflabs.org/bw-mapnik/{zoom}/{x}/{y}.png'​.
Is it due to this change? Is the service still provided?
TheDJ added a comment.
Edited · Mar 18 2019, 9:41 AM
Hi @Don-vip
Yes, as you may note from this ticket, this service has been mostly running unmaintained since 2015 (many tiles were not refreshed since date as well... ). As we are going through several large changes in our cloud platform these services needed migration. Over the past 3 months, as a volunteer, I've taken on the job of getting familiar with this service and starting to rebuild this service. This might lead to reduced availability of maps in the short term as I deal with several large problems in the little time I have available to work on this.
It is my intent that hikebike, osm-bw and osm-no-labels are retained in the long run.
Peachey88 added a subscriber: Peachey88.
Mar 23 2019, 6:24 AM
GTirloni added a project: cloud-services-team (Kanban).
Mar 23 2019, 9:37 PM
Krenair mentioned this in T167293: Nova-network to Neutron migration.
Mar 24 2019, 5:34 PM
Krenair added a comment.
Mar 24 2019, 5:38 PM
@TheDJ: Thanks for working on this. To clarify the status of this task, are the tiles2/tiles3/wma1 instances still serving traffic, running important background processes, or otherwise in use?
TheDJ added a comment.
Edited · Mar 24 2019, 7:41 PM
Tiles2 and tiles3 are mostly standing by doing nothing (i've incidentally used them to boost bulk rendering capacity as ive been rendering 4 years of not updated tiles)
Wma1 is still fully and wholly doing all wikiminiatlas work. I have not done any work on it.
Andrew updated the task description. (Show Details)
Mar 25 2019, 2:06 PM
So, just to confirm my understanding of this... maps-tiles2 and maps-tiles3 can be shut down whenever, and maps-wma1 still needs a new person to work on it?
TheDJ added a comment.
Mar 25 2019, 3:25 PM
@Andrew correct
Andrew added a comment.
Mar 25 2019, 4:08 PM
@dschwen can you please catch us up on your plan for maps-wma1.maps.eqiad.wmflabs? Have you already built a replacement for it in the new region?
Andrew added a comment.
Mar 29 2019, 1:16 PM
@dschwen, what's the latest regarding maps-wma1?
Andrew added a comment.
Mar 29 2019, 1:19 PM
@TheDJ I'm shutting down maps-tiles2 and maps-tiles3 right now. Will delete in a few days if there are no ill effects.
TheDJ mentioned this in T161256: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert.
Mar 29 2019, 8:42 PM
dschwen added a comment.
Apr 1 2019, 2:32 PM
maps-wma1 can be deleted. There is a new maps-wma instance. I still have to switch over some webproxies. But I'll do that today. I ported my tile render code to mapnik 3, but still have to do some minor things like converting my upstart scripts to systemd. This will only affect rendering of new tiles - but most of the world is already cached by now.
Andrew added a comment.
Apr 1 2019, 2:49 PM
This is great! Thank you!
Andrew updated the task description. (Show Details)
Apr 1 2019, 2:49 PM
Stashbot added a comment.
Apr 1 2019, 2:51 PM
Mentioned in SAL (#wikimedia-cloud) [2019-04-01T14:51:46Z] <andrewbogott> shutting down maps-wma1 as per T204506
dschwen added a comment.
Apr 1 2019, 3:10 PM
:-/ I need to delete/re-add proxies to point to the new instance. I have proyx urls like 1.wma.wmflabs.org and label.wma.wmflabs.org. But the new horizon interface wont let me add a two-level subdomain like label.wma (tells me to specify a name without dots in it).
Andrew added a comment.
Apr 1 2019, 3:13 PM
Yep, the only proxies supported are things like <proxyname>.wmflabs.org. You might be able to set up cnames pointing to the existing proxies but I'm not sure how that will work with https
bd808 added a comment.
Apr 1 2019, 3:17 PM
In T204506#5074311, @dschwen wrote:
:-/ I need to delete/re-add proxies to point to the new instance. I have proyx urls like 1.wma.wmflabs.org and label.wma.wmflabs.org. But the new horizon interface wont let me add a two-level subdomain like label.wma (tells me to specify a name without dots in it).
We have scripts that will let these legacy proxy entries on the backend. Support for them has been removed from the normal user facing UI because of incompatibility with our TLS certificates. Make a list here of the (proxy name, backing instance, backing port) triples you need and I can create them. I would also recommend you make "normal" proxies more like "wma-1.wmflabs.org" and migrate your code to use them as soon as you can so that TLS will be possible.
dschwen added a comment.
Apr 1 2019, 3:17 PM
Ugh, this is borked. I cannot delete the proxy entries with the . in the name. I get You have selected: . Please confirm your selection. This action cannot be undone. and after pressing Delete the entry is still there.
dschwen added a comment.
Apr 1 2019, 3:18 PM
@bd808 horizon does not let you use a dash either!
bd808 added a comment.
Apr 1 2019, 3:21 PM
In T204506#5074331, @dschwen wrote:
Ugh, this is borked. I cannot delete the proxy entries with the . in the name. I get You have selected: . Please confirm your selection. This action cannot be undone. and after pressing Delete the entry is still there.
Yes. This is expected as well. From the point of view of the Horizon app these legacy proxies are data corruption. I had to make some command line tools to manage them when we moved the proxies for the tiles hosts in this project as well.
dschwen added a comment.
Apr 1 2019, 3:23 PM
And the dash? (anyhow, just delete all wma proxies containing a dot at your convenience, please)
Andrew added a comment.
Apr 1 2019, 3:27 PM
In T204506#5074336, @dschwen wrote:
@bd808 horizon does not let you use a dash either!
Created T219789 for this. Bryan's script should be able to work around that in the meantime.
Andrew added a comment.
Apr 1 2019, 3:32 PM
btw @dwschen, while you're doing this I encourage you to create standardized urls to replace your old extra.dot.wmflabs.org proxy names so you can move clients over to using the new tls-supporting urls. It's generally bad to use the unencrypted services and I can't guarantee any kind of long-term support for them.
dschwen added a comment.
Apr 1 2019, 3:44 PM
@Andrew , Yes, I am planning to update my code to use non-dot subdomains. But I'm waiting until the dash issue is resolved.
TheDJ added a comment.
Apr 1 2019, 5:23 PM
I didnt mention it in this ticket yet, so for completeness sake: Last week i started redirecting [abc].tiles .wmflabs.org to https://tiles.wmflabs.org for the exact same reason.
bd808 added a comment.
Apr 1 2019, 11:42 PM
Attempting to fix the proxies for @dschwen:
$ sudo /usr/local/sbin/wmcs-webproxy --project=maps listdomain backend​================================================ ========================​0.wma.wmflabs.org http://10.68.16.70:80​1.wma.wmflabs.org http://10.68.16.70:80​2.wma.wmflabs.org http://10.68.16.70:80​3.wma.wmflabs.org http://10.68.16.70:80​4.wma.wmflabs.org http://10.68.16.70:80​5.wma.wmflabs.org http://10.68.16.70:80​6.wma.wmflabs.org http://10.68.16.70:80​7.wma.wmflabs.org http://10.68.16.70:80​a.tiles.wmflabs.org http://172.16.5.154:80​b.tiles.wmflabs.org http://172.16.5.154:80​c.tiles.wmflabs.org http://172.16.5.154:80​label.wma.wmflabs.org http://10.68.16.70:80​maps.wmflabs.org http://172.16.5.154:80​oldwma.wmflabs.org http://10.68.16.70:80​tiles.wmflabs.org http://172.16.5.154:80​warper.wmflabs.org http://172.16.0.158:80​wma.wmflabs.org http://172.16.1.144:80​wma0.wmflabs.org http://172.16.1.144:80​wma1.wmflabs.org http://172.16.1.144:80​wma2.wmflabs.org http://172.16.1.144:80​wma3.wmflabs.org http://172.16.1.144:80​wma4.wmflabs.org http://172.16.1.144:80​wma5.wmflabs.org http://172.16.1.144:80​wma6.wmflabs.org http://172.16.1.144:80​wma7.wmflabs.org http://172.16.1.144:80$ for i in $(seq 0 7); do sudo /usr/local/sbin/wmcs-webproxy --project=maps delete ${i}.wmadone$ sudo /usr/local/sbin/wmcs-webproxy --project=maps delete label.wma:# At this point I had to use Horizon to manually delete dangling DNS records in the wma.wmflabs.org zone and ultimately the empty zone itself.:# Once that was done I was able to re-create the proxy entries pointing to the new backend$ for i in $(seq 0 7); do sudo /usr/local/sbin/wmcs-webproxy --project=maps add ${i}.wma http://172.16.1.144:80; done2019-04-01T23:35:29Z mwopenstackclients.DnsManager WARNING : Creating 0.wma.wmflabs.org.​2019-04-01T23:35:32Z mwopenstackclients.DnsManager WARNING : Creating 1.wma.wmflabs.org.​2019-04-01T23:35:34Z mwopenstackclients.DnsManager WARNING : Creating 2.wma.wmflabs.org.​2019-04-01T23:35:37Z mwopenstackclients.DnsManager WARNING : Creating 3.wma.wmflabs.org.​2019-04-01T23:35:40Z mwopenstackclients.DnsManager WARNING : Creating 4.wma.wmflabs.org.​2019-04-01T23:35:42Z mwopenstackclients.DnsManager WARNING : Creating 5.wma.wmflabs.org.​2019-04-01T23:35:45Z mwopenstackclients.DnsManager WARNING : Creating 6.wma.wmflabs.org.​2019-04-01T23:35:48Z mwopenstackclients.DnsManager WARNING : Creating 7.wma.wmflabs.org.$ sudo /usr/local/sbin/wmcs-webproxy --project=maps add wma http://172.16.1.144:80​2019-04-01T23:36:03Z mwopenstackclients.DnsManager WARNING : Creating wma.wmflabs.org.$ sudo /usr/local/sbin/wmcs-webproxy --project=maps add lable.wma http://172.16.1.144:80​2019-04-01T23:41:06Z mwopenstackclients.DnsManager WARNING : Creating lable.wma.wmflabs.org.$ sudo /usr/local/sbin/wmcs-webproxy --project=maps listdomain backend​================================================ ========================​0.wma.wmflabs.org http://172.16.1.144:80​1.wma.wmflabs.org http://172.16.1.144:80​2.wma.wmflabs.org http://172.16.1.144:80​3.wma.wmflabs.org http://172.16.1.144:80​4.wma.wmflabs.org http://172.16.1.144:80​5.wma.wmflabs.org http://172.16.1.144:80​6.wma.wmflabs.org http://172.16.1.144:80​7.wma.wmflabs.org http://172.16.1.144:80​a.tiles.wmflabs.org http://172.16.5.154:80​b.tiles.wmflabs.org http://172.16.5.154:80​c.tiles.wmflabs.org http://172.16.5.154:80​lable.wma.wmflabs.org http://172.16.1.144:80​maps.wmflabs.org http://172.16.5.154:80​oldwma.wmflabs.org http://10.68.16.70:80​tiles.wmflabs.org http://172.16.5.154:80​warper.wmflabs.org http://172.16.0.158:80​wma.wmflabs.org http://172.16.1.144:80​wma0.wmflabs.org http://172.16.1.144:80​wma1.wmflabs.org http://172.16.1.144:80​wma2.wmflabs.org http://172.16.1.144:80​wma3.wmflabs.org http://172.16.1.144:80​wma4.wmflabs.org http://172.16.1.144:80​wma5.wmflabs.org http://172.16.1.144:80​wma6.wmflabs.org http://172.16.1.144:80​wma7.wmflabs.org http://172.16.1.144:80
bd808 added a comment.
Apr 1 2019, 11:44 PM
I deleted the typo'ed lable.wma.wmflabs.org and created label.wma.wmflabs.org with the same target.
Andrew closed this task as Resolved.
Apr 20 2019, 2:46 PM
Andrew claimed this task.
The last of the shutdown VMs have been deleted, so this is now done!
Andrew mentioned this in T233995: Setup a dedicated HTTPS terminating proxy for maps project.
Apr 30 2020, 7:20 PM
Log In to Comment
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL