So.. the ancientness of the maps servers show their ugly face once more ;)
I'm guessing /data/project NFS is run in root_squash mode, as root users can't seem to do anything there. Unfortunately however, there are multiple files in this directory that are root:root owned. This makes it they cannot be deleted, changed etc...
Anyone know how to fix this ?
www-data:www-data will probably be good. If needed, I can claim ownership of www-data via sudo -u www-data
Possibly exclude /data/project/wma, as that is currently owned by dschwen, and i don't want to interfere with that.
$ ssh labstore1003.eqiad.wmflabs $ cd /srv/maps/project $ ls -lh total 20K drwxr-xr-x 2 root root 4.0K Mar 27 2014 chippy/ drwxr-xr-x 7 root root 4.0K Sep 26 2014 styles/ drwxr-xr-x 10 www-data root 4.0K Mar 12 19:24 tiles/ drwxr-xr-x 4 root root 4.0K Feb 3 2015 warper/ drwxr-xr-x 9 1115 www-data 4.0K Dec 12 2017 wma/
Got clarification from @TheDJ via irc chat that the styles and tiles directories are the most concerning for him at the moment. I will focus on them and exclude chippy, warper, and wma.
Mentioned in SAL (#wikimedia-cloud) [2019-03-12T20:33:24Z] <bd808> Changing files owned by root to www-data:www-data in /data/project/{styles,tiles} from NFS server (T218145)
@Bstorm did some archeology for me and discovered that the root_squash was placed by https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/316713/ about 2.5 years ago. Unfortunately no associated task or comments to help understand why the change was made. It is however a good general practice to enforce root_squash on NFS exports which can be accessed by multiple user groups to help prevent various forms of attack (WP:BEANS).
This took about 20 hours to complete:
$ sudo ionice -c 3 nice -19 find /srv/maps/project/tiles -user root -exec chown www-data:www-data {} + chown: cannot dereference ‘/srv/maps/project/tiles/mapnik-osm-carto-data/mapnik-osm-carto-data’: No such file or directory chown: cannot dereference ‘/srv/maps/project/tiles/mapnik-osm-carto-data/data’: Too many levels of symbolic links
@TheDJ I fear that there are some files left in need to ownership changes, but I think at this point you will need to find them one by one and ask for a Cloud root user to fix them for you.