OSM Tileserver: Difference between revisions
āāTile refreshing: keep forgetting this |
|||
Line 81: | Line 81: | ||
mod_tile looks at the import statement for the planet to determine if tiles are old. This is import statement is basically the modification timestamp of the file <code>/data/project/tilesplanet-import-complete</code>. |
mod_tile looks at the import statement for the planet to determine if tiles are old. This is import statement is basically the modification timestamp of the file <code>/data/project/tilesplanet-import-complete</code>. |
||
Normally this timestamp is left by the osm2pqsl import, but we keep it manually on this tiles server. <code>touch -t 20210615 /data/project/tilesplanet-import-complete</code> |
Normally this timestamp is left by the osm2pqsl import, but we keep it manually on this tiles server. <code>touch -t 20210615 /data/project/tilesplanet-import-complete</code>. We set it via cron every day, assuming that WMF imports work :) |
||
For an individual tile, it is possible to request the current status by appending /status to the url like so: [https://tiles.wmflabs.org/osm/18/136090/86311.png/status a status url]. You can also submit it to be rerendered by appending /dirty instead. All rerender requests are queued, so it might still be a considerable time before the tile is actually rerendered. |
|||
== DSN/Web proxies == |
== DSN/Web proxies == |
Revision as of 08:42, 17 June 2021
This page is for the Cloud VPS project to implement an OpenStreetMap TileServer to serve the same basic data as OSM themselves (synced from them). This is orthogonal to (or in some other sense, a prerequisite for) other OSM efforts going on in Cloud Services for related overlays and databases. Corrections to the information below are welcome, this is all to the best of my knowledge at this time...
Resources
- phab:T204506
- https://tools.wmflabs.org/openstack-browser/project/maps
- https://horizon.wikimedia.org
- maps-tiles1.maps.eqiad1.wikimedia.cloud
- THIS IS THE NEW DEBIAN STRETCH HOST
- This is renderd master
- This server is the only webhost
- maps-tiles2.maps.eqiad1.wikimedia.cloud
This is renderd slave 1
- maps-tiles3.maps.eqiad1.wikimedia.cloud
This is renderd masterThis server is the only webhost
Technology stack
- Apache with mod_tile and mod rewrite
- conf /etc/apache2/sites-enabled/tileserver_site.conf
- https://launchpad.net/~osmadmins/+archive/ubuntu/ppa?field.series_filter=bionic
- lib: /var/lib/mod_tile
- deps for compiling: apt-get install build-essential autoconf libtool libmapnik-dev apache2-dev
- docs for running mod_tile on jessie
- /usr/lib/apache2/modules/mod_tile.so
- Mod tile status page
- puppet: includeĀ ::apacheĀ ?
- Munin: Create graphs of tile freshness https://tiles.wmflabs.org/munin/mod_tile-month.html
- conf:
- output: /var/www/munin
- apt: sudo apt-get install munin munin-node munin-plugins-core munin-plugins-extra
- Renderd
- conf: /etc/renderd.conf
- output: /data/project/tiles/[tilevariant]
- Important: make sure fonts for global scripts are installed.
- Hillshading is not rendered it seems. pre generated tilesĀ ?
- apt: renderd/trusty
- renderd, can have slaves. this is determined by the --slave param when starting the renderd daemon
- Tends to crash.. added
/etc/systemd/system/renderd.service.d/restart.conf
[Service] Type=exec PIDFile=/var/run/renderd/renderd.pid # Make sure apache and other processes have prio Nice=1 RemainAfterExit=no #It crashes a lot and needs restarts Restart=always RestartSec=2s # Restart it every hour, because it leaks memory RuntimeMaxSec=3600
- Mapnik
- Version: 2.2.0
- Styles: /data/project/styles/
- Lib: /usr/lib/mapnik/2.2/input
- apt: libmapnik-dev/trusty, libmapnik2.2/trusty, mapnik-utils/trusty, openstreetmap-mapnik-carto-stylesheet-data/trusty, python-mapnik/trusty
- Stretch: libmapnik-dev, libmapnik3.0, mapnik-utils, python3-mapnik
- Database
- Host: osmdb.eqiad.wmnet
- User: osm
- Database: gis
- Referenced from /data/project/styles/*/*.xml
- OSM Stylesheets
- Seems most of these are not versioned, have not been updated since 2014 and we seem to be missing the .mml (we only have the output)
- likely will have to keep using what we already have generated for now.
- apt: openstreetmap-carto
- Hikebike: hikebikemap-carto
- OSM Default: openstreetmap-carto
- OSM Default no labels:
- bw-mapnik:
- Installed fonts:
- fonts-dejavu, fonts-dejavu-core, fonts-dejavu-extra, fonts-noto fonts-khmeros, fonts-khmeros-core, font-sil-padauk, fonts-sipa-arundina, ttf-indic-fonts-core, ttf-kannada-fonts, ttf-tamil-fonts, xfonts-encodings, xfonts-utils
- stretch: fonts-dejavu fonts-dejavu-core fonts-dejavu-extra
fonts-droidfonts-khmerosfonts-khmeros-corefont-sil-padaukfonts-sipa-arundinattf-indic-fonts-corexfonts-encodings xfonts-utilsfonts-taml fonts-samyak-taml fonts-lohit-taml fonts-lohit-knda fonts-knda - puppet: includeĀ ::mediawiki::packages::fonts
- postgresql-client for debugging
Tile refreshing
mod_tile looks at the import statement for the planet to determine if tiles are old. This is import statement is basically the modification timestamp of the file /data/project/tilesplanet-import-complete
.
Normally this timestamp is left by the osm2pqsl import, but we keep it manually on this tiles server. touch -t 20210615 /data/project/tilesplanet-import-complete
. We set it via cron every day, assuming that WMF imports workĀ :)
For an individual tile, it is possible to request the current status by appending /status to the url like so: a status url. You can also submit it to be rerendered by appending /dirty instead. All rerender requests are queued, so it might still be a considerable time before the tile is actually rerendered.
DSN/Web proxies
This project does not contain any public IPs. Rather, the public IPs for the project are owned by the 'project-proxy' project, and assigned to proxy hosts that are specific to the maps project.
DNS settings in the 'maps' project refer to the proxy host in project-proxy, for example:
wma1.wmflabs.org. A - Address record 185.15.56.55 Active
There are two proxy hosts in project-proxy, and active and a passive node, named maps-proxy-01.project-proxy.eqiad1.wikimedia.cloud and maps-proxy-02.project-proxy.eqiad1.wikimedia.cloud. The list of proxies is configured with hiera and profile::wmcs::proxy::static::proxy_mappings:
profile::wmcs::proxy::static::proxy_mappings: 0.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud 1.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud 2.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud 3.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud 4.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud 5.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud 6.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud 7.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud a.tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud b.tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud c.tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud label.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud maps.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud warper.wmflabs.org: maps-warper3.maps.eqiad1.wikimedia.cloud wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud wma0.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud wma1.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud wma2.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud wma3.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud wma4.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud wma5.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud wma6.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud wma7.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
To add or modify a proxy, first make the necessary DNS change within the Maps project (and point the new domain to the maps proxy IP, 185.15.56.55. Then ask a WMCS admin to adjust the hiera as needed. Hiera is located in the 'maps-proxy' puppet prefix.
Questions
- Can we puppetize thisĀ ?
- Should we switch from renderd to tirexĀ ?
- Where does the code for renderd come fromĀ ?
- Why is maps-tiles2.maps.eqiad1.wikimedia.cloud not serving httpĀ ? (it was a rendering only slave)
- Is there no osm2pqslĀ ??????????
- The styles are 83MB.. but contain the dbname etc. How to not put that in puppet repo, yet have dbname configurableĀ ?
- what is the difference between the https and the apache manifestsĀ ?
- how to force re-rendering of tiles. See this.
Decisions
- since grafana and nagf provide the same info
Build a new cluster on hostname maps.wmflabs.org (currently unused), for easy testingSwitch to Debian Stretch as best supported- Create a repo to maintain all the styles, config files and any non-privileged information
- Get rid of some no longer active wmflabs maintainers from the project
delete overpass-wiki instance. Made by User:Jotpe, but doesn't seem to be in actual use.- Make a debian stretch package of mod_tileĀ ??????? See: APT_repository
See also
- Historic information about this project is on OSM Tileserver/Archive.