Mod_tile disable expiring tiles completely

I set up a tileserver with the following configuration:

mod_tile (apache2) config:

    Header set Access-Control-Allow-Origin "*"
    Header set Access-Control-Allow-Headers "Access-Control-Allow-Headers, Origin,Accept, X-Requested-With, Content-Type, Access-Control-Request-Method, Access-Control-Request-Headers,Authorization"

    # Renderd configution
    LoadTileConfigFile /usr/local/renderd/renderd.conf
    ModTileEnableStats On
    ModTileEnableStatusURL On
    ModTileBulkMode On
    ModTileRequestTimeout 0
    ModTileMissingRequestTimeout 60
    ModTileMaxLoadOld 1000
    ModTileMaxLoadMissing 1000
    ModTileRenderdSocketName /var/run/renderd/renderd.sock
    ModTileCacheDurationMax 60480000000
    ModTileCacheDurationMinimum 908000000
    ModTileCacheDurationDirty 900
    ModTileCacheDurationLowZoom 9 5184000
    ModTileCacheLastModifiedFactor 0.20
    ModTileEnableTileThrottling Off

    #LoadModule wsgi_module modules/mod_wsgi.so
    WSGIScriptAlias /mapproxy /usr/local/mapproxy/config.py
    WSGIDaemonProcess mapproxy-wsgi-daemon processes=16 threads=8
    WSGIApplicationGroup %{GLOBAL}
    RewriteEngine on
    RewriteRule ^/([0-9]+)/([0-9]+)/([0-9]+).png /osm_tiles/$1/$2/$3.png [PT]
<VirtualHost *:80>
    ServerAlias CC-GEO01.eurofunk.com
    DocumentRoot /var/www
    <Directory />
            Options FollowSymLinks
        Order allow,deny
        Allow from all
    </Directory>
    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        Allow from all
    </Directory>
    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>
    <Directory /usr/local/mapproxy>
        Order deny,allow
        Allow from all
        Require all granted
    </Directory>
</VirtualHost>

and renderd.conf:

[renderd]
pid_file=/var/run/renderd/renderd.pid
stats_file=/var/run/renderd/renderd.stats
socketname=/var/run/renderd/renderd.sock
num_threads=20
tile_dir=/usr/local/renderd/cache
MAXZOOM=20

[mapnik]
plugins_dir=/usr/lib/mapnik/3.1/input
font_dir=/usr/share/fonts
font_dir_recurse=true

[default]
URI=/tile/
XML=/usr/local/mapnik/openstreetmap-carto/CARTO_DE_AT_CH.xml
HOST=CC-GEO01.eurofunk.com
TILEDIR=/usr/local/renderd/cache
MINZOOM=0
MAXZOOM=20
[topo]
URI=/topo/
XML=/usr/local/mapnik/OpenTopoMap/mapnik/opentopomap.xml
HOST=CC-GEO01.eurofunk.com
TILEDIR=/usr/local/renderd/cache
MINZOOM=0
MAXZOOM=20

I renderd my extend OSM - Tile Calculator (WGS84: 5.05 45.71 17.75 55.32 or EPSG:3857 562163 5733998 1975921 7424221) one time from Z0 to Z18 with render_list_geo.pl which uses render_list.

Updading works with pyosmium. I update the database every 5 minutes. With this script i also rerender the updated tiles from Z15 to Z18. With this setup i dont want to get my tiles expired since they get updated by my script automatically. With this command rerendering is done:

/usr/local/bin/render_expired -t /usr/local/renderd/cache --map=default --min-zoom=15 --max-zoom=18 -s /run/renderd/renderd.sock < /usr/local/renderd/scripts/pyosmium/dirty_tiles.txt > render_expired.$$ 2>&1

But they get expired. When i jump around the map mod_tile tells renderd to rerender some tiles although they already exist.

What i tried:

  • Updating mod_tile configuration to the tiles never get expired
  • Creating a planet-import-complete file unter /usr/local/renderd/cache/default and /usr/local/renderd/cache/topo with a timestamp from 2015. I found this option in some other threads but it did not help

Can someone tell me how to tell me how to tell mod_tile not to auto expire tiles. :slight_smile: Thanks in advance.

P.S.: one addition to the render_expired command:

at the beginning of my setup the command looked like this:

/usr/local/bin/render_expired -t /usr/local/renderd/cache --map=default --min-zoom=15 --touch-from 15 --delete-from 15 --max-zoom=19 -s /run/renderd/renderd.sock < /usr/local/renderd/scripts/pyosmium/dirty_tiles.txt > render_expired.$$ 2>&1

I assume the --delete-from and --touch-from deleted me the expired files which were missing afterwards. Since i removed this now i hope this command does not delete the expired tiles automatically.

1 Like

render_expired does not render tiles; it only marks them as expired (by modifying their timestamp to be far in the past). The next time the tile is served, mod_tile will notice that it is dealing with an old tile, and re-render it. This is what you are seeing. This approach means that tiles are not unnecessarily re-rendered (if two mappers are edit-warring about a lamp post on a z18 tile and changing it back and forth every 5 minutes, your process will mark the tile as expired every 5 minutes, but it will not be rendered every 5 minutes).

Stopping this re-rendering (which would be possible with a suitably old planet-import-complete) would mean that your tiles are not updated at all.

See also Prevent mod_tile/renderd from re-rendering cached tile, even if tile is outdated - OSM Help which may have some relevant info.

1 Like

thanks for your input - i understood it completely wrong then. I assume that it works as i want I need to feed the list of “need to update tiles” to render_list so updating is done and add a planet-import-complete file which is very old to /usr/local/renderd/cache to prevent mod_tile expiring other tiles also which dont need an update. Do i need to put it to /usr/local/renderd/cache or into the cache folder of the map /usr/local/renderd/cache/default ?

I try the following now → get the dirty_tiles.txt from osm2pgsql and reorder the lines from Z/X/Y to X Y Z so render_list can handle them.

Then startup the render_list command afterwards:

/usr/local/bin/render_list -n 64 -t /usr/local/renderd/cache --map=default -s /run/renderd/renderd.sock < /usr/local/renderd/scripts/pyosmium/dirty_tiles_reorderd.txt > render_list.$$ 2>&1

then i created a planet-import-complete file in the tile directory like this:

touch -t 201501151030.00 /usr/local/renderd/cache/planet-import-complete
touch -t 201501151030.00 /usr/local/renderd/cache/default/planet-import-complete
touch -t 201501151030.00 /usr/local/renderd/cache/topo/planet-import-complete

(i dont know if its enough in /usr/local/renderd/cache or not so i created the file in every tile cache folder)

lets see if this works

mod_tile checks fro the planet-import-complete file in both the main cache directory and the per-style subdirectories.

In general, I think that your approach is not a good idea; your server will spend a lot of time on re-rendering tiles that nobody is looking at. The “serve an old tile and re-compute it afterwards” approach of mod_tile tends to be better, unless it is really really important to you that users always see the latest data - but your approach will not even guarantee that due to caching of tiles in browser.

Also, working with expire lists is flawed for two reasons: (1) many tiles will be reported as “expired” even if nothing has changed on the rendered tile - for example, if someone has added an Esperanto name to a place node, or someone has added a lamp post, or modified opening hours. And (2) a change in a large object that affects many tiles will be dropped by osm2pgsql, for example if someone changes the name of a large admin boundary which would expire thousands of tiles.

In short, not only will your approach render lots of tiles that nobody is requesting, it will also un-necessarily render tiles on which nothing has changed, and at the same time it will overlook some changes in large objects.

The “holzhammer method” of mod_tile - re-render anything that is older than three days when requested - avoids all these problems.

good point -you are right - rerendering all tiles takes ages and is not worth the effort…

I shortly describe for what my setup is - maybe you have a better idea how to solve my issue:

I work at an IT company - my task is to provide a osm map for whole DE_AT_CH to my customers.

In our company I have a server where I prepare the maps and just ship the cache out. On the customer site no rendering is happening since everything is prepared inhouse.

So the process is → prepare map in house → render all zoom levels I need (Z0-Z18) → constantly update tiles 5 minutely → then I can copy out the whole map which is always up to date onto the customer site.

The problem is now that all the tiles get just expred and nobody will look at the map so they get rerenderd.

I suppose I should skip the expiring and just to a render_list of the whole map every 2-3 months when i need a new export to a customer.

Well my first question would be, do you really need/want to ship out a 300GB+ file every couple of months - can your customers perhaps (a) run a tile server that you provide in a virtual machine, or (b) work with z17 instead of z18, or perhaps only use z18 for some special areas of interest (“along motorways”, “in big cities”, etc) - that could already reduce your shipping size and production time by 2/3.

And yes, if you want to continue what you are doing now, I would recommend doing a full import+render every couple of months, and just delete everything afterwards. As a side effect this means you don’t have to reserve a server for it - if you want to, you can fire up some EC2 cloud box to quickly import the data and render the world, or you can run the job on a desktop machine over the weekend.

And thirdly, using vector tiles would greatly simplify the whole process - download extract, run tilemaker, ship mbtiles file, done.

1 Like

thank you. ill change the system with your input. thanks a million for your fast help :). have a good day