render_list stops: Received request for map layer 'ajt'...failed load

Hi

Have followed this:
Manually building a tile server (18.04 LTS)
https://switch2osm.org/manually-building-a-tile-server-18-04-lts/

I have imported some test data using iceland-latest.osm.pbf from geofabrik.de.

To see if tiles are rendered via localhost I then tried this:
http://localhost/hot/11/900/543.png

which is working.

But now I want to test pre-rendering tiles using:
render_list -a -n 2 -s /var/run/renderd/renderd.sock -Z 3

which result in this beein output to terminal:

debug: init_storage_backend: initialising file storage backend at: /var/lib/mod_tile
Rendering client
Starting 2 rendering threads
Rendering all tiles from zoom 0 to zoom 3
Rendering all tiles for zoom 0 from (0, 0) to (0, 0)
Rendering all tiles for zoom 1 from (0, 0) to (1, 1)
Rendering all tiles for zoom 2 from (0, 0) to (3, 3)
Rendering all tiles for zoom 3 from (0, 0) to (7, 7)
Waiting for rendering threads to finish

then nothing more is happening in the terminal, and no tiles have been written to dir: /var/lib/mod_tile

in the subdirectory of that dir, there is dir: ajt
I see that the tiles generated when using localhost are there.

I could need some suggestions on how to proceed to find out why no tiles are pre-generated.
Is there a log-file somewhere which could help me solve the problem, or should I start somewhere else?

UPDATE:

If I try this, note using another option: -m ajt

render_list -n 1 -s /var/run/renderd/renderd.sock -z 0 -Z 10 -m ajt -a

Then I find stuff like this in var/log/syslog

Jul 20 20:12:14 OSM renderd[5529]: Received request for map layer ‘ajt’ which failed to load

Why would map layer ajt fail to load?

In /usr/local/etc/renderd.conf I have this:

[renderd]
num_threads=4
tile_dir=/var/lib/mod_tile
stats_file=/var/run/renderd/renderd.stats

[mapnik]
plugins_dir=/usr/lib/mapnik/3.0/input
font_dir=/usr/share/fonts/truetype
font_dir_recurse=1

[ajt]
URI=/hot/
TILEDIR=/var/lib/mod_tile
XML=/home/src/openstreetmap-carto/mapnik.xml
HOST=localhost
TILESIZE=256
MAXZOOM=11

Restart renderd (e.g. via “/etc/init.d/renderd restart”) and it’ll try and restart, and if there’s a problem it’ll tell you why that map layer failed to load.

Thanks for tips.
But yesterday, after many hours of searching for a solution on the internet, I tried to install another version of Linux, 16.04 instead of 18.04, and followed the recipy on switch2osm.org for that version. But without any luck. Then I reformatted the disk and installed Python, Mapnik, osm2pgsql, and PostGis on Windows. Now I’m able to generate at least some tiles, but generate_tiles.py, but python report runtime error and exception in thread, and after some tiles are generated the command window freeze (hang) every time.

When I look at the generated tiles for the selected zoom-levels it should generate tiles for, in this case 6 - 11, then I can see name of Torshavn, and main roads, but most land area except for some tiny area in Torshavn have the same color as the ocean. I have imported the faroe-islands-latest.osm into the PostGis DB. I’m using the osm.xml that came with Mapnik as the MAPNIK_MAP_FILE.

So now I have to figure out why:
a) generate_tiles.py freeze after generating 274 tiles, as well as report exception in thread and runtime error

Some tiles generated
Exception in thread Thread-1
[…]
Runtime error: bad allocation
Some more tiles generated
Then windows command window freeze

Similar exception and error when NUM_THREADS = 1, but then fewer tiles are generated before the command window freeze.

When NUM_THREADS = 4 then allways 274 tiles are generated.
When NUM_THREADS = 1 then allways 26 tiles are generated.

b) why most land area get same color as the ocean

Is someone has some tips on how to fix those issues, or what procedure I should follow to find out the cause for it…

I looked at CPU and memory usage when generating tiles the last time, and both CPU and memory usage stayed very low. And the tiles that where generated was generated very fast.

Using Intel® Core i7-960 Processor, Socket-LGA1366, Quad Core, 3.2Ghz,
12 GB RAM
Samsung 860 EVO 500GB SSD for the PostGis database and storage for the generated tiles.
Windows 7

Here is how the output look (extract) when NUM_THREADS = 4:

FAREO-ISLANDS : 9 244 143 Empty Tile
FAREO-ISLANDS : 9 244 142
FAREO-ISLANDS : 8 123 71
FAREO-ISLANDS : 9 245 144
FAREO-ISLANDS : 9 245 141
FAREO-ISLANDS : 9 245 143
FAREO-ISLANDS : 9 245 142
FAREO-ISLANDS : 9 246 141
FAREO-ISLANDS : 9 247 141
FAREO-ISLANDS : 9 246 144
FAREO-ISLANDS : 9 246 142
FAREO-ISLANDS : 9 247 143
Exception in thread Thread-1:
Traceback (most recent call last):
File “C:\Python27\lib\threading.py”, line 801, in __bootstrap_inner
self.run()
File “C:\Python27\lib\threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “generate_tiles.py”, line 118, in loop
self.render_tile(tile_uri, x, y, z)
File “generate_tiles.py”, line 100, in render_tile
mapnik.render(self.m, im)
RuntimeError: bad allocation

FAREO-ISLANDS : 9 247 142
FAREO-ISLANDS : 9 247 144 Empty Tile
Exception in thread Thread-4:
Traceback (most recent call last):
File “C:\Python27\lib\threading.py”, line 801, in __bootstrap_inner
self.run()
File “C:\Python27\lib\threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “generate_tiles.py”, line 118, in loop
self.render_tile(tile_uri, x, y, z)
File “generate_tiles.py”, line 100, in render_tile
mapnik.render(self.m, im)
RuntimeError: bad allocation

FAREO-ISLANDS : 10 489 284
FAREO-ISLANDS : 9 246 143
FAREO-ISLANDS : 10 489 285
Exception in thread Thread-3:
Traceback (most recent call last):
File “C:\Python27\lib\threading.py”, line 801, in __bootstrap_inner
self.run()
File “C:\Python27\lib\threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “generate_tiles.py”, line 118, in loop
self.render_tile(tile_uri, x, y, z)
File “generate_tiles.py”, line 100, in render_tile
mapnik.render(self.m, im)
RuntimeError: bad allocation

FAREO-ISLANDS : 10 489 287 Empty Tile
FAREO-ISLANDS : 10 489 288 Empty Tile
FAREO-ISLANDS : 10 489 289 Empty Tile
FAREO-ISLANDS : 10 490 282
FAREO-ISLANDS : 10 490 283

then it keeps on doing many more tiles, without any more exeptions and errors reported,
then it just stop, produce an empty line where the cursor blink,
then I can only copy text from the command window or close it.

Here is the output when NUM_THREADS = 1

N:\mapnik>testgen.cmd

N:\mapnik>set MAPNIK_MAP_FILE=osm.xml

N:\mapnik>set MAPNIK_TILE_DIR=N:\tiles

N:\mapnik>set HOME=C:\Users

N:\mapnik>python generate_tiles.py
render_tiles( (-7.8, 61.3, -5.8, 62.5) osm.xml N:\tiles/ 6 11 FAREO-ISLANDS )
FAREO-ISLANDS : 6 30 17
FAREO-ISLANDS : 6 30 18
FAREO-ISLANDS : 7 61 35
FAREO-ISLANDS : 7 61 36
FAREO-ISLANDS : 8 122 70
FAREO-ISLANDS : 8 122 71
FAREO-ISLANDS : 8 122 72
FAREO-ISLANDS : 8 123 70
FAREO-ISLANDS : 8 123 71
FAREO-ISLANDS : 8 123 72
FAREO-ISLANDS : 9 244 141 Empty Tile
FAREO-ISLANDS : 9 244 142
FAREO-ISLANDS : 9 244 143 Empty Tile
FAREO-ISLANDS : 9 244 144 Empty Tile
FAREO-ISLANDS : 9 245 141
FAREO-ISLANDS : 9 245 142
FAREO-ISLANDS : 9 245 143
FAREO-ISLANDS : 9 245 144
FAREO-ISLANDS : 9 246 141
FAREO-ISLANDS : 9 246 142
FAREO-ISLANDS : 9 246 143
FAREO-ISLANDS : 9 246 144
FAREO-ISLANDS : 9 247 141
FAREO-ISLANDS : 9 247 142
FAREO-ISLANDS : 9 247 143
FAREO-ISLANDS : 9 247 144 Empty Tile
Exception in thread Thread-1:
Traceback (most recent call last):
File “C:\Python27\lib\threading.py”, line 801, in __bootstrap_inner
self.run()
File “C:\Python27\lib\threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “generate_tiles.py”, line 118, in loop
self.render_tile(tile_uri, x, y, z)
File “generate_tiles.py”, line 100, in render_tile
mapnik.render(self.m, im)
RuntimeError: bad allocation

Just a small update:

I finally got generate_tiles.py to run without any errors - after I downloaded a bunch of shapefiles into a directory in my mapnik folder called world_boundaries.

But the generated tiles do miss some information.
I do believe this has something with stylesheet used and maybe some issue with some tables not existing in the database when importing file osm.pbf file from geofabrik.de into database.

Since I’m so faar off in relation to my original question, and the fact that I’m now on a windows machine - I think it’s best to ask a new question ref. my new issues.

Sorry for this mess, but as you can see I’m a totally newbie in relation to these tools needed to pregenerate raster tiles.

I’ve not tried it, but on Windows I’d expect that you’d run into lots of font issues for a couple of reasons:

  1. You’re using “generate_tiles.py”, and by default that needs all fonts referenced in the stylesheet to be installed, not just “enough to work”

  2. On Windows the fonts are likely to be different to on Linux, which at least the OSM Standard style is designed for

  3. Many stylesheets have multiple versions of some fonts to cope with the default fonts on different Linux systems. They have likely never been used much if at all on Windows.

Also, “the osm.xml that came with Mapnik” is probably not a good one to use. It could have any old rubbish in it such as, in your case, a requirement for some “world boundaries” data.

You really can’t mix and match different bits of software and sets of instructions like this, for a couple of reasons - one is that it likely won’t work, and another is that when it doesn’t work no-one will be able to reproduce what you did because you don’t really know yourself.

On Windows, I’d start from https://www.openstreetmap.org/user/SomeoneElse/diary/42617 .

Thanks again for taking the time to answer. I’t seems to me that the recipy you linked to, similar to the one I started with - ends up generating tiles when requested via a tileserver - and not pre-generated raster tiles. I want to pre-generate the needed raster tiles (.png images in 256 x 256 sizes).

I’m now able to import data to PostGis db (using the files from geofabrik.de) and to render tiles without any errors showing up in Windows cmd window - using PostGis, osm2pgsql, mapnik and generate_tiles.py. It does not look like there are any font issues either.

BUT… something is going wrong, because a lot of information in the map is misaligned, roads and road-parts are missing and some areas are marked with NR. I also get some stright lines in pink and yellow across, or in some parts of some of the tiles.

Do you know what can be the reason for this?

When my attempt to pregenerate raster tiles on linux failed, then I followd this recipy for installing PostgreSql, osm2pgsql, mapnik on windows (windows 7):
https://openstreetmapserverwindows.blogspot.com/2015/11/osmopenstreetmap-tile-server.html

I have a copy of the same area - with same zoom level (11) for the map using pre-rendered raster tiles (from “my” server), and from openstreetmap.org. (my tileserver is actually on another domain, and the web-application that is to use openstreetmap etc. is not availble for other than me in the development process. Since I could only show images in this comment that is allready on Internet - I had to upload the images to another domain of mine):

When I import the data to the PostGis db I’m using this:

SET PROJ_LIB=N:\proj\nad
osm2pgsql -U postgres -m -d osm -p planet_osm -E 3857 -S “N:\osm2pgsql\default.style” N:\osm-data-files\iceland-latest.osm.pbf

Then I use this to get the style needed for the generate_tiles.py:

python generate_xml.py osm.xml output_osm.xml --dbname osm --prefix planet_osm --symbols symbols/ --world_boundaries world_boundaries/ --host localhost --port 5432 --user postgres --password postgres

Here is the output from osm2pgsql when importing data:

N:\osm2pgsql>osm2pgsql -U postgres -m -d osm -p planet_osm -E 3857 -S “N:\osm2pg
sql\default.style” N:\osm-data-files\iceland-latest.osm.pbf
osm2pgsql SVN version af61cae663 (64bit id space)

release notes: ‘Windows version built by Dominik Perpeet (http://www.customdebug
.com/osm2pgsql/index.html)’
Using projection SRS 3857 (EPSG:3857)
Unknown flag ‘nocolumn’ line 157, ignored
Unknown flag ‘nocolumn’ line 158, ignored
Unknown flag ‘nocolumn’ line 159, ignored
Unknown flag ‘nocolumn’ line 160, ignored
Unknown flag ‘nocolumn’ line 161, ignored
Unknown flag ‘nocolumn’ line 162, ignored
Setting up table: planet_osm_point
NOTICE: table “planet_osm_point_tmp” does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table “planet_osm_line_tmp” does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table “planet_osm_polygon_tmp” does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table “planet_osm_roads_tmp” does not exist, skipping
Allocating memory for sparse node cache
Node-cache: cache=800MB, maxblocks=102401*zd, allocation method=8192
Mid: Ram, scale=100

Reading in file: N:\osm-data-files\iceland-latest.osm.pbf
Processing: Node(4803k 1601.2k/s) Way(278k 46.44k/s) Relation(6170 617.00/s) pa
rse time: 19s

Node stats: total(4803473), max(2147122507) in 3s
Way stats: total(278645), max(610011980) in 6s
Relation stats: total(6176), max(8467501) in 10s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Writing way (278k)
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Writing relation (6167)
node cache: stored: 4803473(100.00%), storage efficiency: 50.00% (dense blocks:
0, sparse nodes: 4803473), hit rate: 72.20%
Sorting data and creating indexes for planet_osm_point
Analyzing planet_osm_point finished
Copying planet_osm_point to cluster by geometry finished
Creating indexes on planet_osm_point finished
All indexes on planet_osm_point created in 3s
Completed planet_osm_point
Sorting data and creating indexes for planet_osm_line
Analyzing planet_osm_line finished
Copying planet_osm_line to cluster by geometry finished
Creating indexes on planet_osm_line finished
All indexes on planet_osm_line created in 7s
Completed planet_osm_line
Sorting data and creating indexes for planet_osm_polygon
Analyzing planet_osm_polygon finished
Copying planet_osm_polygon to cluster by geometry finished
Creating indexes on planet_osm_polygon finished
All indexes on planet_osm_polygon created in 4s
Completed planet_osm_polygon
Sorting data and creating indexes for planet_osm_roads
Analyzing planet_osm_roads finished
Copying planet_osm_roads to cluster by geometry finished
Creating indexes on planet_osm_roads finished
All indexes on planet_osm_roads created in 2s
Completed planet_osm_roads

Osm2pgsql took 59s overall

N:\osm2pgsql>

As SK53 has said on the help site on https://help.openstreetmap.org/questions/60188/best-way-to-custom-render-osm-png-tiles-in-windows-environment, it looks like something can’t handle 64-bit integers. In order to get to your end goal of rendered tiles, I suspect it’ll be much quicker to follow a “soup to nuts” guide rather than try and get what currently have working.