Osm2pgsql and failed planet import

General talk
1

Hi everyone,

I’ve been attempting to perform a planet import for mapnik rendering for the past few weeks without success. I’ve setup my PostGreSQL database with PostGis correctly, including the various parameter tweaks. I setup PostGreSQL on a separate Windows Server 2008 box with 8GB RAM, in order to separate processes and not have osm2pgsql have missing memory issues. This box has plenty of space for the final database.

I setup a virtual box on our virtualization server using a CentOS distribution, obtained osm2pgsql from source, turned on the 64bit compilation flags and compiled it. This virtual box has 16GB memory.

I’ve attempted several times to import, without success. The last attempt I made was using a boundary box, hoping osm2pgsql would have an easier time just importing North America. I’m using slim mode and a 14GB cache.

Here is the command line and the output:

./osm2pgsql64 -c -s -m -C 14000 --bbox -165,13,-45,90 -U gis -W -H 192.168.1.131 -P 5432 -S ./default.style ./planet-110810.osm.bz2
osm2pgsql SVN version 0.80.0 (64bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
Applying Bounding box: -165.000000,13.000000 to -45.000000,90.000000
Setting up table: planet_osm_point
NOTICE: table “planet_osm_point_tmp” does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table “planet_osm_line_tmp” does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table “planet_osm_polygon_tmp” does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table “planet_osm_roads_tmp” does not exist, skipping
Mid: pgsql, scale=100, cache=14000MB, maxblocks=1792001*8192
Setting up table: planet_osm_nodes
NOTICE: table “planet_osm_nodes” does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index “planet_osm_nodes_pkey” for table “planet_osm_nodes”
Setting up table: planet_osm_ways
NOTICE: table “planet_osm_ways” does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index “planet_osm_ways_pkey” for table “planet_osm_ways”
Setting up table: planet_osm_rels
NOTICE: table “planet_osm_rels” does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index “planet_osm_rels_pkey” for table “planet_osm_rels”

Reading in file: ./planet-110810.osm.bz2
Unknown node type 8
Processing: Node(1173423k) Way(104443k) Relation(1070274) parse time: 175361s

Node stats: total(1173423751), max(1393174131)
Way stats: total(104443727), max(125432779)
Relation stats: total(1070274), max(1706327)

Going over pending ways
processing way (26160k)way_done failed: ERROR: out of memory
DETAIL: Failed on request of size 419430400.
(7)
Arguments were: 75277817,
Error occurred, cleaning up

Now the error occurs with or without 64bits, with or without the boundary box. It seems like the pending ways section of the code is behaving oddly, or I’ve setup something incorrectly. The size of the memory request is odd as well. Anybody have any ideas?

Best regards to all,
Francisco

2

Hallo Francisco, I came across your posting when searching for
“Going over pending ways
pending_ways failed: out of memory for query result”

I use the postgis-osm database since 2008, however, a friend of mine set it up with 32GB Swap space on Debian UNIX/x86, so I just had to import the database dump. I am used to the gazatteer module, programming C, C++ and C# applications. I use SRTM data as well in a separate postgres database.

I myself make surveys in skiing areas (alps) and I post them on http://www.skiraware.de/Ski, written in german. The bounding box is france-poland/denmark-northern spain. That worked quite well.

Now I’m trying to setup a new database with the europe file from geofabrik. I get the same error as you get, now, I got many more errors, so let’s go into datails:

checkpoint_segments: 64
autovacuum: off
Energiesparmodus Festplatte: aus
Avira Free Antivirus: Echtzeitscanner: aus

After setting the parameters to the values above I got no more errors except one: “pending_ways failed: out of memory for query result”.

I recon that there is but one chance to get around that: Debugging the code. You state that you “obtained osm2pgsql from source, turned on the 64bit compilation flags and compiled it.”

I’d like to know, how. Why? because I have a Win 7 x64 Professional at hand with 16 GB Mem, i5-2500K, 2 x 1 TB of disk space minus 124 GB for the decompressed osm-file. I’d like to setup a debug version of osm2pgsql, put a breakpoint to the “dangerous” statement and see what happens. I guess that some lib file does not have the correct version - or that the pending ways are empty, so memory allocation is out of order (uninitialized variable or something turned wrong by wrong bindings).

Can you tell me compiler details, lib versions and so on so we can give it a try?

Sincerely,
GNavigia.

Call me Tom.