Nominatim import of Europe data

Thomas_DK · August 3, 2015, 8:05am

Hi,

I am trying to importthe latest PBF file for Europe using the setup.php script. The whole thing is running in a Debian (virtual machine) with 500 GB harddisk and 14 GB RAM. So far the import was running fine for a couple of days. This morning when I had a look at the machine again I saw that error message:

Processing: Node(1521712k 110.4k/s) Way(185933k 0.63k/s) Relation(193180 31.20/s)error reading blob content
Segmentation fault
ERROR: Error executing external command: /home/thob/Data/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore -C 432 -P 5432 -d nominatim /home/thob/Downloads/europe-latest.osm.pbf
Error executing external command: /home/thob/Data/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore -C 432 -P 5432 -d nominatim /home/thob/Downloads/europe-latest.osm.pbf

I did run the setup.php with the following switches:

./utils/setup.php --osm-file ../Downloads/europe-latest.osm.pbf --all --create-roads --osm2pgsql-cache 18000 2>&1 | tee setup.log

Here are some few lines from the end of the log file:

CREATE TABLE
COPY 32388
CREATE TABLE
CREATE TYPE
Import
osm2pgsql SVN version 0.87.3-dev (64bit id space)

Using projection SRS 4326 (Latlong)
NOTICE:  table "place" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=432MB, maxblocks=55296*8192, allocation method=11
Mid: pgsql, scale=10000000 cache=432
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: /home/thob/Downloads/europe-latest.osm.pbf
Processing: Node(1521712k 110.4k/s) Way(185933k 0.63k/s) Relation(193180 31.20/s)error reading blob content
Segmentation fault
ERROR: Error executing external command: /home/thob/Data/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore -C 432 -P 5432 -d nominatim /home/thob/Downloads/europe-latest.osm.pbf
Error executing external command: /home/thob/Data/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore -C 432 -P 5432 -d nominatim /home/thob/Downloads/europe-latest.osm.pbf

Can some of you help me to get finally the data in? This isn’t my first attempt and it always takes days to finally crash…

Bests Thomas

SomeoneElse · August 3, 2015, 4:40pm

I’d suggest checking that europe-latest.osm.pbf was valid by running something against it that selects a small amount of data from it (osmconvert / osmfilter, perhaps). Also (obviously) check the md5sum matches the .md5 file that I’m sure that you also downloaded.

See also https://help.openstreetmap.org/questions/12516/error-reading-blob-content-segmentation-fault.

I’d also check that the process works with a (much) smaller .pbf file before trying Europe, to make sure that nothing else is broken further into the process.