How long take the import of Europe data?

General talk
1

Hi guys!

I have a PC (4 cores Xeon E31225 3.10GHz, 16GB RAM, two disks WD Caviar Green in RAID10) with Debian 10 (PostgreSQL 11.9) and I’m importing the Europe’s data using the HowTo from https://switch2osm.org/serving-tiles/manually-building-a-tile-server-debian-11/

Currently the PC works since 32 days and it does not completed the first part of the import (osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script …).
I see currently PostgreSQL creates an Index for planet_osm_ways and this since 16/01/2021…

Do someone has an idea how long take, normally, the import of the Europe’s data? Just to know if I have to still wait a couple of day or a couple of months…

Thanks a lot
Luca

2

I have no clue, sorry.

But I recommend to try that import with smaller files first (Maybe you have done that already to test whether your workflow is OK).

If it works with smaller files, you can choose bigger files step by step, and the estimate how long bigger files may take.

3

I already created maps for Germany, Poland, Austria, Switzerland, Czeck Republic and Italy. So I developed my style as I need.
On this PC I’ll not experiment, just render.

Unfortunately:

  1. The PC on which I created the maps above had 32CPUs and 32GB RAM (a friend of mine lend me the PC, and I don’t have it anymore)
  2. I really can’t remember how long it tooked to import the data…
4

Just to know: why is needed the index

planet_osm_ways_nodes

?
I don’t find any statement in the code (got via git://github.com/gravitystorm/openstreetmap-carto.git) using this table…
And in my queries I always use planet_osm_point and planet_osm_polygon…

So, maybe I don’t need this index at all and I can safety kill the CREATE INDEX statement?

Thanks
Luca

5

This index is needed for osm2pgsql when using --slim --update. This is node tracking table. When applying .osc diffs nodes from this table are used if a way references this node.

6

Hi and thank you very much for your answer.

If I understand correctly, I can stop generating this index now, complete the rest of the import, render the maps and, after that and before I’ll apply the diffs (maybe end 2021), create the index, is it correct?

Thanks
Luca

7

Appying diffs over such long timespan probably makes no sense. Too many rows in geometry table will be touched. Some expert could chime in to confirm.

8

OK, then I’ll stop the generation of this index and let the job complete, so that I can start rendering.
End of this year I’ll get the new data of Europe, drop the database and restart import.

Thanks a lot
Luca