Own OSM server - osm2pgsql - import speed

SebastianPa · 2020-05-18T01:58:21.000Z

Hello,

I’m new in OSM world with just few days of experience in buliding own OSM server on the VPS.

Importing Slovakia took just 845s overall …

The I run: osm2pgsql --append --slim -d gis --hstore --multi-geometry --number-processes 2 --tag-transform-script /home/osm/openstreetmap-carto/openstreetmap-carto.lua --style /home/osm/openstreetmap-carto/openstreetmap-carto.style -C 500 /home/osm/poland-latest.osm.pbf

It crashed after few hours asking for more memory so now from 48 hours I have it running with -C 1500 option.

Problem is that it already runs 48 hours just with the message:
Reading in file: /home/osm/poland-latest.osm.pbf
Using PBF parser.

so I do not know what is the progress - and even I could think that it hangs but based on 70% CPU utilisation I assume that its doimf something

Please advise:

How can I at least roughly tell how much is done?
Does it make sense to cancel it and start with flat-nodes option?
For future import of another countries can you recommend any other options to speed it up and better monitor the progress?

Thnak you very much in advance for any replies.

SK53 · 2020-05-18T10:14:13.000Z

It’s generally not a good idea to use the append option to try & add different country extracts at different times. There are multiple reasons for this, but the simplest one is that append is designed solely for updating the same area.

The best approach is to merge the two pbfs using osmosis or osmconvert and import them as a single file. A more involved one is to change the name of the target tables and build planet_osm_* as views on top of them. I’ve tried this and it works reasonably well, but requires some SQL & PostgreSQL knowledge.

Susi1010 · 2024-07-15T13:52:36.211Z

Hello,

I am currently working online with the OSM server. The extent is so far is Germany and EU but can be global in the future. To improve performance, we would like to set up our own OSM server. The posts I could find on this topic were older. I would therefore like to know what the current requirements and standards are for such a server, e.g. how much storage space and RAM should be available as a minimum. Are there any current instructions or own experiences?

Thank you in advance!

mboeringa · 2024-07-15T19:35:44.641Z

The official manual for osm2pgsql, which is the main importer for creating PostgreSQL / PostGIS databases, is here:

osm2pgsql.org

Osm2pgsql Manual - osm2pgsql

OpenStreetMap data to PostgreSQL/PostGIS converter

I strongly recommend you to skip the section about the old “Pgsql output”, as this option will be removed in the near future from osm2pgsql, and focus on the new “Flex output” which is more powerful and flexible in database layout. And to run a recent version of osm2pgsql as there have been major developments in the past four years.

There is some guidance as to needed hardware for Planet in the manual, but if you intend to process even bigger datasets and / or be prepared for them due to future growth of OpenStreetMap (e.g. Facebook Daylight distribution of OpenStreetMap, which unfortunately is also going to disappear in the near future), then some statistics and hardware specs can be found here:

https://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks

Susi1010 · 2024-07-17T12:16:22.610Z

Thanks a lot for the advice!