Optimal budget hardware for postgis + tile Ubuntu server

General talk
1

Hi! I’d like your hardware recommendations for a budget postgis/tile server for planet size maps.

Hardware-wise I’m thinking something like:

Zen 4 motherboard
Ryzen 5 CPU
1TB WD Blue m.2 drive (I already have this, 2.4GB/sec read, 1.9GB write)
32GB of RAM

More details:

I’m setting up a budget “desktop server” from a few parts I already have. It will do some very basic home server tasks, but I’d like to upgrade it to be able to import planet in a reasonable amount of time and run a tile server as a hobby project. I’m aiming for being able to import planet with osm2pgsql in somewhere below 24 hours, hopefully closer to 12, and am trying to figure out which parts to prioritize on and where I can save.

I suppose the 1TB drive will have ample space and speed for running OS, database, and import/update procedure?

Memory: 64GB is reachable within my budget, but I’ll probably not be able to avoid --slim anyway, and with the relatively fast drive I hope that I can get acceptable import and update times with 32 GB to save some money.

CPU: should I go for more cores (6 or 8 instead of 4) or higher single core speed? Or does it matter much at all with the drive and memory I’ll have available?
I might consider a Ryzen 7 2700X if it’ll drastically improve my import speeds, otherwise I’d prefer to go with a cooler running low-end Ryzen 5.

If the m.2 drive will be the bottleneck I suppose I can swap out a slightly faster Samsung 970 (3.5GB/sec read/write) from my gaming rig, or even get a gen4 drive if it has a HUGE performance impact. But that would impact my budget as well.

I have created an Ubuntu tile server with osm2pgsql → postgis and mod_tile/renderd a few years ago, and can admin my way out of a Linux-powered paper bag, so I think I’ll be able to figure that part out.

If I decide I’ll cache large amounts of tiles I have a large magnetic drive I can throw in for that. The number of tiles served will likely be very low.

Sorry for the long post, but I tried to put the most relevant details at the top, and I find that more info is better in most cases.

Any thoughts on this would be greatly appreciated :slight_smile:

Thanks,
Anders
Oslo, Norway

2

Last week, I imported planet-220103.osm.pbf, using --flat-nodes option. Size:

  • 70GB for flat_nodes.bin
  • 1.1TB for db (on my BRTFS it actually compressed to about half that size)

The import took little over 7days on my Xeon® D-1527 4-core 64GB system, 2TB SSD (Evo 870), running in Docker

So, your 1TB drive seems to be too small for planet size

Few lines of the import logs:

sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua --number-processes 4 -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf -C 4096 --flat-nodes /nodes/flat_nodes.bin
2022-01-10 08:24:20  osm2pgsql version 1.5.1
2022-01-10 08:24:20  Database version: 12.9 (Ubuntu 12.9-0ubuntu0.20.04.1)
2022-01-10 08:24:20  PostGIS version: 3.1
2022-01-16 12:32:55  Reading input files done in 533315s (148h 8m 35s). 
2022-01-16 12:32:55    Processed 7 417 817 175 nodes in 5083s (1h 24m 43s) - 1459k/s
2022-01-16 12:32:55    Processed   827 147 780 ways in 395239s (109h 47m 19s) - 2k/s
2022-01-16 12:32:55    Processed     9 550 488 relations in 132993s (36h 56m 33s) - 72/s
2022-01-16 12:32:57  Clustering table 'planet_osm_roads' by geometry...
2022-01-16 13:09:38  Creating geometry index on table 'planet_osm_roads'...
2022-01-16 13:10:52  Creating geometry index on table 'planet_osm_point'...
2022-01-16 13:16:22  Building index on table 'planet_osm_ways'
2022-01-16 13:54:59  Creating geometry index on table 'planet_osm_line'...
2022-01-16 14:38:24  Building index on table 'planet_osm_rels'
2022-01-16 16:21:30  Creating osm_id index on table 'planet_osm_line'...
2022-01-16 16:26:32  Analyzing table 'planet_osm_line'...
2022-01-16 23:20:11  Creating osm_id index on table 'planet_osm_polygon'...
2022-01-16 23:28:40  Analyzing table 'planet_osm_polygon'...
2022-01-17 11:35:47  Done postprocessing on table 'planet_osm_ways' in 80365s (22h 19m 25s)
2022-01-17 11:35:47  Done postprocessing on table 'planet_osm_rels' in 403s (6m 43s)
2022-01-17 11:35:47  All postprocessing on table 'planet_osm_point' done in 7526s (2h 5m 26s).
2022-01-17 11:35:47  All postprocessing on table 'planet_osm_line' done in 14028s (3h 53m 48s).
2022-01-17 11:35:47  All postprocessing on table 'planet_osm_polygon' done in 39356s (10h 55m 56s).
2022-01-17 11:35:47  All postprocessing on table 'planet_osm_roads' done in 2605s (43m 25s).
2022-01-17 11:35:47  osm2pgsql took 616287s (171h 11m 27s) overall.
3

Thanks for your data, SlavikCA, I plan to do the import on similar machine.

However it sounds like maneatingduck can just use 1 TB disk with BRTFS (or ZFS) with compresion on and that would suffice, isn’t it?

There are some more shared benchmarks with importing the whole planet:

https://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks