how big is the planet loaded into postgresql?

1c7718e72f40f9a84070172b2cdcab80 · 2012-04-13T23:57:12.000Z

hi,

how big is the database (bytes) with the hole planet in it?

Lambertus · 2012-04-14T08:24:30.000Z

I’ll reproduce an email from Kai Krueger to the osm-dev list a few days ago:

There are two main components to the storage system of a tile server,
each of which can have different requirements depending on the circumstances

Tile storage cache

For the tile storage usually one needs quite a bit of space, but
performance isn’t quite as critical. For a general purpose world wide
map you will likely need somewhere on the order of above 600 GB. The
full world wide tile set is considerably larger than that, but rendering
on the fly of e.g. z18 ocean tiles is usually possible without too much
problems. I don’t know the exact scaling, but it seems like above
somewhere between 300 - 600 GB the cache hit rate only increases slowly
with size of the cache.

Performance wise, it appears like 1000 tiles/s will generate somewhere
on the order of 300 - 500 iops on the disk system, although that
obviously depends on the size of ram of the server and the distribution
of areas served. This is a level of performance that you can probably
get out a raid array of a few sata disks. The performance requirement on
this part of the disks likely scale fairly straightforward with the
number of tiles served per second. Adding a reverse proxy in front of
the tile server can also help reasonably to distribute load for tile
serving. For most tile servers I have seen so far tile serving hasn’t
really been much of an issue, but in the order above 1000 Tiles/s you
probably do need to consider it as well.

Rendering database

The rendering database is where for most people the disk performance
bottlenecks are. For the full planet, the postgis database with indexes
is around 300 - 400GB in size. This is as others have pointed out where
some people use SSDs for. Quite a bit of performance is consumed in
keeping the database up to date with minutely diffs from OSM. This
performance does not depend at all on how many tiles you serve, but only
the rate of editing in OSM. From what I have seen (and other might
correct me), a 2 - 4 disk sata raid array might not be able to keep up
with edits during absolute peak editing times (e.g. Sunday afternoon
European time), but should catch back up during the night. On top of
that is the actual rendering of tiles. As typically one doesn’t
re-render tiles in advance (other than low zoom tiles), but only once
they are actually viewed. Rendering performance does to some degree
depend on the tile serving performance. If it doesn’t matter how up to
date rendered tiles are, rendering requests can be queued and rendered
during quiet periods, which considerably reduces the performance
requirements on the database.

So overall whether you need an SSD for the database mostly depends on
how up-to-date you want to me with respect to OSM edits. If you want to
be in the order of minutes behind OSM, you probably will need an SSD.
Given that a fast update is important for mappers as reward for their
work, the SSD in osm’s tile server has been a big win. If daily updates
or less are fine, then you might not need one. Once you get down to
monthly updates, you are likely best not using an “updateable database”
but do full reimports, the size of the database reduces typically to
less than half the size.

It also depends on how spatially distributed your requests are. If e.g.
your site has a “bunch of locations” around which it displays local
maps. I.e. the same locations are shown over and over again, the
rendering load is much less than if you offer “Downloading country wide
tiles for offline use in a mobile app” even with the same amount of
serving load.

If you don’t need a world wide map, then hardware requirements also
considerably reduce and once you get down to only e.g. a country like
e.g. Italy or the UK, you possibly don’t really have to worry about the
database anymore at all, as any modern hardware is probably sufficient.

Kai

chris66 · 2012-04-14T09:45:52.000Z

http://wiki.openstreetmap.org/wiki/Planet.osm

The file is 25.000.000.000 bytes.

Database size depends on which data you are importing.

1c7718e72f40f9a84070172b2cdcab80 · 2012-04-14T20:14:46.000Z

thank you,

your answer helps me to estimate the upper bound.

but do you know the size of the postgis db with all nodes, ways, relations and users loaded into ram or the binary file?
and without tiles and no xml format. i think there is a difference between the 250 gb xml and the db binary file or ram usage.