Good morning,
As per title, is it possible to run the
osm2pgsql --create
command and import only tiles for specific zoom levels (say 5-10) ?
The objective being having a small database for accommodating a small disk size?
Thanks
Good morning,
As per title, is it possible to run the
osm2pgsql --create
command and import only tiles for specific zoom levels (say 5-10) ?
The objective being having a small database for accommodating a small disk size?
Thanks
That’s not really how it works - you don’t “import tiles”.
osm2pgsql loads raw OSM vector data into a Postgres database. You then use a series of tools to interrogate that database and make tiles out of it.
If you want to minimise disk usage, then you will need to tell osm2pgsql only to import a subset of raw OSM features. You do this by rewriting the Lua script that osm2pgsql uses to choose data.
But to be honest, if I were looking to make a small-scale map with minimal disk usage, I would probably avoid OSM entirely and use Natural Earth data instead. (Or, of course, use a vector tile solution that doesn’t involve a database.)
Thank you @Richard !
It is probably too late to change the decision to use OSM (not mine) as the front-end integration with the maps is already done (and is currently using the free tile endpoint: tile.openstreetmap.org).
Am I right to understand that, even if I would only ever ask zoom levels between say 5-10, I would still need to have a DB which is several TB if I want to show all the world? Maybe I am wrong in this basic point and it’s probably best to double check (unless of course I modify the lua script you mentioned).
Thanks a lot for your help.
planet.osm → postgis → mapnik → map tiles → tile server
Do you want to render your own maps (tiles) or do you just want to cache or mirror (osm carto) map tiles?
OpenStreetMap Carto - OpenStreetMap Wiki
Tile Usage Policy
Creating your own tiles - OpenStreetMap Wiki
https://switch2osm.org/
My website needs to show maps of all the world and that requires serving tiles between levels 5-12 (like https://tile.openstreetmap.org/5/20/11.png).
Back in the days we used to simply use tile.openstreetmap.org. Then we discovered that it had a fair usage policy (rightly so) so we decided to host it ourselves. I can proxy and cache tileopenstreetmaporg with very little effort (and is what we are doing right now to be less of a burden to tileopenstreetmaporg) but that might still have issues if tileopenstreetmaporg decides for any reason to block us. The only true independent solution was to “host our own tile server”.
Is this wrong? Please do tell!
PS: anything which is not 1:1 compatible with tiles from tile.openstreetmap.org is probably impossible to use as it would require to re-develop the frontend.
Thanks!
PS: I had to remove some links for the post to be published
As Richard mentioned , you could filter out a lot of things that only shows beyond z12 from the lua script used by Osm2pgsql, this way, less data would be imported, and you will end up with a smaller database.
Of course you need to dig a little to understand what you are doing.
In particular, filter out buildings!
Unless the OWG has contacted you, your level of tile usage has not been a problem. We always try to contact users before blocking.
Of course, in the terms we do reserve the right to block anyone and it comes with no SLA. If that is a concern, you need to either contract with a company that has a SLA or host it yourself.
Implicit in the decision to use tiles like the Standard Layer is the choice of using server-side rendering, rather than something client-side like MapLibre.
If you’re going down the route of using OpenStreetMap Carto you can do a regular import and pre-render zoom 0 to 12. You’ve then got a bunch of files you can stick on a web server and have no rendering on demand.
If disk space is an issue when doing an osm2pgsql import with osm2pgsql --create --slim --drop
then you can modify the tag transforms to exclude some data. You’ll gain most of the benefit from excluding buildings. These are objects where the only keys imported into columns are building
and possibly name
.
OK - but how much data do you actually need? As an example, at zoom 10 OSM’s “Standard Style” shows an awful lot of information. It’s entirely possible that you don’t want anything like that amount of detail, and would actually prefer something with less so that you can overlay other information over it.
The resulting tiles can be compatible with whatever you’re using to display them, even if you source data from (say) Natural Earth or use simplified shapefiles for some things.
If you want an example, here is one that uses osmium tags-filter
and then osm-tags-transform
with this to remove some tags from the database. I’d suggest starting with a small area and seeing how much smaller you can make the database by omitting data that you do not need - and perhaps using Natural Earth shapefile data at low zoom (example) too.
Thanks everyone for all your replies!! Appreciated!!
To be sure I understand, are you talking about running something like GitHub - Magellium/osmtilemaker, create tiles and then serve them as static files? Or is there anywhere I can “just download tiles”? I read in the fair-usage that I cannot scrape tile.openstreetmap.org.
I am trying to run a full planet import to see what it really means in terms of disk/time:
osm2pgsql --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua --slim --drop --cache 4000 --number-processes 8 --multi-geometry --flat-nodes nodes.cache planet-latest.osm.pbf
…so far i’m half-way done with “Ways” and at 440GB of DB data.
If I wanted to go down the “modify the lua” route… does that mean I need to modify the openstreetmap-carto.lua file, right?
Thanks!
If you just want to display a map, you should consider ProtoMaps (open source). No need to run a tile server, you just host a single static file, which contains all the vector tiles. 120 GB for full planet, up to zoom 15 (can overzoom to higher levels). The map viewer is MapLibre.
You’ve already started, but I’d suggest actually deciding what you want to show before doing that.
That might be part of it, but the main thing is to remove data from the planet file before importing - see the examples above.
While that would of course be a suitable answer in some cases, based on what OP said (" 1:1 compatible with tiles from tile.openstreetmap.org") it probably doesn’t fit here.
Thank you everyone!
Indeed ProtoMaps seems very cool but would require to redevelop the FE which is not something our FE team wants to do.
To be more precise on our requirement (I just had a call on this matter):
I could:
If the whole world import doesn’t really take forever I might just a single import from the same complete dataset. As of now I’m leaning more towards the “create tiles” and then dump the whole machine which created the tiles. The whole thing can be scripted for when I need to update the tiles.
Again thanks for all the help!
Another option would be to bulk-load pre-rendered worldwide z1-z8 tiles and serve those statically, and only dynamically render the more detailed tiles for your country (i.e. skip #1 and only do #2). The raster tiles for worldwide z1-z8 are about 87,000 files, taking up less than 1 GB.
(Please note: OSM doesn’t allow bulk downloads of tiles, but vendors like ThunderForest or Geofabrik can likely provide these for you).
Thanks @GeoDeskTeam!
Good idea! I could have an apache configuration which serves /1/* - /8/* from a static folder and loads /9/* - /15/* from mod_tiles, which is linked to a db with only my country loaded up. Right? This would be easier for updating my country and wouldn’t require to spin a machine on-demand every time.
I would need to check hw requirements for this since I would still need beefier machines as opposed to only have 20-50GB of static files.
All in all the planet import is not such a problem. It only takes around 10-15hs depending on hardware and needs a 1TB drive as of now.
On the other hand the tile generation (using osmtilemaker) is so slow it’s crazy!
I was thinking of MapProxy, but I’m sure there are multiple pieces of software that can generate this.
A few years ago I created low zoom prerendered tiles but no one cared.
Yes.
If you’re not inclined towards changing this software, I’d just filter out building ways from the OSM PBF.