Getting full OSM data in a Mapbox/Maplibre file format

Caerandir · 2024-04-01T17:41:23.492Z

Hi all,
I am finding my way into Maplibre/Mapbox JSON vector data. I now seem to understand how a style definition works and how I can modify my map rendering to match my needs.
My problem is now: I fail to find a source for OSM data that I can render using such a style and which contains the complete OSM tag set. I am aware e.g. of MapTiler, but they heavily “edit” the OSM data. If I look for something exotic like e.g. “abandoned:highway=track” or “trail_visibility=intermediate”, there is nothing in the data.
My next try will be using Geofrabrik’s PBF files, but I really could use some helpful pointers and hints…
Thanks a lot!

Nakaner · 2024-04-03T12:10:56.362Z

First, Protobuf (PBF) is a format to encode structured data in binary form. PBF is used to encode various information. There is a PBF format for real-time timetables (GTFS Realtime), there is one for raw OSM data (.osm.pbf files from https://planet.openstreetmap.org/ or https://download.geofabrik.de/), and there is a PBF format for Mapbox vector tiles (this is optimized for rendering). Do not confuse them.

Vector tiles contain data optimized for rendering. This means, the contain only those attributes someone (the author of a vector tile schema) thought to be relevant. The attributes are often transformed before they are encoded (i.e. merging highway=* and railway=* into class and subclass attributes).

abandoned:highway=* and trail_visibility=* are OSM tags. I don’t know any vector tile schema that does not discard them. However, the software to create vector tiles is open source. For example, you can create your own tileset using Tilemaker with your Lua processing script. Tilemaker reads the .osm.pbf file and creates vector tiles.

Many map rendering engines using vector tiles support the usage of multiple tile sources in parallel. Therefore, you could filter the OSM data with osmium tags-filter before you create vector tile in order to minimize the raw OSM data.

woodpeck · 2024-04-04T07:41:53.129Z

see also: Demo: Full data vector tile map with basic filtering for QA

Caerandir · 2024-04-08T09:43:54.303Z

Thanks for your reply!
I found that Maplibre as “fork” of Mapbox is also a very viable option - it is open source and free, so no payments like with Mapbox if you exceed basic functionality. Maputnik as editor for styles is not too different from Mapbox studio, and I guess I’ll do my style work with Maputnik - but still in the learning phase, maybe I reconsider at some point.
Regarding getting the data, the tilemaker approach Nakaner pointed me at works nice and well (after some starting hickups - which are expected if a novice like me fumbles his way around).
My current feeling is that I now have all building blocks to do my project - and I’m happy about this. Will report back here when I have success and point to a full writeup.

Caerandir · 2024-04-03T13:41:42.818Z

Thanks Nakaner, very helpful! I also realized that Overpass now supports output as GeoJSON - I suppose that this might also be a road to walk down, but I guess I will first try the methods you suggested.

Caerandir · 2024-04-07T20:26:57.530Z

Hi Nakaner,
just installed tilemaker on Debian bookworm. First tried apt install… - was happy that it is available as precompiled package. Did not work, so I followed github instructions and did make/sudo make install, still does not work (made sure I used the compiled version). Error is always the same - when I run tilemaker with the example .lua script, it complains:
lua runtime error: demo.lua:34: attempt to call global 'Find' (a nil value)
It seems it is missing some library, but I installed all library packages I could get hold of - needed to fetch all the dev packages for successful make anyhow.
Not using “Find” does not seem to be a solution - it is a very fundamental LUA-Function for processing the data…
Any help appreciated!

EDIT: Seems I have found the reason myself: attempt to call global 'Find' (a nil value) · Issue #683 · systemed/tilemaker · GitHub - let’s see if I can get it running with this info…

EDIT EDIT: Using the tilemaker 2.4 .lua scripts solves it → now processing my first mbtiles map yay (which basically means I did not use my self-compiled tilemaker after all… huh? Well, I don’t care, prefer the Debian package anyhow…)

afgb1977 · 2024-04-07T23:23:06.878Z

With MapBox you can create your own styles and add extra data from OSM and/or third parties and/or your own.
The json styles are fabulous and allow customization to each person’s taste.
In MapBox Studio you can upload the data in different formats and represent it however you want in the .json style.
Although it is difficult to add external sources to MapBox, this can be done in .json code and combine various sources.
I am also a fan of .json styles and I am still learning about it, if you need help with basic issues I will be willing to do so to the best of my abilities.

Caerandir:

If I look for something exotic like e.g. “abandoned:highway=track” or “trail_visibility=intermediate”

I would do the following =
I would do a search in OverPass by Bbox or administrative area, download the data in geojson, upload it to MapBox Studio (I think it is 300 MB /month of free upload for external data for accounts free), once loaded I would add them to one of the basic MapBox styles, preferably Outdoors, you can represent this data however you want (colors,…), once edited you can use your new style via .json or xyz and more connection formats.