OSM Vector Tile Capabilities and Advancements as of 2023?

Foundation
1

I’m curious about state of the art of rendering, without direct consequences. The background is that there is discussion in the Engineering Working Group about vector tiles, and I would like to have the best available information. I know that I have might have missed developments every here and there, and others might profit from a thread as information source with a relevant title as well. I’m also aware that most of the information is in principle available, but for all older sources I also appreciate if you simply confirm along the link that the information is still up to date.

I also hope that it is easier this way to compare the various partial solutions that exist and discover where it is difficult for everyone or where existing solutions can be plugged to give a better bigger.

Are there currently services that promise long term availability of vector tiles?
To point people to to set up a website using that tiles?
Scalable enough to use it as an alternative layer, if wanted so or not?

What is the asserted or expected update rhythm?
Is known where the update bottleneck sits?

What vector tile schemes do such services support?
Is there a documentation or the conversion rules public for those schemes, where?

What is the approx size ratio of those tiles in comparsion to raw OSM PBF or XML?

Are there notable processing steps (e.g. generalization or feature selection more sophisticated than tag filtering, relation breakdown, areas breakdown over multiple tiles)?

What front ends exist and which formats can they consume?

Are there frontends (browser frameworks for end users) that support mixing various vector sources?
Really mixing with interdependent rules in a joint style or just overlay?
Can features be moved between different zoom levels (e.g. render on zoom 10 a zoom 11 feature from tile source A conditionally based on information from zoom 10 of tiles source B)?

What is the end user experience on mobile devices?

What support exists to make styles?
Documentation? Design software or websites? Whitelabel styles with sufficiently liberal licenses? Something completely different?

Answers to only some questions are also welcome. I’m trying to complement this way the information I have researched so far to not overlook things that are obvious to other people but not me.

11 Likes
2

These have been well-trafficked, but here’s some of my recent writings:

Also see the following relevant technology:

6 Likes
3

Lots of questions to unpack. I’ll start by pointing out that OSM Seed includes a minutely updating vector tile generator based on imposm and Tegola as part of a broader solution for standing up an OSM-like platform. I haven’t seen it mentioned much in EWG communications or notes, so it might be worth reaching out to the developers behind it for some perspective on how they implemented vector tiles.

In a parallel universe, OpenHistoricalMap has been serving up vector tiles since 2018. It must use vector tiles, because generating raster tiles for the whole world for each day since the dawn of time would be a rather difficult proposition. OHM uses OSM Seed’s tiler in conjunction with forks of many classic OSM components. A mapper can save some changes and refresh the page in a couple minutes to see those changes reflected on the map, just like on OSM.

The map on OHM’s homepage is a Leaflet map that loads these tiles using an older, openly licensed version of Mapbox GL JS and the mapbox-gl-leaflet plugin. This is not unlike MapTiler’s outstanding pull request to add a vector style to the OSM homepage. The main difference is that some client-side JavaScript filters the features dynamically based on a user-selected date. The vector map style also necessitated a fork of Overpass turbo.

Since the the vector tile endpoint and stylesheets are publicly documented and available for use by third parties, there’s also a standalone slippy map powered by MapLibre GL JS that third-party websites can embed. These embeds currently appear on some pages on the OSM Wiki, a regional gazetteer site, and possibly some other sites I’m unaware of.

Now for the caveats. The tile schema is changing continually without any backwards compatibility guarantee. The schema isn’t documented yet, because so far there hasn’t been very much external interest in writing custom styles against it. The default stylesheet is also a work in progress but functional enough to unblock day-to-day mapping routines.

Recently, OHM has experienced some bouts of downtime, but I’m not involved closely enough with the backend infrastructure to know to whether it has anything to do with the tech stack’s scalability or maturity. OHM’s planet is still only a fraction of the size of OSM’s planet (808.7 megabytes in OSM PBF as of writing). There’s a much lower volume of editing activity, but resource constraints can cause the tile generation process to get stuck when there’s a lot of editing going on.

On the flip side, some of the problematic tiles are much larger than any corresponding OSM vector tile would be, due to the sheer amount of data in OHM locally. For example, an OHM tile covering San José, California, at zoom level 10 weighs 11.05 megabytes (1.28 MB compressed) compared to 150.41 kilobytes (108.06 kB compressed) for an OpenMapTiles/Planetiler/Americana tile, even though OHM has hardly anything besides boundaries in this city so far. OSM doesn’t have to worry about such massive volumes of data in vector tiles because of its inherently limited scope compared to OHM.

5 Likes
4

Yes. Protomaps, MapTiler, Mapbox, and Thunderforest immediately come to mind. Mapbox and Thunderforest both have been providing vector tiles for over 9 years.

Generally the providers have documentation, but if you have a Maplibre GL style (which points at a set of vector tiles), setting up a page with a map is no harder than doing so with Leaflet.

The four providers above would certainly handle the traffic of being a featured layer. In fact, two of the current featured layers are using vector tiles server side. Moving to client-side vector tiles would, if anything, improve the scaling for the provider.

In general, anything that pre-renders tiles (e.g. OpenStreetMap Americana) or anything going to pmtiles or mbtiles will scale without any reasonable limitations. I’m sure the OWG could also host a pmtiles or mbtiles file if requested - the hard part is generating them, not serving them.

Variable. Thunderforest updates minutely, others are often approximately weekly. If pre-generating all the tiles from OSM PBF, the limitation is somewhere between every few hours to day. If using a database as backend storage, minutely is possible.

Generally it’s a question of if a system is designed to consume OSM diffs or not, and its entire pipeline supports incremental changes. Many don’t because there is no business need for minutely updates for most providers.

Generally each service supports one vector tile schema. There is some overlap (e.g. Americana and Maptiler are producing tiles with a common schema), but not much.

Variable. Of the examples above, I believe Maptiler and Protomaps are both open source, while Thunderforest and Mapbox are not.

This is not a meaningful question generally - a tile is a set of pre-processed data being delivered to a client where response times should be measured in ms, an OSM PBF is delivered to a server for batch processing where download times are in minutes or hours.

In general the processing steps for vector tiles are similar to raster tiles, except that you’re not using image libraries or applying styling. Techniques more sophisticated are essential for usable vector tile sizes.

Merging multiple features into single multipolygons or longer linestrings is essential for many layer types.

Realistically, these days all the front-ends are consuming a Mapbox or Maplibre GL style, and the style specifies a source of MVT tiles. Maplibre GL is the most popular front-end that’s not tied to a provider, but Leaflet, QGIS, and others will also read the styles.

Each Maplibre GL style layer specifies a source and souce layer, which is a vector tile source and layer. There is no difference between two style layers from the same vector tile source and two style layers from different vector tile sources, except the latter results in more HTTP requests, bandwidth used, and slower performance from having to deal with more data.

Better than raster tiles, as mobile devices tend to make better use of fractional zoom, high DPI, rotation, and the other advantages vector tiles bring. Remember, high resolution raster tiles tend to be quite large. Keeping tile size reasonable is essential for mobile, as poorly designed tiles will cause your app to go OOM and crash.

Maputnik and Fresco are both GUI-driven editors. I wouldn’t use them for complex basemaps, as the GUI approach doesn’t work well with really complex styles. I’ve been using Charites, which pre-processes multiple files into one Maplibre GL style.

I would not write raw Maplibre GL styles in a text editor for a collaborative project because the format does not allow comments and isn’t great with version control.

Maplibre documentation is adequate. Maputnik and Fresco both operate as websites.

Something you haven’t touched on is performance and tile size. To have acceptable performance across a range of devices, you need to work hard to minimise tile size and write styles in a way that keeps the number of style layers down by using what are called expressions.

Raster tiles cannot easily cause performance problems, but vector tiles can easily do so. It is possible to accidentally write definitions that produce vector tiles that are several MB, while even a large @2x raster tile is going to be measured in the 50-250kb region.

The single biggest complaint with anything showing vector tiles is likely to be battery usage, as poorly written styles cause more CPU and power usage, and poorly produced vector tiles require more radio usage, which draws power.

10 Likes
5

Thanks for answering the questions in depth, @pnorman! I don’t disagree with anything you wrote; here are some further observations for those who are just getting acquainted with the technology.

For completeness, here’s Mapbox in 2018 describing their roughly weekly cadence in the context of countervandalism. I don’t think Mapbox’s process is directly relevant to community-maintained infrastructure, because Mapbox reviews changes before deploying them selectively to production, something OSM won’t do as a practical matter. The release cadence is constrained by their ability to review everything, not by rote data processing.

(Mapbox’s full talk at State of the Map U.S. goes into more detail about how they bucket changes by layer, location, and day rather than by changeset. In theory, this would enable them to vary the update frequency between layers and from region to region, though I have no information about whether that’s the case. It also describes an aspect of countervandalism that we’re just now grappling with, automated account creation.)

Composing multiple vector tile sources (and indeed multiple raster tile sources) does have a potential benefit in making tile updates more modular. Providers like Mapbox take advantage of client-side compositing to provide a traffic tileset that’s updated and downloaded much more frequently than the basemap tileset. One could imagine a tileset for coastlines and landcover that updates somewhat less frequently than the tileset for places, roads, and POIs. Some services provide a layer for labels and icons so that users don’t have to download all the translations about everything.

There’s a caveat that prevents mixing and matching arbitrary vector tile sources together. A source defined by TileJSON (required by Mapbox/MapLibre) is divided into layers; no two sources loaded by a style can have conflicting layer IDs.

Postprocessing is also required for cartographic reasons, for example:

  • If the areas in a layer need to be labeled once at each centroid, then there needs to be a separate set of point features corresponding to those centroids. Otherwise, labels will repeat once per tile.
  • If bodies of water need to be outlined, then ideally each tile would contain a separate feature representing the union of all bodies of water within that tile, or perhaps the union of all coastlines/lakeshores/riverbanks, in order to keep the tile boundaries from getting outlined. (There is a workaround, but it’s generally seen as a good thing if well-designed tiles can keep styles from having to implement clever workarounds.)
  • Some map styles depict both boundary lines and colored halos ringing the interior of boundary areas. This potentially requires both linear and multipolygon representations of boundary features.

Note that all these tools compile down into the same declarative JSON format, which allows it to port across platforms quite easily. Mapbox/MapLibre GL and Tangram are all available not only for the Web but also in native application environments on major desktop and mobile platforms. The same JSON style generally renders identically across all these platforms.

Another approach is to assemble the style on the client at runtime using Mapbox/MapLibre GL’s imperative “runtime styling API”. This allows the style to be organized like any other codebase and vary based on user preferences, at the cost of portability; the “style” is actually distributed as application code. OSM Americana is one example of a style built in this manner.

Indeed, the 10-megabyte tile I mentioned earlier is mainly due to how redundant geometries compress poorly in the MVT format or any other Simple Features–compliant format such as GeoJSON. If a single way is part of 1,049 relations, each tile that covers it will need to include its coordinates 1,049 times over as part of 1,049 different multipolygons, whereas this data only needs to appear once in OSM PBF or TopoJSON format. This is an extreme example, but this illustrates how it can be less than straightforward to estimate the size of a tile based on the data in OSM.

3 Likes
6

@pnorman and @Minh_Nguyen have answered pretty much everything comprehensively so these are just a few additional notes.

What is the approx size ratio of those tiles in comparsion to raw OSM PBF or XML?

As well as what’s already been pointed out, note that vector tiles are traditionally only generated up to z14 or z15, and larger-scale viewing is achieved by “overzooming” these tiles. So any tilesize comparison isn’t like-for-like - z16 tiles are effectively 0kb. :slight_smile:

Are there notable processing steps (e.g. generalization or feature selection more sophisticated than tag filtering, relation breakdown, areas breakdown over multiple tiles)?

This very much depends on what stack you’re using. If you use something backed by a PostGIS database, you have all the spatial processing available to PostGIS. If you use a standalone tile generator (tilemaker or planetiler) then you don’t have that but they may still provide some vector tile-targeted geometry processing features. A couple of the issues that Minh mentions, for example, are explicitly solved in tilemaker.

The stack may also provide tag-mangling superpowers - if you serve from PostGIS then you may have loaded your database with osm2pgsql, which has Lua transforms; tilemaker also has Lua transforms for tags; and planetiler is getting a YAML-based ruleset I think.

What is the end user experience on mobile devices?

For cycle.travel’s mobile apps, I didn’t even consider using raster tiles for a moment. The advantages of fractional zoom and rotation/tilt have already been mentioned, but for navigation purposes, on-device vector tile rendering is much more flexible. For example, for the polyline overlay showing the user’s planned route, I’m able to show that above the road layer but below the labels, so you can still see the name of the road you’re riding on.

Vector tiles also make offline maps feasible. An offline download of roughly 1° x 1.5° in cycle.travel’s schema is between 12Mb (rural Scotland) and 55Mb (London) depending on feature density. Downloading an equivalent area of raster tiles to z18/z19 would be larger, unwieldy and server-intensive.

What support exists to make styles?

In addition to the options already mentioned, I’ll flag up glug, which I wrote and use. It’s a Ruby DSL that compiles down to Mapbox/Maplibre GL JSON. One of the nice things about it is that you can use a standard expression syntax for Mapbox’s “data-driven properties” styling (referred to by @Minh_Nguyen as “imperative runtime styling API”). So here’s an example of a contour labels layer:

layer(:contours_text, source_layer: :contours, source: :contours, zoom: 12.5..22) {
	filter (height % 20)==0
	text_field concat( height+100, "m" )
	text_font [ "NotoSansRegular" ]
	text_opacity 0.8
	text_color 0x905500
	text_halo_color 0xFFFDEE
	text_halo_width 2

(etc.)

Finally, you didn’t ask this question but I’ll answer it anyway; don’t think of vector tiles as a like-for-like replacement for the Mapnik raster stack, but rather, a completely different rendering technology with its own approach to web cartography. I’ve heard too many people say “oh, vector tiles are no good because they can’t do generalisation like <this>, or label placement like <that>, or pattern fills like <the other>”, and so on. Sure. There’s also lots of things that Mapnik can’t do, it’s just that we’ve all been using it for 50 years and have forgotten those.

A good cartographer will get decent results from either stack. Cartography isn’t just about the look of a map, it’s also about how the user interacts with it, and the dynamic, interactive nature of vector tiles gives many more possibilities here. (Obvious one: you can click/tap POI icons and make things happen. We’ve wanted that on osm.org for years!)

9 Likes
7

Could vector tiles be an opportunity to replace web Mercator with a projection centred on the point that is viewed and zoom levels with a constant scale panning north and south?

1 Like
8

Just to provide some hard numbers from the most recent render of Americana’s tile server:

  • Input planet.osm.pbf: 78,935,922,649 bytes
  • Output planet.pmtiles: 71,632,747,169 bytes

However, the pmtiles output contains adjacent-tile buffering which is not needed for our use case and we estimate that the planet pmtiles could potentially be shrunk by 17% with a new option added to planetiler.

2 Likes
9

Finally, you didn’t ask this question but I’ll answer it anyway; don’t think of vector tiles as a like-for-like replacement for the Mapnik raster stack, but rather, a completely different rendering technology with its own approach to web cartography.

+1, I am not sure it has been mentioned or is still very relevant in 2023, but vector tiles require more capable hardware (afaik all frameworks require open gl) and put more strain (power consumption for instance) on the end devices (at least if the image is rendered on the client and not on the server), so for a global project we might want to provide fallback raster tiles for non-gl users?

1 Like
10

You can produce raster tiles from vector tiles as a fallback option. OpenMapTiles at least provides this functionality.

3 Likes
11

I think the chances of the osm-carto raster tile stack being switched off in the next five or even ten years are pretty tiny. We’ll still have raster maps.

1 Like
12

By default, Mapbox/MapLibre GL expect vector tiles to cover twice as large an area as raster tiles; consequently, when we say zoom level 15 in a vector tile context, that’s equivalent to zoom level 14 in a traditional raster context.

Mapbox Streets tiles go up to vector zoom level 16. This requires more server-side resources but results in smaller tiles (by file size) at lower zoom levels where the additional detail may not get rendered anyways.

Planetiler’s original configuration mechanism is straight-up Java code, which has access to JTS (the progenitor of seemingly every geometry library). Presumably the relevant functions will be exposed in any forthcoming configuration DSL.

Mapbox GL JS supports several projections. By default, it gradually reduces the effect of the projection as you zoom in, since Web Mercator’s distortion is less perceptible at mid and high zoom levels.

You can also define a custom center for some projections, but changing it constantly could disorient any user who navigates the map by panning around. Someone once showed me a proprietary mobile map application for the Chinese market that automatically switches between different conic projections as you pan to different countries. Each transition was cool but rather jarring.

As an alternative, Mapbox supports a perspective projection (in other words, a globe view) that recenters dynamically as you pan. For legibility, labels are billboarded rather than being draped onto the surface of the Earth. You can try out this globe view at geojson.io.

Mapbox introduced these features after migrating to a proprietary license, so they aren’t present in MapLibre. There are outstanding requests for both alternative projections and a globe view.

3 Likes
13

Not quite every geometry library!

1 Like
14

Wow! This is unfortunate. Do you think there is any possibility that the MVT format could be extended to allow for one to many relationships so that replicating the geometry wouldn’t be necessary? Or is this how it always has to be for Simple Features compatibility?

15

“TopoJSON-format PMTiles” sounds like something you could plausibly hack up in a weekend.

16

Just out of curiosity - Are the renderers used on mobile apps vector tile consumers in the sense of this topic? E.g. my hiking app uses GitHub - mapsforge/vtm: OpenGL vector map library - running on Android, iOS, Desktop and browser. and it looks quite nice and detailed.

One drawback: It has fixed stacking of overlapping landuse/natural/areas. Can other vector tile renders do that - like OSM heuristics - the smaller is on top? Otherwise, this would change mapping habits quite a bit.

17

Yes, and some of these libraries are also available for the Web.

Yes, this is basic functionality in vector maps. In some renderers, the style can automatically sort features above or below each other based on a particular numeric property of the feature (Mapbox, MapLibre). The idea is that the tile generator would include a property that indicates the feature’s full area before tiling. However, this adds a lot of data to the tile that doesn’t compress well, so instead some tilesets presort the features from largest to smallest and the renderer paints them onto the screen in that same order.

2 Likes
18

On that note, Nextzen’s tile service, Tilezen, supports TopoJSON tiles as one of its three formats. It’s based on a GeoJSON vector tile service that @migurski and OSMUS used to operate.

For better or worse, MVT is a more interoperable format than TopoJSON. Backwards compatibility would be relevant to any EWG efforts unless also accompanied by investment in client-side file format support throughout the ecosystem. From informal conversations on OSMUS Slack, it does sound feasible to extend the PMTiles format to contain TopoJSON files. But Tangram is the only renderer I know of that supports TopoJSON tiles natively. For MapLibre, an application would need to convert those tiles to GeoJSON format on the fly; in turn, MapLibre would convert the GeoJSON back to MVT.

19

On the topic of tile size. I need to quote myself:

VTs are essentially unbounded in size, further it is difficult to predict how large a decoded tile will be. A single rogue tile could potentially use up all available memory. Besides offering a gigantic attack surface for DoS attacks, this creates issues with quite legit services, see Mapillary V4 issues #1435 .

See: MVT issues/enhancements · Issue #1396 · MarcusWolschon/osmeditor4android · GitHub

20

First of all, I would you to thank you all for the huge amount of information. I have indeed learned a lot. I’ll continue to follow this thread while I’m trying to make consistent answers to the initial questions (and some follow-up questions that implicitly appeared).

1 Like