Great stuff! I digged myself into this. I also found a few issues. I think this has to do with the previous issues commented, but the geometry simplification algorithm produces strange results (IMO), in that the result looks somewhat blocky:
Here is a piece of the west coast of Norway (when zooming in a low-zoom tile):
While working with this, I found some more issues. Could you or someone else list where issues regarding which part should be reported? Not sure what should go where. E.g. where should the above two things be reported? And where should the following be reported:
issues/feature requests with shortbread schema
issues regarding how the map turns out (e.g. the geometry stuff above, missing vector layers)
Meanwhile, I created a StreetComplete themed map style for the shortbread schema using pnorman’s demo tilekiln tiles (Click to view)
Now, I noticed, it loads very slow, and this is because the vector tiles are huge, especially on low zoom. (This is probably quite related to the geometry simplification algorithms used by tilekiln.)
I zoomed in once from the world map on zoom 0 to Hamburg city center on max zoom and measured the bytes transferred via the Firefox developer tools.
I compared it with versatiles.org, another FOSS map stack that uses the shortbread schema (but tilemaker instead of tilekiln to generate the tiles, maybe a few other things), and JawgMaps, a commercial provider of vector tiles known for their very small vector tiles (probably hard to compete):
The links lead to a StreetComplete-themed interactive map view each using the tiles from the respective provider. Feel free to explore. For the StreetComplete theme at least, apart from the geometry simplification thing, the most visible differences are that barriers and access-restrictions on roads are missing from the shortbread schema.
In the current OSMF tile servers database updates are done by osm2pgsql which passes a list of tiles that may have changed to renderd which updates the metadata for a block of tiles. When a tile which has metadata indicating it is out of date is requested the server attempts to re-render it on demand. Tirex is not used, but it is basically a drop-in replacement for renderd. mod_tile is used as a frontend for all of this and includes some logic to bypass renderd in some cases.
With the setup I have database updates are done by osm2pgsql, which passes a list of tiles to tilekiln which generates them at that time. When running in production tiles are never generated on-demand.
The former system means that if there is no load on the service the server is not very busy, but it scales poorly at higher load. It is also operationally difficult as any issue with the entire rendering chain can cause immediate issues for users.
The latter system has essentially the same load regardless of number of users, as fetching a pre-generated tile is very cheap. It is also operationally simpler because an issue with generating tiles will prevent updates, not serving users.
This looks normal to me. There is no simplification - it’s moving an unsimplified geometry to an integer coordinate space. I don’t know what tile you were looking at or how much it was overzoomed by, but it looked to be about a z5 tile overzoomed by about 5-6 zooms. I’m not worried about this, because it’s not a real use-case and any tile based system will look bad when overzoomed that much.
This is what a z5 raster tile looks like overzoomed by only 4 zooms
Vector tiles hold up better here because the edges remain crisp, but eventually the impact of the integer quantization become obvious.
Thanks for spotting this. I have it noted as a TODO in the source code. It wasn’t present in osm2pgsql-themepark’s table definitions so didn’t get added this in the rest.
Do you happen to have a list of tiles requested? It would be useful for testing purposes. Note that tilekiln is not yet compressing with content-encoding, so the bytes unpacked is the better value to compare. This is a known issue, but I want to solve this correctly to eliminate compressing and decompressing multiple times between the generation, storage, tilekiln, and the CDN.
Issues with the schema itself (as opposed to my implementation) can be raised on the shortbread github. We should be able to fix schema bugs fairly easily, but schema changes will need to happen in 1.1 or 2.0. It’s still good to get a list of feature requests in early so we can start considering them.
This lives in spirit because some code is shared between the shortbread implementation and the spirit implementation.
Tilekiln issues can be raised on github but any errors (e.g. 503 errors) are more likely to be something needing restarting on my setup. There are potentially issues that could be raised against osm2pgsql or osm2pgsql-themepark but those would arise out of issues against other components.
The tiles are from shortbread 1.0 so you can see the tags used there.
Yes, versatiles released a new style and previously some assets on their servers were moved.
You can. Label points for water polygons are in the water_polygons_labels layer. Adding them to the water_polygons layer would not help as it is not possible to label what is represented as a polygon in a vector tile because of how tile boundaries are handled. Without a link I can’t verify if the label is there or not.
The current TileJSON specifies a maximum zoom level of 14. Do you anticipate increasing it in the future? The website currently maxes out at z19, which would incur overzooming by five levels. (This PR proposing the OpenMapTiles port of osm-carto would’ve gone up to z23, which is pretty silly given that tileset’s maximum supported zoom level and MapLibre limitations.)
This is coming from Shortbread, so it’s not up to me. I’m not sure that adding another zoom level would help. It’s worth it if
most of your views are on z17+, or
service roads or buildings are included on z15+ but not z14.
The former might be true for osm.org but it won’t be for many, and the latter is unlikely to be true for any style mappers are interested in.
The answer has two parts. The first is that the zooms aren’t equivalent between the demo site and osm.org. The demo site uses 512 logical pixel tiles like almost all client-side rendering, and osm.org uses 256 logical pixels. z19 on the demo site looks like z20 on osm.org.
The other is a clarification on the original statement. Zoom 5 or similar zooms will look poor on when overzoomed by 5+ zooms when the tileset is following standard practices and the max zoom of the tileset is a higher zoom.
For zooms other than the last zoom you really only need an extent of 1024 or 2048 at most. Zoom 5.999 will be displayed at about 1024 device pixels per tile, and a DPR of 2 like most devices means that one physical pixel is about one vector tile unit with an extent of 2048 and you won’t easily notice errors. Higher DPI with a higher DPI don’t change this, since you can’t really see physical pixels at those resolutions.
In addition to this, at high zooms you are at a low scale and most features will not have the complex geometry that the fjords of Norway do at z5 so any differences are really only visible in a side by side comparison. Because we overlay data on osm.org that is 1:1 with the data in the tiles the side by side comparison case actually matters, but for most people it won’t.
The last zoom level is conventionally of extent 4096, but I could double this depending on goals. I’ll have a peek at the tile access logs to see how much z18 vs z19 traffic there is.
Thanks for the pointers! Will post feature requests at shortbread about barriers and access restrictions on roads.
No. But you can reproduce it by opening the Firefox Web developer tools Ctrl+Shift+I, reloading the page, and then see which tiles are requested when zooming in from 0 to Hamburg.
osm.org has a peak of users who zoom in as much as they can, but even on osm.org the peak is at a lower zoom, so we don’t want to sacrifice much at z14-z16.
I did this. For everyone else’s reference, jq -r '.log.entries[] | .request.url' hamburg.har | sed 's@https://demo.tilekiln.xyz/shortbread_v1/\(.*\)\.mvt@\1@' > hamburg.txt will extract it.
71 tiles were requested. This is as a Galaxy S20+ Android 11 in portrait mode.
I am wondering, will the final style look similarly to the current OSM Standard layer, or are you working on a completely new design? Or is the design even considered at the current stage?
I have also one suggestion about the code itself. When using Python without async, especially for web services, you are leaving a lot of performance on the table. Especially when using FastAPI (which derives most of its benefits from ASGI). The sooner you start writing async, the less painful the transition will be later .
Are you planning on adding compression for the tiles and possibly other optimizations? They are currently quite big in size.
In general, I think the better approach with vector maps (on OpenStreetMap) is to have
a. one base map that only shows the common things you’d expect on any map and then
b. have special overlays for various data that are shown on top of that base map and highlight the given data, e.g. metro and public transport network, bicycle infrastructure, max speeds, hiking routes, building types / 3d buildings, power infrastructure, POIs (shops etc.), …
Point b could be implemented by pre-crafted overpass queries with coloring applied or additional vector layer sources.
This approach has several benefits:
maintainers of the then-official openstreetmap vector map style are not under pressure to perform the balancing act between creating a style that shows as much as possible to demo to the world the detailedness of OpenStreetMap data and a style that is actually clear, not overfraught with details, to end users. This produced a fair bit of contention in the past with osm carto
The default OpenStreetMap map will look better (clearer, less busy) for normies
If the overlays-button is not too hidden and overlays are still pretty and linkable, normies can still use and enjoy the OpenStreetMap detail by using these
OpenStreetMap contributors will have an easy to reach tools on the website website to view, analyse and maybe debug aspects of the map data that is currently not even visible on osm carto (see examples in point b)
Doh, thanks for pointing out this discrepancy – it’s so easy to forget.
On this note, is openstreetmap-website integration with a vector map currently on the roadmap? Is the plan to migrate to MapLibre like the demo site, or continue to use Leaflet but with a MapLibre compatibility shim?
(Click to view)
.