Generating PMTiles for MapWithAI JOSM plugin using tippecanoe

hfs · October 3, 2023, 10:33pm

Summary

What tippecanoe settings do I have to use to generate a valid PMTiles file for use with the MapWithAI plugin in JOSM?

Background info

I’m trying to generate a data set in PMTiles format for use with the MapWithAI JOSM plugin. The plugin recently gained the ability to load data from custom PMTiles.

The input data set are potentially missing addresses in Brandenburg (Germany) as Point geometries in GeoJSON format: brandenburg_missing_addresses.geojson.

The tippecanoe tool can generate vector tiles as PMTiles (also MBTiles, or MVT trees) from GeoJSON files (and other formats).

Test v1: default settings

If I run

tippecanoe --force --output=missing_v1.pmtiles --name=missing \
  --no-tile-compression brandenburg_missing_addresses.geojson

I get a PMTiles file with zoom levels 0 through 14. Zoom 14 contains the full data. Lower zoom levels are thinned out and contain only a sample of the points.

I can load this data in JOSM. If you want to try it yourself, you need to have the MapWithAI plugin installed. Go to Data › MapWithAI › MapWithAI Preferences. In the bottom section with Selected entries add a new entry with the + button:

Service URL: https://gist.github.com/hfs/81492081e966ee9f70490802dea79398/raw/f783f06acee7605c1ef33cfafebbb45bee278762/missing_v1.pmtiles
Name: v1
Type: PMTILES

The data gets loaded, but only a thinned out zoom level. In areas with lots of potential addresses, only a few are loaded.

I’m testing with this location: OpenStreetMap (I don’t intend to add these addresses to OSM, there’s just a high density of reported addresses.)

Screenshot from JOSM: Only a few addresses are loaded into the overlay layer.

To compare this to the data actually contained in the PMTiles file, I’ve converted it back into one GeoJSON per zoom level with the help of tippecanoe-decode.

Here’s a screenshot showing the contents of the PMTiles file per zoom level in QGIS.

You can see the zoom level 14 is complete an not thinned out. These are the address nodes I would expect to get loaded into JOSM.

Instead, if you compare the positions of the actually loaded data with the different layers in the QGIS screenshot, you can see that zoom level 12 is the one that gets loaded into the MapWithAI layer.

Test v2: Only zoom level 12

Ok, after these results I thought that the plugin might be hardcoded to use zoom level 12.

So I created version 2 using

tippecanoe --output=missing_v2.pmtiles --name=missing \
  -z12 -Z12 --no-tile-size-limit --no-tile-compression \
  brandenburg_missing_addresses.geojson

-z12 -Z12 limit the zoom minimum and maximum zoom level to 12. So only this zoom level is generated. --no-tile-size-limit disables pruning of features, so all points should be present.

The result file is available here: https://gist.github.com/hfs/81492081e966ee9f70490802dea79398/raw/f783f06acee7605c1ef33cfafebbb45bee278762/missing_v2.pmtiles

I cannot load this data. Instead, I get this exception:

java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Vector tile layers must have a layer name
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:562)
	at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:591)
	at java.base/java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:672)
	at org.openstreetmap.josm.plugins.mapwithai.actions.AddMapWithAILayerAction.actionPerformed(AddMapWithAILayerAction.java:116)
	at java.desktop/javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1972)

I don’t understand why v2 results in this error, while v1 was created with almost the same options. Both used --name to set a name.

Test v3: Zoom 0–12, set a description

After v2 I thought, maybe providing only zoom 12 is a problem. And there’s also a --description option, maybe that sets the name the exception is complaining about.

tippecanoe --output=missing_v3.pmtiles --name=missing \
  --description="Potentially missing addresses in Brandenburg, Germany" \
  --attribution="GeoBasis-DE/LGB (2023): Georeferenzierte Adresse" \
  -z12 --no-tile-size-limit --no-tile-compression \
  brandenburg_missing_addresses.geojson

The result file is available here: https://gist.github.com/hfs/81492081e966ee9f70490802dea79398/raw/f783f06acee7605c1ef33cfafebbb45bee278762/missing_v3.pmtiles

This results in the same exception as v2.

Questions

After all this I’m very confused. My questions are:

What tippecanoe settings do I have to use to generate a valid PMTiles file for use with the MapWithAI plugin in JOSM?

Does the MapWithAI plugin use the information about the minimum and maximum zoom levels in the PMTiles file?

Does it make sense to provide thinned out lower zoom levels, or should I only generate one zoom level with the full data? Which zoom level?

What’s the difference between v1 and v2/v3 resulting in the exception? I don’t understand why v2 is missing some “name” that v1 has, despite the command line options to tippecanoe being so similar?

bdon · October 4, 2023, 6:28am

It looks like your outputs from tippecanoe are valid. Maybe it has to do with the tileset display logic in the JOSM plugin? There’s a link to an issue tracker here: GitHub - JOSM/MapWithAI

Tiled vector data is useful if you want to have visible overviews at low zoom levels for a large amount of data, more than can be fetched at once by the client. Generating a single zoom level doesn’t accomplish that. For 5MB of point data a single GeoJSON file might be a better fit.

vorpalblade-kaart · October 4, 2023, 12:07pm

PMTiles support in the plugin currently depends upon an algorithm that takes the bounds to download and converts it to a tile. The tile that “covers” the entire bounding box (from given OSM link) is at z12. I’ll look into whether or not it makes sense to make requests based off of the max zoom that the tileset supports.

Anyway, the issue tracker for the plugin is JOSM Trac (use the Plugin mapwithai component, autofilled if you click the link).

vorpalblade-kaart · October 4, 2023, 1:47pm

OK.

I’ve got a fix for v1 which I’m in the process of validating (the same code path works with MVT tiles, so I need to check that).

v2 and v3 appears to have an off-by-one error. I stepped through my code to check and see if it is my fault (yep), but it was surprising since I haven’t seen this with anything else.

The bytes that I’m reading from the tile is 1 more byte than needed. I’m reading 16018 bytes from the pmtiles source, which is why something “bad” is happening. It looks like I messed up the HTTP range request by one byte.

There should be a fix out in the next hour or two.

hfs · October 5, 2023, 7:26pm

Thanks a lot for looking into it! I’m happy to compile and test development versions if you want.

I imagine that tiles for editable objects need to be treated a bit different than tiles for map display. One would always want the highest available resolution. Lower zoom levels can contain fewer features, simplified geometries and rounded coordinates. On the other hand you don’t want to overload the client.

Maybe one approach could be to select which tiles to load starting from the center of the current map view going outside to the borders of the map. Since one needs to calculate which range requests to make anyway, one knows how much data will be loaded before making the actual request. Maybe setting an upper limit of how much data to load makes sense and then loading loading tiles up to the limit.