Does mapnik support pyramids/overviews?

Marcos_Dione · February 19, 2024, 8:09pm

I’m an idiot, that’s 1 or 3 orders of magnitude bigger, depending how one counts…

yvecai · February 19, 2024, 9:44pm

I think I get it: I remember now gdalwarp is not multithreaded.
Try gdal_translate vrt to tif, then gdaladdo - ro that should write new, smaller tiffs.
You’re working on an SSD at least, right?

Marcos_Dione · February 20, 2024, 7:51pm

But then I would have to vrt the overviewed TIFFs, and vrt does not support them, it only exposes the ‘main’ TIFF.

Yes, of course…

yvecai · February 20, 2024, 8:28pm

Why would you “vrt the overviews”? Declare new layers with each raster overview as tifff the appropriate zoom levels in Mapnik.

Marcos_Dione · February 20, 2024, 9:36pm

Ugh.

Also: how would gdaladdo would be faster than gdalwarp? Is 'warp so stupid it hasn’t figured out it’s only resizing and it’s somehow applying projection conversion too?

yvecai · February 20, 2024, 9:51pm

No, it’s just a screwdriver and you need a hammer, that’s all

Marcos_Dione · February 21, 2024, 5:42pm

github.com

yvecai/DEM-scripts/blob/master/2023/hillshade_mem.py#L11


      
          from osgeo import gdal
          from osgeo import ogr
          from osgeo import osr
          from osgeo import gdal_array
          
          from osgeo import gdalconst
          import sys
          import os
          import pdb
          
          ## Crop a global epsg:4326 DEM in 10° squares, compute hillshading and 
          ## wrap output to epsg:3857. No edge effects
          ## takes ~9h on 8 cores, nvme disk for a 1 arcsec / 30m DEM
          
          gdal.SetConfigOption('GDAL_NUM_THREADS', '8') # speedup warping
          
          infile=sys.argv[1]
          input_DEM=gdal.Open( infile, gdal.GA_ReadOnly )
          src_res=0.0002777777777778966603
          # ~ for lat in range(80,90,10):
          	# ~ for lon in range(-180,180,10):

You’re saying that you can crop, hillshade and warping/reproject the whole world in 10x10° tiles in 9h? Just warping Europe is taking ages here. And I have done this before, it didn’t take that long. That’s why I’m so surprised.

I thought of using plain ImageMagick to downsample, but it can’t read VRTs. I probably should ask in a gdal forum.

Marcos_Dione · February 23, 2024, 6:02pm

OK, according to WISH: Add overview support to VRT format · Issue #9280 · OSGeo/gdal · GitHub and VRT -- GDAL Virtual Format — GDAL documentation, VRT files dot explicitly declare overviews, but are clever enough to use them when available. I’ll try to do some tests this weekend.

Marcos_Dione · February 23, 2024, 8:20pm

This is amazing! It’s unbelievably fast to build and unbelievable fast to render afterwards! Technology is amazing!