New 25cm Aerial Photos available from Chief Directorate: National Geo-spatial Information (CD:NGI)

[Firefishy]

It was recently announced on the OSGeo Africa mailing list that CD:NGI is hoping to have complete 25cm Ground Sample Distance (GSD) Ortho-Rectified aerial imagery coverage of South Africa by the end of March 2023.

Previously myself, @Adrian_Frith, @Gerhardus_Geldenhuis and others have collected copies/updates of the 50cm aerial imagery from CD:NGI in Cape Town. I have previously mostly been responsible for processing and hosting it.

The 25cm Imagery is 12TB and is made up of around 45000 GeoTIFF 280MB files.

The imagery will likely need to be reprojected to Web Mercator projection, tiled and overviews generated to be usable in JOSM, iD and other OpenStreetMap editors.

Most likely the source imagery will be converted to Cloud Optimized GeoTIFF with embedded overviews and served using TiTiler’s MosaicJSON support but this still needs to be finalised. @Adrian_Frith has previously used this method for NGI’s 1:50k topographic maps, but there may be scaling issues when running this against the significantly larger 25cm imagery dataset.

At the moment we are not sure where the imagery will be processed and what will be used for the final hosting. The OpenStreetMap Foundation do have a Dev server, but this “only” has 14TB of available storage.

[Firefishy]

CD:NGI normally provide the imagery for free. To collect the imagery one provides the “sales” desk @ NGI in Cape Town with an external hard disk and they copy across the imagery over a few days. CD:NGI have the option of charging time and materials, but this has been wavered for us in the past.

[Firefishy]

My ideal would be to upgrade the Cape Town based OpenStreetMap server “angor” with larger disks to host the processed tiled imagery in Cape Town.

But large server disks are expensive. 16TB SAS disks are around R6800 each and 4x are required.

It would be great to find a sponsor for the larger disks.

[Glennwithtwons]

We at SAEON have some plans for the WC 25cm imagery that will involve a similar workflow (cogs and titiler). We have an existing server that accepts http range requests we use to host and share cogs. I also have some project funds that could cover the costs of adding additional disk space to the server.

This type of service is probably within the scope of our mandate (we are publicly funded), so I should be able to get this approved.

Thoughts?

[Firefishy]

@Glennwithtwons Sounds fantastic!

What would be most practical? Options…

• A. You get the source files from CD:NGI and you process them to COG?
• B. We send you copies as we receive them from CD:NGI and you process them to COG?
• C. We process them to COG and send you copies?

Would you also host a public “titiler” yourself? Or just COG range request access?

[Glennwithtwons]

I would probably need help fetching the data from NGI. It would be a schlep to do it all myself.

Otherwise B or C would work. B might be smarter as we can do the compute on the same rack as the storage.

I would also consider creating a stac catalog of all the cogs. Having both titiler and a stac api that points to the cogs would be the best situation to make the data widely accessible. We could perhaps also host these services.

Let me send some emails to our infrastructure team. If they are amenable we can talk next steps.

[Firefishy]

LINZ (New Zealand National Mapping Agency) have a basemap service which is built using gdal, open source and their own open source custom software. It looks very impressive.

The documentation is limited (they have additional internal documentation), but they have some interesting pieces, like this example on COG Resampling with reprojecton and overview creation.

[Firefishy]

Estimated hosting costs on AWS (without any cost optimisation) is around US$500/month. AWS is unlikely to be a route unless we can get sponsored under the AWS Open Data Program.

I’ve also spoken to OpenAerialMap, they don’t think they have the resources (sponsored) to handle the size of the imagery. Not a firm no, but a strong unlikely.

[Glennwithtwons]

The infrastructure team at SAEON is happy with this and we have the disk space available, it is just a matter of provisioning it. We can also run the HTTP server to share the cogs and titiler service. I can work on creating a stac catalog and eventually get a stac api running too.

Do y’all want to move forward with this?

[Firefishy]

We’ve not yet been able to collect a copy of the source imagery files from NGI. Logistics of people taking time off work to get there.

The source imagery will then likely be uploaded to OpenStreetMap dev server. Estimate is likely 2 or 3 weeks to upload the imagery, but maybe copies could be sneakernet shared with others in Cape Town to allow faster parallel upload (bittorrent?).

Once we have a copy of the imagery on OpenStreetMap dev server, I and likely @Adrian_Frith (and others?) will try different COG options. The LINZ example looks like a good starting point to me. Likely take at few weeks to convert the imagery to COG.

The OpenStreetMap dev server does not have sufficient space to store a full copy of the converted imagery. I may be able to find additional temporary storage.

Once all the imagery is in COG format then happy to share with you via whatever works. RSYNC? Bittorrent?

[Glennwithtwons]

This is great. I would be happy to be part of the sneakernet. I would also like to play with optimising the creation of the COGs.
Once we get around to uploading the COGs to SAEON, we will give you access to the host machine and you will need to connect with our VPN. Rsync will work, not sure about BitTorrent

Let know when the data gathering will commence, and which tiles I should grab. You can reach me at:
glenn at saeon dot ac dot za
too if needed

[Glennwithtwons]

One open question that I need answered regards CD:NGIs policy on data resharing. Are they happy for us to redistribute their data? I can’t find any info about the licence they use

[Firefishy]

NGI have previously never placed any licensing restrictions on usage of the imagery. I personally consider the usage terms to effectively be Creative Commons — Attribution 4.0 International — CC BY 4.0. NGI outright own the imagery and initially started making the imagery available under the Promotion of Access to Information Act (PAIA).

[Firefishy]

Progress has been slow. We are struggling to get sufficiently sized external USB Harddisk(s) to collect the data from NGI.

@Glennwithtwons will likely visit NGI once we’ve been able to source the disks.

[Firefishy]

Hard disks have been dropped with NGI to collect the data! Thank you to @Glennwithtwons

[Firefishy]

I have a set of sample files now and have been testing different options for processing and compressing the imagery for production usage:

Using 3125AC grid square as an example.

WebP compressed Cloud Optimised GeoTIFF @ 70% quality:
4999343024 bytes (4.7GB).

WebP compressed Cloud Optimised GeoTIFF @ 50% quality:
3904456172 bytes (3.7GB).

WebP compressed Cloud Optimised GeoTIFF @ 99% quality: 18671775096 bytes (18GB). Sample not included.

Source files for 3125AC are 25 files JPEG compressed @ 97% (95% to 97% typical) as supplied are 8789910414 bytes (8.2GB).

[Firefishy]

The disks with 12TB of 25cm imagery have now been collected from NGI, and we’ll start the upload soon.

[Firefishy]

Upload has started! Processing the imagery will take a few weeks.

[Firefishy]

Unfortunately we have not received a full copy of the imagery. Another collection will be arranged.

[Firefishy]

Preview of the imagery is now available here: http://goldbox.firefishy.com:8080/
The preview site is slow and will be much faster for final version. The high zoom level quality will likely improve too.

Additional imagery will be collected from NGI next week.