Technical updates to the tile.openstreetmap.org service (OpenStreetMap.org Standard Layer)

Foundation
,
44

Yes. To take another example, with a tile server of mine (not an OSMF one, very little used) I’ve found it more useful to look at disk usage per raster tile layer, which is as follows:

1684192	15
1433544	17
1332140	16
1117856	18
1020884	14
515068	13
188708	12
127452	21
113420	19
109020	20
82692	22
67552	11
27188	23
26380	10
16144	24
10340	9
4168	8

Higher zoom tiles are smaller (and on this server tiles >= 16 are purged more often). I suspect that the usage pattern is very different to OSM though - less use as a “low res background”, less scraping, higher zoom levels supported, etc.

Edit: made clear “… not an OSMF one …”.

2 Likes
45

Thank you bunches, the one missing data puzzle to update the tile_disk_usage wiki page that I could not gather from the outside. Now I need a rainy evening night to put all together and bring back some usefulness into the wiki page.

46

Thanks for the data. Here’s a quick and dirty plot:

image
image457×281 14.2 KB

So a bit tricky to extrapolate to Z20. If we assume that the number of stored metatiles for Z20 is the same as Z19, this corresponds to a 28% increase in storage needed to serve Z20 tiles.

I would guess that Z20 tile would be quicker to render than Z19, so the increased demand on the rendering pipeline would be less than 28%.

47

On a system where I’m rendering z20-z24, I’m not seeing much difference between the the total elapsed time** to render z20+ as z19

** not the same as “demand on the rendering pipeline” - I’m not sure how you’d measure that. It’s also not the same style - it’s one that’s quicker to render, but more detailed at these zoom levels.

48

Over 80% of traffic to the tile.openstreetmap.org backends now uses IPv6 :nerd_face:

The rest is for servers where the hosting institutions is not able to provide IPv6.

5 Likes
49

Is it “uses IPv6” or “supports using IPv6” ?

Become former seems to imply only 20% of users still use IPv4-only (and IPv4-preferred) networks – which would be fantastic news, but e.g. Google’s IPv6 adoption stats show that only 46% of users use IPv6, and test-ipv6 similarly shows numbers close to 50-50.

So if we are really talking about actual IPv6 usage (number of connections or bandwidth usage), then it sounds that either:

  • OSM tile users are much more IPv6-advanced then the rest of the regular world, or
  • The results do not represent the users themselves (e.g. it can be that OSM servers see 80% IPv6 traffic because CDN always uses IPv6 when communicating with OSM server, regardless if users contacting CDN used IPv4 or IPv6[1])
  • something else?

BTW can we see OSM’s IPv4 vs. IPv6 usage stats somewhere (or could you show us snapshot at least) ?

  1. but in that case the number should be much closer, if not equal, to 100% ↩︎

50

I believe this was talking about connections between the CDN and the tile servers, not from users.

1 Like
51

The 80% figure was specifically about CDN to Backend Render Server traffic.

Here is approximate client traffic (end user ↔ CDN), regions are not exact:

Europe:
IPv4: 78.04%
IPv6: 21.96%

Asia:
IPv4: 71.37%
IPv6: 28.63%

Americas:
IPv4: 69.71%
IPv6: 30.29%

3 Likes
52

ysera has now had its BIOS settings tuned.

Summary:

  • CPU Configuration / Extended APIC → Enable
  • Advanced Power Management Configuration / Power Performance Tuning → “OS Controls EPB”
  • Advanced Power Management Configuration / Hardware PM State Control → Native
  • Advanced Power Management Configuration / Turbo → Enable
  • Chipset Configuration / Sub NUMA Clustering (SNC) → Auto
  • Memory Configuration / Enhanced Post Package Repair (PPR) → Enable
  • Memory Configuration / Operation Mode (PPR) → Test and Repair
  • Memory Configuration / Post Package Repair Type → Hard
  • Boot Configuration / Boot mode select → UEFI

The one-time longer Enhanced Post Package Repair found 3x ECC DIMMs with failing DRAM row/column/cells and successfully remapped these failing sections.

Initial load test showed a modest performance improvement of around ~12% when fully loaded. Primarily because linux now control over CPU Energy Performance Preference (EPP). When lightly loaded the per core improvement should be greater. Tuning in OS was confirmed using x86_energy_perf_policy and set using: x86_energy_perf_policy --all balance-performance. System now posts faster now that LEGACY option ROMs are no longer needed.

6 Likes
53

This post would be the right place to report the following?

2 Likes
54

Thanks for the report. The abusive tool has now been blocked. I unfortunately suspect they will try a cat/mouse game which makes running a public tile service more difficult.

7 Likes
55

There will likely be an influx of sites reporting Denied tiles…

It is mostly due the website using Referrer-Policy=no-referrer or Referrer-Policy=same-origin which blocks browsers from sending a Referer request headers. Without a Referer request header we are unable to easily identify who is using the tiles.

7 Likes
56

I have now temporarily removed the referer check for tile.openstreetmap.org. I will bring it back a bit more gently later. It had a larger than expected impact.

5 Likes
57

Thanks for removing the block. Can I request that you consider folks who run Leaflet etc from local html files in the browser? There is no way to set either the referrer or user-agent headers as modern browsers prevent changing these headers. I’m guessing the code is seeing the absence of the referrer header as meaning the caller is an app, but that isn’t the case when browsers run local files.

I completely agree with your goals and hope you can find a way around this knotty issue.

3 Likes
58

just GitHub · Where software is built got 16 reports (I rewrote Blocked tiles - OpenStreetMap Wiki message since then)

but given that operations get probably more in other channels - maybe sending all people with blocked tiles to one public repo/forums would reduce workload on ops?

and encourage interested osm community members to reply there? and reply with link there to any “please unblock my site” messages send to operations vie security issue email, linkedin messages, private mail and other misused contact channels?

from what I see, browser will send browser user agent in such case

59

Right, but the standard browser user agent is (was) not allowed by OSM. I think according to the terms of use that the name of the App is expected in the user agent when there is no referrer header. (I was able to get the tiles to load by temporarily changing the user agent in developer tools - but that’s cheating. :slight_smile: It isn’t supported in normal browser operation). I’m requesting a method to allow local files use with identification through a method supported by browsers.

60

No action is required at the moment. I rolled back the HTTP request referer enforcement because it broke too many things like as you notice file:// HTML documents.

1 Like
61

I have updated the Tile Usage Policy in an attempt to make it clearer, in part adding a few examples.

62

For anyone who, like me, was inadvertently breaching the Tile Usage Policy by globally setting Referrer-Policy: same-origin, at least leaflet has an option to set the Referrer Policy for tile images, avoiding the need to change it for a whole site.

I have set the leaflet option to strict-origin in my uses and can see that the referrer header is now being sent as required.

3 Likes
63

Since the OpenStreetMap community is multilingual, could the Attribution examples page provide examples in various languages to indicate that this is possible (ie OpenStreetMap-Mitwirkende, Contributeurs OpenStreetMap, colaboradores de OpenStreetMap, المساهمون في)