The "OSM Standard tile layer" looks wrong (white lines, abusive comments etc.)

Understood, esp. with streets being moved way out of their normal base.

But looking at those prometheus graphs I see culebre and nidhogg not having any problems in re-rendering tiles, but odin and ysera have, so maybe even if those tiles are invalidated at e.g. odin, that machine is currently not able to catch up on all those invalidation jobs?

The backlog in the dirty queue gets rendered in the evening, see the “Queue Lengths” panel:

Queue Lengths panel for renderers serving Europe (click “dirty” bottom left)

As I understand, if the dirty queue is full (8K requests), like on odin and ysera during the day, additional tile requests get dropped and are re-issued with the next request, but may land in the queue or get dropped again.

Earlier today, the status of the z15 tile for odin was expired, but now it is rendered:

  • Last rendered at Sun Apr 11 08:14:19 2004.
  • Last rendered at Thu Apr 11 09:46:41 2024.

https://odin.openstreetmap.org/15/16786/10839.png/status
https://tile.openstreetmap.org/15/16786/10839.png

So a hard refresh with Ctrl-F5 should not only update the tiles in the Browser cache, but also at least your current CDN instance as it sends no-cache headers with the request.

1 Like

At this point, maybe more than just 12 months would seem appropriate

5 Likes

I’ve redacted the changes by the 3 vandal accounts used so far. That doesn’t cover “partial revert” ones, of which there are a few (such as https://www.openstreetmap.org/node/434940423/history - no offensive name there but you can see that the “initially reverted location” was incorrect).

2 Likes

HI, I just stumbled into this issue by chance, just browsing the map around Busto Arsizio, Italy:

Moderation: Map image with inappropriate and offensive street names removed

Exact location is: OpenStreetMap

Also, “alien” highways are visible in the surroundings of that area, and only if zoom factor is set to 16 (if I change it to 15 or 17, they disappear)

Hope this can be fixed soon.
Best things,
Max

See OSM Standard tile layer hacked? - #12 by SomeoneElse

:thinking: I guess it’s this same page…

Have you tried a hard refresh with Ctrl-F5 to update your tile caches?

2 Likes

Whilst I’m sure that most of the problems that people are seeing** are due to local browser cache, from where I’m sitting anything that you can do to improve the CDN cache invalidation would be gratefully received :slight_smile:

** at least 30 DWG tickets at the time of writing.

Edit: 34 now I think…

3 Likes

Surprising, since this is a classic easy, long-since-solved problem in computer science! Step up your game guys!

16 Likes

What would also be useful would be something written by the people in charge of the OSM infrastructure** explaining what actually happens between someone editing something in OSM and it appearing as the “standard” layer in their browser.

We (the DWG) are still getting lots of new tickets coming through for “vandalism”. In some cases in a new private browser I can see it too, but of course a shift-refresh loads new data that was generated behind the scenes and things are “fixed”.

** Not just a wiki page. Lots of people write wiki pages and not all of the wiki page authors are particularly familiar with what they’re writing about or are very good at communicating; I mean something like a blog post that we can link to that is (a) correct and (b) going to stay correct. Maybe someone from the CWG could help?

7 Likes

The changes are published in the minutely replication feed which is consumed by each of the eight (currently) render servers which update their databases and mark any tiles which are affected as dirty - all that will normally happen within a few minutes of the change being made.

Once a day the low zoom tiles (zooms 1 to 12) are re-rendered by each server regardless of whether they are dirty or not.

Everything else is actually driven from the client when you browse an area - your browser makes a request to a fastly CDN node and that will either return a cached tile or will ask a render server for it.

If it’s a low zoom tile, or a tile that exists and is not marked as dirty, then the render server will just return what it has.

If it’s a zoom 13 or higher tile that the render server doesn’t have a copy of or which is marked as dirty then it will queue it for rendering and then wait for a few seconds - if after that time the render has not finished then it will return the dirty tile or a 404 response for a missing tile.

The rendering continues in the background though so the next time a CDN node asks it will likely get the new version unless the render server has been so busy that it has been dropping render requests - missing tiles take priority over dirty ones so are less likely to suffer from being dropped.

9 Likes

Thank you for the detailed explanation.

However, there must be some fine print.

curl 'https://tile.openstreetmap.org/19/272529/174905.png' >a.png

delivers as of now a rendered ugly spike in the footway that has been
fixed in changeset 149801286 on 2024-04-10T03:41:50Z.

Could be that a dropped tile never returns to the rendering queue or
that marking tiles dirty work comprehensive but not entirely immaculate.
Or something else.

Given that Vector Tiles are under way, I do not this is worth busting an
arcane bug somewhere. But before people getting nuts seeing dirty tiles
even after having cleared out all browser caches, it might be fair to
say that most likely some corner cases simply exist.

Well that will depend on which combination of servers you are hitting - it is fine for me for example:

image

Rather than using curl you’d better testing in a browser where you use shift+reload to force the CDN cache to refresh - either than or figure out the switches to curl to do that because that example is likely just going to get a cached version.

https://odin.openstreetmap.org/19/272529/174905.png/status - marked as expired (2004):

Last rendered at Sun Apr 11 12:56:52 2004.

https://nidhogg.openstreetmap.org/19/272529/174905.png/status - rendered just recently:

Last rendered at Fri Apr 12 11:48:42 2024.

You can see what render server and cache the tile is served from in the custum x-* response headers, e.g. (or F12 > “Network” tab in the browser):

curl -sI https://tile.openstreetmap.org/19/272529/174905.png | grep -E '^x-'

x-tilerender: nidhogg.openstreetmap.org
x-served-by: cache-muc13967-MUC
x-cache: HIT
x-cache-hits: 1
x-timer: S1712926946.195807,VS0,VE1

The renderers odin and ysera are overloaded during daytime, see also #22.

@SomeoneElse I can help with this. I can write it up for the blog and make sure it’s posted on all social media and we can send it to the weekly OSM, too.

Can you give me the basic info? Or is there someone else who can? I don’t have a clear enough grasp to gather the most meaningful information, but I can absolutely make it coherent and clear for the general audience, both OSM’ers who might not know the nuance and people outside of OSM who notice the issue.

If you give me some paragraphs and send them to the CWG list, I can go from there. Or start a HackMD document. Or tell me someone I can quickly talk to who can explain it all to me.

1 Like

It’s basically what people including Tom wrote above, which was in answer to a question along the lines of “I noticed some vandalism in OpenStreetMap map tiles. People say the data is fixed, but I still see a problem in my browser. Why is that?”.

1 Like

Is there someone who can translate this slightly more into lay man’s terms? For this to be effective in the way that I think we want it to be effective, it needs to be more generally understandable. It’s fine if the person wants to tell me verbally. I can quickly hop on a zoom meeting.

2 Likes

I’d prefer to hold fire. We are investigating on a technical solution.

1 Like

In my area someone extended a Scarborough Terrace all the way through. Maybe related to socira/IOospa/kziessen vandal?