Cyber attacks in the OSM space

In my opinion, the description of

is pretty much accurate and close to what’s in the Yellow Pages thereunder.

If there are studies that go beyond, it is certainly not crowding the market place, perhaps some “basic research”?

So true. My point is one step further: we have to accept harm to our the data, because we do want to keep free and open public editing. Intentional abuse cannot be fully prevented without closing the door, and taht would be the end of OpenStreetMap as we know it.
The question is: (How) can we prevent that seriously compromised data reaches end user applications, causing unacceptable displays (current case) and functionality (less obvious but just as easy).

End user application providers face more or less the same problem: currently, they have to assume that serious errors can be in their data source; if they refresh very often they have all the improvements fast but are vulnerable to major errors; if they delay the refreshes they miss actuality (important for routers).

Note that at this point there is no difference between intentional and unintentional harm to the data. The criterium is: how bad/harmful/unacceptable is it for the end user applications using the data.

I think providing a cleaner, more protected source would be the best overall solution. Besides providing the opportunity to prevent distribution of harmful data, there are other benefits, which are discussed in amother topic, I believe.

The Foundation has started to seek out financial support for a
three-tiered solution:

  1. A tool to isolate and throw out all of the edits of one or more users
    in one step
  2. A downstream database that keeps only those objects and object
    properties (tags vs coordinates) that are stable over time X. With X
    being a day, a week, or a month.
  3. More tooling to track ongoing edits

I expect those things to take two to three years to come to fruition.
The good thing about the federal nature of OpenStreetMap is that
everybody is encouraged to produce third party tools quicker.

For example, I’m working on that tool to throw out users in one go,
robust against arbitrary user stacking, partial reverts, and any
changeset shaping that is based on Overpass spitting out an OSM file
that can be opened and applied by JOSM. That is not a great UX but
definitely the best step to figure out which finer semantics makes sense
when the time for a feature in openstreetmap-website comes.

Again, that is expected to take one to three months before something
becomes usable. We have already read a lot about the general mood, and
it might make sense as a next step to actually implement things. The
more third party tools are present to be scrutinized the better we
understand the actually useful semantics.

18 Likes

I would note two points

  • we already have a mechanism to redact edits at a very large scale (larger than anything that has happened in more than a decade), instead of creating something new, what about an effort to clean those tools up and make them run in the current environment.

  • we have plenty of existing tooling to detect bad edits, it just has the same problem any newly developed tooling will have, it wont be run by the OSMF.

2 Likes

What exactly does “stable” mean? Unchanged?

While “throw out” is rather vague; if you mean “revert”, we’ve already got one of those. The challenge tends to be that people using individual changeset-based revert tools that inadvertently cause more problems than “doing it properly” in the first place. If we think that a community-based revert has been more successful than it actually has (as was the case with the vandalism in the eastern Mediterranean last year) then subsequent reverts are more difficult.

If you mean “redact”, I think we have a good solution there too - I found it easy to script “redact all changes by users X, Y and Z” the last few times I’ve had to do it.

For the benefit of those that are lucky enough not to have been involved in OSM in 2012, this is the “redaction bot”. It was used for removing all changes made by certain users from the database and all dependent changes. Technical changes would be needed to make that Oauth2 complient (this is something that’s already been flagged up within the DWG). However, iIf you catch vandalism early, there tend not to be many dependent changes, and “reverting to the status quo ante” prior to a redaction is already supported by the existing tools.

However:

  • I’d probably have expected to see an initial approach to find out what we as an organisation already had before doing that?
  • It sounds like you are proposing to do option 4 as described here. Is that something that the board have decided upon and do they intend to put in place all the other requirements for that beyond the merely technical (support etc.)?

How that will (or even could) work really needs discussion in its own thread…

Edit: s/west/east above.

9 Likes

This comment suggests that the OSMF (or maybe the EWG?) is coming up with vandalism-prevention solutions without actually talking to the DWG, who sits on the front line of combating this stuff.

2 Likes

It isn’t just that, anything that would/could be inserted in the flow of data post editor and pre-diff/dump distribution is only going to fly if the buy in from the OWG and operations exists. Without that, it it is even less likely to be deployed than OSM-NG, aka “not at all”.

1 Like

Andy, thank you for the quick feedback. If there is a surprise then this is always a reason to proactively sort out and clear any potential misunderstandings.

When it comes to being able to respond to large scale bad edits,
you confirm that you have enough tooling to clear out the database from all edits of bad mappers in short time?

The redaction bot is (beside of OAuth2) fully operational and sufficient to yield the necessary reverts to get a cleared database replicated through the minute diffs? The redactions take place strictly after the reverts, and as such it is no problem that those redactions do never get replicated?

The thread around the Mediterranian event in 2023 shall not be interpreted in a such way that the DWG has felt overwhelmed? That is good to hear, because of an overworked DWG has made a couple of people felt nervous.

I will then encourage Brian (or anyone else) to rewrite the wiki page in a way that suggests is it still a useful and used peace of software. Or do it myself if it is not going to happen. (Brian has shown competence in writing documentation and is apparently following the thread.)

Do you realistically see the raster tile stack to persist still long enough that more than CDN level tuning is justified? Paul has made good progress in showcasing that a minute diff vector stack is possible. The uniform answer about vector tiles has been that they are also production ready in any other regard. Hence I would presume that the OSMF will no longer distribute raster tiles in any form in 2027 (and maybe earlier, with no promise).

In particular many people have expressed concern that the OSM Carto style has not exactly an abundant number of active maintainers and adviced rather against repair attempts.

The part of the message that I have apparently not made clear enough: the board sees it within its mission to get development funded where no volunteer activity takes place but development is needed. This and other threads give the quite clear impression that vandalism is indeed a problem where community members expect the Foundation to help protect.

This does not mean that the board has a micromanaging plan how to implement that. It is more like that at a point the board will approach the working groups and ask them what to do with the now available resources if they come into sight.

The other possibility would be to ask the working groups to stockpile ideas just in case some money comes around. With the back story of the stockpiled vector tiles, the stockpiled GDPR improvements, the stockpiled overhaul of the data model, and other stockpiled ideas this sounded like exactly the kind of idea stockpiling that most people consider unappealing and demotivating.

So yes, I’m absolutely interested in hearing further ideas. But I fully accept that other people rather want code than talk, and therefore will restrain from throwing ideas in with no prospect of implementation.

A downstream database that keeps only those objects and object
properties (tags vs coordinates) that are stable over time X. With X
being a day, a week, or a month.

How that will (or even could) work really needs discussion in its own
thread…

That deserves definitely a thread in its own. In particular if one of the most competent people about OpenStreetMap I know (Andy, you) has doubts that it is possible.

5 Likes

Speaking as an OpenStreetMap Carto maintainer, we have never advised anyone on vandalism. We provide a stylesheet that turns OSM data into a rendered map. Issues like incorrect names, data, and vandalism is nothing we can do anything about.

There are issues with maintenance, but these are more complex and tie in to the state of the CartoCSS and Mapnik ecosystem. As these are off-topic for a discussion about vandalism, I will not go in to them here.

The redaction bot works fine right now. It will stop working when OAuth 1 goes away, but that’s not a huge issue - it needs a whole bunch of updates really. The problem is that the software is so specialized that there has generally been zero to one people who run it. For 99% of redactions other software does just as well and is easier to use. For the other 1% it’s easier to struggle with software you know than get the redaction bot running.

Redactions have never been replicated. This is a largely theoretical issue for content that we can’t distribute, but not an issue for vandalism.

This is also a side issue. The redaction bot is designed for redactions, which are not necessary for vandalism. A revert is sufficient. What the redaction bot was designed to do, disentangling edits from each other, is not necessary for a revert. Large-scale vandalism does not go undetected for years with lots of useful edits on top of it.

5 Likes

Hypothetically, suppose someone managed to inject a high volume of edits in a short amount of time (let’s assume sleeper accounts or perhaps that a large number of regular accounts that were past the new-user edit volume constraints were compromised). I’m talking to the point where the inflation of the full-history planet file and/or changeset diffs became problematically large. Is that a real potential problem and if so do we have tools to combat it?

It is not a problem for diff size or the full history planet file. There are potential problems around complexity for downstream consumers of diffs - some consumers find some types of data hard to handle efficiently, but size by itself is not normally a problem.

Given how the full history file is 127GB compressed, to make a big impact to its size you’d need to have a load thousands of times our current API load. It grows by about 0.7% a month right now. A vandal would have to fit in 14 months worth of changes before they are detected to make the full history 10% larger.

1 Like

A bit of an aside about redactions, because not everyone reading this will be familiar with the process:

Normally, vandalism is just reverted - essentially, changed back to the value immediately before the vandalism. Here is an example of that - someone changed the tagging on something so that information was lost, and I changed it back to how it was before. The intermediate value can be seen.

In some cases, the intermediate version is something that can’t stay in the database, perhaps for licence reasons, or because someone added personal data that should be removed. In this case, after a revert, the intermediate version is redacted too. Here is an example of this. The redaction used in this case was this one. The full list can be seen here.

In that list you can see a couple of “License Change Redaction” entries that date from around the time of the licence change. This change was made by the redaction bot that was referred to above. As @pnorman says, it is rarely used nowadays (I have never found a need to).

3 Likes

Ask the many people who have made their first forum or mailing list post in the last couple weeks whether they believe the surprising map content was intentional or unintentional. :wink: I think it’s a good point to reflect on, because some vandals will be more persistent than even the most clueless of well-meaning mappers.

Countervandalism isn’t just about keeping the content clean; it’s also about keeping the troublemakers out so they don’t keep making trouble for us to clean up after, or I suppose your idea of quarantining them so that they waste their time. We can block vandals today, but how easy is it to evade a block, and how quickly do we catch block evasion? As I recall, we had an incident last year involving rapid account creation. Do we see sockpuppetry as an ongoing risk, or is it safely under control now?

The redaction bot did run at an awesome scale. It’s still an after-the-fact mechanism though. Some of the people who have come in to complain about this wave of vandalism have been asking how we don’t prevent obvious vandalism from entering the database in the first place. Maybe that’s an unreasonable expectation, but I can see where they’re coming from. It doesn’t take a rocket scientist to realize that Andy Townsend isn’t notable enough for his name to dash around the world like Santa Claus in December.

This flexibility to choose what to reject on the spot is very powerful. Folks have been quick to reject the idea of blocking edits based on a “naughty word list”, but I think this assumes a static word list, matched literally, that has to be developed out in the open at the same pace as all our other software.

As one of our wiki’s administrators, I appreciate the ability to set and modify rules on the fly. As with your script, the rules can be kept private and don’t have to be easy for the vandal to figure out. But unlike your script, the rules can also perform a variety of automated actions in response to tripping a filter, such as quietly logging a violation, notifying the user, blocking the edit from saving, or even the nuclear option of immediately banning the user and anyone with their IP address. Unfortunately, I think the last time I floated something like this for OSM, someone reckoned that evaluating every upload in real time would be atrociously unperformant, so I haven’t given it a lot more consideration. I’d love to be proven wrong though!

(By the way, if you want to talk cybersecurity™, one of the wiki’s abuse filters was inspired by an infamous case of accidental election interference a few years back. I’ll let you guess who.)

5 Likes

My movie plot (as famous cyber security expert Bruce Schneier calls them) cyber attack – I create a fresh account, sit down for less than an hour in front of zoomed out global view, draw a path with a few hundred nodes, tag it trail_visibility=no, name it “You are lucky today!”, upload that in a millisecond, inflate planet file by 0.000 permille and make tile renderers grind for more than a week even though it would be reverted in less than the time to create it.

OAuth2 version of redaction bot

3 Likes

Allow a limit to be set on expiry of line segments by tomhughes · Pull Request #2185 · osm2pgsql-dev/osm2pgsql · GitHub though only a fraction of tile renderers run world-wide will even use expire lists - most people who run a tile server, other than the OSMF, will simply rely on mod_tile’s “3 days is old” logic.

Is this why I can’t see the GPS traces layer on the map? I have both the traces and notes enabled but only see notes.

Of course there is a difference, but at the point where the data is flawed, the cause is no longer relevant for how to handle the flawed data towards distribution. The severity of the errors determines how to handle the data, and the cleaning/filtering/blocking mechanism doesn’t need to know the cause.
Intentional severe damage does have a higher likelihood of repetition, which makes measures at the side of data input important. It also stresses the need for a propagation blocking mechanism that can be applied in degrees of strength.
And I would add, the mechanism should not block the full propagation, just changed versions of objects that might be tainted.

I am thinking of a per-object “cleared for distribution” mechanism in the database, to which policies can be applied from automatic clearing to full block. That would enable full unblocked editing with direct feedback for mappers, while offering data users more secure data. Without setting up a separate “clean” database.

No, that seems to be an unrelated error. The GPS trace layer loads in a different map, but the request from osm.org seems to be broken.