Cyber attacks in the OSM space

General talk
21

While “throw out” is rather vague; if you mean “revert”, we’ve already got one of those. The challenge tends to be that people using individual changeset-based revert tools that inadvertently cause more problems than “doing it properly” in the first place. If we think that a community-based revert has been more successful than it actually has (as was the case with the vandalism in the eastern Mediterranean last year) then subsequent reverts are more difficult.

If you mean “redact”, I think we have a good solution there too - I found it easy to script “redact all changes by users X, Y and Z” the last few times I’ve had to do it.

For the benefit of those that are lucky enough not to have been involved in OSM in 2012, this is the “redaction bot”. It was used for removing all changes made by certain users from the database and all dependent changes. Technical changes would be needed to make that Oauth2 complient (this is something that’s already been flagged up within the DWG). However, iIf you catch vandalism early, there tend not to be many dependent changes, and “reverting to the status quo ante” prior to a redaction is already supported by the existing tools.

However:

  • I’d probably have expected to see an initial approach to find out what we as an organisation already had before doing that?
  • It sounds like you are proposing to do option 4 as described here. Is that something that the board have decided upon and do they intend to put in place all the other requirements for that beyond the merely technical (support etc.)?

How that will (or even could) work really needs discussion in its own thread…

Edit: s/west/east above.

9 Likes
22

This comment suggests that the OSMF (or maybe the EWG?) is coming up with vandalism-prevention solutions without actually talking to the DWG, who sits on the front line of combating this stuff.

2 Likes
23

It isn’t just that, anything that would/could be inserted in the flow of data post editor and pre-diff/dump distribution is only going to fly if the buy in from the OWG and operations exists. Without that, it it is even less likely to be deployed than OSM-NG, aka “not at all”.

1 Like
25

Andy, thank you for the quick feedback. If there is a surprise then this is always a reason to proactively sort out and clear any potential misunderstandings.

When it comes to being able to respond to large scale bad edits,
you confirm that you have enough tooling to clear out the database from all edits of bad mappers in short time?

The redaction bot is (beside of OAuth2) fully operational and sufficient to yield the necessary reverts to get a cleared database replicated through the minute diffs? The redactions take place strictly after the reverts, and as such it is no problem that those redactions do never get replicated?

The thread around the Mediterranian event in 2023 shall not be interpreted in a such way that the DWG has felt overwhelmed? That is good to hear, because of an overworked DWG has made a couple of people felt nervous.

I will then encourage Brian (or anyone else) to rewrite the wiki page in a way that suggests is it still a useful and used peace of software. Or do it myself if it is not going to happen. (Brian has shown competence in writing documentation and is apparently following the thread.)

Do you realistically see the raster tile stack to persist still long enough that more than CDN level tuning is justified? Paul has made good progress in showcasing that a minute diff vector stack is possible. The uniform answer about vector tiles has been that they are also production ready in any other regard. Hence I would presume that the OSMF will no longer distribute raster tiles in any form in 2027 (and maybe earlier, with no promise).

In particular many people have expressed concern that the OSM Carto style has not exactly an abundant number of active maintainers and adviced rather against repair attempts.

The part of the message that I have apparently not made clear enough: the board sees it within its mission to get development funded where no volunteer activity takes place but development is needed. This and other threads give the quite clear impression that vandalism is indeed a problem where community members expect the Foundation to help protect.

This does not mean that the board has a micromanaging plan how to implement that. It is more like that at a point the board will approach the working groups and ask them what to do with the now available resources if they come into sight.

The other possibility would be to ask the working groups to stockpile ideas just in case some money comes around. With the back story of the stockpiled vector tiles, the stockpiled GDPR improvements, the stockpiled overhaul of the data model, and other stockpiled ideas this sounded like exactly the kind of idea stockpiling that most people consider unappealing and demotivating.

So yes, I’m absolutely interested in hearing further ideas. But I fully accept that other people rather want code than talk, and therefore will restrain from throwing ideas in with no prospect of implementation.

A downstream database that keeps only those objects and object
properties (tags vs coordinates) that are stable over time X. With X
being a day, a week, or a month.

How that will (or even could) work really needs discussion in its own
thread…

That deserves definitely a thread in its own. In particular if one of the most competent people about OpenStreetMap I know (Andy, you) has doubts that it is possible.

5 Likes
26

Speaking as an OpenStreetMap Carto maintainer, we have never advised anyone on vandalism. We provide a stylesheet that turns OSM data into a rendered map. Issues like incorrect names, data, and vandalism is nothing we can do anything about.

There are issues with maintenance, but these are more complex and tie in to the state of the CartoCSS and Mapnik ecosystem. As these are off-topic for a discussion about vandalism, I will not go in to them here.

The redaction bot works fine right now. It will stop working when OAuth 1 goes away, but that’s not a huge issue - it needs a whole bunch of updates really. The problem is that the software is so specialized that there has generally been zero to one people who run it. For 99% of redactions other software does just as well and is easier to use. For the other 1% it’s easier to struggle with software you know than get the redaction bot running.

Redactions have never been replicated. This is a largely theoretical issue for content that we can’t distribute, but not an issue for vandalism.

This is also a side issue. The redaction bot is designed for redactions, which are not necessary for vandalism. A revert is sufficient. What the redaction bot was designed to do, disentangling edits from each other, is not necessary for a revert. Large-scale vandalism does not go undetected for years with lots of useful edits on top of it.

5 Likes
27

Hypothetically, suppose someone managed to inject a high volume of edits in a short amount of time (let’s assume sleeper accounts or perhaps that a large number of regular accounts that were past the new-user edit volume constraints were compromised). I’m talking to the point where the inflation of the full-history planet file and/or changeset diffs became problematically large. Is that a real potential problem and if so do we have tools to combat it?

28

It is not a problem for diff size or the full history planet file. There are potential problems around complexity for downstream consumers of diffs - some consumers find some types of data hard to handle efficiently, but size by itself is not normally a problem.

Given how the full history file is 127GB compressed, to make a big impact to its size you’d need to have a load thousands of times our current API load. It grows by about 0.7% a month right now. A vandal would have to fit in 14 months worth of changes before they are detected to make the full history 10% larger.

1 Like
29

A bit of an aside about redactions, because not everyone reading this will be familiar with the process:

Normally, vandalism is just reverted - essentially, changed back to the value immediately before the vandalism. Here is an example of that - someone changed the tagging on something so that information was lost, and I changed it back to how it was before. The intermediate value can be seen.

In some cases, the intermediate version is something that can’t stay in the database, perhaps for licence reasons, or because someone added personal data that should be removed. In this case, after a revert, the intermediate version is redacted too. Here is an example of this. The redaction used in this case was this one. The full list can be seen here.

In that list you can see a couple of “License Change Redaction” entries that date from around the time of the licence change. This change was made by the redaction bot that was referred to above. As @pnorman says, it is rarely used nowadays (I have never found a need to).

3 Likes
30

Ask the many people who have made their first forum or mailing list post in the last couple weeks whether they believe the surprising map content was intentional or unintentional. :wink: I think it’s a good point to reflect on, because some vandals will be more persistent than even the most clueless of well-meaning mappers.

Countervandalism isn’t just about keeping the content clean; it’s also about keeping the troublemakers out so they don’t keep making trouble for us to clean up after, or I suppose your idea of quarantining them so that they waste their time. We can block vandals today, but how easy is it to evade a block, and how quickly do we catch block evasion? As I recall, we had an incident last year involving rapid account creation. Do we see sockpuppetry as an ongoing risk, or is it safely under control now?

The redaction bot did run at an awesome scale. It’s still an after-the-fact mechanism though. Some of the people who have come in to complain about this wave of vandalism have been asking how we don’t prevent obvious vandalism from entering the database in the first place. Maybe that’s an unreasonable expectation, but I can see where they’re coming from. It doesn’t take a rocket scientist to realize that Andy Townsend isn’t notable enough for his name to dash around the world like Santa Claus in December.

This flexibility to choose what to reject on the spot is very powerful. Folks have been quick to reject the idea of blocking edits based on a “naughty word list”, but I think this assumes a static word list, matched literally, that has to be developed out in the open at the same pace as all our other software.

As one of our wiki’s administrators, I appreciate the ability to set and modify rules on the fly. As with your script, the rules can be kept private and don’t have to be easy for the vandal to figure out. But unlike your script, the rules can also perform a variety of automated actions in response to tripping a filter, such as quietly logging a violation, notifying the user, blocking the edit from saving, or even the nuclear option of immediately banning the user and anyone with their IP address. Unfortunately, I think the last time I floated something like this for OSM, someone reckoned that evaluating every upload in real time would be atrociously unperformant, so I haven’t given it a lot more consideration. I’d love to be proven wrong though!

(By the way, if you want to talk cybersecurity™, one of the wiki’s abuse filters was inspired by an infamous case of accidental election interference a few years back. I’ll let you guess who.)

5 Likes
31

My movie plot (as famous cyber security expert Bruce Schneier calls them) cyber attack – I create a fresh account, sit down for less than an hour in front of zoomed out global view, draw a path with a few hundred nodes, tag it trail_visibility=no, name it “You are lucky today!”, upload that in a millisecond, inflate planet file by 0.000 permille and make tile renderers grind for more than a week even though it would be reverted in less than the time to create it.

32

OAuth2 version of redaction bot

3 Likes
33

Allow a limit to be set on expiry of line segments by tomhughes · Pull Request #2185 · osm2pgsql-dev/osm2pgsql · GitHub though only a fraction of tile renderers run world-wide will even use expire lists - most people who run a tile server, other than the OSMF, will simply rely on mod_tile’s “3 days is old” logic.

34

Is this why I can’t see the GPS traces layer on the map? I have both the traces and notes enabled but only see notes.

35

Of course there is a difference, but at the point where the data is flawed, the cause is no longer relevant for how to handle the flawed data towards distribution. The severity of the errors determines how to handle the data, and the cleaning/filtering/blocking mechanism doesn’t need to know the cause.
Intentional severe damage does have a higher likelihood of repetition, which makes measures at the side of data input important. It also stresses the need for a propagation blocking mechanism that can be applied in degrees of strength.
And I would add, the mechanism should not block the full propagation, just changed versions of objects that might be tainted.

I am thinking of a per-object “cleared for distribution” mechanism in the database, to which policies can be applied from automatic clearing to full block. That would enable full unblocked editing with direct feedback for mappers, while offering data users more secure data. Without setting up a separate “clean” database.

36

No, that seems to be an unrelated error. The GPS trace layer loads in a different map, but the request from osm.org seems to be broken.

37

How much of a performance hit such an evaluation incurs depends heavily on how wide the surrounding context is that is used to check the data.

  1. Only checking the data that is present in the upload itself
    This can theoretically be done before even performing a single database query and can therefore be very fast. The problem here is that you only know the new state of objects and have no knowledge about previous coordinates, tag values, or any coordinate information for ways or relations. You do not know if or which tags or coordinates have been changed. A check like this is suitable for simple word filters for slurs or similar. It could potentially also detect simple malicious data changes such as new world-spanning ways being created if everything happens in the same upload API call, but this could be quite easy to circumvent by e.g. splitting up the upload API calls or re-using existing objects.
  2. Including the existing state of modified objects
    This would give you more information, such as the current tags, coordinates (for nodes) and member IDs (ways/relations).
    The current upload process only performs limited database queries for the previous state of objects that are required for consistency (e.g. ensuring that modified/deleted objects actually exists, that way nodes/relation members exist, etc…). To get access to information about the current state of objects that goes beyond their mere existence, more database queries would have to be performed and a lot more data would have to be fetched from the database than before. This would obviously incur a performance hit, but this should still be very possible without any big issues.
  3. Including the existing state of member objects
    Due to the nature of the OSM data model, ways (and relations) do not include any coordinate information by themselves. Without this data, it is not possible to do any checks for e.g. modified existing ways that are stretched across the world (as used in recent vandalism attacks). Any checks that require access to this data would require fetching additional information from the database. Like in the previous case, these queries are already partially performed, but would need to be expanded to accomodate this use case. This is where it has the potential get tricky, performance-wise, depending on the amount of additional data that has to be pulled in.
  4. Including unrelated connected objects (sharing the same nodes)
    This would be required to do semantic checks for e.g. highway or coastline connectivity. Besides checks like this being hard to implement in a way that catches only vandalism and still allows normal edits, the performance costs are likely prohibitive (for the required database queries as well as potentially the checks themselves).
  5. Including unrelated un-connected objects (in the same area)
    Required for e.g. checking for nodes floating in the ocean or for ways crossing over other objects without connecting to them.
    Performance: No. Don’t even think about it.
  6. Across different upload requests
    Maybe in a very limited fashion for mostly metadata checks. Likely not really feasible to do in a stateless way and would require keeping additional state in the database (main or in a separate dedicated one). Complex to implement.

For context, API call response times averaged over 8 minute chunks for a 24h period from the public dashboard:

  • min: 35.9 ms
  • median: 90.9 ms
  • average: 121.26 ms
  • 90th percentile: 174 ms
  • max: 1640 ms
6 Likes
38

As others have noted earlier, some data consumers have resorted to a similar mechanism on their side out of necessity. The ability to block individual objects, changesets, or some other synthesized subset of changes can mitigate some classes of problems powerfully, but it isn’t a panacea. For one thing, there’s still the question of how to evaluate these policies, which potentially runs into the same performance limitations as the abuse filter idea I’ve floated or, worse, something akin to the coastline problem.

These systems always need humans to review the backlog of blocked changes for false positives and unblock them, to avoid increasingly jarring discrepancies between the raw database and the distributed dataset. Anyone who uses OSMCha regularly will be familiar with the labels that get applied by these review systems and how frequently they get misapplied. This is not an insurmountable problem, but maintaining good working order might for example require the OSMF to raise funds to pay freelancers or staff as reviewers.

It’ll also create a second stream of feedback about changes that need to be rereviewed. When I worked at Mapbox, I was constantly fielding complaints from mappers that this or that year-old changeset wasn’t showing up.[1] Thankfully, I had access to a tool for letting a blocked feature through the system, but I could easily see these dual work streams taking up more time from the DWG than they experience today. Something like AbuseFilter wouldn’t be immune to this problem either, but at least a well-meaning mapper wouldn’t be caught unaware for weeks that the neighborhood they remapped is in a fragmented, inconsistent state with respect to public distribution.

Ironically, just by setting up any kind of review system, we would create an even higher expectation of quality among downstream users than they have of OSM today. We would no longer be able to lean on “Hey, it’s just raw, crowdsourced data” as an excuse. We might even get into an unfair contest with Overture about who can filter out changes the best.

Despite my professed enthusiasm for AbuseFilter as a guardrail, I think MediaWiki’s other countervandalism measures are more important in that environment and possibly more applicable in ours. Fundamentally, we still rely on a relatively small group within the community to combat abuse of every kind. Not enough mappers know how to revert even straightforward problems. One would think that a project that believes so much in decentralization and common effort would lower the barrier to entry for cleanup, too, not just craft mapping.

  1. I wasn’t on the data team and it wasn’t my job, but for many mappers I was the only Mapboxer they knew. ↩︎

1 Like
39

Small comment: we do get all current tagging for an object when modifying / creating a new element which would make checking for “bad” tag values reasonably straightforward*. As you point out anything else gets very messy and I doubt if doing any of outside of superficial checks (aka bounding box size and a maximum distance for geometry changes) “pre-acceptance” is viable as a short term measure.

* that we don’t have finer grain tag changes causes other, non-vandalism related, issues though.

1 Like
40

Sure. It doesn’t repair and it doesn’t prevent, but it can give repair time and time for preventive countermeasures, while blocking distribution of serious errors, and at the same time still allowing normal mappers to edit freely.

Requiring review clearing for all changes would be possible, but not necessary. Avoiding backlog is easy: after a certain time the objects versions are considered cleared. Review can only take so long, if no-one cleared it, it goes on to distribution. When facing a mild threat or accident, this time could be set relatively low. When facing a very seriously damaging event, set the max clearing time higher; if repairs cannot be done within this time, only then stop distributing changed data completely.

Somewhere in between there is room for a review system. The reviews should only do the triage, not a judgement of mapping or tagging. This could be organised by the mapper community. For some things clever bots could probably do the work, or prioritize waiting reviews by probability of being harmful.

If you maximize the clearing time, you cannot have unlimited backlog. If the community and other reviewing measures can not clear the changes in time, you are no worse off than in the current system where you have immediate distribution of harmful data, but you will still have gained time to take other measures.

41

How would people feel about a bounding box restriction on users that meet a certain risk criteria, e.g., newly created or fewer than 75 edits or has recent blocks?

13 Likes