Editing 3000 POIs happen more often than you’d think. 
(please do not spam +1 or similar comments there, you can use Github reaction feature)
4 Likes
This one is pretty fresh.
Thanks. I’ve now blocked another 150 or so suspect ones (the first dozen on a phone from the top deck of a bus) - that’s 528 over the last day or so. However, as noted almost exactly a month ago we seem to be doing nothing to prevent this cycle repeating. It shouldn’t need the DWG to spot vandal accounts by detecting a “vandal” modus operandi in changesets - by that time they’ve already registered a new account in OSM and will likely have created further problems.
I’m frankly quite surprised that the OSMF board isn’t giving it a higher priority.
11 Likes
@osmf-board Can you give a comment on this? What are our plans for the defence?
What is currently preventing restrictions on new accounts? At least this Add support for rate limiting signup requests by tomhughes · Pull Request #4198 · openstreetmap/openstreetmap-website · GitHub
Knowing the difficult nature of osm.org maintainers I would like to draw your attention to the problem and get a response from you.
DWG and the community has been wasting effort for a week now. It takes a day to fix the consequences. We should be wary of the same vandalism on a worldwide scale. And the vandalism may take a form that will be difficult to eliminate with existing tools.
this is not an official OSMF board statement. I wrote this just before going to sleep.
You linked one of obvious pieces. In long term, increase capacity of osm.org website development is a goal and given what happened better security and moderation would be almost certainly among priorities.
(ongoing fundraising is a piece of that)
Code needs to be reviewed before it is deployed. This is done to prevent exciting events such as “we lost all our data” or “now noone understands code of the project”.
Note that currently OSMF staff (listed at Contractors and employees - OpenStreetMap Foundation ) includes:
- Site Reliability Engineer (not a programmer!)
- iD developer (distinct from Ruby on Rails used by website)
Neither develops GitHub - openstreetmap/openstreetmap-website: The Rails application that powers OpenStreetMap - this is done entirely by volunteers.
Personally I am not really qualified to even review such pull request. If you are not either, please do not comment on Add support for rate limiting signup requests by tomhughes · Pull Request #4198 · openstreetmap/openstreetmap-website · GitHub - especially complaints “why it is not done yet” variety. This is not productive way to motivate volunteers.
Yes, there are some attempts to improve situation, especially long-term.
If there is some obvious quick-acting thing that OSMF board can do and it missed it then I would gladly learn about it. Sadly just making something high priority does not result in desired effects.
Some steps were taken with coordination (deliberately not mentioning specifics, not sure how much of that should be shared).
Yes, I am aware of some low hanging-fruit. Like Prevent blocked users from signing up with the "same email address" again and again · Issue #4206 · openstreetmap/openstreetmap-website · GitHub or rate-limit changeset comments · Issue #4196 · openstreetmap/openstreetmap-website · GitHub (that resulted for example in Add rate limiting for changeset comments by tomhughes · Pull Request #4202 · openstreetmap/openstreetmap-website · GitHub - nice to see that apparently my comment turned out to be sort of useful. Feel free to blame me if legitimate changeset comments will fail once this is merged.)
Hiring someone on short notice to implement this is not really feasible.
I had some ideas, but they turned out to be not viable.
6 Likes
Depending on the seat of the OSMF, criminal charges might apply to such blatant vandalism? Although this can be seen as solely acting against the terms of service by perhaps a single skilled Individuum and not necessarily by a state-like-actor, the political tilt may turn this into something where the administration can be tasked with? Much like with data-breaches and such? One more lever…
I am aware of attempts to use such legals tools. For example in case of a wikipedia vandal spending several hours each day to vandalise wikipedia pages and harass editors, for several years.
Sadly… As far as I know it was not really effective and was considered as not worth it. And that was in case where identify of vandal was at least partially known and was within the same country.
“we lost all our data”
That’s understandable. But
“now noone understands code of the project”
causes concern. You can find not a few simple issues that are not implemented because you need to do something else to improve something else. For non-urgent issues it is justified. But the current problem takes a lot of community effort and it seems that according to the comments in Pull Request some more beautiful solution is being searched for, which may take longer to implement.
Some steps were taken with coordination
That’s a good thing. That’s partly why I don’t feel like bothering the developers with questions right now. But I want to understand where we are now.
2 Likes
There are quite a lot of opportunities to explore, I’d say.
What this situation has shown is that there are various parts of OSM that are fragile and in need of attention. This is what the board can, and probably should, give strong guidance towards.
We learned that account-signup is handled by the same codebase that powers the rest of the osm website. Which is done in a language only small number of people know and is getting less. (Ruby: 6.7% in 2021, 6.2% in 2023). And the main dev there made himself the bottle-neck as he so nicely explains on github.
That problem will not get better over time. So, for the board, the first and most important part is to separate the account-creation from the rest of that system and actually be able to innovate with more developers and less friction. Decide on a tech (Java for instance), hire a dev for just this job and make the account-creation component live. I doubt it needs to be open source.
It bears repeating what we are currently seeing here. This attack is likely the cause of a very small number of actors, may simply be one that is a decent programmer. The result is that the OSM community has been mobilized more and more. it has been affecting a very large section of our users and mappers. After the first wave we have seen a lot of people stepping up and helping the DWG doing things like reverts, which is great! But this is now the new norm. The DWG would likely not be able to fight this off alone. This means that this attack is now burning through good-will of a very large section of the OSM community. This has to stop. The community resources are too precious!
It is not just the DWG, it is community leaders, it is people that do reverts and it is a lot of mappers in those areas that are in need of reverts as they can’t map lest their work gets reverted too. And naturally downstream.
Because of that, more dramatic but temporary solutions should be considered until the real problem is solved. Where the real problem is the one I described above.
So, immediately start blocking account creation (probably fastest to do at the firewall level) all IPs that were used to create the accounts that are getting banned. Yes, this will block a lot of VPNs, probably tor exit nodes too.
But are we ready to tell a much larger group of mappers/users that the organization can’t fight one person with a bot and a willingness to do damage?
I have to say that I agree with the premise of this topic. This is something I am not afraid to blame the foundation and the board about too, this is not a fresh young project and giving random people on the Internet full, unlimited write access to everyone and all data is just plain stupid. There are thousands of projects out there that have all come up with some way to do some limiting of accounts until they gather karma.
It is quite irrelevant to blame this on volunteer developers not having time to do it, this should have been made a priority years ago. Back then it would have been able to do it without haste and without burning through funding. Now you lost that option and burning through funding is by far the best choice you have to keep the project safe. That’s the foundation’s job, isn’t it?
5 Likes
I’ve merged this without waiting for a full review because we really need it live - happy to deal with any further review comments via followup PRs.
New set of measures become active, more is planned.
Thanks to @TomH
15 Likes
Just to give everyone here a bit of feedback - the new measures have been working - a number of “likely problem signups” have been blocked tonight. However, if anyone spots any problems please do report them so that we can investigate.
Edit 28/8/2023: No new problems overnight.
6 Likes
Vandalism is a big issue that needs serious attention. However, we need to be careful not to break the spirit of opensource… what about limiting new accounts to only add but not delete data?.. once their contributions attain a certain verification threshold they can be awared with an editor’s badge. Just like it is in this forum. Users attain different badges the more they contribute
1 Like
Unfortunately, I don’t think that would work. We’ve already seen previous “vandal attacks” that only added data (such as features arranged to spell out certain words). Allowing “new accounts” to only add data would not prevent that.
Discussion is ongoing how to detect “large quantities of unexpected changes in a short period of time”, among other issues, but anything needs to be done in such a way that it causes as little inconvenience as possible to “genuine” mappers, new and old.
3 Likes
Many social platforms share this, the discourse platform being a prime example. I tried in my topic Earning trust as Newbie to put some light on this. Unfortunately, openstreetmap is missing even the basics for something like that. Never mind the permissions (ACLs?) following up.
There’s a valuable concept that every negative event must be turned into a lesson so this or a similar event would less likely to happen in the future. And here’s something we need to realize and remember well:
Before this happened, for years, various people were saying exactly this about smaller-scale issues (e.g., the infamous Pokemon Go cheaters adding fake features). And there were other people, willing to flip the situation upside down and say that such issues aren’t “burning through the good will”, and everyone should just be super-polite and patiently remove the rubbish objects, hoping that one of those who added it would, all of a sudden, become a valuable mapper. And they were calling those who opposed it all kinds of names.
It also reminds me (and this analogy is not accidental) how some European governments finally realized that not all governments (and people) want a peaceful life, so they are now reopening the rearmament programs and spending millions on defense. While just “yesterday” they blamed their a little more skeptical and less idealistic neighbors for imaginary “xenophobia” and unwillingness to destroy the borders, etc.
Malicious intent exists. And the least wise thing that can be done about it is to pretend it doesn’t and effectively put a burden of dealing with it on regular people. That applies to opensource projects, international politics, local government efforts against the crime - anything of that sort.
6 Likes
Idea: block account creation for 2 weeks. That may make the vandal get bored and go away.
The biggest issue is that it’s a multi-dimensional supervised pattern-detection problem. And even that, if undesirable changes have a good mix of commonality and entropy, can completely miss them. But if the objective isn’t to build a proactive detection method, making a discovery method for finding more instances of the same action is possible. Say, in this particular case, harmful edits had 0% geometry changes and 100% name=* changes. 100% of the resulting name values were in capital letters and no substring overlaps with the previous value. User names had untypically high entropy of small and capital characters mixed.
Would a vandal be able to change if this pattern was discovered? Likely, yes. But if it’s combined with an effective revert tool allowing to erase the changesets by list completely, the incentive for vandalism would be pretty low.
1 Like
Add rate limiting for changeset comments by tomhughes · Pull Request #4202 · openstreetmap/openstreetmap-website · GitHub was also merged
new accounts cannot make 10 000 changeset comments within an hour anymore
this makes harder to spam people from new accounts and should reduce other bad consequences
2 Likes
Sorry, but that’s not good enough.
The board needs to recognize three things:
- This cyber attack has exposed vulnerabilities in our cyber security posture
- We will continue to see cyber attacks over time because our data is valuable and many competing groups care very deeply about what goes in the map for political and commercial reasons
- We don’t have the expertise we need to fix the problem
As long as we are reacting to specific attacks, we will continue to see a pattern of new attackers and delayed responses while we scramble to make a fix. In the meantime, people like @SomeoneElse and his colleagues on the DWG will continue to be stuck fighting wildfires with garden hoses.
As long as we have a lot of smart people with IT, programming, and digital cartography expertise, and no smart people with expertise in cyber security, this problem will persist.
Please consider putting out a call to action to get the right people on the team, or hiring someone to fill this critical gap in expertise.
4 Likes