On replacing Basic Auth with OAuth 2.0

The rate limits in question are for reading from the API and have nothing to do with the new upload rate limiting (being authenticated wont help you with the later). In normal use you are extremely unlikely to hit the limit, and relaxing it for authenticated users was added for the fairly narrow edge case of mapping parties when you have a larger group of people behind a single (NAT) IP address.

Supporting OAuth2 is the least of the issues you will have if you want to run a bot against the API, see also Automated Edits code of conduct - OpenStreetMap Wiki

Non-existent use case (there is no API for that)

Currently only the private information of the authenticated user is restricted. That might change one day (not holding my breath) because the current situation is clearly not legal, but that applies to scrapping the API for user information in an automated fashion in any case -right now-, so not a use case that is going to win anybody over.

1 Like

Since I can’t properly react, please join me in my living room for some popcorn while I pontificate in long-form prose. But don’t read the extif data off the image file and show up at my doorstep. That would just be weird.

Before I go on, I just want to clarify that you’re about to read the words of a complete moron who definitely qualifies as an unqualified developer. At least, as far as web authentication goes. My intelligence on other topics remains up for debate.

I’ve actually had to implement OAuth2 once, when I built StreetFerret, which authenticates with Strava. I stumbled through writing the code, and then once I got it working, promptly forgot how it worked and prayed that I’d never have to touch the code. So, I’m well-acquianted with being too dumb to implement OAuth2.

I actually think that even Basic Auth is too complicated for new and inexperienced developers.

Yikes! Even more gobbedlygook! I actually had to implement refresh tokens also for StreetFerret, and bear in mind that this was back in the days when you had to write shoddy code yourself rather than having AI chatbots do it for you.

This is also my approach to software development. But once in awhile the chatbot gets the code right.

:popcorn: :popcorn: :popcorn:

Thrown shade right there, but maybe you’re onto something?

Total oversight, and I’ve got the solution.

I agree with @NorthCrab that we should eliminate OAuth2 for easier scripting, and I agree with @pnorman that we should eliminate OAuth 1.0 and Basic Auth. And, I agree with @02JanDal that we should reinvent the wheel, which we are so good at as noted.

I propose SIMPLE web authentication, which stands for Super Insecure, Moron-Proof Login Experience.

The way it works is very simple, you just add your username and password into the querystring of any authenticated request you want to make to openstreetmap.org.

For example, a login request might look like this:

https://www.openstreetmap.org/login?username=ZeLonewolf&password=PopcornLover69

So simple, even a moron could figure out how to curl some sweet, sweet API write requests. It’s also perfectly secure since we’re using HTTPS. To be really user-friendly though, we should consider implementing this functionality on plain-old HTTP as well.

I stand with @NorthCrab in protesting the needless complexity of modern, secure authentication protocols and yearn for a simpler time when passwords were just the names of our pets, and the most advanced hacking tool was a sticky note left under a keyboard1.

Hear hear!

Pass the :popcorn:.

I expect a long-form response to my very serious proposal.


1pretty solid punchline for an AI chatbot I think.
2 Likes

Don’t tell him how many client (re-)implementations of OAuth 2 OSM currently has :sunglasses:

You didn’t get the hint. I mean, the barrier in the form of inconvenient authorization is the wrong protection against misuse of the API. For Example, this did not save us from automatic vandalism in Russia and Israel. We shouldn’t scare off conscientious developers at the very beginning.

This is not an excuse for inconvenient authorization.

It is sad. But it still doesn’t mean that we are limited to just editing the map.

@NorthCrab is making the argument that not supporting basic auth creates a significant hurdle to using the API. I’m just questioning his statement that that is somehow bad because it stops “New and inexperienced developers”.

You jest, but perhaps things can be made more secure while being simpler to use. How about implementing WebAuthn ?

1 Like

We need a “whataboutery” emoji :slight_smile:

Being serious, you’d have to have been living under a rock to have avoid being bombarded with stuff about passkeys (as WebAuthn is also known) in recent weeks. The problem is that while it is a solution to a common problem, I don’t think that it would help with scripted API access, which seems to be the main issue above.

Having said that, the only scripts that I do use against the OSM API already do support Oauth2, so I don’t see a problem

I’ve been developing software for almost 30 years now, and there are - obviously - still concepts/languages/… that I have no experience in. I had to learn tons about OSM in the last decade, and I am guessing that there are more people out there like me who have to actually learn how something works rather than just knowing it by the grace of the FSM.

Given the Etiquette/Etiquette Guidelines - OpenStreetMap Wiki from 2022 explictly state “Be welcoming. OpenStreetMap aims to be a community that welcomes and supports people of all backgrounds, cultures and identities. […] Remember that people may be new to OpenStreetMap. We should be tolerant and supportive towards new members.”, you questioning how “stopping new developers” is somehow bad, for me, just doesn’t fit.

If you want to prevent new developers from screwing up the live database (costly vandalism that doesn’t care about anything OSM stands for, aside), IMHO, there’d need to be a system for checks on what API users are doing, that’s far more than just using a secure Authentication Mechanism.

If the only concern for scrapping BA is password safety (as hinted to by @pnorman in his diary entry) than this whole discussion becomes user-centric (Does OSM trust their users to use secure passwords and not the same for every account? Does the user maybe want to use BA rather than re-issuing OA1 tokens every x months?) and the experience and capabilities of the developer have nothing to do with anything.

K

2 Likes

In hindsight, judging a support for an idea after giving a post mere 5 hours on Sunday evening (or night, depending on your TZ) was bound to to produce, ah, somewhat suboptimally representative statistical sample :smile_cat:

The original github post was actually on Jan 11th.

I was talking about this thread on discourse (where your claim of “no support” was written)… Not much regular people follow all OSM github issues or random OSM diaries. While I would guess I follow way more OSM random discussions on several mediums (including GitHub) than the average Joe, even for me this thread was first sight of the issues @NorthCrab raised awareness of.

1 Like

That’s funny that you think it’s the technology that’s the biggest barrier to (checks notes) new, inexperienced developers. In my experience, the more common complaints are about the people that maintain the software packages involved not…how can I put this nicely…fostering the sense of community that would facilitate an optimal onboarding of a new developer. Of course, we could make the same argument about openstreetmap-ng, which doesn’t even pretend to be a community effort.

It was @NorthCrab that invoked the “new and inexperienced developers” in his suddenly discovered concerns for his fellow developers. I was just enquiring why that these people that in his eyes are obviously incapable to employ ready made libraries that support OAuth 2, should be trusted with writing to the OSM API, given that doing that needs some reading and familiarizing.with the concepts and data models, which might be quite alien for somebody new to the project and there is potential for serious upset if something goes wrong.

I, for my part, have expressed the opinion that OAuth 2 support shouldn’t be an issue, as there is widespread support and it is just one of those things that you will figure out once, and then probably copy-paste for the next 10 years.

Um, really? That level of simplification seems to work well for everything:

“hello world” program SkyNet Artificial General Intelligence
1. build powerful enough computer 1. build powerful enough computer
2. write the program 2. write the program
3. execute the program 3. execute the program

See, those are basically the same too; now where are those self-driving cars already :smile:

To compare:

  • creating new PAT is simple click of single button. That’s it.
    • Registering Oauth2 application requires reading a lots of docs, entering the correct redirect URI, implementing HTTP(s) daemon which will listen on it and handle it correctly, etc.
  • using the API with PAT is simple matter of changing URI from https://example.com/api/xyz to https://example.com/api2/${PAT}/xyz. One trivial string change from unauth API to authenticated one. That’s it.
    • Using the API with Oauth2 requires either reading even more docs and writing several thousand lines of hard to debug code, or making your code depend on random libraries of unknown quality (assuming they exist for your platform/language of choice - mine is wget $url | sed oneliner in POSIX shell, eh? How do I add oauth2 to that oneliner?), writing a dozen (or few) lines of needed glue and painstakingly debugging the result.

In my universe those Oauth2 vs. PAT (or Oauth2 vs. Basic Auth) are definitely not the same, not even close.

5 Likes

Note that not all the things that scripts do are able to be respresented in .osc file (which I presume you mean by “JOSM upload the changes via the API”). Many are read only (but user related), and some are write but unrelated to node/way/relation changes.

For example the last ad-hoc scripting I’ve had few days ago were related to user preferences. Or it could be private GPX Traces accesses or something else

1 Like

The thing that your post completely ignores is the existence of abstractions.

In my post I wrote the steps that a developer using a decent library would take. Let’s compare some ideal Python code:

from my_osm_api import OSM

client = OSM(client_id="XXX", client_secret="XXX")
client.authenticate()
client.nodes.get(523455)
from my_osm_api import OSM

client = OSM(pat="XXX")
client.nodes.get(523455)

Seem sufficiently similar to me (one could even get rid of the client.authenticate() call which would make them essentially identical). (the fact that there is no package for Python that exposes such an interface is annoying, but can be relatively easily fixed)

If one ignores abstractions such as these, one would also have to start to include the steps of “serializing JSON”, “composing HTTP headers”, “do a TCP handshake” in the steps for using a PAT.

I’m genuinely curious what scripts you write in a Bash-like language. Can you post a link?

It would be fully possible to write a osm-get tool that would solve that issue anyway.


One thing that curiously hasn’t been mentioned yet in this thread is the development cost and maintenance burden of adding PATs as an authentication mechanism. Would @NorthCrab, @Matija_Nalis, @TrickyFoxy or anyone else advocating for PATs be ready to write a PR for openstreetmap-website adding this support, as well as promising that they will help maintain that part of the code for the next 5-10 years including handling any security concerns exposed through this new potential attack vector?

My overall take is that I’m happy with OAuth 2, it works for my usecases, and presumable so are the maintainers of openstreetmap-website. I can’t speak for the later (but know the usual concerns maintainers have about additional features), but I personally wouldn’t care much either way so I wouldn’t care much if anyone else wants to add and maintain support for PATs.


Also note that interestingly access tokens are currently set to never expire: openstreetmap-website/config/initializers/doorkeeper.rb at cef7d82c729ffcd1ddd05eabfef8314ce884789d · openstreetmap/openstreetmap-website · GitHub

Seems like a security oversight to me that should be fixed, but while this is the case it would be possible to write a simple webapp that does the OAuth dance and gives you a permanent token that you can use in your scripts as much as you want.

3 Likes

At Geofabrik we have, for many years now, limited the download of “full-metadata” regional diffs to people with an OSM account (for GDPR reasons). The web server requires a cookie (which is more or less the PAT in the context above) and sends you through an OAuth flow to create the cookie if you don’t have one. Since a scripted approach with wget, curl, and friends would not work with OAuth, we provide a simple commandline client sendfile_osm_oauth_protector/doc/client.md at master · geofabrik/sendfile_osm_oauth_protector · GitHub that lets you generate such tokens. Not rocket science. (And yes, it’s currently OAuth 1 only but we’ll fix that in due course.)

4 Likes

Why do you need, for example, StreetComplete to constantly update the token?! Let’s also force account passwords to be changed every year.

Well, after such a change, the convenience of OAuth over Basic Auth for developers is completely killed.

Maybe first we won’t break the convenient Basic Auth since there is no equally convenient alternative? Let me remind you that initially it was planned to abandon only OAuth 1.0 Remove support for OAuth 1.0a and Basic Auth · Issue #286 · zerebubuth/openstreetmap-cgimap · GitHub

For example, I think I saw a link on GitHub from @mmd that explores ways to simplify OAuth when using scripts. It seems that this still requires registering an OAuth application, but without the extra fuss with redirects.

As already said, the ideal solution is one button on osm.org/give_me_token, which issues a token. And under the hood, she can create an application herself and receive an OAuth token from the current user. In this scenario, the developer won’t even know about OAuth 2.

That is only “required” in the case when you are writing your own OAuth 2 client implementation, that is totally unnecessary for any of the use cases that have been mentioned here.

Just as an example at least some of the libraries that I looked at for potential use in Vespucci only require a line or two of code. I note that in the end I ended up actually reading the spec and adding yet another OSM specific client to the 3 ? 4? implementations that exist. Which is interesting as a case study why these things happen, but that’s about it.

And a further example for python GitHub - Zverik/cli-oauth2: Helper library for OAuth2 in command-line tools

3 Likes

There’s a very important distinction between passwords and tokens:

  • Passwords should never be written down anywhere (they may be stored in passwords managers however), and as long as a user follows that rule it will be basically impossible for another actor to get their hand on the password
  • Tokens are routinely stored on the device (access tokens usually just in memory, refresh tokens often on disk) and can thus easily be compromised (this can be made a lot more secure by using encryption, but that either requires another password the user has to remember (inconvenient) or using a OS-provided secret store (sadly not available on all OSes and often hard to use for the developer))

So while passwords don’t have to be exchanged regularly (if managed correctly be the user) and common recommendations even are against such policies (though there’s a lot of conflicting opinions about that), tokens should be exchanged/refreshed more regularly.

Yes, there are use cases where that would be the easiest solution for you as a API client. And I encourage you to go and submit a PR for that as I previously mentioned, provided the maintainers are happy to accept it (and if not, it would be perfectly possible to host that as a service yourself).

That’s because Basic Auth has some very bad security implications:

  • Many people use insecure passwords (with tokens, including PATs, you force the user to use a well generated password)
  • It gives full access to your account, including to delete it (with tokens, including PATs, you can choose what the token gives access to)
  • It might be sent in clear text without you knowing about it (many such cases have been fixed in the tools and servers involved, but many more still likely exist)

This is easier to understand when one takes a few steps back in time, to when basic auth was a common way to give a third party service access to your account in a system. At that time, it was sufficiently common to be a problem to have software that either stole your login details outright (in which case they’d not only have your login to that one system, but usually also to most others since most people don’t use unique passwords) or lost them for you (by saving them as plain text or similar). That’s the problem that OAuth was designed to solve; since the third party software now never actually managed the password it couldn’t steal nor lose it, and if the access tokens were stolen or lost the damage could be limited.


Just to be clear, my arguments are mostly that OAuth 2 isn’t as inconvenient as OP claims as long as you follow good software engineering practices (which you should anyway). PATs do help if you don’t, as well as in a few edge cases, but supporting them is additional code that someone has to write and maintain (and again, it’s easy to topple that argument by just volunteering to do so). Basic Auth is code that needs to be maintained and that is inherently insecure.

Working within the framework of OAuth 2 I think the best would however be to enable the client credentials flow, since it would remove the need to spin up a webserver for the OAuth dance. I’m not that familiar with the library openstreetmap-website uses, but it might just be a single line of configuration and a few small tweaks to the UI.

2 Likes