Requiring user authentication has been discussed as mentioned above and people are interested. Someone has to design an architecture and implement it. None of the “out of the box” solutions will work with osm.org as an authentication provider and decentralized providers of other servers checking authentication and performing authorization.
Options like API keys require checking keys near query time. JWT and pre-signed URLs don’t, but require token rotation by the user of the service behind authorization.
It probably needs pointing out: the current public Overpass and Overpass-Turbo instances are all run as private initiative (the OSMF runs an internal instance for the query feature, but that is not directly publicly available). I’m not quite sure what the moaning here is supposed to achieve, shaming the current operators in to spending more, not-your-money and not-your-time, or what?
As I’ve pointed out before we already knew that running a free-for-all read only OSM API is a dicey proposal and the surprising thing is that it was possible to keep things working for so long. Alternatives are subscribing to a paid service for example from Geofabrik (very price worthy), using one of the scope limited instances (they have up to now not suffered as much), or running your own or convincing somebody else to do that.
If the later is your goal then you probably should state who you want to run the service, and how it is going to be financed.
If a requirement to contribute to OSM is having to setup their own private Overpass server, they’ll just move on to other things.
Well obviously it isn’t. I’m sure that most people contribute to OSM without using any sort of “what keys are in the database” check - they rely on editor presets, or what they know people around them use. There are also Overpass alternatives that haven’t yet fallen foul of these problems (e.g. Postpass). Where something like Overpass is needed is for explicit OSM-object vs geometry queries (with Postpass you’ll get the latter) and for metadata and attic data.
So for those who still would like to use this particular type of service…
and others don’t share the desire.
… well we can refer those straight to Arkell vs Pressdram.
some don’t have needed hardware
As part of writing a diary entry about this, I explicitly did a low-end test. For a smallish region (Britain and Ireland) I found I could get by with a €12 a month VSP (that was from the Cloud provider I regularly use; I’m sure if you shopped around you could reduce that). Once split between a group of people who might be using it, €12 per month is really NOT a lot of money in the global north.
Few people have skills to setup such a service
Setting up most OSM-based services (raster maps, Nominatim, vector tile generators among them) is really just a case of following the instructions - if you follow those in a mainstream environment it will work. Overpass is more complicated (and more flexible), but it really isn’t rocket science - managing it at the end of the day is still just adjusting a bunch of shell scripts. The challenge I found with Overpass was “under what circumstances it worked reliably and under what it fell over in a heap”. Detecting that “it has fallen over in a heap” is, of course, straightforward. With Overpass, I had two excellent resources to draw on (previous diary/wiki entries).
It may not be obvious (with Overpass Turbo, it wasn’t to me) but there may be a way of telling some existing service to use a different Overpass API endpoint. In the case of relatify, it seems to be a line in a script that you could modify if you ran it locally. With PTNA it seems to be a bit more hardcoded, but the author may be amenable to parameterising it.
Well, it depends if you make it public or not. I chose to make mine public and document it on the wiki page, but that’s a personal choice.
I tried “private.coffee” (aka: “kumi.systems”) but they were not really up-to-date and “private.coffee” is really slow now. PTNA currently needs global scope
Hard coded in 2 *.js files for the GTFS versus OSM comparison, 1 *.js file for OSM relation details.
Those 3 *.js files query each a single route_master or route relation (and its children) by their ID.
So, if I would be nasty, I could even use the editing API. But I refrain from that: in this case using the editing API is discouraged. Accepted, period!
Just for the statistics:
Looking at the “access.log” files of the web server for the 3 *.js files: less than 5.000 Overpass API queries in average per day over the last 14 days, triggered by users checking and editing public transport route relations.
Ultimately what this comes down to for me is that in recent past, Overpass was a free and relatively reliable service, and a bunch of useful tools for OSM editors and mappers were built based on it. There’s links to Overpass Turbo from Taginfo pages. JOSM has a little wizard for getting data from Overpass. Etc.
Some of those could possibly be relatively easily retrofitted if Overpass was to add some sort of an auth token requirement.
I understand that coding an auth token and rate limit scheme takes effort.
If current situation continues unchanged, legitimate use of Overpass will die away, some of the tools currently using Overpass will switch to other services, and some will die away.
I dunno, from my perspective as a mapper I wish the recent responses in this thread were a bit less “well it’s free and you haven’t sent a PR to fix yet” and a bit more “yes this sucks but it’s difficult and unfortunately currently no one has time or money to fix it.”
Again, I am looking to use it as a mapper. I know I could, but I don’t want to set up my own Overpass server to find mistagged pedestrian crossings in my region that I want to fix. Pressing retry 10 times on the current server is also not very appealing.
Just discovered this topic. I’m the main dev of cartes.app. We used the .de instance, until we found out about it being drowning. So we started our own. Then we went to mutualize our instance with the French community on osm-fr servers, discussed on the equivalent forum.
Now we’re drowning with both our load (that we’re optimizing right now), and the external load from multiple actors
Setting up the instance wasn’t hard using the online guides, but we’re suspecting it doesn’t use most of the VM’s resource.
We’re especially suffering from the meta requests being heavy (2 seconds vs 20 ms on the same simple node request). Searching for these keywords led me to this thread
Erratum : it’s working. First I had disk permission problems, then I understood that the direct link show this error message but using the download_clone script works, my nodes.bin is getting filled at ~ 1Gb/min.
I’m wondering if the “fast startup” option in the quick installation guide still works. I’m getting a Forbidden You don't have permission to access this resource message.
To avoid putting more load on the .de instance, I’m using direct scp to copy the database folder to bootstrap the new server. Not sure it’ll work, but I see no reason why. Minor version mismatch should not be a problem. Then the systemd processes should catch up with this ansible script (that doesn’t handle the fast startup phase) here
It’s about 10x faster than downloading from the .de server.
Edit : But it doesn’t work. I suspect files get unsynchronised during the file-by-file transfer. The suggestion made then was to use zfs snapshots.
There appears to be at least 1 official repo and two forks.
drolbr’s original repo, used by the .de and .fr instances, that is way behind both other forks mmd-osm’s fork that has several performance improvements but with no commit in the last 2 years and diverged from source
b1tw153’s fork that is just an up to date version of drolbr’s, without the improvements of mmd’s, but with other improvements.
Given that there is no improvement of the situation in sight in terms of users being a bit more mindful with their use of the Overpass API, I’ve started to put heavy banning rules in place on the servers. Large scale users will be cut off until the server is responsive again.
This may do some collateral damage. If you want to use the Overpass API, make sure to follow the same rules as for all the other free services around OSM:
When called from the browser, a HTTP referrer must be included in the request.
Scripts must set a custom HTTP user agent which identifies them well. No stock UAs, no UA faking, no UA rotation.
No parallel running of scripts to “speed up” processing. One user, one script. Do not work around that rule by distributing load over multiple servers or using multiple “cloud workers”.
Keep calls to a minimum and make sure to rate-limit them.
If you need to use Overpass a lot, consider downloading and working with an extract instead.
If you use the service in a commercial setting, please look for alternatives (self-installation or commercial providers).
The goal here is to get at least Overpass Turbo to run again reliably. Then we’ll see if banning can be a bit more directed instead of using the lawn mower approach.
If you have 50 €/month on your project and want/need reliable OSM data access, please take a day and set up your own server instead of relying on the open overpass instances.
If you are performing numerous/complex queries, or your query area is extensive, consider using GeoDesk. You can build a local database from any PBF file (whole planet takes up about 100 GB; easy setup, 20 minutes to import on a modern machine) and perform queries from Python, Java, or C++, or via a command-line utility.
That’s the problem with the alternatives to Overpass that we were told about : they all either are not easier to setup, either are limited in some way (metadata, query complexity, etc.).
I recently wrote an extension for DuckDB called duckdb-osmium which lets you query OSM PBF files using SQL, without extracting them into a separate database first. For example:
-- find pizzerias with unknown opening hours
SELECT type, id, tags['name'] AS name, geometry
FROM 'example.osm.pbf'
WHERE tags['amenity'] IN ('restaurant', 'fast_food')
AND tags['cuisine'] = 'pizza'
AND tags['opening_hours'] IS NULL;
It uses libosmium internally so it can build geometries (Point, LineString, Polygon, etc) from OSM elements, including multipolygon and boundary relations. OSM PBF does not support fast random access, so querying this way will be slower than loading the data into a proper database, but for an extract (like a state or country from Geofabrik) it’s pretty usable. When combined with DuckDB’s built-in spatial extension it also lets you export the query results to GeoJSON, GeoParquet, and other formats.
Like the other options mentioned above, it’s certainly not a complete replacement for all of the features offered by Overpass, but for certain use cases (like finding all elements in a given area with a specific combination of tags) it may be a viable alternative. I have used it myself to create MapRoulette challenges and as the backend system for building Layercake data layers.
