Overpass API performance issues

[Mateusz_Konieczny]

@lonvia

When the quota is used up, the effected IP (or any IP using an effected app) gets hard banned from the server until the quota has filled up.

if I used Bad App in past and stopped - will my IP be banned again when their quota is used up again?

[Kai_Johnson]

@TrickyFoxy

btw, have you considered reducing the default value of [timeout:XX]? If I’m not mistaken, it’s still 180 seconds.

The default timeout value of 180 seconds is hard coded, not configurable. And it’s essentially irrelevant here. Any query can request any timeout value regardless of the default. And the vast majority of queries complete in under 2 seconds.

@TrickyFoxy

… to exceed the quota, the user must complete ~100 requests per day.

Most of the big offenders are executing thousands or tens of thousands of queries per day. At some point, that’s just inefficient client architecture because there has been no incentive to do anything other than firing an HTTP request to Overpass every time OSM data is needed.

[Kai_Johnson]

@ToniE

Is there a way to report the CPU usage of the query in the response?

Not without modifying the Overpass source code.

@ToniE

On the other side, would sending the CPU usage in the response expose other (security) problems - just asking?

Not really. In theory someone could use the data to craft queries that consume huge amounts of CPU time. But it’s already trivial to create Overpass queries that consume huge amounts of CPU time even without that information.

[trial]

@Kai_Johnson

Most of the big offenders are executing thousands or tens of thousands of queries per day.

Decreasing the priority depending of the recent usage could be an option?

[Kai_Johnson]

@trial

Decreasing the priority depending of the recent usage could be an option?

Honestly, the better solution – which @lonvia has already mentioned – is documenting alternative methods of using OSM data for specific use cases without relying on Overpass.

The suggestion to use GeoDesk from @GeoDeskTeam is very helpful. And I’m particularly hopeful about use cases that use PMTiles as an alternative data source.

[TrickyFoxy]

@Kai_Johnson

And the vast majority of queries complete in under 2 seconds.

Then the reason for the decrease in the number of successful Overpass requests in the Munin graphs is still unclear…

I just can’t help thinking that behind the noise from the large number of requests, some other problem is not visible, for example, with RAM and disks. There is clearly some explanation for the gradual decrease in the number of successful requests from 6,000 per minute to 600. Well, either you need to study and group the queries during the day and go to the specific query bans :)

p.s. just in case, I don’t mean anything imperative, just ideas

---

@Kai_Johnson

Most of the big offenders are executing thousands or tens of thousands of queries per day.

It’s worth noting that 10 000 requests per day is a number that is still mentioned in the Overpass wiki and documentation ¯\_(ツ)_/¯

[TrickyFoxy]

btw, I’ll share a case that I’m now solving without Overpass.

We’re currently working on standardizing lake names in the Russian community. We needed to analyze all the names. The Overpass API isn’t currently useful, but there’s another cool approach:

• Geofabrik publishes daily (!) country-by-country data. For Russia, this totals just 4 gigabytes of data.
• But we only need bodies of water. osmium tags-filter .... natural=water And we get only 400 megabytes.
• Let’s throw out the geometry, which we don’t need for analyzing the names. And we’ll store the data itself in Parquet. And DuckDB can do all this: lakes-names/export_water.sql at master · deevroman/lakes-names · GitHub
• I receive only 3 megabytes of data, with full tags, polygons centers and their areas
  Now let’s just put this on GitHub Pages, because DuckDB can run right in the browser! And now we have a tool that works instantly and with powerful SQL capabilities: Названия озёр в OSM

My page is very specific to our task, but replace natural=water with type=multypolygon and you’ll get a daily geometry validator for your region (which will likely be less than 4 gigabytes). Overall, there’s a huge scope for experiments.