Hi!
The script
http://tools.oberpfalzwiki.de/cache/test_outbound.php
tries to connect to overpass-api. It doesn’t work, but it works for google.de. Can you tell me why?
Diedda
Are you following rules documented at Overpass API - OpenStreetMap Wiki and announced in Overpass API performance issues ?
what this script is even doing? These sites are not like each other.
Your IP keeps being banned because that wiki is sending bad requests (error code 400) at a very high rate.
Hi! What is the problem with our posts? It has been working for a long time.
which posts?
Sorry, not posts, requests :-) !
What is wrong with our requests?
hard to say more without knowing what requests you are sending
what is the overpass query you make?
see Overpass API performance issues for why new limits were applied
Subject: Blocked IP on overpass-api.de from production server
Hello,
our production server appears to be blocked by overpass-api.de.
From our server, requests to overpass-api.de fail with “Connection refused” on both destination IPs:
- 65.109.112.52:443
- 162.55.144.139:443
Example commands:
curl -4 -v --max-time 15 https://overpass-api.de/api/status
curl -v --resolve overpass-api.de:443:65.109.112.52 --max-time 15 https://overpass-api.de/api/status
curl -v --resolve overpass-api.de:443:162.55.144.139 --max-time 15 https://overpass-api.de/api/status
DigitalOcean tested the same endpoint from another FRA1 droplet and confirmed that routing is clean and the endpoint returns HTTP 200. They suggested that our specific public IP may have been blocked by the Overpass API operators.
Our server public IP is:
64.226.68.63
We use Overpass to retrieve building geometry around specific latitude/longitude coordinates. We are reviewing our request rate and adding caching, throttling and fallback endpoints to reduce load on the public Overpass instance.
Could you please confirm whether this IP is blocked, and if so, whether it can be unblocked or what usage limits we should follow?
Thank you.
@lillonet I’ve merged your question to this existing topic because it seems like a very similar question.
Please read the previous comments above and see if any of them help.
Have you tried any other Overpass servers, including paid options? Have you considered setting up your own?
Thank you for merging the topic.
Yes, we tested other public Overpass servers from the same production server:
-
overpass-api.de status
Result: Connection refused from our production IP. -
overpass.private.coffee status
Result: connection opens but the request times out, and sometimes returns 502. -
maps.mail.ru Overpass status
Result: HTTP 200, and some interpreter queries work.
At the moment we have added a fallback to the maps.mail.ru Overpass instance, but we understand that public instances are not intended to be a guaranteed production dependency.
We are also reducing our request volume by adding caching and better throttling. Our use case is retrieving building geometry around specific latitude/longitude coordinates, not bulk downloading.
We are considering paid options such as Overspan or Geofabrik, or setting up our own Overpass instance if needed.
Our main question is: can someone confirm whether our production IP 64.226.68.63 is currently blocked by overpass-api.de, and if so whether it can be unblocked after we reduce the request rate?
Thank you.
[formatting corrected by mod]
Yes, the IP had been blocked for ignoring the 429 return. See my message above. We are at capacity, so everybody needs to back off a bit more. If you get a 429, wait at least 30s before sending the next query.
You should seriously consider options other than Overpass here. If all you need are buildings around a point, then vector tiles might be a better answer. It is supereasy in the age of tilemaker and planettiler to make your own set of “just buildings” tiles and get the information you require from there.
Or play around with parquet files. That might work as well.
Overpass is very likely the least efficient of all options.
Hi! Thanks for your reply, but this will end up in more requests to overpass than I did before. I’ve got a caching-mechanism in my previous solution.
Vector tiles result in no requests to Overpass.
Thank you for confirming.
We understand the issue: our application was not backing off correctly after receiving 429 responses and continued trying nearby radius queries. We are changing this behavior.
The new policy will be:
- when overpass-api.de returns 429, we stop the current request flow;
- we wait at least 30 seconds before sending another Overpass query;
- we will avoid continuing with additional radius attempts after a 429.
Our use case is only to retrieve the building footprint around a given point, so we understand that Overpass may not be the most efficient long-term solution.
Thanks again for the clarification.
Good morning
I’m fine with the new mitigation measures took on public overpass instances to reduce the load and ensure a good service for all.
We’ve got several instances of Podoma that has been blocked due to a flawed retry strategy on our side we are currently working on to fix according to documentation.
Can you tell me how long lasts an IP ban please? Is it permanent and needs to be asked for removal or does it expire after some days/weeks?
The IP bans are completely automatic and will disappear after a while.
Bans by user agent/referrer are manual and you need to ask about them. Those are the ones where you get a 406 returned (before being IP banned when ignoring the error).
Hello
Is there a duration or a rule for this duration documented somewhere?
Do retrying before the end of the ban resets any timer or something?
Currently Podoma instances retry every day and I see failing attempts beginning on August 18th without any ban clearing so far.
this may serve as at least temporary band-aid, and as having key-based limits has chance of being more stable
(at least until ban lifts itself)