Overpass API requests timing out in FME

Hello,

I am having an issue accessing the Overpass API from FME using the HTTPCaller.

The endpoint I am using is

https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];(way[“leisure”=“park”](41.279903,-73.516389,41.294408,-73.434042););out body;>;out body geom;

FME returns the following error:

“HTTP transfer error: Timeout was reached”

It also indicates:

“Please ensure that your network connection is properly set up.”

I have tested different bounding boxes, and the requests continue to experience timeouts.

Could someone please let me know:

  1. Is the Overpass API currently experiencing any problems?
  2. Is there a recommended Overpass instance I should use instead of overpass-api.de?
  3. Are there any recommended timeout or HTTP settings when accessing Overpass from FME?
  4. Is there a recommended way to query OpenStreetMap park data for larger areas such as towns, states, or the United States?

I would appreciate any guidance on whether this is an API/server issue or something I need to configure differently in FME.

Thank you for your help.

Best regards,
Madhusudan

(help and support mod note)

I’ve lightly edited** the post above so that the endpoint undo occur after the relevant text. I’ve removed some of the formatting from the query itself and indented out to remove Markdown formatting, but I’m not convinced that the (previously formatted) query matched what was being run, since it’s not currently a valid overpass query.

Note that posts here are formatted with markdown and indenting lines by 4 spaces like

this

formats code better

** and the submitter has edited it again to make it clear that it’s essentially one http get with a defined area in Connecticut.

(to try and answer the question)

I’m guessing that FME is this, and I’m not familiar with it. I’m surprised that the overpass query that it is using does not have any constraint on the area that the search is taking place on.

A similar search generated from taginfo looks like this:

/*
This query looks for nodes, ways and relations 
with the given key/value combination.
Choose your region and hit the Run button above!
*/
[out:json][timeout:25];
// gather results
nwr["leisure"="park"]({{bbox}});
// print results
out geom;

There, the {((bbox}}) implies “search within only the area hat I can see on screen”. Where this isn’t appropriate there are other ways to specify areas; see here in the OSM wiki.

However, as you note sometimes the server that you are using is “just too busy” (because everyone is using it). You can see a list of alternative Overpass servers in the wiki here. Some are free, some not.

Other options that might work include postpass (a different API to overpass; one that actually matches OSM data types better), or, if you really want data from a larger area “just downloading data from planet.osm.org or an extract and extracting data with osmium etc.”

https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];(way[“leisure”=“park”](41.279903,-73.516389,41.294408,-73.434042););out body;>;out body geom;

I am using this.

If you get a timeout, then your IP has been temporarily banned from the servers. This might have had different reasons: not sending a custom user agent, sending a user agent that was banned for overusing the servers or you’ve simply sent too many requests.

It looks like FME can natively process OSM PBF planet files. I suggest switching to that integration.

There isn’t a stable/useful Overpass API ATM.

The main endpoint in Turbo:
https://overpass-api.de/api/

Returns this ~70% of the time:

An error occurred during the execution of the overpass query! This is what overpass API returned:
Error: runtime error: open64: 0 Success /osm3s_osm_base Dispatcher_Client::request_read_and_idx::rate_limited. Please check /api/status for the quota of your IP address.

My actual OSM editing time has reduced to ~20% due to these ongoing problems either trying to find ways around the problem or just giving it up as a bad job. I even found watching Mission Impossible !! more enjoyable than fighting OSM’s servers/.