Overpass/OSM DB missing data

ChrisPJS · 2019-11-12T13:38:09.000Z

Hi,

I’m not sure to be in the correct subtopic for my question but I didn’t found better.

I try to use Overpass API over Europe OSM data. I want to get all European country (+ their islands). Example, for France : Metropole + Corsica + Guadeloupe + Martinique…

To do it, I downloaded europe OSM datafile from geofabrik : https://download.geofabrik.de/europe-latest.osm.bz2 because I saw that in Europe folder I can get all what I wanted. Example : Europe/France/Corsica+Guadeloupe+Martinique… and same things for others countries.

I followed Overpass API installation : https://wiki.openstreetmap.org/wiki/Overpass_API/Installation with success.

I tried to request on different countries with success but can’t get data for many islands. Example : I can get Corsica data but CAN’T get for others French islands (Guadeloupe, Martinique…).

Request example :

[out:json]; node(around:50,16.24263,-61.535662); <; out geom;

→ this returned values using https://overpass-turbo.eu test tool, but nothing with my custom overpass API (tried on different request and different GPS loc).

Also, I checked import log files generated during import of .osm.bz2 into db/ folder, I saw many (more than hundred) lines like :

compute_geometry: Way 83225893 used in relation 10106318 not found.

→ is it related to my problem ?

Can the europe-latest.osm.bz2 be corrupted (missing data) ?
Can I import islands on my current /db to not re-download all data ?

Thank a lot !

SomeoneElse · 2019-11-13T11:09:08.000Z

Some relations for countries mostly in Europe (for example France) have parts that are outside Europe, so it’s quite possible that a “European” extract will be missing some parts of some countries.

What problem are you actually trying to solve?

ChrisPJS · 2019-11-13T12:42:18.000Z

My initial goal was : to get European countries (geographic) + all islands associated to them (example : France is an European country, so I want also their islands : in Europe like Corsica + outside Europe like Martinique, Guadeloupe…).

SomeoneElse · 2019-11-13T15:13:04.000Z

ChrisPJS:

France is an European country

Strictly speaking, France is a country that is mostly in Europe (see https://en.wikipedia.org/wiki/Overseas_France for details). it sounds like you’ve got the European bit of France, but not the rest.

The actual error message you’ve got is because you’ve got part of the Euphrates river https://www.openstreetmap.org/relation/10106318 (the European bit, I guess) but not https://www.openstreetmap.org/way/83225893 , which is presumably outside Geofabrik’s definition of Europe.

How to fix this probably depends on what your goal is; that’s why I asked that.

muralito · 2019-11-13T20:19:31.000Z

SomeoneElse:

Strictly speaking, France is a country that is mostly in Europe

In fact, France largest border is with Brazil…

ChrisPJS · 2019-11-14T08:40:51.000Z

Thank you for details.

Finally, someone has an idea on how to achieve my goal ?

SomeoneElse · 2019-11-14T08:42:03.000Z

ChrisPJS:

Finally, someone has an idea on how to achieve my goal ?

Unfortunately, I don’t think that we know what that is yet…

PHerison · 2019-11-14T09:37:58.000Z

I assume you will need to import data for other relevant parts of the globe (oceania, south-america, …) as well.

ChrisPJS · 2019-11-14T10:57:35.000Z

How can I import two continents ? Example : Europe + America only (not the whole planet)

escada · 2019-11-15T04:22:58.000Z

Do you just need the boundaries of those countries? In that case, you can use https://wambachers-osm.website/boundaries/
If you need other data such as roads, POI, etc. you need another method.

So please explain which data you mean when you write “I want to get all European country (+ their islands)” It might be obvious to you what you need, but it is not for the people that want to help you.

ChrisPJS · 2019-11-18T11:09:09.000Z

Hi, I need all data like roads, POI…

Thank you for taking time for my problem !

SomeoneElse · 2019-11-18T11:14:10.000Z

ChrisPJS:

Hi, I need all data like roads, POI…

You’ve downloaded all that - so you’ve already got it. What you haven’t said is what you are trying to do with it.

ChrisPJS · 2019-11-19T15:46:35.000Z

For example, I try to get all node in one island with :

[out:json]; node(around:100,16.24263,-61.535662); <; out geom;

I don’t get any data returned.
Same request on overpass-turbo returns me data.

Also, same request with lat,lon in European country (Germany for example, or France) returns me data
=> so I have a problem (missing data) in my DB

alphensebezorger · 2019-11-19T20:17:15.000Z

so I have a problem (missing data) in my DB

That is correct. The file you downloaded from geofabrik doesn’t contain any objects outside the polygon shown on http://download.geofabrik.de/europe.html. That is why you can’t find any data for the overseas islands. You will have to download more OSM data somehow.
The Europe extract intentionally does not include any data on the overseas parts of European countries, which are not inside the polygon.

ChrisPJS · 2019-11-20T16:08:23.000Z

Yes I understand, bu do you know how to combine two polygon into one DB ?

SomeoneElse · 2019-11-20T16:30:28.000Z

ChrisPJS:

Yes I understand, bu do you know how to combine two polygon into one DB ?

Typically, you’d merge them together before loading into a database. “osmium” is one way of doing that, “osmosis” is another. One thing that a map server that I run does is splits a downloaded file into several pieces, translates each piece into a different language, and merges them all back together. The script that does all this (using osmosis) is here: https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/update_render.sh .

There’s also documentation on osmosis and osmium in the OSM wiki and linked from it.

ChrisPJS · 2019-11-21T09:55:59.000Z

Thank you for response, I’ll check in this way.

ChrisPJS · 2019-11-21T15:13:47.000Z

I tried, it worked fine with multiples small files but can’t merge huge file (europe :33GB) with others small files (memory error).

Is there a solution to add .osm.bz2 files into an existing database ?
The idea is to initial load with Europe and after: import in the existing DB others files.

Thanks

SomeoneElse · 2019-11-21T16:32:50.000Z

ChrisPJS:

I tried, it worked fine with multiples small files but can’t merge huge file (europe :33GB) with others small files (memory error)

What was the “it” that you tried? I made two suggestions…

What actual command line did you use? How much memory do you have? Do you have the opportunity to configure more (e.g. via swap)?

ChrisPJS · 2019-11-22T11:08:49.000Z

I tried with osmium :

osmium merge little.osm.bz2 europe.osm.bz2 -o merged.osm.bz2 --input-format=osm.bz2 --v

I have 8 GB RAM + 8GB SWAP

=> Do I must have SWAP >= little+europe (xxxMB + 33GB) ?