Generating shapefiles from OSM data

[Anthony_Ap]

Hello everyone,

I’m quite new to OpenStreetMap and my purpose is to generate shapefiles for every country of the world. However, I’ve noticed how difficult it is to find good data.

Until today, I was using geofabrik and wambachers web application to retrieve shp. I’ve also tried Overpass Turbo, Osmosis, osm2shp.

Problem: data provided within ESRI Shapefile are not always true. Let me explain: there are regions and departments that no longer exists, administrative boundaries may be outdated or gathering water areas (Netherland for example). Also, regions are not always on admin_level = 4 (same for departments). This is upsetting as I’m willing to make a generic program.

Is there a way to generate myself shp in order to get administrative land boundaries for

• country level?
• region level?
• department level?

Thanks in advance for your help!

[kocio]

I guess you can download boundaries for any area in different formats (including SHP) from here:

https://wambachers-osm.website/boundaries/

[Anthony_Ap]

Thank you kocio for your quick response.

https://wambachers-osm.website/boundaries/, that’s where I used to download my shp and where i found the problem.
For instance, netherlands shp includes seas and water areas which I want to exclude.

To be more specific: is there a way to generate administrative boundaries shp using osm file?

[alester]

@Anthony Ap

…there are regions and departments that no longer exists, administrative boundaries may be outdated…

If you know there are errors in the database or the data is outdated, the best solution would be to fix those in the OSM database.

@Anthony Ap

For instance, netherlands shp includes seas and water areas which I want to exclude.

As for excluding sea areas of administrative boundaries, that’s much more difficult. The boundary of the Netherlands (and any other country bordering the ocean) extends out into the water in reality, so that’s what the OSM database contains. If you don’t want that area included, you’d need to manually build your own boundary following the coastline in your own database. There’s probably a way to do this part programmatically by following the connected natural=coastline ways between the two points where the boundary intersects the coastline.

[smaprs (Sérgio)]

@Anthony Ap

is there a way to generate administrative boundaries shp using osm file?

You can easily do it with QGIS. Install in it QuickOSM plugin (or open data something like that too), so you can import in .osm or .geojson. You can also download all of .OSM objetcts at http://download.geofabrik.de/. I guess there are only boundaries anywhere too. Or select boundaries or anything that you want by Overpass https://wiki.openstreetmap.org/wiki/Overpass_turbo/Wizard and import to QGIS. Then save what you want as SHP.

[wambacher (Walter Nordmann)]

@Anthony Ap

For instance, netherlands shp includes seas and water areas which I want to exclude.

Just check LAND in the buttom area and you are done.

much better: Use the CLI and set &ExportArea=land

walter

[Anthony_Ap]

If you know there are errors in the database or the data is outdated, the best solution would be to fix those in the OSM database.

I’m willing to become an active contributor as OSM concerns are also mine

There’s probably a way to do this part programmatically by following the connected natural=coastline ways between the two points where the boundary intersects the coastline.

Can’t we correct the admin boundaries directly into OSM?

@smapsr I’ve tried both QuickOSM and overpass, great tool but sadly same problem comes by about water areas. Maybe there’s a way to filter it from overpass but i can’t figure it out

[Anthony_Ap]

@wambacher

@Anthony Ap

For instance, netherlands shp includes seas and water areas which I want to exclude.

Just check LAND in the buttom area and you are done.

much better: Use the CLI and set &ExportArea=land

walter

I’ve follow the instructions on your website about the cli command to get shp and i think it’s the most suitable tool! However, the seas within the borders are not excluded. I’ve managed to merge entities so it doesn’t include water area but I’d like to import it into osm data

[wambacher (Walter Nordmann)]

@Anthony Ap

I’ve follow the instructions on your website about the cli command to get shp and i think it’s the most suitable tool!

thanks.

However, the seas within the borders are not excluded.

Yes, only seas with international boundaries are clipped. And some realy big ones too.

insert into water_polygons values(100001,100000,(select st_multi(way) from planet_osm_polygon where osm_id=-1156846),'Bodensee 1156846');
insert into water_polygons values(100002,100001,(select st_multi(way) from planet_osm_polygon where osm_id=-3474227),'Rio de la Plata-Mündung 3474227');
insert into water_polygons values(100003,100002,(select st_multi(way) from planet_osm_polygon where osm_id=-1374224),'Tanganjikasee 1374224');
insert into water_polygons values(100004,100003,(select st_multi(way) from planet_osm_polygon where osm_id=-108807),'Mwerusee 108807');
insert into water_polygons values(100005,100004,(select st_multi(way) from planet_osm_polygon where osm_id=-318675),'Karibasee 318675');
insert into water_polygons values(100006,100005,(select st_multi(way) from planet_osm_polygon where osm_id=-1969016),'Turkanasee 1969016');
insert into water_polygons values(100007,100006,(select st_multi(way) from planet_osm_polygon where osm_id=-332617),'Genfer See 332617');
insert into water_polygons values(100008,100007,(select st_multi(way) from planet_osm_polygon where osm_id=-5901404),'Westlicher Aralsee 5901404');
insert into water_polygons values(100009,100008,(select st_multi(way) from planet_osm_polygon where osm_id=-5901405),'Westlicher Aralsee 5901405');
insert into water_polygons values(100010,100009,(select st_multi(way) from planet_osm_polygon where osm_id=-1269323),'Sarykamysch-See 1269323');
insert into water_polygons values(100011,100010,(select st_multi(way) from planet_osm_polygon where osm_id=-6515),'Laguna Merín 6515');
insert into water_polygons values(100012,100011,(select st_multi(way) from planet_osm_polygon where osm_id=-1125603),'Chankasee 1125603');
insert into water_polygons values(100013,100012,(select st_multi(way) from planet_osm_polygon where osm_id=-3000330),'Titikaka-See 3000330');
insert into water_polygons values(100014,100013,(select st_multi(way) from planet_osm_polygon where osm_id=-1206310),'Ontario-See 1206310');
insert into water_polygons values(100015,100014,(select st_multi(way) from planet_osm_polygon where osm_id=-4039900),'Eriesee 4039900');
insert into water_polygons values(100016,100015,(select st_multi(way) from planet_osm_polygon where osm_id=-4228055),'Saint Clair-See 4228055');
insert into water_polygons values(100017,100016,(select st_multi(way) from planet_osm_polygon where osm_id=-1205151),'Huron-See 1205151');
insert into water_polygons values(100018,100017,(select st_multi(way) from planet_osm_polygon where osm_id=-4228055),'Saint Clair-See 4228055');
insert into water_polygons values(100019,100018,(select st_multi(way) from planet_osm_polygon where osm_id=-4039486),'Oberer See 4039486');
insert into water_polygons values(100020,100019,(select st_multi(way) from planet_osm_polygon where osm_id=-1205149),'Michigan-See 1205149');
insert into water_polygons values(100021,100020,(select st_multi(way) from planet_osm_polygon where osm_id=-2606941),'Victoriasee 2606941');
insert into water_polygons values(100022,100021,(select st_multi(way) from planet_osm_polygon where osm_id=-1110965),'Albertsee 1110965');
insert into water_polygons values(100023,100022,(select st_multi(way) from planet_osm_polygon where osm_id=-940893),'Eduardsee 940893');
insert into water_polygons values(100024,100023,(select st_multi(way) from planet_osm_polygon where osm_id=-2631184),'Malawisee 2631184');
insert into water_polygons values(100025,100024,(select st_multi(way) from planet_osm_polygon where osm_id=-4557289),'Dojransee 4557289');
insert into water_polygons values(100026,100025,(select st_multi(way) from planet_osm_polygon where osm_id=-2905510),'Prespasee 2905510');
insert into water_polygons values(100027,100026,(select st_multi(way) from planet_osm_polygon where osm_id=-2905270),'Ohridsee 2905270');
insert into water_polygons values(100028,100027,(select st_multi(way) from planet_osm_polygon where osm_id=-392575),'Skutarisee 392575');
insert into water_polygons values(100029,100028,(select st_multi(way) from planet_osm_polygon where osm_id=-1298088),'Bilećasee 1298088');
insert into water_polygons values(100030,100029,(select st_multi(way) from planet_osm_polygon where osm_id=-36970),'Urmia Lake 36970');
insert into water_polygons values(100031,100030,(select st_multi(way) from planet_osm_polygon where osm_id=-5486417),'Asowsches Meer 5486417');
insert into water_polygons values(100032,100031,(select st_multi(way) from planet_osm_polygon where osm_id=-2791995),'Asowsches Meer Erweiterung 2791995');
insert into water_polygons values(100033,100032,(select st_multi(way) from planet_osm_polygon where osm_id=-1206317),'Großer Salzsee 1206317');
insert into water_polygons values(100034,100033,(select st_multi(way) from planet_osm_polygon where osm_id=-1240556),'Okeechobeesee 1240556');
insert into water_polygons values(100035,100034,(select st_multi(way) from planet_osm_polygon where osm_id=-9072),'Kiwusee 9072');
insert into water_polygons values(100036,100035,(select st_multi(way) from planet_osm_polygon where osm_id=-1348022),'Abbesee 1348022');
insert into water_polygons values(100037,100036,(select st_multi(way) from planet_osm_polygon where osm_id=-280282),'Nassersee 280282');
insert into water_polygons values(100038,100037,(select st_multi(way) from planet_osm_polygon where osm_id=4741633),'Totes Meer 4741633'); -- keine rel sondern way
insert into water_polygons values(100039,100038,(select st_multi(way) from planet_osm_polygon where osm_id=-2905510),'Prespasee 2905510');
insert into water_polygons values(100040,100039,(select st_multi(way) from planet_osm_polygon where osm_id=-2905270),'Ohridsee 2905270');

technically no problem for me to add more - but where is the end? Which should be added and which not?

Regards
walter

[alester]

@Anthony Ap

Can’t we correct the admin boundaries directly into OSM?

Yes, definitely. Keep in mind, though, that a country boundary extending out into the ocean is already correct. Changing it to follow the coastline in the public database would be quickly reverted because it wouldn’t reflect reality.

[wambacher (Walter Nordmann)]

@Anthony Ap

Can’t we correct the admin boundaries directly into OSM?

definitely no - because they are not wrong.

btw: Clipping boundaries by coastline is not an easy job. My programs are running some seconds up to 30 !! hours to clip all boundaries of one single country depending on coastline complexity

And yes, i’m proud of it

Regards
walter

[smaprs (Sérgio)]

@smapsr I’ve tried both QuickOSM and overpass, great tool but sadly same problem comes by about water areas. Maybe there’s a way to filter it from overpass but i can’t figure it out

Don’t know if I got it well, but the filters for overpass wizard are for example:

boundary=* and natural!=coastline in AnyCountry

for only boundaries, all of them; and none of coastline.
(I don’t remember if these examples match with reality, just examples)
More in its wiki.

[Anthony_Ap]

Hello Walter,

Since the beginning of the year, I’m wondering how to regenerate all shp and data of every country using OSM data. It’ll be much more simpler (for users and me) to correct data directly into OpenStreetMap. This is the ranking of websites providing (I believe) accurate shapefiles in most cases:

http://overpass-turbo.eu/
https://wambachers-osm.website/boundaries
https://gadm.org/data.html
http://download.geofabrik.de
https://extract.bbbike.org
http://www.diva-gis.org/gdata
https://osmaxx.hsr.ch/orders/new/existing_excerpt/
http://www.statsilk.com/maps/download-free-shapefile-maps

And yes, I’m proud of it

and I think you can be pretty proud of it. Yours is only second because when the shapefile is generated, attribute table contains less information than overpass’ - alternateNames for instance.

I’ve been studying how to generate my own shapefiles using osmosis but I often encounter difficulties like the one I’ve been describing in my post. How do you manage to export land boundaries on wambachers-osm?

Warm regards

[Anthony_Ap]

@wambacher

technically no problem for me to add more - but where is the end? Which should be added and which not?

ok i get the issue right here - for this particular case i’ve been editing the shp with qgis to fit my needs. Is there a solution to filter it automatically instead of using qgis?

@smaprs

for only boundaries, all of them; and none of coastline.

Maybe you meant “natural=coastline”? I didn’t retrieve any results for Emirates neither Netherlands. I’ll keep looking into the wiki - thanks!

[wambacher (Walter Nordmann)]

@Anthony Ap

Yours is only second because when the shapefile is generated, attribute table contains less information than overpass’ - alternateNames for instance.

There are some technical restictions for shape files:

see https://desktop.arcgis.com/en/arcmap/latest/manage-data/shapefiles/geoprocessing-considerations-for-shapefile-output.htm

• Field names cannot be longer than 10 characters.
• The maximum record length for an attribute is 4,000 bytes. The record length is the number of bytes used to define all the fields, not the number of bytes used to store the actual values.
• The maximum number of fields is 255. A conversion to shapefile will convert the first 255 fields if this limit is exceeded.
• The dBASE file must contain at least one field. When you create a new shapefile or dBASE table, an integer ID field is created as a default.
• dBASE files do not support type blob, guid, global ID, coordinate ID, or raster field types.
• dBASE files have little SQL support aside from a WHERE clause.
• Attribute indexes are deleted when you save edits, and you must re-create them from scratch.

Because of that i have to

• change attributes like “name:en” to “enname” (no colon allowed in dBase)
• change attributes like “official_name” to “off_name” (length)
• limit the number of attributes to a subset (maximun 255 allowed)

Since there are currently 2896 !! different attributes (tags) in use, I had to make a selection. It’s of course possible, to add more attributes (up to 255 of course).

see https://wambachers-osm.website/images/osm/data/count_tags_usage.txt

There are 2 solutions:

• tell me, what you need - and wait some time
• export GeoJson and convert to Shape on your site. GeoJson has none of those restrictions and does contain all tags. (hope so ;))

Regards
walter

btw: which lakes should be clipped additionally?

[Anthony_Ap]

@wambacher

tell me, what you need - and wait some time

I think this is the best solution, I’ve tried to convert geoJSON to shp and you were right about restrictions.
Actually my needs are essentially alternate names for foreign languages.

btw: which lakes should be clipped additionally?

Actually ijsselmeer and Zeeuwse meren:
https://giphy.com/gifs/k5zUV76AnFY56h8JX8