JOSM QGIS Arc Pro - .OSM Extraction

I have the newest planet.osm file and I can’t seem to do anything with it. I have ArcPro, QGIS and JOSM. JOSM and QGIS crash when I try to do anything with such a large file and as we know Arc Pro is for some reason entirely unsupported especially since planet.osm’s only shape files are in Antarctica.

How do I get the data from the .osm into something I can actually use? Ideally I’d just like to download a country, but I can’t do that. Even GeoFabrik is by continent.

There are command line tools that will do regional extracts from the overall OSM planet file. I think Osmosis will do it, but I’ve never tried myself.

A variety of sites offer pre-made extracts, not just Geofabrik.

If you need support for a commercial product you should probably contact the vendor. OSM was not created to be a distribution means for proprietary file formats.

Most of the instances I’ve heard of people using large amounts of OSM data in traditional GIS have involved first loading it into some sort of database suitable to the work they’re doing first and then connecting to that database. The PostgreSQL page on the gives several means of getting OSM data into it for further use. There may be people on this forum that can offer specific advice if you describe what you’re actually trying to accomplish but this is a very broad post otherwise.

I just went through GEOFABRIK and grabbed a country, imported it to QGIS, and then exported to .shp - the attribute tables are basically empty. Nothing that’s on OSM transferred over which makes this data useless.

Those extracts are useless to me without the metadata and attributes that are on the organic OSM website itself.

Tags like “route” instead of road, rail, etc are worthless for GIS work. It’s broken down on OSM, but there’s no way to get it.

Have you tried reading the documentation they have for the shape files that they provide?

Can you give some details as to what versions of their files you are actually using and how you are attempting to make use of them? If we knew what you were actually trying to accomplish someone here might be able to offer constructive suggestions.

1 Like

GeoFabrik strips the shapefiles (they only exist in Antarctica which is useless) and the .pbf or .osm files of all of the original OSM attributes which makes them worthless.

Things I’ve tried -
Planet OSM most recent wold file (144gb) download
GeoFabrik - ASIA .bz2 download
GeoFabrik - Taiwan .pbf download

Ways to open I’ve tried-
QGIS - open file export to .shp
JOSM (just crashes)
Arc Pro (doesn’t do anything because the only importers are for desktop which hasn’t been mainstream for nearly a decade.

End state goal -
This all started after finding this: https://www.openrailwaymap.org/ and wanting to get the data for Taiwan for a project. It seems impossible. The website itself won’t answer any emails. This data is from OSM but of course you can’t download a country’s worth of data and no one in their right mind is going block by block through an entire country to get all of the data (with the attributes, because otherwise it’s useless).

I am about to just call it a known data gap and a loss because I can own all of the outputs but if I cannot use it in any way then it’s just taking up space.

I also just tried Overpass Turbo:
/*
This has been generated by the overpass-turbo wizard.
The original search was:
“railway=* in Taiwan”
*/
[out:json][timeout:25];
// gather results
nwr"railway"; // Area ID for Taiwan
// print results
out geom;

But it returns empty files.

When I run:

/*
This has been generated by the overpass-turbo wizard.
The original search was:
“railway=* in Taiwan”
*/
[out:json][timeout:25];
// fetch area “Taiwan” to search in
{{geocodeArea:Taiwan}}->.searchArea;
// gather results
nwr["railway"](area.searchArea);
// print results
out geom;

I get a warning:

Large amounts of data

This query returned quite a lot of data (approx. 20 MB).

Your browser may have a hard time trying to render this. Would you really like to continue?

Which I said no to because It really is a bit much for a single browser tab, but it doesn’t sound like that’s “no data”.

The equivalent “quick query” in the QuickOSM plugin for QGIS gives four layers and massive attribute tables.

Hurrah! It only took 4 posts before we actually found out what you want to do. :smile:

It the way you phrased your requests is anything like the way that you’ve posted here, then I wouldn’t be surprised they went into the virtual circular container.

(in a Reg Presley voice) … But you have done it. What you haven’t done is managed to process what you’ve downloaded from GeoFabrik. OpenRailwayMap shows 5 different map styles over 2 different backgrounds, with another optional overlay. The OpenStreetMap tags used by the project are described here. There are a lot of them.

As an example, let’s pick a bit of railway in Taiwan. The tags that you can see there will exist in the “planet” file, and in the Taiwan data that you’ve downloaded (compare with that object as XML). In order to do something useful with it, you’ll need to define what data you want from it and process it. The source code for OpenRailwayMap has a Readme here which links to a map style here. If you wanted to deploy an instance of that map style, that’s where you’d start. However, you probably want something slightly different, perhaps “all current railway=rail in Taiwan and some (but not all) tags”. You may then want to throw that into a database to do more processing on it, or create a map from it, or something else.

You know with the snarky sarcasm in your post you’d think you’d be able to sniff the question out of my original post. Let me point it out for you:
“How do I get the data from the .osm into something I can actually use? Ideally I’d just like to download a country, but I can’t do that. Even GeoFabrik is by continent.”

I haven’t done anything. I pulled the .osm by using the API extent extraction for the bottom, middle, and top of Taiwan. I brought that into QGIS and extracted it to .shp’s (what I needed). I brought that data into Arc Pro and changed the symbology on rail with embedded tags and filtered through what was displayed.

People keep pointing to GeoFabrik like it’s the answer to everything, it isn’t. ESRI and AGOL are the largest commercial supporters of GIS companies and yet our OSM doesn’t work with it unless you want to jump through hoops. Wasting time is wasting money.

At least InsertUser came with some great information and helped me as I worked towards something I could use. Go crawl back in your hole SomeoneElse.

If you don’t know how to answer a question, maybe next time just don’t post. Nothing in what you provided is an answer, at all.

2 Likes

And the award for most entitled user goes to…

You are getting snarky responses because your posts (can’t really call them questions) make it clear that you didn’t bother to do even a little bit of reading or research on the topic before coming here to ‘ask’ unspecific non-questions and call everything that you don’t understand “useless”.

I would suggest getting your eyesight (and/or attention span) checked.

1 Like

It’s funny that you trolls keep coming with the same misinformation and argument. Yes that opens up a list of countries in the region that you want to look at. Yes it also gives you an OSM PBF or BZ2 file and not a shape file. So for circling back to the original issue which is the question of what do we do with an OSM file if we work primarily under esri circles with ArcMap and Arc Pro? Your answer still does absolutely nothing to provide feedback on that but thanks to your limited understanding of literacy we see that you have called me entitled and provided nothing so thank you once again for calling me back here to waste my time. :pinched_fingers:t2:

  1. Download .pbf file from Geofabrik
  2. Decide how you want to map OSM’s elements and tags to GIS feature types and attributes, create your_osmconfig.ini accordingly.
  3. Run the conversion
ogr2ogr --config OSM_CONFIG_FILE your_osmconf.ini -skipfailures -f "ESRI Shapefile" output.shp input.pbf  -overwrite

See OSM - OpenStreetMap XML and PBF — GDAL documentation for more info.

1 Like

Both the table that links to and the Taiwan page link to taiwan-latest-free.shp.zip, which as the later page says “yields a number of ESRI compatible shape files when unzipped”. This is all written in plain English on their website and readily understood by anyone vaguely literate. QGIS is able to open directly from this zip file just fine and I would presume Esri software can also read it as they invented the file format.

As I mentioned above if you have a specific problem with the file handling capabilities of a commercial piece of software you should probably take it up with them; that’s what support contracts are for.

Now if you had more specific requirements for the data extract you require we could happily have advised you if you had offered us the common courtesy of describing what you actually required in a reasonable level of detail. Instead you came to a community run project’s support forum, run by volunteers, and expected us to waste significantly more time trying to guess what you needed rather than you just telling us.

I would also note that the company who is generously hosting copies of OpenStreetMap data for free says in multiple places on their site that they are able to offer custom extracts for a fee which should have jumped out to you as a cost effective option if you really do value your time as much as you claim to.

3 Likes