Tool to manipulate OSM data

PeeWee32 · 2012-02-17T20:01:50.000Z

I’m a looking for a free/opensource tool to delete, change and add OSM data. These manipulations are not meant for changing the OSM source data but to achieve easier post processing of OSM data in order to create a Garmin Map. For creating a Gamin map I use MKGMAP. Input files are standard OSM file formats such as . osm. .pbf or .bz2 . I am looking for a database tool that can read these files, manipulate them and again produce a . osm .pbf or .bz2 file.
I use a windows PC. I am accustomed to using MS access databases but have no experience with GIS databases whatsoever.

The type of processing I am looking for is:

Adding/deleting tags to “ways” based on (a combination of) other tags.
Adding/deleting tags to “ways” based on the location of (part of the) way in a polygon/area.
Adding/deleting tags to “ways” based their being a member of a “route-relation”
Etc. etc

Is there a tool that can do (more or less) these things? Any suggestions?

Cheers Peewee

Vclaw · 2012-02-18T00:50:13.000Z

You can probably do most of that using Osmosis with the TagTransform plugin. http://wiki.openstreetmap.org/wiki/Osmosis/TagTransform

stephan75 · 2012-02-18T10:26:34.000Z

… or you can have a look at the source code of http://wiki.openstreetmap.org/wiki/Osmfilter and http://wiki.openstreetmap.org/wiki/Osmconvert

… or http://wiki.openstreetmap.org/wiki/Osmium

PeeWee32 · 2012-02-19T06:54:30.000Z

Stephan and Vclaw thanks a lot. I will have a look at the links. I have already done a little bit with OSMconvert but have not examened all of it’s potential. Enough info for the next weeks I suppose

Cheers PeeWee

Bernhard_Hiller · 2012-02-19T09:24:47.000Z

Actually the actions you want to do are actions defined in the “style”. You can create new tags there, change existing ones, add tags when not yet existing, overwrite values, etc.
Some examples:
In my map for skiing, the relations file contains e.g.
piste:grooming=classic { apply { set loipetype=classic } }
That means: when the relation contains a piste:grooming tag, and that tag has the value classic, I add my tag loipetype=classic (there is no loipetype defined elsewhere, I created it in my style files for processing here only).
In the lines file, that gets translated to a custom line:
loipetype=classic & loipediff=easy [0x10f01 resolution 16]

In a different map, I show dirt roads with a different style than paved roads:
highway=unsurfaced {add tracktype=normal_track;}
highway=* & (surface=gravel | surface=pebblestone | surface=unpaved) { add tracktype=normal_track }
highway=* & (surface=compacted ) { add tracktype=normal_track }
highway=* & (surface=surface=grass | surface=ground | surface=sand ) { add tracktype=bad_track }
tracktype=grade2 {set tracktype=normal_track }
tracktype=grade3 {set tracktype=normal_track }
tracktype=grade4 {set tracktype=bad_track }
tracktype=grade5 {set tracktype=bad_track }
highway=major & tracktype=* & tracktype!=grade1 { set highway=major_dirt }
highway=minor & tracktype=* & tracktype!=grade1 { set highway=minor_dirt }
highway=major [0x03 road_class=3 road_speed=4 resolution 16]
highway=major_dirt [0x11 road_class=1 road_speed=2 resolution 18]
highway=minor [0x04 road_class=2 road_speed=3 resolution 18]
highway=minor_dirt [0x12 road_class=0 road_speed=1 resolution 20]

My conclusion: you do not need extra software, just get used to the rules of the mkgmap style files, and adjust your TYP file with e.g. MapTk.

PeeWee32 · 2012-02-19T19:16:38.000Z

Thanks Bernhard for your reply. I am afraid that I have not been clear in my original posting. I am aware of (most/some) of Mkgmap’s possibilities but as said I am also familiar with MS access. Querying databases (SQL) is something I understand better than Mkgmap style rules. I was just hoping there would be some sort of SQL environment for querying/changing OSM database tables. I’m not familiar with GIS database so maybe I am looking for something that just is not there. It could also be that there are possibilities but that they go beyond mine .

I’ll give you an example that I think cannot be solved with Mkgmap but maybe could be solved with another tool.

In the Netherlands a lot of “highways” are tagged as “unclassified” that could / should be tagged as “residential”. For post processing I would like to change all “unclassified” highways to “residential” if they are (partly) within an area of landuse=residential. If Mkgmap can solve this it would be great but I doubt it. If another tool can it would also be great and it might broaden my horizon with possibilities that I am not aware of right now.

Mattheus · 2012-02-19T20:06:57.000Z

PeeWee32:

In the Netherlands a lot of “highways” are tagged as “unclassified” that could / should be tagged as “residential”. For post processing I would like to change all “unclassified” highways to “residential” if they are (partly) within an area of landuse=residential. If Mkgmap can solve this it would be great but I doubt it. If another tool can it would also be great and it might broaden my horizon with possibilities that I am not aware of right now.

I just had a look at the mkgmap style sheets:

highway=residential [0x06 road_class=0 road_speed=2 resolution 22]
highway=minor [0x06 road_class=1 road_speed=3 resolution 21]
highway=unclassified [0x06 road_class=0 road_speed=3 resolution 21]

As you see the mkgmap standard style sheet does the job you require. If your style sheets are different, simply change the number into 0x06 if you use the standard mkgmap style sheets.

chris66 · 2012-02-19T21:46:03.000Z

PeeWee32:

In the Netherlands a lot of “highways” are tagged as “unclassified” that could / should be tagged as “residential”.

This should be fixed in the database and I already fixed one town in NL.

PeeWee32 · 2012-02-20T18:28:57.000Z

Mattheus:

PeeWee32:

For post processing I would like to change all “unclassified” highways to “residential” if they are (partly) within an area of landuse=residential.

I just had a look at the mkgmap style sheets:

highway=residential [0x06 road_class=0 road_speed=2 resolution 22]
highway=minor [0x06 road_class=1 road_speed=3 resolution 21]
highway=unclassified [0x06 road_class=0 road_speed=3 resolution 21]

As you see the mkgmap standard style sheet does the job you require. If your style sheets are different, simply change the number into 0x06 if you use the standard mkgmap style sheets.

But how do I know if the highway=unclassified is within a Landuse=residental area?

Mattheus · 2012-02-21T07:54:44.000Z

PeeWee32:

But how do I know if the highway=unclassified is within a Landuse=residental area?

Try changing the 0x06 into 0x10 if you use the mkgmap lines style sheet. This should renderer the line into a cyclepath instead of an unclassified road. In this way you can check whether this works.

edit:
I just looked up the lines in the standard mkgmap lines style sheet. There is no 0x10 line in it mentioned. So the line will not rendered or will be visible as a line. You can try 0x16 (footpath) or 0x0a (track).

Bernhard_Hiller · 2012-02-21T18:23:28.000Z

I understand your problem. But as far as I do know, it cannot be solved with mkgmap styles. Also not with “simple” database operations.
The complicated point is that a “way” need not fully be inside an “area”, and in such a case you’d have to split it into way(s) outside the area and a way inside. But even for the simplified case that at least one point of the “way” should be inside the “area” to be considered inside, there is no solution with mkgmap, and with Microsoft Access you’d have to do lots of VBA programming (and with the thousands of ways and areas in the Netherlands, the iterations would be inacceptably slow).

PeeWee32 · 2012-02-22T17:05:31.000Z

Bernhard Hiller:

I understand your problem. But as far as I do know, it cannot be solved with mkgmap styles. Also not with “simple” database operations.
The complicated point is that a “way” need not fully be inside an “area”, and in such a case you’d have to split it into way(s) outside the area and a way inside. But even for the simplified case that at least one point of the “way” should be inside the “area” to be considered inside, there is no solution with mkgmap, and with Microsoft Access you’d have to do lots of VBA programming (and with the thousands of ways and areas in the Netherlands, the iterations would be inacceptably slow).

I understand the complication but I was hoping some sort of GIS database tool (PostgreSQL ??) could do the trick (and a lot more). MS Access is no option because I have no clue how to get OSM data in an Access database and besides I also do not think Access is suitable for GIS data.

ligfietser · 2012-03-07T16:35:01.000Z

I think mkgmap could handle this when you use the locator function.

A small test with the Luxembourg abstract seemed to work.

Add all landuse=residential to a multipolygon relation with admin_level=11 & boundary=administrative & name=residential_area in JOSM and save this osm file
Prepare boundary files from this extract: http://wiki.openstreetmap.org/wiki/Mkgmap/help/options#Using_precompiled_bounds_for_the_address_index

osmfilter luxembourg.osm --keep-nodes= --keep-ways-relations=“boundary=administrative =postal_code postal_code=” >lux-boundaries.osm
java -Xms1000m -Xmx1000m -jar mkgmap.jar --createboundsfile=lux-boundaries.osm

3)In the mkgmap settings you have to set the boundary file now to the new custom bounds.
4) In the line styles from mkgmap you can now manipulate the streets within those landuse=residential areas:

(highway=unclassified | highway=residential) & mkgmap:admin_level11=residential_area { set highway=cycleway }

I have set it to cycleway (red lines) to see if it works:

Step 1) must be automated with a script if you want to manipulate larger areas but I dont know how (osmosis?). Also if admin_level=11 is already in use and needed you must use another locator tag (postal code?).

Bernhard_Hiller · 2012-03-08T16:56:42.000Z

That’s an interesting idea, but still too complicated. Such a query could be useful for routing related data, e.g. setting max_speed to 50 when roads are in villages/towns.

ligfietser · 2012-03-09T09:13:33.000Z

Is there a tool to add a collection of polygons (in this case landuse=residential) into one relation?
Tried it with JOSM with a filtered extract (just those polygons) but this makes still no sense because its too much data.

ajoessen · 2012-03-09T09:58:16.000Z

ligfietser:

Is there a tool to add a collection of polygons (in this case landuse=residential) into one relation?
Tried it with JOSM with a filtered extract (just those polygons) but this makes still no sense because its too much data.

Yes, you named the problem: too much data. The database has to keep track of the changes of every relation member. Large relations slow down the databases for rendering, uploading and downloading.

So it is better to leave the landuses separately. Or use a common tag and search/filter for that.

Greetings,
ajoessen