How to get countrywide list of streets with name of city

TBOX1231 · April 2, 2018, 5:59pm

Hi Community,

for an online project I need a list of all streets in Germany and their cities.
I downloaded the .pbf Data from https://download.geofabrik.de for each federal state of Germany and tried to extract the data with osmconvert and osmfilter.

With

for f in *.pbf; do
  echo $f
  osmconvert $f | osmfilter - --keep="addr:country= and addr:city= and addr:street=" --drop-version --ignore-dependencies --drop-relations --drop-ways |osmconvert - --csv="@oname @id @lon @lat addr:country addr:city addr:street" > $f.csv && rm x.osm && rm $f
done

I can get a list of all streets, but I don’t know how I can assign them to the corresponding city. I tried for two days, but I seriously don’t know how I can achieve this.
Do I have to download a shape file of all cities and look for the LAT/LNG Data of the streets?

Many Thanks in Advance.

hadw · April 3, 2018, 10:11am

Streets don’t generally have addresses in OSM. Only buildings and some other features do. Also OSM coverage of addresses is much more patchy than its coverage of streets.

I don’t know an easy way of assigning cities to streets. Nominatim seems to attempt this, but I don’t know there is any way of extracting this information in bulk, without working at the source code level. Also, there are quite a few questions on this forum form people complaining that Nominatim identifies the wrong locality.

GerdP · April 3, 2018, 11:30am

In mkgmap [1] this calculation is done to allow address search. It handles several special cases like roads building city boundaries,
roads crossing city boundaries and places where the name in addr:street doesn’t match the name of the highway=* way.

If you don’t care about these special cases it should be rather easy to use mkgmap to create a list of highway=* ways and a city name
that is likely to be correct. If you are interested I can give you details.

Gerd
[1] http://www.mkgmap.org.uk/

TBOX1231 · April 3, 2018, 12:32pm

Thank you both for the fast replies
I will take a look at mkgmap and try to understand how to use it! Maybe I should explain my Idea a little bit more detailed:
I’m working on an autocomplete field which suggest streets + cities in a search form. E.g. you type “Ma” and it should suggest
“Main Street, Wiesbaden”, “Main Street, Hamburg”, “Max Street, Berlin”, “Monster Street, Heidelberg”.
So, it is not that important that the data is 100% correct as the user still can insert the street manually.

I know this can be done with OSM, because some other platforms like pizza.de (I think I saw OSM in the imprint) and http://www.zeit.de/interactive/strassennamen/ use OSM Data, so it should be possible to connect streets to cities.

I will take a look at mkgmap and tell you about my results!

GerdP · April 3, 2018, 1:56pm

Some hints to create a list like this:

Way 31981987 [highway=residential, mkgmap:admin_level10=Schönwalde I, mkgmap:admin_level2=DEU, mkgmap:admin_level4=Mecklenburg-Vorpommern, mkgmap:admin_level6=Landkreis Vorpommern-Greifswald, mkgmap:admin_level8=Greifswald, mkgmap:city=Greifswald, mkgmap:country=DEU, mkgmap:postal_code=17491, mkgmap:postcode=17491, mkgmap:region=Mecklenburg-Vorpommern, mkgmap:residential=yes, name=Joliot-Curie-Straße] 
Way 285741470 [highway=service, mkgmap:admin_level2=DEU, mkgmap:admin_level4=Mecklenburg-Vorpommern, mkgmap:admin_level6=Landkreis Vorpommern-Greifswald, mkgmap:admin_level7=Anklam-Land, mkgmap:admin_level8=Neetzow-Liepen, mkgmap:admin_level9=Neetzow, mkgmap:city=Neetzow-Liepen, mkgmap:country=DEU, mkgmap:postal_code=17391, mkgmap:postcode=17391, mkgmap:region=Mecklenburg-Vorpommern, mkgmap:residential=yes, name=Am Schlosspark, oneway=no]

For mkgmap you need Java 1.8 or later, splitter, mkgmap and bounds.zip
and enough disk space for the list (it will be 1G - 1.5G)

The latter can be downloaded here:
http://www.mkgmap.org.uk/download/mkgmap.html
http://www.mkgmap.org.uk/download/splitter.html

Next, download germany.osm.pbf ,e.g from geofabrik:
http://download.geofabrik.de/europe/germany-latest.osm.pbf

Create a directory, e.g. e:\citylist
Now, unzip mkgmap and splitter in this directory
copy bounds.zip into this directory
make sure that Java Runtime 1.8 or later is installed

change to the directory and execute splitter with a command like this:

java -Xmx4G -jar splitter-r591\splitter.jar --output=pbf --max-nodes=2000000 --write-kml=splitter.kml f:\osm\germany-latest.osm.pbf > spliter.log

This will run a few minutes. In the meantime you can create a so called style.
For your purpose this is very simple. The mkgmap package contains an example style called default.
(citylist\mkgmap-r4143\examples\styles\default )
Create a copy of the directory , name it e.g. style_citylist
Change to the directory of the copy, remove these files :
points, polygons ,options, and relations
Next, open the file lines in an editor and replace the content by these two lines:

include 'inc/address'; 
highway=* & name=* & mkgmap:city=* {echotags ""}

The echotags function tells mkgmap to print one line for each way with highway=* and a name tag to stderr.
When splitter has finished its work you should find several files in directory citylist.
Change to that dir and use this command to produce the list:


cd \citylist
java -Xmx6G -jar mkgmap-r4143\mkgmap.jar --style-file=mkgmap-r4143\examples\styles\style_citylist  --bounds=bounds.zip --max-jobs -c template.args 2> citylist.txt

Now, be patient. I suggest to test this with a smaller input, e.g. the file for bremen.
The list contains a lot more tags than needed. I assume you are able to filter what you need.

edit: list size

TBOX1231 · April 3, 2018, 4:57pm

Great! Thank you for your effort.

I tried to follow your steps, but it looks like sth went wrong

I used “Saarland” as an example file, running these commands (I’m using a Mac, so it’s “/” instead of ""):

$ java -Xmx4G -jar splitter-r591/splitter.jar --output=pbf --max-nodes=2000000 --write-kml=splitter.kml sl.osm > spliter.log
$ java -Xmx6G -jar mkgmap-r4143/mkgmap.jar --style-file=style_citylist --bounds=bounds.zip --max-jobs -c template.args 2> citylist.txt

I copied style_citylist to the main directory and edited the file as you described.

The file citylist.txt now contains:

SCHWERWIEGEND (BoundaryUtil): 63240001.osm.pbf: boundary directory/zip does not exist: bounds.zip
SCHWERWIEGEND (LocationHook): 63240001.osm.pbf: LocationHook is disabled because no bounds files are available. Dir: bounds.zip
SCHWERWIEGEND (LocationHook): 63240002.osm.pbf: Disable LocationHook because bounds directory is unusable. Dir: bounds.zip
Way 48238030 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue Foascht, oneway=yes]
Way 109097054 [highway=pedestrian, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Place Fernand Kons]
Way 109101089 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Quai de la Moselle]
Way 124317416 [fixme=name?, highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Impasse St François]
Way 124317419 [fixme=name?, highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue Enz]
Way 124317420 [fixme=name?, highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue Saint-Cunibert]
Way 124317411 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Route du Vin, ref=N 10]
Way 124317412 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue du Pont, ref=N 2]
Way 124317413 [fixme=name?, highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Um Gréin, oneway=yes]
Way 124317414 [fixme=name?, highway=service, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Place de la Résistance]
Way 124317415 [fixme=name?, highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Vieux Quartier]
Way 124317426 [fixme=name?, highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue des Pêcheurs]
Way 55123140 [access=agricultural, addr:city=Weiskirchen, highway=service, mkgmap:city=Weiskirchen, name=Batschweiler Weg]
Way 291581636 [highway=cycleway, is_in:city=Remich, mkgmap:city=Remich, name=PC 3]
Way 291581622 [highway=cycleway, is_in:city=Remich, mkgmap:city=Remich, name=PC 3]
Way 74059487 [highway=pedestrian, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue de Macher]
Way 74060059 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue Enz, ref=N 2]
Way 542275360 [highway=footway, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Ruelle des Tanneurs]
Way 542275361 [highway=footway, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Ruelle des Tanneurs]
Way 139160842 [highway=footway, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Ruelle des Tanneurs]
Way 139161249 [highway=footway, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=D’Piirtchen]
Way 159103143 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, name=Esplanade, ref=N 10]
Way 60536897 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Quai de la Moselle]
Way 126082616 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Route du Vin, ref=N 10]
Way 35381782 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue du Camping]
Way 74186076 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue Enz, oneway=yes]
Way 139731004 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue Dauvelt]
Way 5016986 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue Dauvelt, oneway=yes]
Way 5016982 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue Neuve]
Way 441250133 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue du Pont, ref=N 2]
Way 401926288 [addr:city=Saarbrücken, addr:street=Dudweilerstraße, highway=bus_stop, mkgmap:city=Saarbrücken, mkgmap:street=Dudweilerstraße, name=Fernbusbahnhof]
Way 169164330 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue de l’Hospice]
Way 25511345 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, name=Route du Vin, ref=N 10]
Way 219561763 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Route du Vin, ref=N 10]
Way 10408451 [highway=secondary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Place du Marché, ref=CR 152C]
Way 10408556 [highway=secondary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Place du Marché, ref=CR 152C]
Way 26141323 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue Saint-Nicolas, oneway=yes]
Way 23002341 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue du Pont, ref=N 2]
Way 153552635 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, name=Rue de Macher]
Way 153552632 [highway=residential, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Rue de Macher, oneway=yes]
Way 318715220 [highway=footway, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=D’Piirtchen]
Way 318715219 [highway=footway, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=D’Piirtchen]
Way 139974524 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Esplanade, ref=N 10]
Way 256893079 [highway=primary, is_in:city=Remich, mkgmap:city=Remich, name=Route du Vin, ref=N 10]
Way 256893074 [highway=secondary, is_in:city=Remich, mkgmap:city=Remich, mkgmap:residential=yes, name=Place du Marché, ref=CR 152C]
Way 438598142 [addr:city=Blieskastel, addr:postcode=66440, addr:street=Auf dem Han, highway=service, mkgmap:city=Blieskastel, mkgmap:postal_code=66440, mkgmap:residential=yes, mkgmap:street=Auf dem Han, name=Auf dem Han]
Way 122505273 [addr:city=Waldmohr/Pfalz, addr:country=DE, addr:postcode=66914, contact:phone=06373/9428, highway=services, mkgmap:city=Waldmohr/Pfalz, mkgmap:country=DEU, mkgmap:phone=06373/9428, mkgmap:postal_code=66914, name=Raststätte Waldmohr]

TBOX1231 · April 3, 2018, 5:06pm

Edit: Sorry, I see that I’m missing the bounds.zip. Trying to generate this now and will tell you my results.

TBOX1231 · April 3, 2018, 5:32pm

Ha! It worked! Thank you so much, you saved me a lot of time!
Pretty sure a lot of other people will find this useful as there seems to be no other tutorial for this task.

GerdP · April 4, 2018, 8:02am

Great

BTW: You can compile the bounds.zip on your own, see
https://wiki.openstreetmap.org/wiki/Mkgmap/help/options#Using_preprocessed_bounds_for_the_address_index

If you plan to compute this list on a daily base you might want to create your own fork of mkgmap and use only the routines
that compute the address details for a given way. Contact me in a pm if you want more details.

Gerd