Userfriendly Garmin map builder for Windows

General talk
1

Note to readers: this post is primarily intended for people with good knowledge of the Garmin map requirements and tools.

One of my goals with OSM is to provide a userfriendly way for people to create their own Garmin maps. The garmin.openstreetmap.nl website is the first version of this idea but it has drawbacks, amoung which a few big ones:

  • Unable to install more than one map from this same website (without using 3rd party tools)
  • Precompiled tiles don’t allow redically different end-usage (e.g. my generic routing tiles cannot serve as source for the specialist cycle maps from user Ligfietser)
  • A few sponsored servers need to host every personalized map in every flavour.

Now I’m trying to work around these key issues and this is what I have in mind:

In short
The user selects the tiles of interest through an OpenLayers interface similar as now. The user also provides the preferred product ID, mapname, stylesheet, TYP and all the other relevant Garmin specific options. After that the tiles are downloaded from a server in raw OSM format, processed on the user pc into a map and installed.

A bit longer
Because most users are still Windows users I figured that an NSIS installer might do the job good enough (I previously thought about developing a Java application that would do all of the above, but it’s much more work I reckon). I haven’t found a browser plugin for NSIS so the user has to use the pc’s browser to select the tiles and download a metalink and feed it to NSIS, but everything else can be handled in NSIS itself (afaik). NSIS accepts the Garmin map options, a list of tiles to download from one of the servers, proceeds to download the files, then use Mkgmap to process the tiles into the requested map and add the registry keys for MapSource.

The metalink for a map containing two tiles would look something like this:


<?xml version="1.0" encoding="UTF-8" ?>
<metalink version="3.0" xmlns="http://metalinker.org">
   <files>
     <file name="63240004.osm.7z">
	 <verification>
	   <hash type="md5">f6a0e19d81311b8916e0413c551603b2</hash>
	   <hash type="sha1">780c6d79d78f4546c765f2e11b03287bad8cdd1a</hash>
	 </verification>
	 <resources>
	   <url type="http" location="nl" preference="50">http://planetosm.oxilion.nl/garmin/routable/01-04-2012/63240004.osm.7z</url>
	   <url type="http" location="de" preference="50">http://osm.pleiades.uni-wuppertal.de/garmin/routable/01-04-2012/63240004.osm.7z</url>
	 </resources>
     </file>
     <file name="63240032.osm.7z">
	 <verification>
	   <hash type="md5">faacb525068bfd6bdc382263f4a9213c</hash>
	   <hash type="sha1">3ea6b486aba5e7ef1510bde12b30a8a6612c9518</hash>
	 </verification>
	 <resources>
	   <url type="http" location="nl" preference="50">http://planetosm.oxilion.nl/garmin/routable/01-04-2012/63240032.osm.7z</url>
	   <url type="http" location="de" preference="50">http://osm.pleiades.uni-wuppertal.de/garmin/routable/01-04-2012/63240032.osm.7z</url>
	 </resources>
     </file>
   </files>
</metalink>

(In this example the OSM files are compressed with 7z which gives a great compression ratio).

With a bit of careful implementing I think this installer could be used by many map providers (not only mine). In this case, all one has to do is providing a definition of the tiles in OSM format (sha1/md5 sums, tilename, download location) and stylesheet and/or TYP file.

NSIS is able to contain Mkgmap, to detect if Java is installed and even install Java when it’s missing (after asking permission ofcourse).

So, how about the benefits?

  • Using raw OSM source files allows users to generate specialty maps using (and learning to use) one installer.
  • Have many different specialty maps installed simultaneously.
  • Reduce the CPU expensive combining and compressing and require much less storage on the server by offloading these to the end-user.
  • Instant map download and install procedure, instead of having to wait for the server and the email to arrive.

One obvious drawback: I still need to upload the precompiled tiles to the server and create custom maps there for Mac OSX, Linux etc. users because the NSIS installer doesn’t work on non-Windows operating systems. This means I have to upload the precompiled tils in img format as well as the OSM source files.

What do you think?
Before I spend a lot of time on this, I would like to hear what you think of this. Will it work? Did I miss something? Ideas? Tips? Is it a stupid idea?

2

Apart from the osm files, the user will need:

  • a TYP file
  • the corresponding style file (you must not freely combine TYP and style files - that does not work!)
  • the corresponding command line options for mkgmap
    When a user is content with your maps, you could provide him with this extra data. But when someone wants to create a different map or even his own map, he’ll need that data from somewhere else. And I think that that is far more complicated than getting the osm data from some source like geofabrik.
    And for many users - the vast majority who are not computer freaks - will find it too complicated to install and use nsis. Better spoon-feed them with ready made maps.
    When it comes to creating your maps, you could probably reduce the load on your servers a little. But when it comes to different maps, I do not believe that it will work.
3

Thanks Bernhard for having a critical look at the idea. :slight_smile:

You are correct about the data needed and it’s coherence, the TYP file and stylesheet used by Mkgmap are closely related. It will be necessary provide the user a choice out of a few TYP/stylesheet combo’s and download these from the website (or prepackage them in the installer).

I think it will be possible for the user to create their own map style by using the pre-splitted tiles in OSM format, but this would be something for the future. Initially I would focus on providing only a few data- and style sources.

I don’t really understand your problem with using NSIS, perhaps you misunderstand what I have in mind, I don’t know. NSIS is used to create an application (the ‘installer’, but you could also name it a “map manager/builder”) that works like a wizard in which the user specifies it’s preferences, chooses the style and tells the “map manager” where it can find the metalink file. The “map manager” then opens the metalink file, downloads the OSM data tiles specified within the metalink file, installs Mkgmap, executes Mkgmap with the required commands and finally installs the resulting map. The ‘only’ thing the user sees is an application that asks a few questions.

I just discovered that Ligfietser already has something that’s performing some of the steps I mentioned above. I think his NSIS script would be a great place to start from.

I’m only providing one map at the moment, but I would love to be able to also provide a worldwide bicycle map (based upon Ligfietser’s work) a Nautical map and contour versions for all of these. If I rely soly on (sponsored) servers then this will not be possible while proving the same service as I do now. If the map manager application is simple enough, and provides some advantages over the website, then it will be interesting to use for mapusers.

4

Hi Lambertus,
Your idea sounds interesting. But I doubt it will be very user friendly for the end user, a lot of things still in mkgmap can go wrong / have to set up right on the users pc in order getting a decent map. I think maybe this will be more interesting for other mapmakers like me (so I think you will offer something like a semi-fabricate). Do I understand that you will split the planet file in tiles and provide it to the user, so that they can run mkgmap and make a map from it? This will save me a lot of work, I dont have to download the geofabrik tiles anymore, only a selection of your tiles and only have to run mkgmap.

My tool only has a very minimal use of nsis, just to register the map for windows and check if a previous version is installed. The rest I have stripped out. My tool only can make another map from already made img’s which are compatible and use the same style/typ files.

Maybe its a good idea to share your thoughts on the mkgmap mailing list as well or with GPS wizards like Javawa?

5

Mkgmap runs without supervision on my servers and I hardly ever get complaints about map build failures. So: if the map manager application sanitizes the parameters that are fed into Mkgmap then I don’t see a lot of problems there. But I agree that running a complex application like Mkgmap on an end-user pc is asking for (some) troubles (like Java incompatibilities). These can be dealt with and I don’t expect perfect results all the time. Judging by comments on other websites an in emails, the current website is far from perfect either.

Sure, the tiles that I upload to the server have been ‘test compiled’ already (using the generic stylesheet) and therefore are likely to work with most stylesheets. But this is one of my big assumptions that, if proven wrong, undermines the whole idea. And, ofcourse, the map manager idea would be useful for (casual) map providers as well.

So, your tool does some Garmin specific registry work and invokes Mkgmap. That’s infinitely more then the not-yet-existing “map manager” NSIS script… :smiley:

Good idea, I’ll try to write it up in an email one of these days.

Now that I’m reading more into NSIS documentation I do notice that NSIS isn’t really suited for this advanced work. Yes it has easy dialogs, registry functions, compression, XML handling, uninstaller, metalink download capabilities and all that, but it severely lacks in proper programming functions. I mean, using push and pop commands to store variables on a stack? Come on, it’s 2012 not 1981 anymore! It might be user friendly, it certainly is not programmers friendly :rage:

6

I also see some problems:

The user needs java installed, in most cases java 64bit. He needs enough RAM. 8gb is state of the art, but not everyone has such a pc.

I also thought about converting my batch-file to an exe with GUI, but haven’t enough time. I don’t understand the advantage of downloading splitted tiles. I think splitting a extract isn’t a problem at all.

7

Correct. Or a Java runtime must be provided. I have some examples that do this, e.g. Oracle SQL developer (yes, the installed base is huge :frowning: ).

Are you sure? For rendering tiles and creating a gmapsupp.img? I’ll fire up my old 32bit XP laptop tonight to check this. I would be a serious problem if this is correct.

One of my servers has only 3 GB ram and has no problems with creating gmapsupp.img files.

On the contrary:

  1. splitting the OSM data into tiles IS a problem. It’s why I implemented a recursive tile rendering script to create the tiles as large as possible.
  2. the current extracts focus on countries or regions and are not suitable for recombining, so a user has to download the complete planet every time he want’s to create a map that falls outside any one of these extracts.
8

Maybe you can adapt and improve my script a bit to use it for recombining the img’s on the clients desktop?
This doesnt need too much memory (works on my old laptop with XP and 512Mb ram) .
For creating a larger gmapsupp, it needs a little bit more memory (1-1,5 Gb).

9

32bit on windows only provides 1300MB of RAM for a single process (give or take 100). That’s enough if you don’t need an address index. If you want one, it can crash on Germany alone… (needs around 1.1GB I’ld assume).

Mapsource / Mapinstall can create address index without lots of RAM, but only has about 1/10 of the speed.

10

It’s true we need some newer tools for getting the maps. The better our data and our maps get, the more users will try to get maps from us - and mainly more users who just know to click here or there on a web page and expect everything else to run automagically.
From my experience (with not too big maps), really high performing computers are not necessary for generating the maps. But a new user might try to get a map of whole Europe with an index.
The extracts which are available from Geofabrik etc. often do not cover the region I want to get. For generating a map of Bavaria and Czech republic, I’d have to split Europe. When using 2 maps, routing often fails near the border, even when no cross-border routing is required (I have to disable the other map to get routing working again).
I am afraid that I cannot provide a better idea now.

11

Lambertus, maybe these optimizations are only necessary if you cover a hugh area. For normal areas it isn’t really important if you have 100 or 150 Tiles for Germany. On the other hand I think it could be a easy way for consumers, but you wont save any traffic. ( pbf-tile >> img-tile ).

I think the best think for regular maps is to download once the needed extract, cut it with osmosis or osmconvert/osmfilter and then update this file with osmosis/osmupdate.

Ok, I use mkgmap with 5 threads. If you only use one you wont need so much memory but it will take much more time.

12

I’ve combined the Germany tileset into a gmapsupp.img on my old first generation 32 bit Centrino laptop with 512 MB ram. The process completed successfully but after many hours (heavy swapping ofcourse) and memory usage was close to the max JRE setting. The gmapsupp includes an index which, I think, everyone wants. So extremecarver is correct in his assertion that larger areas like Western-Europe is out of reach for 32-bit users.

I’ll park this idea for now, perhaps work on it causally to learn the stuff surrounding Java programming (Eclipse, Java Dialog classes, including and executing other JAR’s, etc.). I’m not going to use NSIS to implement a map manager application now that I’ve studied it more closely.

Thank you all for the valuable input!

13

Well, so if memory usage was close on Germany already without contourlines, with contourlines it would likely tip over, or just wait a few weeks…

Rule of thumb: you need about 10% more memory available than the combined size of all *.img files, if you want to create the map with address index. If without also look at gmaptool instead of mkgmap, it might be faster/easier as it doesn’t require Java JRE.

14

But contour lines dont affect the adress index, or do they? :wink:

Greetings,
ajoessen

15

Yes, they do. Every tile counts…

16

As it happens this morning I just started my windows program to collect tiles from Lambertus site and to generate a garmin map out of them. Then I saw this topic and knew we had discussed this before:
http://forum.openstreetmap.org/viewtopic.php?pid=145643#p145643 That was februari last year.

So at the moment it downloaded the 168 tiles selected (from Ukrain all Russian Federation to Wladivostok). Then sendmap20 was used to combine them to one gmapsupp.img file. Also gmaptool I used for this.

The timeconsuming part is the download. About 7-20 seconds for a tile. Total threequart hour.

Combining to one 983 MB file with gmaptool is fast in 10 seconds. (Sendmap20 7 minutes). Both programs produced a file of 1.031.585.792 bytes.

With gmaptool one can supply a FID, PID and a Map name.

Lambertus: If you want to test my program on your XP machine, be my guest.