Uk data onto Garmin - problems

brgmx5 · 2008-09-28T17:39:46.000Z

Hi

I’m a newbie - only contributed to OSM for the last 3 months - and this is my first post.

Is it just me, or is it getting much harder lately to get up to date UK map downloads for Garmin?

When I first got involved back in June, there was an updated UK gmapsupp.img file on the /OSM_Map_On_Garmin/Download page about every 2 weeks. But the latest one there now is dated 30 July, 2 months old.

In August, a new page by user Jclark2906 appeared, promising an updated .img download “most weeks”. Only 2 files have ever appeared, dated 9/8/2008 and 17/8/2008.

So I’ve tried to brew my own. In the past I’ve successfully downloaded UK extracts from planet.osm from nick.dev.openstreetmap.org/downloads/planet/ and used mkgmap to create a .img file. But when I try this with the latest uk-080924.osm file, mkgmap just hangs. I have a modest 512MB Win XP PC, BTW.

Maybe uk-080924.osm is corrupt in some way?

Or alternatively I read on the OSM_Map_On_Garmin page that mkgmap won’t process arbitrarily large input files. The documentation unhelpfully doesn’t say how large such arbitrarily large files are, nor does it describe the symptoms of giving it one that’s too big. So I can’t be sure that this is the cause of my problem either. But I am suspicious that this might be the case, and it might explain why the other .img files have stopped appearing, if the folks who generate them are using the mkgmap route too.

If the UK extract has now got too big for mkgmap, are there any plans to fix mkgmap? I found one wishlist for the prog somewhere (can’t find it again) which I don’t recall made any mention of this as a todo. Or split the extracts into 2 files?

There are 2 workarounds suggested. One is to use osmcut, but this is only available as a .c which needs to be compiled and built, and I don’t have the appropriate resources to do this. (There is a java version which is deprecated on the grounds that it doesn’t work and isn’t maintained; I tried anyway, without success, but the only help I could find for the thing was in German and so I probably didn’t understand it too well).

The other workaround is to download tiles and combine them using sendmap20. I’ve done this in the past successfully too. But although tiles updates are promised to be weekly, they weren’t updated last week (the latest files are dated 17-Sep-2008). Hopefully this was just a one-off blip, but if this data source is going to dry up too, then it would seem there is no viable route any longer to get UK map data onto Garmin.

emj · 2008-09-29T08:53:18.000Z

First of all, it’s very hard to do stuff every week. As soon as something more important comes up, and you don’t have any personal gain from doing it, people will stop doing it. cloadmade planet.osm dumps are done regularly but they do it on company money so it’s easier.

It would be cool if you could include links in your post because I don’t know where to find this osmcut tool.
*

splitting .osm files from openstreetmap.*
you can use osmosis.

*2. UK openstreetmap maps for Navigation on Garmin and Navit
*
*3. Corrupt osm file
*I don’t think so but anything is posible.

*4. mkgmap on low memory
*I’m not sure that will be fixed, but you could try to add a ticket to openstreetmap bug database. It might be hard to fix easily.

brgmx5 · 2008-09-29T17:21:44.000Z

emj,

Thanks for your reply.

I do understand that most stuff is done by volunteers, and it’s good that they do the work that they do. I tried hard not to be critical, but just state the position in as neutral a way as I could.

The osmcut tool is mentioned here: http://wiki.openstreetmap.org/index.php/OSM_Map_On_Garmin
I did give the partial URL of this page. (BTW, how do I put hotlinks into these posts?)

The cloudmade stuff looks useful - thanks. I hadn’t found this, and I will check it out. I see it is referenced from the http://wiki.openstreetmap.org/index.php/OSM_Map_On_Garmin/Download page, but I guess the reference to “OSM Error Edition” put me off it - what does this mean?

I’ll look at osmosis too; hadn’t found that either. But to be honest, raw .osm (XML) files are almost beyond my capabilities to process. My internet connection is just ADSL with a 5GB per MONTH cap. The UK .osm file is presently 1.3GB and growing at about 20MB/week - at best I could manage to get one of these a month (provided nothing went wrong with the download). The compressed .bz2 versions where available are better at 100MB currently, but it’s not going to be too long at present growth rates before these become impractical too.

I don’t think that it would be useful to post a bug report on mkgmap without better evidence that it is a file size issue. I’m really only guessing. Since nobody else has commented that they’ve had similar problems, maybe it’s something wrong with my setup (though mkgmap works for me on other (smaller) .osm files). Perhaps nobody else tries to process this stuff on such a pathetically small Windows PC?

extremecarver · 2008-09-29T22:56:42.000Z

Osmcut is available here compiled: http://smash-net.org/openstreetmap/software/osmcut.0.5.jar.gz
Here you can see some other approaches to get the map onto your garmin unit: http://wiki.openstreetmap.org/index.php/OSM_Map_On_Garmin/Cycle_map
http://wiki.openstreetmap.org/index.php/OSM_Map_On_Garmin/Cycle_map/TYP_files (you have to undo the uncommenting of the osmcut lines in the perl text) - rendering is nicer than on standard cyclemap however.

I don’t know why anyone would download uncompressed instead of bz2 ???

extremecarver · 2008-09-29T23:05:13.000Z

Oh and I download my raw mapdata here http://download.geofabrik.de/osm/europe/
it’s updated daily. Rendering for Austria (25MB BZ2) needs about 3-4 minutes on my Pentium M 2.26 Ghz laptop, so I don’t mind. Rendering with mkgmap yourself has the big advantage of being able to choose yourself how the maps will look like on your unit and at which zoom levels a certain element first shows up!
Great Britain is only 96MB BTW right now, Germany has already 212MB.
The resulting gmapsupp.img are around half size of the bz2 download.

brgmx5 · 2008-09-30T19:09:06.000Z

I’ve tried again on a different machine: a Vista PC with 1GB RAM and a 1.73GHz CPU. I’m running mkgmap version 590 with Java version 6 update 7 (build 1.6.0_07-b06).

I used the england.osm.bz2 file from http://download.geofabrik.de/osm/europe/great_britain/ as suggested by extremecarver. This file is 874.3 MB unpacked, so smaller than the 1.2 GB uk-080924.osm file from http://nick.dev.openstreetmap.org/downloads/planet/
that I was using previously. (Why are these files dated only 5 days apart so different in size, with the later one smaller?)

The new machine looked better: the CPU ran at nearly 100% until it crashed after a minute or so, but at least it gave error messages as follows:

C:\Users\brgmx5\Documents\My Garmin\mkgmap>java -Xmx512M -jar mkgmap.jar england.osm
SEVERE (BufferedWriteStrategy): Map is too big and will not work

[message repeated many times]

SEVERE (BlockManager): overflowed directory with max block 202, current=203
Directory overflow. Map will not work

[message repeated bumping current= by 1 until…]

SEVERE (BlockManager): overflowed directory with max block 202, current=265
Directory overflow. Map will not work
C:\Users\brgmx5\Documents\My Garmin\mkgmap>

At the time of the crash it has created a 32MB 63240001.img file

–

I now think that my original PC with 512MB RAM is just too small to do this stuff, and that what I originally interpreted as mkgmap hanging was probably the thing continually thrashing its swap file and getting nowhere.

It would be really helpful if someone could take a quick look at these messages and confirm that this is the input file too big issue. It would also be useful to know what the input file size limit is, if it’s not too complicated a function of operating environment/system configuration/physical memory/swap file size etc.

Could people also quickly say what size .osm files they’ve been able to process with what kind of spec Windows PCs?

extremecarver · 2008-09-30T23:26:00.000Z

I have 2Gb and give 1256MB to mkgmap. It never uses more than around 200MB for Austria. Try with Osmcut. that should work. I don’t know why the size difference exists? Maybe some pre-filtering of unneccessary objects?

brgmx5 · 2008-10-01T16:39:08.000Z

The size diff is because they’re different areas: england is < UK. Doh!

I realised this as soon as I had pressed the submit button.

Lambertus · 2008-10-02T07:35:00.000Z

The solution here is to cut the area (UK) into smaller sections. This is what Garmin does as well. Tools for cutting the planet data into smaller pieces are Osmosis and OSMCut. Feed those chunks to mkgmap and it will produce the map sections which you can send to your GPS using e.g. the sendmap tool.

As you have problems with osmcut, I suggest using Osmosis (Adapt to reflect UK differences):

osmosis 
  --read-xml SloveniaGarmin.osm --tee 4
  --bounding-box left=15 top=46 --write-xml SloveniaGarminSE.osm
  --bounding-box left=15 bottom=46 --write-xml SloveniaGarminNE.osm
  --bounding-box right=15 top=46 --write-xml SloveniaGarminSW.osm
  --bounding-box right=15 bottom=46 --write-xml SloveniaGarminNW.osm

The values of left, right are longitudes and top, bottom are lattitudes. Have a look at informationfreeway.org (in the lower right corner) to find the right lat/lon values.

Then run mkgmap with the resulting .osm files as commandline parameters. Mkgmap should not use excessive amounts of ram for these operations.

brgmx5 · 2008-10-02T17:28:56.000Z

Thanks everyone. I’m happy to report success with osmosis. I managed to chop the england.osm file into 4 (different sized) fragments, largest fragment size = 313KB. (I actually figured this out late last night before Lambertus’ post, using the example he quotes as a model, but it was too late then to stick something up here.)

The good news for me is that both the osmosis step and the subsequent mkgmap step now both run easily on my smaller machine, taking about 5 mins each. This is much more convenient than borrowing the larger machine each time I want to make a map. Also, I don’t like the rendering of the cloudmap “error edition”, which I also tried.

I have to say that I’m a little concerned to discover that the main tool we have for doing this (mkgmap) is designed such that it doesn’t scale. In the context of a continually growing database this doesn’t seem very sensible design. The osmosis command line is very clunky too; mine was 194 characters long even using command abbreviations. There must be some limit on command line length in Windows cmd, don’t know what it is off the top of my head; in DOS it was 127 chars. If in due course more fragments are needed… But this should be easy to fix (eg accept the command line args from a file). Or fragment the fragments in 2 (or more) osmosis steps…

I think it would be useful to update the http://wiki.openstreetmap.org/index.php/OSM_Map_On_Garmin page to reference osmosis explicitly and make the chopping up step clearer, and maybe remove references to osmcut at least for Windows systems. As a newbie I lack the confidence to do this myself (not even sure how to edit wiki pages). I would not have found osmosis without help from this forum.

Two final things: I built my map using the --gmapsupp option on mkgmap, and I copy the gmapsupp.img file to the GPS either using “USB mass storage” mode, or by taking the SD card out and putting into a card reader on my PC.

Lambertus hints that sendmap might send the mkgmap output files directly without needing to combine them - is this correct?
I’ve been taking the SD card out because I’m a bit concerned that the micro-USB port on the GPS (it’s an Etrex legend HCx) doesn’t look very robust and might not survive too much connecting and disconnecting of the cable. Is this a known problem?

Lambertus · 2008-10-02T18:27:52.000Z

brgmx5:

Thanks everyone. I’m happy to report success with osmosis. I managed to chop the england.osm file into 4 (different sized) fragments, largest fragment size = 313KB. (I actually figured this out late last night before Lambertus’ post, using the example he quotes as a model, but it was too late then to stick something up here.)

313KB for one quarter of the UK seems way too small for me. I bet those numbers are wrong

I have to say that I’m a little concerned to discover that the main tool we have for doing this (mkgmap) is designed such that it doesn’t scale. In the context of a continually growing database this doesn’t seem very sensible design. The osmosis command line is very clunky too; mine was 194 characters long even using command abbreviations. There must be some limit on command line length in Windows cmd, don’t know what it is off the top of my head; in DOS it was 127 chars. If in due course more fragments are needed… But this should be easy to fix (eg accept the command line args from a file). Or fragment the fragments in 2 (or more) osmosis steps…

The chopping up has nothing to do with Mkgmap not scaling, it’s just that maptiles cannot be larger then 32MB afaik. Garmin itself does splitting of maps into chunks as well.

Regarding the commandline: We don’t live in the DOS era anymore. Linux takes up to 32k of characters on the commandline and Windows NT+ should be fine too. I suggest you make a script containing all the commands needed to create the images so you only have to build the Osmosis commandline just once. That way it doesn’t matter if it’s difficult to read or not.

I think it would be useful to update the http://wiki.openstreetmap.org/index.php/OSM_Map_On_Garmin page to reference osmosis explicitly and make the chopping up step clearer, and maybe remove references to osmcut at least for Windows systems. As a newbie I lack the confidence to do this myself (not even sure how to edit wiki pages). I would not have found osmosis without help from this forum.

It’s a wiki and you are the newbie who knows what’s missing there, so please add your ideas and tips so it helps other newbies too.

Two final things: I built my map using the --gmapsupp option on mkgmap, and I copy the gmapsupp.img file to the GPS either using “USB mass storage” mode, or by taking the SD card out and putting into a card reader on my PC.

That’s indeed the idea of the gmapsupp option.

Lambertus hints that sendmap might send the mkgmap output files directly without needing to combine them - is this correct?

sendmap can combine images into a gmapsupp file and send it to the GPS immediately. It’s just that you don’t have to put the gmapsupp file on the GPS by hand if you use sendmap.

I’ve been taking the SD card out because I’m a bit concerned that the micro-USB port on the GPS (it’s an Etrex legend HCx) doesn’t look very robust and might not survive too much connecting and disconnecting of the cable. Is this a known problem?

No, I have not heard of any such problem. But putting the SD card into a cardreader helps speeding the filetransfer (especially if the files are big), but could shorten your SD card life because of possible frequent ESD (electrostatic discharge). But I wouldn’t bother about both…

brgmx5 · 2008-10-03T08:57:11.000Z

313KB for one quarter of the UK seems way too small for me. I bet those numbers are wrong

OK, OK, 313MB…

Regarding the commandline: We don’t live in the DOS era anymore. Linux takes up to 32k of characters on the commandline and Windows NT+ should be fine too. I suggest you make a script containing all the commands needed to create the images so you only have to build the Osmosis commandline just once. That way it doesn’t matter if it’s difficult to read or not.

I already built a script… there’s no way I can type in ~200chs straight off without making mistakes. I’ve been poking around a little, and there doesn’t seem to be a simple answer to the limit on WinXP - different tools have different limits - but the smallest I can find is 8K so it probably isn’t a serious issue.

The chopping up has nothing to do with Mkgmap not scaling, it’s just that maptiles cannot be larger then 32MB afaik. Garmin itself does splitting of maps into chunks as well.

I don’t understand the relationship between tiles and mkgmap input file sizes, but that’s probably the advanced course and can wait for another day.

It’s a wiki and you are the newbie who knows what’s missing there, so please add your ideas and tips so it helps other newbies too.

Well I might have a go then, but don’t hold your breath. Is there a practice area anywhere?

No, I have not heard of any such problem. But putting the SD card into a cardreader helps speeding the filetransfer (especially if the files are big), but could shorten your SD card life because of possible frequent ESD (electrostatic discharge). But I wouldn’t bother about both…

SD cards are cheap, GPS units are expensive by comparison. I think I’ll keep taking the card in and out.