How to extract data from full history file and convert it to database?

Hi everyone,

What I am trying to do is to extract all road networks in Canada from the full history dump file in all time periods and finally convert them into postgreSQL or Oracle SQL database. I am using a Windows laptop (i7-4700HQ, 12GB ram, 256GB SSD). Which software should I use? How long is it going to take? Will the software crash?

I’ve looked into Osmium, but I am not sure how to install it (and the required libraries) and get it running. Could anyone from the OSM community help me? I will be monitoring this post constantly until the problem is solved…

Thanks,
A struggling geography student

What parts of http://wiki.openstreetmap.org/wiki/Planet.osm/full have you tried already?

Maybe you should start your efforts with a smaller continent extract which contains all the history data.

And you should consider to try your workflow on linux … because most tools are best tested in linux rather than on windows systems.

To add a little bit to what stephan75 said above, I definitely wouldn’t start with the full history file, at least not initially. According to http://planet.openstreetmap.org/planet/full-history/ it’s 73Gb as a bzip2. Down at the bottom of http://wiki.openstreetmap.org/wiki/Planet.osm/full there’s an “extracts” section, and there there are links to previous history extracts. At http://osm.personalwerk.de/full-history-extracts/2014-12-22/continents/ I can see that central-america.osh.pbf is < 400Mb, so I’d probably start there :slight_smile:

There’s a tutorial for a “osm history renderer” over at https://github.com/MaZderMind/osm-history-renderer/blob/master/TUTORIAL.md , and that includes a “loading a database with the data” section which will be presumably be useful to you even if you don’t want a “history animation”. That seems to cover up to Ubuntu 12.10; I’d be tempted to test things out initially using exactly that version running in a virtual machine under Windows. Different versions of some of the tools will be available on later Ubuntu / Debian releases, and if you’re not especially familiar with Linux to start with things could get very confusing.

Note that I haven’t actually done any of the above, and I suspect that people who have might be quite rare in the forum. People who have will probably read the “dev” mailing list https://lists.openstreetmap.org/listinfo/dev , and may be around in either the #osm or #osm-dev IRC channels over on OFTC (see http://wiki.openstreetmap.org/wiki/IRC - but after asking any question you’d need to hang around a while as people won’t necessarily be in a position to answer immediately).

All in all - good luck!