I was unable to unzip the file because it said it was corrupt.
bzip2 -sdk planet-latest.osm.bz2 > osmbzip2.txt 2>&1 &
Numerous times and had the same results when I run bzip2 -t it said the file was corrupted.
I re-downloaded the file and it said the same thing.
I am thinking it is roughly the same spot, kinda hard to tell, just checked the file size periodically.
I read somewhere that large files like this can cause memory segment faults when running bzip2.
I read up on memory management in man bzip2, but it says it does not apply to decompression.
It is using all my memory.
I do not want to keep re-downloading the file, cause I do not want to keep hitting the server, and I a limited on my band width.
Wget’s output said everything was ok both times.
Has anyone else had this problem. The downloaded file size as 51 G both times.
The last time I downloaded the file a couple of years ago, I know I had this problem and had to run the command numerous times to get it to work. Worried that the file is too big, is that possible or is the file corrupted twice in a row.
To use the -c option of wget, you must be sure that the file has not changed in the server. If you continue from a different file the resulting file can have a first piece from the former file, and the last piece from the last file.
I did check the md5 check sum and it tested that the file was ok.
I suspected that the bzip2 was taking all of my resources up on the computer and killing off bzip2 as it is a none essential service.
The only thing I could think to do was to try renice. I set the renice to 19. I honestly did not think it would work, however I am now currently at 352 Gb decompressed. I will let you know if this method works, as I should know at around 9pm mountain time.
352Gb is over twice as far as I have gotten previously, so I must have been right that is was choking out my machine.
The thing I do not understand is why and is it going to happen when I go to run osm2pgsql.
I keep thinking there must be an option I could use on bzip2. Maybe memory management level, but as far as I understand it it does not apply to decompression.
Maybe I could have increased /proc/sys/vm/swappiness.
The bigger question is why did nothing show up in my var/log/syslog or any other logging files.
I am starting to think that maybe this is a bzip2 bug.
I will let you know what happens.