GOL tool gets stuck in analyzing stage when building planet

General talk
1

When trying to build a gol file from the latest planet, gol-tool-0.1.8 very quickly progresses through the “Analyzing” stage until it reaches a percentage in the 90s, then ceases any visible progress:

# ./gol build planet planet-latest.osm.pbf --tag-duplicate-nodes=yes
Building planet.gol from planet-latest.osm.pbf using default settings...
Analyzing... 92%

I have this issue on two different machines (both running Ubuntu 22.04 and openjdk 17), although the displayed percentage isn’t identical (92% vs. 97%). At that point, htop doesn’t show meaningful load anymore.

Any ideas on what might be amiss or how I could debug it?

2

Thanks, I’ll have a look at this right away.

Just to confirm, is this the planet file from Nov. 3?

Does progress vary on different runs, or does it always stop at 92% and 97% respectively on the two machines?

In the meantime, could you try the following:

  • Delete the planet-work temporary directory (if any) that may have been left behind by a previous run
  • Run gol build again with option -d (to show debug-level logs)
1 Like
3

Yes, from Nov. 3, although I did obtain it from the mirror at https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-latest.osm.pbf

I’ve seen it reach 92% and 93% on the same machine.

I’ve done that now, although I do not see any additional output on the console.

1 Like
4

I started the download of the planet.

How much RAM do your systems have?
One thing you could try in the meantime is to explicitly increase the heap size in the gol startup script (for the call to java), e.g. -Xmx8g

(In any case, I’ve logged this as a bug, because the Analyzer should always report any out-of-memory conditions, if that is the cause)

1 Like
5

One system is a virtual Hetzner Cloud server with 32GB RAM, the other a root server with 64 GB.

However, I’m now observing different behavior with a new download of the planet file, so I wonder if it might instead be an issue with a corrupt download. The checksum does indeed point in that direction. Please let me check that cause before you put any more time into debugging.

1 Like
6

Ok, a fresh download works just fine. :sweat_smile:

I’m not sure how I managed to get a broken file twice by running wget on two separate machines (connectivity issues…?), but the one I used on the root server is clearly damaged. It’s far too small, for starters.

In case you want to use it for something, I’ve put the broken file here and will leave it there for a while.

Anyway, thank you for your support and – as always – for the awesome tool! :heart:

2 Likes
7

Your file was very useful, I managed to reproduce the bug (race condition in OsmPbfReader that prevented a clean shutdown in case of corrupt input). A fix is already in main, it will become part of Release 0.1.9.

Thanks again for your help & your kind words!

1 Like