[SOLVED] generate_tiles error after running normal for days

TomasC · 2010-01-25T10:37:03.000Z

Hello all,

I was generating a map of the Benelux with Mapnik 0.7.0 with data from Postgresql with PostGIS 8.4 using four threads (which is the default), zoom levels 6 up to and including 18. I started Thursday and more than 30 hours later it was still churning along without a problem.

Now it’s monday, and I find that somewhere along the weekend the generate_tiles.py script crashed. I posted the output of the script and any other information I could think of below.

Anyone any ideas? Even if you don’t know what happened, do you know of a way I can restart the process at the point it crashed (i.e. on zoom level 18, tile map 133176)?

Thanks in advance,

Tomas

This is the output:

[...]
Benelux : 18 133176 86273   Empty Tile
Benelux : 18 133176 86274   Empty Tile
Benelux : 18 133176 86275   Empty Tile
Benelux : 18 133176 86276   Empty Tile
Benelux : 18 133176 86277   Empty Tile
Benelux : 18 133176 86278   Empty Tile
Benelux : 18 133176 86279   Empty Tile
Benelux : 18 133176 86280   Empty Tile
Benelux : 18 133176 86282   Empty Tile
Benelux : 18 133176 86281   Empty Tile
Benelux : 18 133176 86283   Empty Tile
Exception in thread Thread-7:         
Traceback (most recent call last):    
  File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()                                                          
  File "/usr/lib/python2.6/threading.py", line 477, in run              
    self.__target(*self.__args, **self.__kwargs)                        
  File "./generate_tiles.py", line 106, in loop                         
    self.render_tile(tile_uri, x, y, z)                                 
  File "./generate_tiles.py", line 89, in render_tile                   
    im.save(tile_uri, 'png256')                                         
RuntimeError: Could not write file to ../www/images/tiles/18/133176/86284.png

Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()                                                          
  File "/usr/lib/python2.6/threading.py", line 477, in run              
    self.__target(*self.__args, **self.__kwargs)                        
  File "./generate_tiles.py", line 106, in loop                         
    self.render_tile(tile_uri, x, y, z)                                 
  File "./generate_tiles.py", line 89, in render_tile                   
    im.save(tile_uri, 'png256')                                         
RuntimeError: Could not write file to ../www/images/tiles/18/133176/86287.png

Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()                                                          
  File "/usr/lib/python2.6/threading.py", line 477, in run              
    self.__target(*self.__args, **self.__kwargs)                        
  File "./generate_tiles.py", line 106, in loop                         
    self.render_tile(tile_uri, x, y, z)                                 
  File "./generate_tiles.py", line 89, in render_tile                   
    im.save(tile_uri, 'png256')                                         
RuntimeError: Could not write file to ../www/images/tiles/18/133176/86285.png

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "./generate_tiles.py", line 106, in loop
    self.render_tile(tile_uri, x, y, z)
  File "./generate_tiles.py", line 89, in render_tile
    im.save(tile_uri, 'png256')
RuntimeError: Could not write file to ../www/images/tiles/18/133176/86286.png

As you can see, all four threads crashed on subsequent tiles. I do not know why the script suddenly could not write the tiles to file anymore. Things I checked:

Disk space:

~/slippy_map/www/images/tiles> df -h
Bestandssysteem       Grtte Gebr Besch Geb% Aangekoppeld op
/dev/sda6              20G  6,7G   13G  36% /
udev                  2,0G  260K  2,0G   1% /dev
/dev/sda7             136G   40G   90G  31% /home
/dev/sda1             301G  106G  196G  35% /windows/C

Ability to write a file to the exact file location it crashed on:

~/slippy_map/www/images/tiles> dd if=/dev/zero of=../www/images/tiles/18/133176/86286.png bs=1024k count=1
1+0 records gelezen
1+0 records geschreven
1048576 bytes (1,0 MB) gekopieerd, 0,00139983 s, 749 MB/s
~/slippy_map/www/images/tiles> stat ../www/images/tiles/18/133176/86286.png
  File: ‘../www/images/tiles/18/133176/86286.png’
  Size: 1048576         Blocks: 2048       IO Block: 4096   normaal bestand
Device: 807h/2055d      Inode: 2097345     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   tomas)   Gid: (  100/   users)
Access: 2010-01-25 11:26:00.026286710 +0100
Modify: 2010-01-25 11:26:00.027292076 +0100
Change: 2010-01-25 11:26:00.027292076 +0100

dmesg output contains nothing relevant, as far as I can see.

TomasC · 2010-01-25T18:54:23.000Z

The situation has changed:

~> LC_ALL="en" touch test
touch: cannot touch `test': No space left on device

A proper file system check (e2fsck -f -D -C 0 /dev/disk/by-id/ata-WDC_WD5000AAKS-60A7B2_WD-WCASY3646118-part7) fixes the problem temporarily, meaning that I can then generate a few files, after which the problem pops up again.

I believe this is an ext4 file system problem.