Rendering GPX trails with mapnik

pberck · 2009-04-03T05:35:58.000Z

emj:

The coastlines are rendered from shapefiles, try to look at your shapefile in other tools first and compare in what way they are different. I can confirm that mapnik can render shapefiles fine, since I’ve created my own, I had no problems so it would be interesting to know what you problem was.

I looked a bit closer at the options of gpx2shp - and I found a “edge” option which seems to do what I want!

Tack så mycket!

-peter

emj · 2009-04-03T19:26:37.000Z

np good thin you gt it working. Any pics?

the_winch · 2009-04-03T20:21:58.000Z

I have used mapnik to render my gpx traces, you can see the result here http://lin.pinkbile.com/gpx/
I rendered transparant tiles with a black dot for every track point. Openlayers then overlays my tiles over the osm tiles.

I wrote a simple ruby script to do the gpx → osm conversion. It just made a node tagged with amenity=post_box for every track point in the gpx files.

Then imported the osm file to postgis with a vanilla osm2pgsql. Modified the mapnik stylesheet. Then finally rendered the tiles with generate_tiles.py. I added a line to generate_tiles.py to delete blank tiles.

pberck · 2009-04-04T05:41:39.000Z

emj:

np good thin you gt it working. Any pics?

Here’s one: http://pixelz.se/map_bjare.jpg My tracks rendered in HotPink Converted with gpx2shp, and drawn with a few lines of code in the mapnik python script. (A lot in NV Skåne still needs to be mapped.)

the_winch:

I wrote a simple ruby script to do the gpx → osm conversion. It just made a node tagged with amenity=post_box for every track point in the gpx files.

Nice! Are you willing to share the ruby script?

-peter

emj · 2009-04-04T13:10:27.000Z

pberck:

Nice! Are you willing to share the ruby script?

-peter

Nice looking map, the pink is really stellar. But are you willing to share the commands use to create this?

pberck · 2009-04-04T15:14:42.000Z

emj:

Nice looking map, the pink is really stellar. But are you willing to share the commands use to create this?

Of course

First of all, convert a .gpx file from my 60CSx to a shape file:

gpx2shp -e 20080808.gpx

Then in my python code to draw a map, add some rules:

s = Style()
r = Rule()
r.symbols.append( LineSymbolizer( Color('#FF1493'), 2 ) )
s.rules.append( r )
m.append_style( 'My Style', s )
lyr = Layer('1')
lyr.datasource = Shapefile(file='/home/pberck/GPS/20080808_trk_edg')
lyr.styles.append('My Style')
m.layers.append(lyr)

And render then rest of the map as usual

-peter

springmeyer · 2009-04-04T20:07:15.000Z

Hey guys, if you install Mapnik 0.6.0 (the latest release) you can render directly from the GPX files.

Mapnik can do this through the OGR plugin, so just compile Mapnik with…

$ python scons/scons.py PLUGINS=all

which will enable the ogr and other plugins (only shape,postigis, and raster are default)

Then create a new OGR datasource like:

lyr.datasource = OGR(file=‘/home/pberck/GPS/20080808_trk_edg.gpx’,layer=‘tracks’)

you can get a sense of how to control with gpx layer is used by running the ‘ogrinfo’ command on your gpx file, eg:

$ ogrinfo test_point_line.gpx
Had to open data source read-only.
INFO: Open of test_point_line.gpx' using driver GPX’ successful.
1: waypoints (Point)
2: routes (Line String)
3: tracks (Multi Line String)
4: route_points (Point)
5: track_points (Point)

pberck · 2009-04-05T08:07:15.000Z

springmeyer:

Hey guys, if you install Mapnik 0.6.0 (the latest release) you can render directly from the GPX files.
Mapnik can do this through the OGR plugin, so just compile Mapnik with…
$ python scons/scons.py PLUGINS=all
which will enable the ogr and other plugins (only shape,postigis, and raster are default)
Then create a new OGR datasource like:
lyr.datasource = OGR(file=‘/home/pberck/GPS/20080808_trk_edg.gpx’,layer=‘tracks’)

Now you tell me

This works great!

I don’t seem to have the ogrinfo command though. (I have a mapnik svn checkout).

Thanks!
-peter

(edit) Example of a track rendered with Ogr datasource: http://pixelz.se/mymap_edges.png, as a red dotted line.

emj · 2009-04-06T06:51:51.000Z

pberck:

springmeyer:

Hey guys, if you install Mapnik 0.6.0 (the latest release)

Now you tell me

Hey! it was released after you asked your question…

sly · 2009-04-07T12:16:28.000Z

Even if you found your solution, in case someone is reading this thread later, I propose another solution to draw gpx tracks by inserting them in postgres/postGIS

( The advantage I see Is that you can create more complex queries to draw what you want, draw more tracks faster, …)

Convert all your gpx tracks to shape files
Use shp2pgsql to import them in your database
define a mapnik style+layer with srs=“+proj=latlong +datum=WGS84”

you’r done.

Here was my script to do the whole import (in a directory with 6,000 paragliding tracks in IGC format :

( First you need to prepare the table with -p option of shp2pgsql )

#!/bin/bash

for x in *.igc ; do
FILE=echo $x | sed s/".igc"//
gpsbabel -i igc -f $FILE.igc -o gpx -F $FILE.gpx
gpx2shp $FILE.gpx > /dev/null 2>/dev/null
shp2pgsql -a -g way “$FILE”_trk.shp cfd 2>/dev/null | psql gis_france > /dev/null 2>/dev/null
done

Here is my simple style+layer :

#bd42f6 1 0.5 cfd postgis LOGIN PASSWORD DB (select way from cfd) as cfd -180,-89.99,180,89.99

Here is the result :
http://beta.letuffe.org/?zoom=11&lat=45.75594&lon=6.27156&layers=0000000B00FFFFFFT

pberck · 2009-04-07T14:37:55.000Z

sly:

Even if you found your solution, in case someone is reading this thread later, I propose another solution to draw gpx tracks by inserting them in postgres/postGIS

( The advantage I see Is that you can create more complex queries to draw what you want, draw more tracks faster, …)

Convert all your gpx tracks to shape files

Use shp2pgsql to import them in your database

define a mapnik style+layer with srs=“+proj=latlong +datum=WGS84”

you’r done.

That was actually what I wanted to do initially (almost, I tried to go from gpx via osm to pgsql directly with osm2pgsql but that didn’t work). I shall try your method tonight!

sly:

Here is the result :
http://beta.letuffe.org/?zoom=11&lat=45.75594&lon=6.27156&layers=0000000B00FFFFFFT

Cool!

-peter

pberck · 2009-04-08T12:54:22.000Z

sly:

Convert all your gpx tracks to shape files

Use shp2pgsql to import them in your database

define a mapnik style+layer with srs=“+proj=latlong +datum=WGS84”

Grmbl, can’t get it to work. No errors, just no tracks on the map (probably means it is something simple )

I take a Garmin .gpx track, convert it to a shape, and import it into the DB. Selecting “way” from the created tables has lots of entries in there, so everything appears to be working. Hmmm… how can I debug this…?

-peter

sly · 2009-04-08T13:10:48.000Z

check if there is something in your table after the import
SELECT * FROM your_gpx_table
check postgres logs to see if there isn’t an access denied
confirm there is a “access denied” to your new database/table for the mapnik user you created
remind that you fogot to :
ALTER TABLE your_gpx_table OWNER TO username;

That’s just a guess, but that’s what I forgot and exactly got nothing like you, without errors.
I’m discovering postgres and don’t know why my user doesn’t have access to all tables in the database…

pberck · 2009-04-09T06:01:23.000Z

sly:

check if there is something in your table after the import
SELECT * FROM your_gpx_table

check postgres logs to see if there isn’t an access denied

confirm there is a “access denied” to your new database/table for the mapnik user you created

remind that you fogot to :
ALTER TABLE your_gpx_table OWNER TO username;

That’s just a guess, but that’s what I forgot and exactly got nothing like you, without errors.
I’m discovering postgres and don’t know why my user doesn’t have access to all tables in the database…

There is data in the DB:

gis=# select count(*) from cfd;
 count 
-------
  7059

Hmm, there are errors in the log:

2009-04-08 06:27:33 CEST ERROR:  column "the_geom" does not exist at character 17
2009-04-08 06:27:33 CEST STATEMENT:  SELECT AsBinary("the_geom") AS geom from 
          (select way from cfd) as cfd WHERE "the_geom" && SetSRID('BOX3D(12.545 56.17817209103231, 13.0856.47480622614124)'::box3d,-1)

Does that mean I didn’t prepare the tables in the correct way? As you see, I used the same tablenames as in your example.

sly · 2009-04-09T09:57:02.000Z

pberck:

2009-04-08 06:27:33 CEST ERROR:  column "the_geom" does not exist at character 17
2009-04-08 06:27:33 CEST STATEMENT:  SELECT AsBinary("the_geom") AS geom from 
          (select way from cfd) as cfd WHERE "the_geom" && SetSRID('BOX3D(12.545 56.17817209103231, 13.0856.47480622614124)'::box3d,-1)

Does that mean I didn’t prepare the tables in the correct way? As you see, I used the same tablenames as in your example.

I was wrong but this error might help greatly, when you prepared the DB with -p you didn’t specified the geometry column so “the_geom” (default value with shp2pgsql) was choosen, but the query (select way from cfd) tries to get the column “way” wich was the one I choose (to keep it like the other mapnik requests)

To prepare the table you should use :

shp2pgsql -p -g way one_shp_file.shp | psql your_db
^^^

pberck · 2009-04-09T12:09:49.000Z

sly:

I was wrong but this error might help greatly, when you prepared the DB with -p you didn’t specified the geometry column so “the_geom” (default value with shp2pgsql) was choosen, but the query (select way from cfd) tries to get the column “way” wich was the one I choose (to keep it like the other mapnik requests)

To prepare the table you should use :

shp2pgsql -p -g way one_shp_file.shp | psql your_db
^^^

Hm, I must be misunderstanding something totally If I do:

shp2pgsql -p -g way 20090407_trk_edg cfd | psql gis
shp2pgsql -a -g way 20090407_trk_edg cfd | psql gis

It still gives me the same error:

2009-04-09 14:04:05 CEST ERROR:  column "the_geom" does not exist at character 17
2009-04-09 14:04:05 CEST STATEMENT:  SELECT AsBinary("the_geom") AS geom from 
          (select way from cfd) as cfd WHERE "the_geom" && SetSRID('BOX3D(4.883334414476785 51.303,5.959665585523214 51.97099999999999)'::box3d,-1)

So it still seems to want the “the_geom” column somehow?

sly · 2009-04-09T15:23:54.000Z

Hm, I must be misunderstanding something totally If I do:
shp2pgsql -p -g way 20090407_trk_edg cfd | psql gis
shp2pgsql -a -g way 20090407_trk_edg cfd | psql gis

Looks like it’s perfectly good, but there still is some crap somewhere that prevent it from working. There must be a reference to that unexisting column “the_geom” somewhere.

What’s in your geometry_columns table ?

Mine is :

select * from geometry_columns;
 f_table_catalog | f_table_schema |    f_table_name    | f_geometry_column | coord_dimension |  srid  |      type
-----------------+----------------+--------------------+-------------------+-----------------+--------+-----------------
                 | public         | cfd                | way               |               2 |     -1 | MULTILINESTRING
                 | public         | planet_osm_point   | way               |               2 | 900913 | POINT
                 | public         | planet_osm_line    | way               |               2 | 900913 | LINESTRING
                 | public         | planet_osm_polygon | way               |               2 | 900913 | POLYGON
                 | public         | planet_osm_roads   | way               |               2 | 900913 | LINESTRING

pberck · 2009-04-09T16:03:40.000Z

sly:

What’s in your geometry_columns table ?

This is mine (some experimental entries left):

gis=# select * from geometry_columns;
 f_table_catalog | f_table_schema |    f_table_name    | f_geometry_column | coord_dimension |  srid  |      type       
-----------------+----------------+--------------------+-------------------+-----------------+--------+-----------------
                 | public         | planet_osm_point   | way               |               2 | 900913 | POINT
                 | public         | planet_osm_line    | way               |               2 | 900913 | LINESTRING
                 | public         | planet_osm_polygon | way               |               2 | 900913 | POLYGON
                 | public         | planet_osm_roads   | way               |               2 | 900913 | LINESTRING
                 | public         | way                | way               |               2 |     -1 | MULTILINESTRING
                 | public         | cfd                | the_geom          |               2 |     -1 | MULTILINESTRING
                 | public         | cfd                | way               |               2 |     -1 | MULTILINESTRING

(thanks for all your patience and help by the way!)

-peter

sly · 2009-04-09T20:46:47.000Z

pberck:

sly:

What’s in your geometry_columns table ?

This is mine (some experimental entries left):

gis=# select * from geometry_columns;
 f_table_catalog | f_table_schema |    f_table_name    | f_geometry_column | coord_dimension |  srid  |      type       
-----------------+----------------+--------------------+-------------------+-----------------+--------+-----------------
                 | public         | planet_osm_point   | way               |               2 | 900913 | POINT
                 | public         | planet_osm_line    | way               |               2 | 900913 | LINESTRING
                 | public         | planet_osm_polygon | way               |               2 | 900913 | POLYGON
                 | public         | planet_osm_roads   | way               |               2 | 900913 | LINESTRING
                 | public         | way                | way               |               2 |     -1 | MULTILINESTRING
                 | public         | cfd                | the_geom          |               2 |     -1 | MULTILINESTRING
                 | public         | cfd                | way               |               2 |     -1 | MULTILINESTRING

(thanks for all your patience and help by the way!)

-peter

okay, try to flush all entries related to “cfd” and restart the “prepare table” from shp2pgsql

( there shouldn’t be the “the_geom” geometry_column in that table

pberck · 2009-04-10T05:24:07.000Z

sly:

okay, try to flush all entries related to “cfd” and restart the “prepare table” from shp2pgsql

( there shouldn’t be the “the_geom” geometry_column in that table

That worked! Now I get my track! I guess the old stuff left in the DB was the cause of the error.

Thanks!
-peter