Routing website using Gosmore routing engine

Lambertus · 2012-11-30T08:16:04.000Z

I’ve read a bit about Varnish and it’s mostly used to cache (semi-)static requests from users. Because most (>99.9%) route requests are unique I don’t see any added value for a reverse proxy. Or do I see this wrong?

ict4schools · 2012-11-30T09:31:20.000Z

No you are right. Maybe Varnish isn’t that helpfull after all. It’s only helpful for same requests yes.

The advantage you can get from running PHP in FPM mode + NginX is not the speed per request, but on how much resources per request your server uses and therefor how much requests per second the server can handle. Apache with running PHP as mod_php is rather resource inefficient.

Indeed running gosmore as binary each time won’t help either with the speed of the request.

I will try to install the netherlands part, internal at our side, because we use only the netherlands and I noticed we were spamming your servers. (we are the 93.186.184.x). So by that I will also take a look on how it can be optimized, because I think it could be done. Although I’ve already taken a look at the source of gosmore, but that’s a serious mess.

From our point of view, caching is possible. Because we calculate distances for many of the same routes. We use it as autocomplete to calculate distances between many same addresses. So I’ll build a local cache on our part so that your servers won’t get that much of requests from our side. And will use our server as first API. If that doesn’t work yournavigation will than be a fallback.

But I can understand that at your side, caching is not an option. What you can do however is add a robots.txt to disable spiders from crawling your API. That will remove some requests from happening.

I will email you about the info on installing the server with gosmore with optimalisations.

Lambertus · 2012-11-30T10:32:34.000Z

Thanks for your thoughts.

I will try to install the netherlands part, internal at our side, because we use only the netherlands and I noticed we were spamming your servers. (we are the 93.186.184.x).

I searched the logs for the last few days (including the day which saw 350.000 requests) for your IP address, but there is no match. You’re welcome to use the API service, just remember to add an email address or app name to the request so that I can contact you when excessive use is detected, see the API documentation in the OSM wiki. Ofcourse you may setup your own server and I offer to assist where possible.

The advantage you can get from running PHP in FPM mode + NginX is not the speed per request, but on how much resources per request your server uses and therefor how much requests per second the server can handle. Apache with running PHP as mod_php is rather resource inefficient.

Ok, I understand. Apache/PHP isn’t optimal but, considering that I’m not a trained linux server admin, I’m glad to have Apache/PHP running well. The API frontend server has 2GB ram and 2 cores which are doing nothing mostly (relaying requests to the API backend server(s)). The API backend servers are 16GB ram 4-8 core machines and Gosmore is by far the most memory/cpu hungry application on those servers. So my primary question remains: how much does a more complex chain really help?

Regarding Gosmore, it indeed requires some extra time to startup on each request compared to an already running service, but there is a special feature where an external tool keeps the routing database locked and shared in RAM (I don’t think it’s in SVN already). The gosmore instances immediately have the entire routing database available. So I think that the startup overhead per route request is quite low already.

The remark about a robots.txt is interesting I do see some crawler activity, although not significant. But a robots.txt is simple to add

Lambertus · 2012-12-24T14:38:14.000Z

Lambertus:

I’ve read a bit about Varnish and it’s mostly used to cache (semi-)static requests from users. Because most (>99.9%) route requests are unique I don’t see any added value for a reverse proxy. Or do I see this wrong?

I must apologize to ict4schools because my assertion was incorrect. Yesterday I implemented a very simple cache using tempfs on the frontend webserver and the logs show that 30-40% of the requests are cache-hits. Apparently some applications and/or users produce multiple identical requests within a relatively short time? I don’t know how it’s possible that there are so many non-unique route requests, but anyway, I’m very happy with the results.

The first version of the cache held about 1000 requests, but the final (current) version simply discards requests that haven’t been used for more then 1 hour. The effects are clearly visible on the route backend server. Cache size usage for 1 hour (about 4000 routes) is less then 50 MB currently and varies slightly during the day.

ict4schools · 2013-01-02T08:00:35.000Z

Lambertus:

Lambertus:

I’ve read a bit about Varnish and it’s mostly used to cache (semi-)static requests from users. Because most (>99.9%) route requests are unique I don’t see any added value for a reverse proxy. Or do I see this wrong?

I must apologize to ict4schools because my assertion was incorrect. Yesterday I implemented a very simple cache using tempfs on the frontend webserver and the logs show that 30-40% of the requests are cache-hits. Apparently some applications and/or users produce multiple identical requests within a relatively short time? I don’t know how it’s possible that there are so many non-unique route requests, but anyway, I’m very happy with the results.

The first version of the cache held about 1000 requests, but the final (current) version simply discards requests that haven’t been used for more then 1 hour. The effects are clearly visible on the route backend server. Cache size usage for 1 hour (about 4000 routes) is less then 50 MB currently and varies slightly during the day.

No problem. I admire your humility.

I guess it’s because of the implementation of the applications requesting the routes. I guess the route won’t be cached in the application and will be requested again if the information is needed again.

crimean_us · 2013-02-17T11:34:30.000Z

Lambertus:

I’ve put together a website showing off the combined routing capabilities of Gosmore, the Namefinder indexer, the Route altitude profiles generator and OpenLayers slippy map framework. It’s purpose is the combine the current off-the-shelf 3rd party software that is developed for OpenStreetMap into a usable route planning website.

Excellent work!
I have compared yournavigation website results with another rooting services. I think it gives the best Foot/pedestrian/Bicycle results in Crimea(Ukraine) region. So I want add this functionality to my Crimean tourist guide website. I have not PHP experience. I prefer Java back-end and Java-script front-end (GWT, ZK, or another Java-Javascript framework). I try to get results by easiest for me way. But I can’t see easy to use Java-script examples like competitors have:
MQA Optimized Routing examples
google maps Optimized Directions
Can you give me close easy examples how to use yournavigation API easiest way on my site or another recommendations?

crimean_us · 2013-02-18T13:39:21.000Z

Hello Lambertus,
I admired by yournavigation website service!
But I can’t use any API revisions remote access by javascript. I try to use www\api\dev\yours.js I see differences trunk and http://www.yournavigation.org/api/dev/yours.js versions. I have tried different urls and versions in yours.js:

//Have tried var apiUrl = “http://www.yournavigation.org/api/dev/”;
//Have tried var apiUrl = “//www.yournavigation.org/api/dev/”;
//Have tried var apiUrl = “//yournavigation.org/api/dev/”;
var apiUrl = “http://www.yournavigation.org/api/1.0/”;
…
//Have tried var url = apiUrl + ‘route.php?’ + this.search;
var url = apiUrl + ‘gosmore.php?format=kml&’ + this.search;
var self = this;
$.get(url, {}, function(xml) {
self.parseKML(xml);
}, “xml”);
/*Have tried
this.request = $.get(url, {}, function(xml) {
self.request = undefined;
self.parseKML(xml);
}, “xml”).error(function(jqXHR, textStatus, errorThrown) {
self.route._segmentError(self, textStatus + ": " + errorThrown);
});
*/
But I have got just error message:
XMLHttpRequest cannot load http://www.yournavigation.org/api/1.0/gosmore.php?format=kml&flat=44.952139…at=44.61145030000031&tlon=33.52241819999991&v=motorcar&fast=1&layer=mapnik. Origin null is not allowed by Access-Control-Allow-Origin.
What should I do to parse remote API responses in java-script from my local computer and in future from my http://crimean.us site subdomain?
Thank you,
Konstantin.

Lambertus · 2013-02-19T19:59:45.000Z

Hello Konstantin, your browser will block cross site scripting calls because of possible security issues. Therefore you’ll need to send the request to yournavigation.org via proxy script on your own server.

It would look something like this:
Var apiurl = “proxy.php?url=http://www.yournavigation.org/api/dev/”

There is an example proxy in the YOURS repository named ‘transport.php’.

RouteXL · 2013-03-12T12:39:46.000Z

I’ve been rebuilding our Gosmore maps every month or so, but the latest updates didn’t work out. I’ve build the program from svn and downloaded a fresh planet. The rebuild doesn’t end in errors and delivers pak’s, but these are much smaller than normal:


11473176 Mar 10 07:09 america.pak
45057416 Mar 10 04:56 eurasia.pak

It may very well be an error in our setup, but before I go into low-level debugging: is anyone else having trouble rebuilding Gosmore .pak’s lately?

stephan75 · 2013-03-12T16:40:38.000Z

hello pmars,

may that be related to http://wiki.openstreetmap.org/wiki/64-bit_Identifiers ??

Lambertus · 2013-03-12T18:11:53.000Z

It’s very possible that stephan75 is correct. Pmars, do you perform the rebuild on a 32bit or 64bit OS?

RouteXL · 2013-03-13T07:40:23.000Z

@Lambertus I’m on 64-bit (Ubuntu 12.10)

@stephan75 Great suggestion. Checked my toolchain and most probably I need to update Osmosis, I’m currently running version 0.40.1. Will do so and report back to let you guys know.

Thanks!

RouteXL · 2013-03-15T11:21:58.000Z

Well, updating Osmosis wasn’t enough. it seems that the option “idTrackerType=Bitset” in the update script is not working. Without it, Osmosis uses dynamic id-trackers and the resulting file sizes seem to explode. As a result the machine was in swapping madness this morning and I had to reboot it.

I will do another run over the weekend allowing for more runtime but if anyone found a solution I am very interested.

BTW: @Lambertus, how’s the YOURS website routing maps update? Can we learn from that?

RouteXL · 2013-03-18T10:26:11.000Z

My toolchain ran succesfully over the weekend. I had changed the Osmosis idTrackerType to Dynamic and added some filters too:

idTrackerType=Dynamic --tf accept-ways highway=* --used-node

The resulting *.pak files have “normal” sizes again:

5989025172 Mar 18 01:43 america.pak
9802347044 Mar 17 10:13 eurasia.pak

Testing some routes went fine. So I guess my problem is solved. Thanks for you help.

Lambertus · 2013-03-18T10:49:29.000Z

Hi pmars,

I just read your first post and wanted to say “I haven’t ran an update because I’m afraid it breaks the routing” but you solved it in your second post. Great! I’m really glad it wasn’t a Gosmore problem.

As an alternative to Osmosis you can use osmupdate to keep a local planet up to date. It’s a bit faster and uses less resources compared to Osmosis.
As an second alternative you can use osmconvert to extract the bounding boxes for Eurasia and America’s. Same as osmupdate, osmconvert is a bit faster and uses less resources compared to Osmosis.

On sarel.openstreetmap.org (which only serves Eurasia) I’m using the following update script:

wget -q -O - http://planet.openstreetmap.org/pbf/planet-latest.osm.pbf | buffer -m 8M | ~/osmconvert - -b=-32,-89,180,89 --out-osm | buffer -m 8M | ./gosmore rebuild
mv ./gosmore.pak eurasia/

It’s really quick, the buffers are used to spread the load over multiple cores and ensures that gosmore can fully utilize the core on which it runs (gosmore rebuild is single-threaded)

RouteXL · 2013-03-18T13:00:41.000Z

Aargh, komt ie nu pas mee…

** My apologies for using Dutch. It says: “Thank you. What a great suggestion!” **

Lambertus · 2013-04-10T20:48:27.000Z

So there are currently still only two servers available, one for the America’s and one for the rest of the world (Eurasia). And the number of users are increasing. This month I’ve seen days with over 300.000 route requests. What a marvellous feat for a routing application (Gosmore) developed for mobile devices!

But as the load on the Eurasia server is increasing, and the unfortunate habit of Gosmore to run for long periods and consume huge amounts of RAM for long routes, there are unfortunately periods where the server starts to drop requests. To combat this while waiting for extra servers to become available a simple load balancing function is implemented that simply redirects requests to the American server when the Eurasia server is working on two or more route requests. I hope this increases the responsiveness. It will become handy when the new servers come online as well.

voschix · 2013-07-22T13:41:17.000Z

Tried bicycle routing on http://www.yournavigation.org for the first time.

My test was

Padova - Vicenza (in Italy):

bicycle (routes) + fastest: the proposed route uses the motorway A4
bicyle + fastest: the proposed route uses cycle paths and cycle routes
bicylce + shortest: the proposed route uses the state road SR11, which has heavy traffic and very few cycle paths or lanes

(1) is illegal
(2) is a nice route, but by far the slowest and longest
(3) is the fastest and shortest

Volker

(Padova, Italien)

Andre_Engels · 2013-10-06T00:10:34.000Z

I was doing some routing near my parents’ place, and noted something odd. When choosing the ‘cycling-fastest’ option, it provided me with a route that was not only longer but also slower than the ‘cycling-shortest’ gave. Why this behaviour? (if you want to check: It’s in the village of Drouwen, going from the three-way crossing Hoofdstraat-Lemenweg-Stobbenweg in the east of the village to the crossing Lemenweg-Gasselterstraat-Drouwenerstraat, further to the (north)east. The shortest route was 1.86 km, taking 307 seconds, the fastest route 2.40 km, taking 615 seconds).

Lambertus · 2013-10-09T21:02:55.000Z

’ Fastest’ is a different algorithm then ‘shortest’, as it uses time penalties to determine the fastest route instead of distance to find the shortest route.

The speeds on different road types are configured and for cars the higher the road class the higher the speed. But I did not want to do that for bicycle routing, I configured cycleways as the fastest road type, then unclassified, tertiary, secondary etc. Therefore, in bicycle mode, the router prefers cycleways and unclassified ways.

Does this explain the behaviour you’re seeing?