PTNA: news for Public Transport Network Analysis

NeatNit · February 17, 2025, 2:06am

I see. In that case, I should update the import script. Right now it tries to make the smallest unique key, but it should probably bring in all the information it can get and let PTNA figure out what information to ignore. That way, that extra info can be a hint for mappers.

Should be a quick fix, but it’s sleepy time now… Or it was 4 hours ago.

Edit: done.

ToniE:

Need to stop now: the bug in the parser keeps me busy and frustrated
while ( $text =~ m/xxx/g ) { }
while ( $text =~ m/yyy/g ) { }
does ignore first ‘yyy’ in the $text = “yyy xxx yyy” because searching in the 2nd while-loop starts after ‘xxx’ and I did not yet figure out how to reset the start-position back to 0;
I like Perl and Regex, but this is driving me nuts.
There is certainly an easy solution for 2 while-loops but for 8 while-loops with very complex regex?

Damn, I kinda wish I didn’t find that bug now. Sorry! If I had any experience with Perl I might have tried to figure it out with you, but alas, I don’t… The best I can do is wish you luck.

ToniE · February 17, 2025, 10:27pm

Some few hours ago I released a bug fix reading the CSV-Data from the OSM-Wiki.

@NeatNit thanks for pointing to that and also thanks for pointing to Perl’s “Text::CSV_XS” module which now does the magic parsing.

I ran local tests over all existing analysis configurations before releasing it.
The tests uncovered many bugs in the OSM-Wiki CSV data which have been tolerated by the old code.

Example:

219;bus;This bus is called "South Side Shuttle";North;South;Operator

where the string >> This bus is called "South Side Shuttle" << should have been enclosed in double quotes and the double quotes in the text should have been “escaped” with double quotes.

Correct version:

219;bus;"This bus is called ""South Side Shuttle""";North;South;Operator

I fixed the CSV data of the affected 19 OSM-Wiki pages except for DE-BW-RVF which I’m going to fix now.

The module “Text::CSV_XS” reports parsing errors like these:

‘EIF - Loose unescaped quote’
‘EIQ - QUO character not allowed’.

NeatNit · February 18, 2025, 10:33pm

It was an uphill battle, but I’ve done it: the GTFS->CSV importer in Israel now also imports train routes. Please get the latest version when you get it automated: NeatNit/ptna-gtfs-import - Codeberg.org

CSV output is now more robust, to pair with the improved parsing on your end

I renamed the scripts: israelGtfsRoutesInShape.py to analyze the GTFS data and create routes.json, and ptnaFillCsvData.py to fill this data into CSV. Naming is hard. I think these script names are fine. The first script is very much tailor-made for Israel, the second one is designed to work for any future imports from different sources without having to modify it.

I’m pretty confident there’s nothing left to improve now in either one of those, so these are probably their final versions!

I guess the only thing left now is better documentation. But I want to see it working (automated) first.

Is there any other specific GTFS feed that you think should be imported in the same manner? I could take a look. Keep that momentum going.

NeatNit · February 18, 2025, 10:57pm

You know, I’ve been tinkering with PTNA a lot but I don’t actually use it much. In fact, I never mapped a route!

So I tried today. The button in PTNA that makes JOSM download all the stops for a trip, is amazing. I found an old, badly mapped route and tried to fix it. But when I try to compare GTFS and OSM route, I get this error:

Some cells may not show valid data, there was missing data.

‘There are no OSM-platforms’

What causes this error?

ToniE · February 18, 2025, 11:46pm

In short: GTFS stops are equivalent to PTv2’s platforms.

If a route relation does not have platform members, PTNA cannot compare.
OSM / GTFS comparison is based platforms versus stops.

skyper · February 19, 2025, 3:40pm

Well I messed it up. Anyway, would it be possible to hide certain OSM-routes (columns), then?

ToniE · February 19, 2025, 4:05pm

Shall I hide columns (OSM route relations) which show empty score fields?

Those route relations are not complete in terms of PTNA’s (my) understanding of PTv2 - platforms are missing and should be added.

PTv2 defines for stops and platforms: “recommended if available”.
PTv2 defines for ways: “mandatory”.

For GTFS it is the other way round: stops (besides the road = like a platform) are mandatory, shapes are optional.

If PTNA wants to compare GTFS with OSM, the OSM relations need to have platforms.

ToniE · February 19, 2025, 4:24pm

Thanks a lot for your work.

I’ll take them ‘as is’ and store them

israelGtfsRoutesInShape.py + create-venv.sh in the ‘bin’ folder of the gtfs repo
ptnaFillCsvData.py in the ‘bin’ folder of the ptna repo

There are similar GTFS feeds (single *.zip for the whole country) for NL, NO, LU, and CH: let’s see whether we can reuse it. LU is quite small, would be the same area for the 5 analysis - one catalogue only? Filters (@operator=TICE …) need to be applied.

NeatNit · February 21, 2025, 12:20pm

Where is LU’s feed exactly? I couldn’t find it in either PTNA or the OSM wiki.

I might take a look this weekend or next week. I think if there is a “typical” presentation for GTFS data (like one route_id = one OSM route_master), then code for that would be reusable for a lot of different feeds.

ToniE · February 21, 2025, 2:42pm

Oh yeah, my bad re. LU. There is not GTFS, but i can sesrch for that.

NeatNit · February 21, 2025, 3:54pm

Would be best if you just point out a GTFS feed that you use which is known to use the best practices. I’ll write code for that, which can be reused for lots of feeds, and maybe modified slightly as needed for specific quirks.

I think much of the code for the Israeli conversion is not reusable because of our feed’s many weirdnessed.

ToniE · February 21, 2025, 4:48pm

@NeatNit I’ll check and also ask for volunteers

@skyper @Patchi @mga_geo @AgusQui @bergaufsee @Nielkrokodil @mcliquid @gnlpfth @adamos

Anyone interested in being alpha/beta tester for automated update of OSM Wiki CSV data from GTFS sources?

I see

DE-BY-MVV = regional buses around Munich (‘operator’ is of no use though)
DE-HH-HVV = Hamburg

both as huge sources, there should be smaller ones, I’ll check.

mcliquid · February 21, 2025, 6:08pm

I’d love to! I am currently checking DE-BW-bodo and AT-VVV. I’ve updated DE-BW-bodo some days ago and saw difficulties in the GTFS generated file, because it is not “sorted” or only alphabetically sorted and then the individual sections for the different bus regions are missing later, right?
I would find that difficult to read, because these groupings of lines form regional clusters that you can focus on. Example:

ToniE · February 21, 2025, 6:51pm

Yep, I’ll check them whether they allow an easy adaption.

ToniE · February 21, 2025, 11:02pm

Right: Sorted by by number if the first character is a digit, all others sorted alphabetically.
If the geographical regions have distinct number ranges, the approach of @NeatNit can help a lot.

See the regional busses of DE-BY-MVV 2xx, 3xx, … 9xx, They were more or less assigned to counties until, last year, new counties came into the game. We (I ?) arranged the report according to the counties then but we came back to arranging them by number ranges, because looking for bus 285 (e.g.) was tedious in the CSV data and the report.

ToniE · February 21, 2025, 11:48pm

Seems that we can take them as blue-print for this, (@NeatNit: ) without using ---shape
They seem to have all traps and variations

NeatNit · February 22, 2025, 12:05am

So, just to make sure I got this right:

all route_ids in the data
one route_id = one CSV line
operator = agency_name
from, to = ???
gtfs_feed = DE-BW-bodo? AT-VVV?
comment = ??? (you can put anything here that can be determined programmatically, for Israel I put a link to a website that shows statistics for the route)
Any other hidden values to add that will be useful for splitting the output into different headlines, and can’t be filtered with the existing properties using regex? For example for Israel I want to add a city field that will be a comma-delimited, alphabetized list of all cities the route stops in. That way, @city=חיפה will show all routes that stop only in Haifa, and @city~\bחיפה\b will show all routes that stop in Haifa even if they stop in other cities too. So any such useful fields that could be determined from the GTFS data are possible. Of course, this can all be added later too. I didn’t implement city for Israel yet. I think it will be very useful for splitting the analysis pages into manageable bits.

And of course, where do I download the GTFS data from to work with?

Edit: I also recall seeing somewhere in your GitHub a file that lists all GTFS route types and the OSM route types they correspond to. Can you find that file for me?

ToniE · February 22, 2025, 12:25am

For DE-BW-bodo it’s quite easy: from NVBW, the company (?) owned by the Ministry of Transportation in Baden-Württemberg, Germany (one of the 16 states in DE)

https://www.nvbw.de/fileadmin/user_upload/service/open_data/fahrplandaten_mit_liniennetz/bodo.zip

For AT-VVV you need an account, so let’s skip this for today

Confirmed

all route_ids in the data
one route_id = one CSV line
operator =agency_name
ref = route_short_name
from, to = ??? - not relevant here?
gtfs_feed = DE-BW-bodo
comment = route_long_name

This one, and the following lines?

github.com/osm-ToniE/ptna-www

script/gtfs.php

master


      
              //OffsetPosition, decimal degrees
              $bbox['top']    = $lat + $dLat * 180/pi();
              $bbox['right']  = $lon + $dLon * 180/pi();
              $bbox['bottom'] = $lat - $dLat * 180/pi();
              $bbox['left']   = $lon - $dLon * 180/pi() ;
          
              return $bbox;
          }
          
          
          function RouteType2String( $rt ) {
          
              $route_type_to_string["0"]   = 'Tram, Streetcar, Light rail';
              $route_type_to_string["1"]   = 'Subway, Metro';
              $route_type_to_string["2"]   = 'Rail';
              $route_type_to_string["3"]   = 'Bus';
              $route_type_to_string["4"]   = 'Ferry';
              $route_type_to_string["5"]   = 'Cable tram';
              $route_type_to_string["6"]   = 'Aerialway';
              $route_type_to_string["7"]   = 'Funicular';
              $route_type_to_string["8"]   = 'undefined GTFS route_type: "8"';

NeatNit · February 22, 2025, 12:33am

Great!

Do you mean ref is unique? In that case, great.

Yup, that’s what I saw! Together with the next function, they convert to an OSM route. I guess I should port these two functions wholesale to Python.

Maybe afterwards, when the code is running on the server, we can switch from processing raw GTFS files to fetching the data from SQL. Should be much faster, and benefit from any post-processing you already do to the data, including (presumably) conversion to OSM route type, making the ported function unneeded.

ToniE · February 22, 2025, 12:42am

No, but ‘operator’ should be different in most cases and that is then sufficient.

Argh! Bus 1 appears twice with same operator in different cities.
The first work in route_long_name translates to “citybus”/“localbus”, does not help as ‘from’ though.
Ideally, from and to can be derived from the 1st and last stop_name of the trips? But this could result in a long, long list of names in both strings from and to.
I have no idea.