Updates on loose end detection

stephankn · 2017-10-04T19:41:30.000Z

You might still remember the service on my machine to detect dangling highway ends and highlight those which have aerial imagery available to continue them.

http://compare.osm-tools.org/loose.html

This service used manually maintained polygons of the Bing coverage.

This is no longer state of the art. While it worked well in the past and also provided the additional benefit of giving ideas on where to derive more features from the imagery, it does not scale.
As of today we have not only Bing imagery, but also imagery from Mapbox Satellite, Esri, and two layers from Digital Globe.

I have not yet found areas without coverage. So the times for imagery polygons are gone.

Still leaves us with a lot of major highways suddenly ending. In the areas I focus on with the osm-tools.org services there are nearly 10.000 of those dangling ends.
If concentrating on Thailand, we speak about 450 loose ends still waiting to be connected.

To make it easier to concentrate on the quick to solve parts, I use computing intelligence to detect which of the imagery providers has high-resolution images available.

This is then used to classify the imagery into grades. The dots with the green circle should have the highest quality imagery available, the ones with the yellow border still good quality alternatives. But even the ones without border should be usable to continue highways.

For all dangling ends there is at least one imagery provider available with zoom 16 imagery. Many of them offer you to choose from multiple providers. This allows to deal with cloud coverage and also makes it possible to select the imagery with the smallest offset.

Speaking about offset: This is something I have to strongly highlight. The imagery differs greatly in the positional accuracy. This also varies by region. So imagery from one provider could be horrible aligned in one area, but the best in another.
Before using imagery, check the positioning accuracy by loading GPS tracks and comparing with the imagery. If the area in question has no tracks, zoom out and check nearby areas. Be careful for visible boundaries of the imagery as this can indicate different capture dates which can come with different offsets.
I made it a habit to always load GPS track and recommend to do the same.

As a test-drive I will only display the data for Thailand for a few days before enabling the full coverage of South-East Asia.

I hope we can get those remaining dangling ends resolved quickly. Just doing a hand full every day will get it resolved in short time.

More details of the tool can be found here:
http://www.osm-tools.org/loose.html

Stephan

PS: is there any use of the outdated Bing boundary polygons? I suggest to remove them if no use exists for them any more.

Russ_McD · 2017-10-05T00:06:13.000Z

A good tool … I use many times.
And I agree, the coverage boundaries have served their cause … but time to delete.
Russ

AlaskaDave · 2017-10-05T07:02:50.000Z

Stephan,

I started working on this but noticed that the tool classifies ways ending in “noexit=yes” as though they were loose. Can you correct that somehow?

Also, agree that the Bing multipolygons can be removed.

Cheers,
Dave

Veiviseren · 2017-10-05T20:21:48.000Z

I used Overpass Turbo to find about 6-7 unconnected tracks in the Nane area. Been riding and verified a few. The rest to follow.
Can not remember the exact search string I used…

stephankn · 2017-10-05T21:21:39.000Z

@Dave: The tool only checks major highways. Just ending sounds quite strange for such a road category. I checked the DB and there are in total 28 nodes flagged as loose end having noexit=yes.

I looked at the first hit: node 475188116
While it is clearly visible that for motor-vehicles the road ends there, can we be certain that not a foot-path continues from the road, probably leading to the waterfall? At least here it looks like a footpath continues. Also steps visible in the background:
https://www.google.com/maps/@14.2381538,101.3441581,3a,78.4y,2.39h,87.56t/data=!3m6!1e1!3m4!1syNGHM7Z3mFXcxEj2QVHi3Q!2e0!7i13312!8i6656?hl=en
Maybe Russ can help out with some details as he mapped the area two years ago.

So this specific node i would not exclude. What I can do is to exclude specific nodes based on their ID if they are clearly false alerts.
Here is the full list of all 28 nodes. I hope someone can have a closer look on them. Truly false alerts can be hidden (if they exist).

475188116,1215735969,1313850592,2161376287,3037992001,3092761029,3370217379,3658426347,4947196478,1694010926,4096487566,648755635,3797952462,2456268344,2589792792,3019238301,3337745160,3658426346,3797951936,3797951934,4843740431,4460372579,4603793688,4734284733,4096487081,3797952463,1079916269,3132334291

@Veiviseren: Thanks to fix some. Unconnected tracks might be also something different to this tool. It does focus specifically on major highways. In the database exist a lot more of “nearly connected” ways and even “islands” of completely disconnected ways not reachable by routing.
You can check out osmose.osm-tools.org which directs you to a large number of suspicious data, classified by various categories. Please try to understand the issue first before starting to work on them. Some might be false alerts or tricky to fix. My server calculates daily updates to nearly all countries in Asia.
The OSM Inspector of Geofabrik highlights routing problems when you activate the routing layer:
https://tools.geofabrik.de/osmi/?view=routing&lon=100.75818&lat=14.17945&zoom=9

Another thing which pops up is highway=construction ending unconnected. I am not certain whether to include or exclude. I saw one added by Paul which not even has the slightest sign of construction started but leading straight through buildings. So maybe for such cases it is better to list them as there might be more broken than just the dangling end.

Veiviseren · 2017-10-06T18:01:29.000Z

Thanks Stephan.
I used Geofabrik for it, just thought I used Overpass. Exported all ‘hits’ as GPX and uploaded to my GPS.

Paul_012 · 2017-10-09T10:42:07.000Z

stephankn, thanks for the update.

stephankn:

Another thing which pops up is highway=construction ending unconnected. I am not certain whether to include or exclude. I saw one added by Paul which not even has the slightest sign of construction started but leading straight through buildings. So maybe for such cases it is better to list them as there might be more broken than just the dangling end.

I guess you’re referring to Motorway 6, which is the only major under construction highway I remember recently adding. Actually I initially tagged it as highway=proposed, but someone changed it to highway=construction. That isn’t incorrect, though. Construction has certainly begun on the ground, in certain segments of the road at least.

Usually I make it a point to leave under-construction roads dangling, since in reality the connections don’t yet exist, and adding them early would involve guesswork, which could very well turn out to be incorrect. This could turn into a problem if someone updates the tags from construction to completed without checking the connections. That’s something the tool would certainly help with.

The way I see it, under-construction highways are a persistently open issue unto themselves, but one that can’t be immediately fixed and closed—they’ll need eventual updates as work progresses. Ideally we’d track them separately, but I don’t think there’re enough highway=construction tagged ways to warrant the work. I’m just a bit worried that someone might try to “fix” the dangling ends and add inaccurate connections, turning the open issue into a falsely resolved one.

stephankn · 2017-10-10T21:00:13.000Z

Hello Paul,

thank you for sharing some thoughts. Based on this I decided it would be better to not list construction/proposed ones to avoid potential bad connections which might cause routing problems if that software is not taking care.

Open issues are just below 400 loose ends now. Please keep up in connecting them to get a more complete major road network.

Russ_McD · 2017-10-15T09:52:20.000Z

Although I have ridden in the Khao Yai NP a fair bit, I think I plotted the 3288 from aerials. Ironically I was using the noexit=yes tag to indicate the road actually stopped. Most times I will draw a small piece of the next connecting road or path to prevent the loose end from showing up.
Yes, technically here, a path & steps should have been drawn but they were probably not visible.

stephankn · 2017-10-15T16:16:50.000Z

Hi Russ,

sorry, my fault. I just daw you as last editor of the toilets and restaurant and thought you added it. Should have checked the full history.

While in the north and north-eastern areas loose ends nearly always can be continued using different imagery providers, I notice that more south the mis-tagged ways have a higher percentage. So obviously more to fix in those areas than just the loose end. We can take the loose end as an indicator of an area being suspicious.

For Thailand we are down to302 loose ends. During the next days I will extend the reporting to full SEA. Still hoping that we can put priority on improving the mapping in Thailand.

Russ_McD · 2017-11-11T00:56:28.000Z

Briefly, I’m in total agreement with Paul in that as OSM mappers, we just want to be the first to stick on a “new” road as soon as we see the diggers out … as I did some Chiang Rai updating recently putting in some islands before the cement had even hardened ! So, yes, construction roads should have dangling ends until the exact junction location is opened.

I do think if Stephan can exclude any tert / sec ways that end with either a noexit or turning circle nodes, we can use that to exclude them once verified.

I also see a lot of ways now in Thailand that seemed to have been added by AI, and called tertiary. Unless the imagery supports a major route, and checking on the DOH site reveals no road ref,. then I do downgrade to unclassified, especially if not a through route.

Russ_McD · 2017-12-29T03:22:20.000Z

Was wondering why in excess of 8,000 loose ends are being reported … just zoom out and look at Vietnam.
Seems to me like its been used as some sort of playground for automated imports. Lots of new roads, wrongly tagged, unconnected, & hundreds of nodes on a straight piece of road… of course all added in a short period by a mapper with no history or identity.
Just one example here:
http://www.openstreetmap.org/user/Binh%20Hoang/history#map=13/10.1982/105.3635

In his changeset 44298298, one clearly residential road goes from service, to secondary bridge, to tertiary, and then then stops. It clearly continues.

Lots of change sets, lots of anonymous mappers … poor Vietnam. Its going to take the users there many hours to sort that mess out. So, I have had my moan, but it does really mean that achieving zero “loose ends” in Asia will never happen now.

Should the tool be restricted just to Thailand now ?

stephankn · 2017-12-29T09:25:50.000Z

Hi,

Yes, Asia seems to be considered by different parties in need for imports or remote mapping.

I hope that the just growing community there is able to sort it out.
If you spot large scale bad imports or organized mapping, please report to the international mailing list in the hope someone can take care of it.

Since 21st I get strange postgres errors. Will try to recover it. Data is currently not updating.

ERROR: array size exceeds the maximum allowed (134217727)

I consider creating a special view just for Thailand. With the road network here in an increasingly good shape it is worth filtering it.

Stephan

stephankn · 2018-01-13T10:20:27.000Z

I am still a bit undecided whether to do a separate reporting for Thailand.

In Thailand would be currently 118 loose ends of major highways being reported. With so little left, maybe not needed to report dedicated. Would be quite obvious if a new loose end appears.

Most loose ends currently happen to be located in southern Thailand. I can’t say whether it is related to the Facebook activities or not.

Having an easy way to spot suck loose ends also helps to quickly spot bad mapping. So probably better leave it active for the surrounding countries.

Stephan

Russ_McD · 2018-01-13T11:38:17.000Z

Perhaps a drop down menu to select Country ?? Or is that too much work