Adding bridge information in Kentucky and other states

[tekim (Mike)]

@Maneesh-Mahlawat

@PHerison @tekim Do you see an issue with getting this information from FHWA instead of looking at the satellite imagery

1. Make sure you know that the definition of “Structure Length” matches what OSM generally considers the length of the bridge.
2. There is no method in JOSM that creates a way of a specified length, so you would have to manually shorten or lengthen the way as needed by dragging one of its end nodes until its length matches the FHWA data. In any event, even if you are going to get the data from FHWA you are going to have to look at imagery to see where the bridge starts, and its orientation.

@Maneesh-Mahlawat

I have a follow up question: what if I were to automate this and add bridges in Kentucky (wherever they don’t exist) in one-go, do you see any problems?

To do this properly this would be difficult. One issue is that you have to snap to existing geometry in OSM. To properly add bridges in OSM you need to split the existing way at the start and end of the bridge, and then change the tagging (bridge=yes at a minimum) on the new way that represents bridges. You can’t just import the points from FHWA. In any event, this would be an import and must be documented and discussed with the community first, see Import/Guidelines - OpenStreetMap Wiki

[PHerison]

@Maneesh-Mahlawat

@PHerison @tekim Do you see an issue with getting this information from FHWA instead of looking at the satellite imagery

As @tekim already pointed out, there is no lenght-tag in OSM, so you have to calculate coordinates of the end-node by using the other end-node as an “anchor-point”. Not worth in my opinion, as somebody else might change the node-positions and hence change the length later.

[Minh_Nguyen (Minh Nguyễn)]

I would probably suggest focusing on the bridges that aren’t mapped as bridges yet in OSM (so excluding the example along KY 258). If there’s an existing bridge, an import could start out by adding missing tags, but if any bridge needs tags or geometry to be changed, it could instead flag the bridge for manual review. Another reason for caution around existing bridges is that almost 200 man_made=bridge areas have also been mapped. A bridge area is supposed to be connected to the roadway at either end, so you have to consider topology as you make any edits.

[Kai_Johnson]

@Maneesh-Mahlawat

@Kai_Johnson looks like with this approach, we don’t have to worry about licensing. Let me know

As far as I know, the NBI should be public domain like many of the other Federal government data sets.

One think to keep in mind is that per the notes on NBI data: “All data in the inventory and operating fields is downloaded in metric tons even if rating factor is indicated. The factor has been converted to metric tons.”

[Maneesh-Mahlawat]

I received the following reply from Kentucky Transportation Cabinet. Is this enough or do we need to get something in writing?

Thank you for reaching out about using our spatial data to improve the accuracy and comprehensiveness of OpenStreetMap. The Kentucky Transportation Cabinet has no objections to your use of the publicly published data (properly cited) for the public good.

We have an automated process that publishes our spatial data (with metadata) to the Kentucky Geonet Portal on a weekly basis. Here you will find a variety of data that may be relevant to you including, data on bridges and roads. The KY Geonet Portal can be accessed by everyone from https://kygeoportal.ky.gov/geoportal/catalog/main/home.page.

1. You can search by keyword for available data sets

2. You can also Browse the catalog by content type or topic category

3. In the Results of your search, select Open to be directed to BOX where you may download the shapefile.

Please be advised that if you download the data it will need to be redownloaded weekly to stay up to date. Due to this maintenance, we recommend consuming services if you are able to do so.

I hope this advances your goal of improving the accuracy of mapping data and making roads safer, something we care about as well. If you have any other questions that I can answer please let me know.

I plan to email / call each state to get their permissions. Do you have any suggestions? Thanks!

[Minh_Nguyen (Minh Nguyễn)]

This is helpful and thank you for reaching out to them. Their response is vague enough that I guess it depends what you asked them. Proper citation on our “Contributors” page on the wiki is doable, but we can’t guarantee the comprehensive attribution that, say, ArcGIS maps tend to display in the corner.

Every state probably has slightly different expectations. To save you some time, state DOTs in several states like Massachusetts, Ohio, and California have already released their GIS layers into the public domain. I don’t know if we have a list somewhere that keeps track of these things, but you can often figure out existing permissions by searching the wiki for previous import proposals from DOT sources.

[Maneesh-Mahlawat]

@PHerison Did you look at National Hydrography Dataset (NHD) to import all waterways in one-go? We were looking to match the coordinates of bridges from NBI hitting OSM roads to tag missing bridges and found that this information provided us with accurate bridge locations. Let me know if you have any questions

[Maneesh-Mahlawat]

@Minh_Nguyen @Kai_Johnson We created an import page to outline our approach to importing bridge information and then tagging bridges with truck routing tags. The current page only talks about the first page - import bridge information - but we will update it once start working on truck restriction tags. We are still refining and testing the import process including the code. We hope to have it ready for first import next week. Do you have any comments / suggestions?

[tekim (Mike)]

HI @Maneesh-Mahlawat - I took a look at your import page on the wiki. Regarding:

This approach [JOSM Approach 2 - split way and remove old way.py] utilizes the Jython engine of the JOSM scripting plugin. It retrieves the active data layer and edit dataset from JOSM, then splits existing ways at bridge locations, adds bridge tags (bridge=yes ), and removes the old way. It accurately represents missing bridges by interpolating the start and end points of the bridge segments along the existing way segments.

After splitting a way you do not have to, and you should not, “delete the old way.” Splitting the way in most cases will result in three ways, one of which should be an edited version of the original way (probably the longest one). This preserves the history of that way in the OSM database.

[Maneesh-Mahlawat]

@tekim Thank you for pointing this out. We are thinking of not using “Approach 2” since we are not very confident of keeping “relation” accurate using code. We got a similar feedback from others. We will use “Approach 1” since this does not delete anything

[Kai_Johnson]

I’d suggest that you skip all roads that are tagged with tiger:reviewed=no because there’s a good chance that these roads are not in the right place.

Having dealt with the consequences of zealous mappers putting ford=yes on every intersection of a TIGER road and an NHD waterway, I can attest that the original alignment in the source data sets was often poor. And this resulted in a lot of misplaced or imaginary fords.

There’s no sense in putting in a bridge in the wrong place because undoing that gets more complicated.

Also, the Bridge and Truck Restrictions Import page does not link to any actual processed data. I see the link to the Github source, but it would be good to see some sample results so that it’s clear that the processing actually works as expected.

[Minh_Nguyen (Minh Nguyễn)]

@Kai_Johnson

I’d suggest that you skip all roads that are tagged with tiger:reviewed=no because there’s a good chance that these roads are not in the right place.

I’m familiar with this problem, having fixed many roads that still ran along former river crossings but in reality got bypassed many years ago, including in Kentucky. Unfortunately, tiger:reviewed=no is not a reliable indicator of whether or not a TIGER-imported road has been reviewed. All it means is that it was originally imported from TIGER. In some cases, mappers have updated this tag as they clean up the data, but there’s no meaning to this tag that generalizes to a whole state, let alone a whole country. I routinely encounter roads without tiger:reviewed=* or even with tiger:reviewed=yes that are in worse shape than those with tiger:reviewed=no.

@Kai_Johnson

Having dealt with the consequences of zealous mappers putting ford=yes on every intersection of a TIGER road and an NHD waterway, I can attest that the original alignment in the source data sets was often poor. And this resulted in a lot of misplaced or imaginary fords.

I don’t think this proposed import is comparable to the phenomenon you’re describing. There’s little question that a bridge exists along the road near a given point. Is the concern that NBI or the state DOT has a reasonably accurate location for the bridge, but that we’re tossing it away by snapping to the TIGER-imported road? If so, would establishing a maximum snapping distance alleviate this concern?

[Kai_Johnson]

@Minh_Nguyen

I routinely encounter roads without tiger:reviewed=* or even with tiger:reviewed=yes that are in worse shape than those with tiger:reviewed=no.

True. You know how bad TIGER roads can be.

@Minh_Nguyen

Is the concern that NBI or the state DOT has a reasonably accurate location for the bridge, but that we’re tossing it away by snapping to the TIGER-imported road? If so, would establishing a maximum snapping distance alleviate this concern?

I don’t know if NBI or the state DOT coordinates are reasonably accurate. I guess that’s a hypothesis that the proposed import still needs to test. But we do know that there are a lot of TIGER roads that aren’t in the right place.

Establishing a maximum snapping distance between the bridge coordinates and the road in OSM could be a good check.

I guess the import also needs to consider some edge cases:

• Roads that have intersections within the apparent span of the bridge
• Roads that are split into separate ways within the apparent span of the bridge
• Multiple roads that are adjacent to the apparent span of the bridge and where the placement of the bridge might be ambiguous.

It also wasn’t clear to me from looking at the JOSM scripts whether the processing would be able to make its changes without breaking any parent relations.

[Maneesh-Mahlawat]

@Kai_Johnson @Minh_Nguyen We have been busy integrating, testing, and making sense of the data. Now we are at a stage where we know most of the missing bridges in certain areas First of all let me address some of your earlier comments:

I’d suggest that you skip all roads that are tagged with tiger:reviewed=no because there’s a good chance that these roads are not in the right place.
Having dealt with the consequences of zealous mappers putting ford=yes on every intersection of a TIGER road and an NHD waterway, I can attest that the original alignment in the source data sets was often poor. And this resulted in a lot of misplaced or imaginary fords.
There’s no sense in putting in a bridge in the wrong place because undoing that gets more complicated.

It is like this: the basis of the bridge is NBI data. We are not automatically adding bridges assuming there should be a bridge over the water body. We are also getting bridge span width.

Also, let me address your other concerns:

Accuracy of NBI and State DOT Coordinates:
We see that the coordinates from NBI and state DOT are not the most accurate. However, we have implemented an additional method where we calculate bridge points using a formula that converts coordinates from a specific fixed-point format into decimal degrees. To know more, please scroll to the DATA LINEAGE SUMMARY section to see the formula that yields more accurate geo coordinates. This method has yielded more accurate bridge locations and effectively eliminated duplicate bridge points.

Establishing a Maximum Snapping Distance:
Yes, we agree that establishing a maximum snapping distance is crucial. We are using a 30-meter buffer along the water bodies data from National (NHD) to ensure that the bridges are accurately placed in their actual locations.

As we dived deeper into finding a correspondence between the missing bridge locations in OSM and bridges in NBI databases, we saw several edge cases. Here is how we are addressing those edge cases.

Addressing Edge Cases:

• Intersections within the bridge span: This issue typically occurs at freeway interchanges. Most of these locations already have bridges marked in OSM. For any unmarked locations, we use the layer tag in OSM to identify and manually handle these bridges.
• Roads split into separate ways within the span of the bridge: We are aware of this situation and are currently working on a solution. We will provide an update once we have a finalized approach.
• Multiple adjacent roads near the bridge: We have encountered this issue as well. To address it, we use the intersection between water bodies and OSM ways, along with the distance between the given NBI bridge point and the nearest way, to determine the correct bridge location on the appropriate OSM way.

Regarding the concern about the JOSM scripts and parent relations, we are ensuring that the processing does not disrupt any existing parent relations. Our team is testing edge cases to maintain the integrity of the data.

I think one of the challenges I see here is that you need to trust my statements. That doesn’t have to be the case. We are currently working on making whole of the data, all intermediate data files, and final data available for anyone to test, reproduce the steps, and comment. We should have that available in a few days and at that time you can test the data and make a better judgement.

[Kai_Johnson]

@Maneesh-Mahlawat

We are using a 30-meter buffer along the water bodies data from National (NHD) to ensure that the bridges are accurately placed in their actual locations.

Hmm. The current NHD data sets are great if you want to analyze the connectedness of various water features. They’re not so great if you want to know exactly where the waterways are. I’m sure I’ve seen cases where the NHD flowlines are off by tens of meters.

Comparing TIGER roads to NHD waterways is sort of the worst case of imprecise data being compared to imprecise data.

[Maneesh-Mahlawat]

@Kai_Johnson Maybe you are right. We will make the data available to you before uploading and point to us a location (where there are issues with TIGER roads and NHD waterways) where you see any issues. Or point to us some areas in Kentucky and we can make those available to you right away.

[Maneesh-Mahlawat]

@Kai_Johnson Here are the steps to reproduce our method. You can see input data here. You can see all the output files here. If you are only interested in the final files, you can see final associations among NBI bridges and OSM ways here. Looking forward to your feedback

[tekim (Mike)]

Hello @Maneesh-Mahlawat

Over the last day I have been taking a look at 01-Kentucky-OSM-Bridge-Association-Final.csv So far I have the following comments.

Some of the “bridges” in this file are actually culverts, e.g.:

Some others are “box culverts” 0. Note the vegetation between the edge of the pavement and the lip of the culvert.

There are bridges in the dataset that have the same, or nearly the same, coordinates as other bridges in the dataset:

In some cases the bridge already exists:

(The algorithm may have gotten confused because the bridge crosses water and a road)

In some cases the bridge point is quite far away from the actual bridge. In the example below, the bridge point is about 66 meters from the actual place where the road crosses the waterway. Also, this crossing has already been mapped correctly as a culvert as shown in the second screenshot.

Here is another case where the waterway-road crossing has already been mapped as a culvert:

Another example of the crossing having already been mapped as a culvert:

Another example already mapped as a culvert:

[Maneesh-Mahlawat]

@tekim This is super helpful. We will look at these right away and see what we need to do here. Also, by definition, we don’t need to add any bridges less than 20 feet in span since they are culverts. Is that what you mean by pointing out culverts? What if those are covered in NBI database?

I am glad we are able to share all this before making any change to OSM. We will make the algorithm work before making any changes to OSM

[tekim (Mike)]

@Maneesh-Mahlawat thanks for your quick reply.

@Maneesh-Mahlawat

Also, by definition, we don’t need to add any bridges less than 20 feet in span since they are culverts.

I don’t think you can use Bridge_Length to definitively say whether something is a culvert or a bridge.

This is an example where Bridge_Length > 20 feet (6.4 meters ~= 21 feet), yet it is clearly a culvert based on available imagery.

This is another example where Bridge_Length > 20 feet, yet the satellite imagery suggests it is a culvert.

@Maneesh-Mahlawat

Is that what you mean by pointing out culverts?

I mean that these are things that, based on the satellite imagery, are clearly culverts, yet they have been included in the file 01-Kentucky-OSM-Bridge-Association-Final.csv. If you add a bridge to OSM at the location indicated it will be incorrect. If a culvert doesn’t already exist in the OSM data at the location, it could be added.

@Maneesh-Mahlawat

What if those are covered in NBI database?

You will have to find some way of filtering them out. NBI seems to include culverts as well as bridges.