one option is to try doing it in sections, rather than everything at once
Unfortunately the validator only catches nodes directly on top of one another. Not ones that as veeeery slightly not on top of one another.
Matt, downloading from OSM’s data “along an edge,” whether it is horizontal (sorta easy if “just a latitude line”) or vertical (sorta easy if “just a longitude line”) can and is done in circumstances like this. (Import plus a validator check being run).
It can get challenging with a GPX or an irregular boundary / edge (like a river, a county / state / country border along a river or coastline…). Downloading those is piece by piece: not too much that you choke your editor, not so wispy-thin little that you miss something. “The Goldilocks amount” comes with a bit of practice. I have confidence you can do it, but do know it can be slow and tedious. I find when I’m doing these (less and less, but I’ve done a number of them over the years) that what is rewarding is getting through it all without mistakes and “it looks right.” (And IS right, to the extent you can, should and do QA your work or somebody else helping you does). A pretty “high bar standard” in OSM is when at least two people have “looked over each other’s shoulders and checked each other that the work is both complete and correct as specified.” OSM likes that, as it paves solid road for this process to be repeated in the future as updates in the real world arrive (and they do). This is what I (sometimes) mean when I say that “wiki chases data chases wiki chases data,” but in this case there is some wiki involved as an import proposal. So far, so good.
It might seem slow going, but you are listening, learning, doing a lot of correct things here (if not everything, but I am not in your head and can’t detect how you learn). I like the progress I see, so keep up the good work! This is how our data really improve with imports: when we listen, learn, follow the community guidelines, get some hand-holding at a place like here when there are questions, the answers arrive, the learning continues and all the pieces come together.
That is really interesting, thank you! I haven’t interacted with this feature of JOSM before.
Thank you, I really appreciate all of this!
1 Like
This is disputed see Should river lines be mapped through lakes, estuaries, gulfs, and other large water bodies?
Thank you skyper, that is fascinating!
At a minimum, all of the ways I’m touching will still be tagged as ArtificialPaths, so they could still be deleted in the future if that is what is decided. One issue with this particular import, is in some places the ArtificialPath is sticking out of the body of water before meeting a stream. A solution could be to connect the existing streams to the body of water and delete the ArtificialPath. But since these ways already exist, I thought it safer to connect and re-tag them instead.
But if I’m wrong, re-tagging or deleting the ways inside water bodies will still be possible.
There’s no consensus for removing such waterways OR for discouraging folks from adding them.
For the United States, it is absolutely common for small water bodies that have a river/stream that goes “through” them also have corresponding waterway= ways. There’s contention about specific cases for large bodies and how best to tag things in various cases but I don’t think this specific work goes against any pattern common to where it is being done.
I did it! Or, I did what what I intended to do, the ways are now connected. Here are the changesets.
Thank you all for your patience with me. I hope I didn’t go too far with my activity.
I also created a river relation for fun.
One thing I didn’t do was node merges to just connect waterways to instances of natural=water, when there wasn’t another waterway to connect to also. Maybe there’s more connections to make? For instance: way/80114906 runs between two small ponds, but connects to nothing. Its ends are 4.3cm and 1.8cm away from nodes belonging to the ponds.
When you’re merging waterways with “ponds,” yeah, I’d join nodes that are mere cm apart. This is what I did, as I mentioned I’m coastal, as many streams (eventually) drain to the Pacific ocean. These kinds of questions are good ones, as we are “saying out loud” that it is a good idea for the geometry to follow the geography where warranted. In the case of “waterway to pond” or “waterway to ocean,” (coastline, actually) yeah, join 'em; it is warranted.
Justifying things as “flows into this” or “drains that” are real sayings that humans say about waterways helps us do this. One thing in OSM that is what I’ll call “allowed and encouraged” is to “make data smarter” like this where we are smart enough as a human to do this. In this place, you are likely nodding your head yes that you, too, can give yourself permission to be smart like this, as you and we are together.
Thanks for asking: join 'em.
1 Like
I noticed that the northwest portion of the confetti blob is still very confetti looking. Is that expected? I may have time to poke around next week if you haven’t yet.
1 Like
Thank you both!! I really appreciate everything you’re saying.
I’ve been gathering more information.
Yeah, the northwest part, those lakes don’t have AritificalPaths running through them so there wasn’t anything to connect, just looking at waterways. I don’t know if WaterwayMap considers small bodies of water when connecting streams and rivers? Even if it doesn’t I’m looking at connecting ways in Oklahoma like this: w61823627,w61945487,w61858734. The nodes are there, I’d just have to merge them.
One thing I’m not sure of though, is cases like way/138483691. It’s inside a small pond by itself, it is already connected to the way in the direction it’s flowing out of the pond (north in this case), but then it’s first node is at the back of the pond not connected to anything. Should I connect that, since the nodes are there to do so? Or leave it?
I found 4,125 instances of that, waterway starting at the back of a pond but not connected to it there, no other waterways nearby to connect to. And then 12,905 instances of every other case (waterway flowing out of or into pond while positioned outside the pond; OR waterway inside the pond and flowing up to the edge and stopping with no other waterway to connect to). Then, for the nodes I’d merge this time, 0.5% are at identical coordinates. 40% are closer than 3cm, and 60% are between 3 and 5 cm.
Thank you for your time!
Again, you are “connecting waterways” and so, please connect them. If your mind is not imagining the “knit together” waterway system in this area, it should be. There are ebbs and flows. There are ponds for standing water (often with a stream right through it). The USA and OSM have these, they are “understood to exist in certain ways, with certain methods of flow and connectivity” (like “ways point downstream”). Two inches or less? Yes, join / connect them. That appears to be in the neighborhood of “rounding error,” so you likely are merging things together which should be, but you should “check” (maybe the Validator is doing part of that, I’m not perfectly clear on your workflow, though your results do appear to be “an inch or two off between nodes” in thousands of instances).
How much of your review is manual, how much is automated? I don’t know, but be careful not to automate too much, especially if you are not sure of or how strongly you can trust the logic used to make a determination. Most of the time, actually nearly all of the time, what Validator reports is quite accurate and carefully presented to you. An error should always be fixed, a warning can be ignored, but should be looked at if you know what you’re doing, and you are wearing a hat that says “I know what I’m doing” as you import.
To your best extent possible, please integrate waterways into a “whole fabric” that includes the sort of connectivity that very adjacent nodes (inches, “mere centimeters”) enjoy with actual node-connectivity.
So far, you are showing us that you can map by your so-far-so-good example, though there appears to be more polish and shine ahead. Yup, especially when things are working well and as they should, that’s how we do things around here. Keep it up! (Your interactivity with us here helps).
2 Likes
Thank you. When I commited the changes over the weekend, I went through all of the validator messages. I fixed all the errors, and almost all of the warnings. I looked at every warning one at a time, for sure, leaving the ones where I didn’t think there should be a change. I don’t remember what the case was where that happened.
I’m automatically downloading and parsing the information from Overpass, and then dumping what my program finds to a file. I’m looking at what the waterways would become connected to if its end node(s) got merged with the nodes within 5cm. I’ve now narrowed it down to:
- 16.8k instances of the waterway way connecting to a single closed way tagged natural=water
- 145 instances of connecting to a way belonging to a relation tagged natural=water
- 5 cases of connecting to a way just tagged landuse=reservoir, which I would change to natural=water manually
- Less than 5 instances each of a waterway connecting to a way with natural=water and also one of: a natural=wood, a natural=wetland, or a second natural=water at the same position
- Finally, about a dozen instances of waterways with ends touching that I didn’t find before because I only looked for AritificalPaths and connectors.
There had been some cases of administrative boundaries, with a node at the same coordinates, but I don’t touch anything like that automatically.
Once I was comfortable with what I was going to do, I loaded everything in JOSM and ran a script that goes to each group of nodes to merge and runs the functionality behind the “Merge Nodes” function on them.
2 Likes
Excellent. I’m doing a “handoff” here to @watmildon who is also watching closely and others here who are more quiet. Buff it to a polished shine, folks. Peace, out.
1 Like
waterwaymap only looks at things tagged as waterway= so all of the natural=water features are completely invisible to it. I’m sure there’s some extra preprocessing that could, for example, collapse all water= features to points so all of the in and outflows are “connected” but that requires features like w61823627 to get patched up as you suggest.
Setting aside that this area needs a redraw, I think it’s unusual to have waterways begin at the back of a water= feature. I wouldn’t say it is wrong just unusual for how folks map them. I don’t think connecting them up is wrong but it’s also not meaningfully more correct than leaving it disconnected. Honestly, I’d be tempted to delete those “originator” ways so the import seemed a bit more consistent with OSM “style”. But again, I don’t feel strongly about them.
1 Like
Oh wow, I hadn’t looked at the satellite. I’ve fixed those manually now. I deleted the ArtificalPath too.
I had read the “lakes are basically water-roundabouts” idea before but I never though of collapsing them to one node, that’s really interesting.
Thank you for continuing to bear with me. I’ve been trying to compare OSM to the NHD to confirm these ways inside ponds are really the beginning of a stream, that there isn’t a segment missing in OSM, outside the pond and flowing into it that should be connected to the ArtificialPath. I did find and fix a few instances manually today, where a small connection between two ponds or under a roadway probably got deleted at some point.
But then I found way/139502473, it doesn’t currently connect to anything with its first node, but I see a way dated 2019 in the NHD that leads right up to it:
- NHD query of the immediate area, layer 6 - flowlines
- The waterway newer than when this was imported, its “fdate” is 2/20/2019:
{
"type": "Feature",
"id": 8456072,
"geometry": {
"type": "LineString",
"coordinates": [
[
-99.6095370210525,
35.7899612207287,
0,
9.648375
],
[
-99.6097273773557,
35.7902229606474,
0,
4.469875
],
[
-99.6097743272835,
35.7904827525027,
0,
0
]
]
},
"properties": {
"OBJECTID": 8456072,
"permanent_identifier": "{46333995-D382-42E4-8188-7BA840803DAE}",
"fdate": 1550620800000,
"resolution": 2,
"gnis_id": null,
"gnis_name": null,
"lengthkm": 0.06329655,
"reachcode": "11090201009372",
"flowdir": 1,
"wbarea_permanent_identifier": null,
"ftype": 460,
"fcode": 46007,
"innetwork": 1,
"mainpath": 0,
"visibilityfilter": 24000,
"Shape_Length": 77.7378083086172,
"globalid": "{F4B011AB-2297-4D56-871E-9A4C4E3E7FF8}"
}
},
I understand getting rid of useless ArtificialPaths, but I kinda want to leave 'em for now… or at least I don’t know that I want to be responsible for deleting them, there’s about 4000 of these cases. Another thing I came across, is if I were to delete them, and their last node connects to a waterway heading out of the lake, that waterway could be left connected to nothing, since many of the imported waterways had already been connected without including the edge of the surrounding water.
So in short, I tried to make it easier for me to make a decision, and ended up just making it harder. 
I can work on figuring out how to delete these ArtificialPaths inside ponds and making sure anything connected to their last node gets connected back to the water’s edge. And if anybody does an import of newer waterways in the future, they already have to worry about connecting things and can figure it out then.
Or, I can still leave them and just connect everything else outside of the ponds as planned. I am definitely going to sleep on it.
Thank you!
1 Like
One thing that is useful to think about it how easy would it be to “undo” this if we discovered 6 months from now we have decide we want to backtrack. Given that this is all sourced from NHD, it’s pretty hard to do enough damage that a dedicated person couldn’t (through reverts or redo of import) patch things to how they were. It’s even easier to backtrack for solely tagging changes.
In contrast, updating all the geometry for a ton of varying and connected feature types over many weeks that would need to be hand entered? Much more to worry about.
They’ve been there a while, having them in for a while longer doesn’t hurt anything. As you’ve noted, there are some that may be useful to keep around for if the import is redo/extended etc. I’m glad you’re thinking about what happens for the next folks to come along. I don’t think the map is damaged hugely either way.
2 Likes
(just to comment on the original question)
To be honest, that looks like a bug in the initial import? Now that you’ve fixed it, this stream and this lake look OK to me.
1 Like
Yeah, this is a super common issue with older NHD imports.
1 Like