Tiger cleanup - tags

watmildon · 2023-01-29T07:55:25.297Z

I wrote a bit about some novel TIGER name cleanup you all may find interesting. watmildon's Diary | Using the US National Address Database to assist TIGER tag cleanup | OpenStreetMap.

As for the original question, I don’t find any TIGER tags particularly useful once the review is done. cfcc should be better specified under highway=, tiger:county= while historically interesting when tools weren’t as mature is generally better served using the now robust Overpass service, tiger:name_* just leaves a mine filed of nonupdated tags if a roadway changes it’s name later.

stevea · 2023-01-29T11:52:08.272Z

While I chose to respond by specifically clicking on the @aighes icon, this reply addresses more than simply one person’s comments here.

I remove tiger:reviewed=no when I’ve substantially improved either or both of naming or geometry of the way (road or rail, and I improve both). Other TIGER keys like :cfcc or :county I generally leave alone. And although it is true what @ElliottPlack says, digging through histories is fraught with more difficulty than it is to see the tag right here, right now. I don’t believe a consensus has emerged about :cfcc (which I have found to be useful in some quirky cases) and :county really can be useful (as @Carnildo said) “when zoomed at a particular level” (seems like a “weak” reason, but OK, I’m nodding my head).

Now, ZIP codes simply don’t belong in OSM at all, whether associated with a TIGER tag or not. ZIP codes are not even geographic areas (they don’t delineate these, which is why drawing their boundaries is so problematic). ZIP codes are more like a routing algorithm for efficient postal mail delivery. Their intersection with OSM should be zero.

I read @watmildon’s Diary as linked; good stuff there; thank you. And welcome to posting in Discourse.

15 years after TIGER, there still remains a LOT to clean up from this, and we are only a bit closer to consensus on what to do with the tags: it does seem to be more-universal to delete :reviewed when the way actually has become “reviewed,” rather than the “uh, some do this, some do that…” non-consensus we had five years ago. Maybe by about 2045 we’ll have cleaned up all aspects of TIGER. Maybe later.

Again, I really miss the old ITO World rendering that helped fix TIGER, that was great and highly effective. But we can continue to blue-sky good ideas here and in our wiki TIGER Edited Map - OpenStreetMap Wiki , where the ITO World rendering is the topmost example. For example, there are fairly nice OT queries (for local areas, a county-at-a-time query doesn’t cripple the servers too bad). I can continue TIGER cleanup in my county with these, but I feel OSM can do much better “visually” (as in ITO World-like renderings). We’ve had a few “stabs” at quicky renderers that attempt to do TIGER review, anybody feeling ambitious enough to whip up an ITO World-clone?

But we might decide on good strategies on “what to do with tiger: keys” as a continuing discussion here.

And as I don’t want to write seven more chapters to this book (here and now), I’ll stop.

aighes · 2023-01-29T13:22:39.501Z

stevea:

I remove tiger:reviewed=no when I’ve substantially improved either or both of naming or geometry of the way (road or rail, and I improve both).

Additional it should be fine as well to verify the existing geometry/name/highway is sufficient. If something is already correct, there is no need for any substantially improvement.

Where du you see the benefit of the tiger:cfcc? So far it seems for me to be same thing, but less precise than our highway classification.

Richard · 2023-01-29T15:09:31.620Z

aighes:

Additional it should be fine as well to verify the existing geometry/name/highway is sufficient

And surface. Unpaved highway=residential TIGER A41s are the bane of bike routing in the USA.

DUGA · 2023-01-29T19:26:51.473Z

https://wiki.openstreetmap.org/wiki/TIGER_fixup

Most of the TIGER tags are junks and should be deleted. That being said, what I’ve done is:

Fix the geometry.
Verify if the name is correct or not.
Check tags which may contain historical information.
After then, remove everything related to TIGER.

tiger:cfcc:Useless information.
tiger:name_base: You need to verify and correct name if needed, then remove it.
tiger:reviewed: You need to remove it after everything is set.
tiger:separated: Automatically removed if you iD.
tiger:source: Automatically removed if you iD.
tiger:tlid: Automatically removed if you iD.
tiger:zip*: Useless information, USPS is the source of the truth.
tiger:upload_uuid: Automatically removed if you iD.

aighes · 2023-01-29T19:38:08.114Z

As a cyclist I would personally agree, though I think it’s nothing mandatory anyone have to add in order to remove the tiger:reviewed=no. Let’s hope the recent changes to OSMcarto will give a boost to surface-usage

stevea · 2023-01-29T21:05:32.545Z

aighes:

Where du you see the benefit of the tiger:cfcc? So far it seems for me to be same thing, but less precise than our highway classification.

It was years ago, but I do recall using some of the tiger:cfcc tags when doing some rail classification and “better tagging” (as in existing rail tagging conventions that didn’t exist or weren’t paid attention to when TIGER data were imported, or those that evolved in OpenRailwayMap tagging conventions years after the import), as I have cleaned up thousands of miles of USA Rail that came into OSM by the TIGER import, and that tag proved useful in certain wide-scale OT queries: quite helpful, I recall. I agree that for the most part, tiger:cfcc tags seem “junks” or “useless” (as @DUGA says; welcome to Discourse, DUGA). But as can be true with such data, we don’t know all possible cases for how or when they might be useful. And as in some cases (rare, admittedly, but in the case I note above, actually do happen), they are useful, because they (surprisingly) can be.

Until, as with tiger:reviewed and then we review the data and it becomes “fine enough for OSM,” and then it is absolutely proper that we delete it. I agree with @aighes that “if it is already correct, it doesn’t need to be further corrected.” The other data? Let’s continue to discuss, as “which tags might be disposed of has begun.” We should be careful, though: we must try to imagine use cases, but that is fraught with the danger we don’t imagine everything possible, quite a likely occurrence.

I’d say tiger:zip* could be deleted with a mechanical edit (and yesterday). There is no reason for these data to be in OSM, whether they are, might be or are not correct. It’s simply incorrect for them to be in OSM; no need to make a determination if they are “correct.”

watmildon · 2023-01-29T21:29:00.750Z

Someone in another forum mentioned renaming the TIGER zip tags to postal_code tags which I think is also likely not helpful?

stevea · 2023-01-29T21:34:43.161Z

Renaming “zip” to “postal_code” simply obfuscates / further confuses their origin, which is “mail delivery numeric algorithm acting as imposter data for geographic area.” Making such geographies (polygons) is impossible to do, but people continue to try to “map” (logically and geometrically) ZIP codes to geographic areas, always as “estimated” or as is stated to be more blatantly true, “incorrectly.”

I’ll repeat my strong opinion: tiger:zip is a no-brainer for “can be deleted with a mechanical edit,” but of course, we’d need to achieve wider consensus on that before doing so. Other tiger: keys are more difficult to make such a determination, but they all lean heavily towards “let’s do what we have to do, even taking years to get there if we need to, so that we can show these tags the exit door out of OSM.”

The tricky part, and why we are (still) 15 years into this discussion (and discussion, and discussion…) is to “wring out” the maximal amount of “mapping value” out of these tags before their demise. That’s pretty hard, as we can’t possibly imagine every use case. And simultaneously, the data DO need substantial review and/or improvement.

watmildon · 2023-01-29T21:50:19.667Z

It seems easy to conflate two discussions together:

What tags are safe to remove after a “thorough” review of an object?
What tags do various folks think should just be removed from the DB?

Focusing only on the first class, it seems that there’s some set of folks that derive some (maybe small) value from county tagging and that cfcc is somewhat helpful if the surface isn’t well tagged. Everything else seems to be getting the axe by most folks. Fortunately this seem to match the wiki.

For the second class there’s probably some relatively easy housecleaning among the top TIGER:* tags. Looking at you: tlid (2.7mil entries), source (2.7mil), upload_uuid (2.4mil). And then some perhaps less easy but maybe not too contentious cleanup… maybe: zips (200-300k)?

Everything on tag info with fewer than 100k entries is totally unfamilar to me so I won’t even speculate! I was surprised to see the wider range of TIGER:* tags listed there.

stevea · 2023-01-29T21:54:07.230Z

aighes:

Let’s hope the recent changes to OSMcarto will give a boost to surface-usage

It’s kind of a side-topic to this thread, but it is absolutely true that wider paying attention to “the entire data path” of OSM nodes, ways and relations turning into a rendered map is a real requirement for how the feedback loop of effective mapping happens. Yes, this gives rise our “don’t tag for the renderer” no-no, but it also rears its head when Carto begins to support a tag as a rendered feature: in effect, this says “it is important to get the tagging right on such features, as NOW, they render” in our “standard” renderer, which by doing so, carries some clout on “what is important for Contributors to tag, and how.”

This is a complex topic, as OSM does a certain amount of “trying to hide” the complexities and difficulties of “what is rendered” (OSM is not a map of specific rendering, it is a database), especially by saying “don’t tag for the renderer,” yet at the same time, it (appears to?) influence the way that specific tagging does or will happen, by making choices in the renderer. That feedback (at the end of the pipeline, “a rendered road surface,” for example) reaches all the way back to the mapper, by saying “it is somewhat important to be careful tagging this feature” (because it is rendered). A very tricky balancing act, people like Paul Norman, Joseph Eisenberg and other author-contributors to Carto know quite well.

Again, this is a side topic; back to the main thread.

stevea · 2023-01-29T22:02:36.487Z

For the tags that are safe to remove after a “thorough” review on a datum, I think it is accepted that tiger:reviewed=no should be removed when a conscientious OSM editor (aren’t we all?!) feels the datum meets the requirement of “good enough to enter into OSM.” It’s the equivalent of “whether it came in from TIGER or it came in because I created it from scratch, there is no reason for tiger:reviewed=no to remain on this datum: it is ‘high enough quality to be in OSM’ and a tiger:reviewed=no tag either directly contradicts that or leaves it questionable or ambiguous.”

For the tiger:zip_code data, let’s continue to discuss whether a mechanical edit to delete these is appropriate, I’d say yes to that.

For the others, I’m glad we are discussing these, but let’s not be too glib or easy, as there really may be some use cases we haven’t (yet?) imagined that could make them useful. If they are, let’s “wring out” as much semantic usefulness as possible (with new, better tagging, or better position, or whatever) and make them contain in an OSM-correct method whatever those tiger:* tags purported to impart, and then delete the tiger:* tag. This won’t be a quick, socially-lubricated-as-easy process. It will be fraught with chin-scratching and a wide variety of opinions that fall widely upon a spectrum of what is best we should do.

15 years in, hm…another 15? We could shoot for a finish line in 10, though I think 5 years is ambitious.

Richard · 2023-01-30T12:59:31.826Z

watmildon:

Focusing only on the first class, it seems that there’s some set of folks that derive some (maybe small) value from county tagging and that cfcc is somewhat helpful if the surface isn’t well tagged.

I use tiger:reviewed for the latter (for cycle.travel), not :cfcc.

watmildon · 2023-01-30T19:05:42.244Z

Absolutely sensible.

SherbetS · 2023-01-30T19:19:17.018Z

Yeah. I’d definitely look for tiger:reviewed to find roads that don’t have trustworthy surface info. cfcc is pretty useless considering every road that you’d have to worry about adding surface data to would just be… A41

In due time I’d love to consider a mass edit to just remove the tags to stop confusing mappers.

stevea · 2023-01-30T19:52:13.785Z

I’m beginning to nod my head in agreement, but let’s be careful here before we all scream for tag deletion bots running deep into the night. What it appears we’ve “built” with TIGER data (tags) are a whole bunch of assumptions that keep stacking ever higher. For example, yes, @Richard using tiger:reviewed for making more than a coin-flip’s worth of determination about surface is sensible, but what about the converse: when there is no such tag, it may still be true that if the surface isn’t well tagged, it’s still a big unknown w.r.t. surface. You’d have to dig into the history of the datum to see if tiger:reviewed ever WAS on it, and when, and perhaps what might have happened so that it was removed, with only poor-worthiness “guesswork-level” logic, most likely.

Stated succinctly (nearly impossible with TIGER, unless you go very deep, and I won’t here): you can’t always remove tiger:* tags willy-nilly without some consequences, but eventually we want to do exactly that.

Adamant1 · 2023-01-30T19:52:17.300Z

SherbetS:

Yeah. I’d definitely look for tiger:reviewed to find roads that don’t have trustworthy surface info.

I’ll delete tiger:reviewed if I add surface tags, but I usually keep it if the edit is just a minor adjustment.

SherbetS · 2023-01-30T20:35:17.894Z

What it appears we’ve “built” with TIGER data (tags) are a whole bunch of assumptions that keep stacking ever higher.

Absolutely. IMO the best move is to make focused guidance on how to handle the tags, so we can methodically separate ourselves from the tiger:* stuff.

There’s another thread on the US forums where they’re using MR to add surface data in California.

I feel that going forward, this kind of work may be an important part of shedding all the old tiger:* tagging.

–SherbetS

Carnildo · 2023-01-30T22:46:56.219Z

Worse than “not helpful”, it’s “actively misleading”. “tiger:zip” codes aren’t ZIP codes, they’re ZCTA codes. These are a Census-derived approximation to ZIP codes, useful mainly for interpreting Census-derived statistics.

aighes · 2023-01-31T00:53:43.317Z

stevea:

For example, yes, @Richard using tiger:reviewed for making more than a coin-flip’s worth of determination about surface is sensible, but what about the converse: when there is no such tag, it may still be true that if the surface isn’t well tagged, it’s still a big unknown w.r.t. surface.

How the rest of the world is doing it without the tiger:reviewed Consider there is no more tiger:reviewed, so a mapper removed it. Either there he set a surface, then there should be no issue. Or there is no surface defined, well then there might be assumptions done in the renderer. The assumptions possible by tiger:cfcc have been made already during the import. For me nothing you going to loose here. Anyway the issue of missing surface you will have for everything not taken over from TIGER, more or less like everywhere else. So deleting everything but the tiger:reviewed would make sense and keep tiger:reviewed as a machine-readable status indicator, whether the object has been reviewed.