I consider addr:country=* to be an exception in that it’s usually understood to be the containing country. This reflects real-world practice, which is influenced by the postal system. When I send a letter overseas par avion, I’ll end my return address in “USA”, but never when sending mail domestically. So when people enter addresses to navigate to, or when they enter an address into a form, they’re a lot less likely to include the address than, say, the state or ZIP code. After all, postal systems and address formats are organized at a national level. But I recognize that this is a fuzzy distinction, so I don’t go out of my way to remove addr:country=*.
There are some exceptions, like this library that straddles an international border: its centroid lies within Canada, but its physical/mailing address belongs to the American side. McMurdo Station has a post office well outside of any country. These cases obviously need addr:country=US. Still, I’d naïvely expect exceptions to be quite rare in practice. (Prove me wrong!)
You’re right that the Census Bureau’s ZCTAs aren’t ZIP codes, but that’s not what I was referring to. We often tell newcomers who ask for our postal boundaries that ZIP codes are “just delivery routes” to get them off our case. Actually, a ZIP code is a boundary, akin to a pizza parlor’s delivery service area, but for a post office. While the exact shape of the boundary is internal to the USPS and possibly a little fuzzy, it is possible to know the ZIP code of a given street address, even if it doesn’t receive mail directly.
You’re also right that some extremely remote areas probably aren’t covered by any ZIP code boundary. But Yosemite National Park is a good example of something else: ZIP code 95389 corresponds to the park, specifically its post office. If you look up the ZIP code on the USPS website, they give Yosemite National Park as the recommended city name, with several recognized alternatives. There are no other street addresses anywhere else in the park. Can a nothing have a ZIP code or not? It’s a moot point.
Some ZIP codes correspond to no postal delivery point whatsoever. For all practical purposes, 618 Escondido Road has a ZIP code of 94305, as does every address on the Stanford campus. The USPS’s ZIP code lookup tool doesn’t recognize 94305. Yet elsewhere, they do list the post office on campus with 94305 as part of the address. So does the county assessor, who serves as the local addressing authority. If an address doesn’t receive mail, they indicate this fact with the ZIP+4 suffix -0000. Fortunately, we’d put it in a generic addr:postcode=* key, not addr:zip_code™=*.
I don’t claim that this situation generalizes to other countries. Apparently it’s different in Canada, which is fine by me.
Inevitably, you’d find cases where multiple features have the same address and also addr:primary=yes. A church next door to its office and rectory/parsonage – which is the primary POI associated with the address? That’s a trick question: it’s the landuse=religious place=plot area surrounding them, right?
Some countries uniquely identify delivery points that share the same address, by adding some extra digits to the postal code. In the U.S., this is the 5-digit ZIP code, plus the ZIP+4 extension, plus a two-digit DPC extension. I do map the ZIP+4 when possible, to the chagrin of geocoder developers everywhere, but I never map the DPC for an address. That would require me to snoop around in someone’s mailbox and decode the barcode that the post office applied to an envelope inside, or alternatively shell out a fortune for CASS software. It isn’t part of the written address format and would be useless to any renderer or geocoder.
Snooping around the parallel German thread, I gather that the original motivation for object:housenumber=* was that someone tagged addresses on a bunch of fire hydrants on sidewalks in front of addressable buildings. I’ve never added addresses to fire hydrants before. You can already associate a hydrant with the nearest property spatially.
Address tags are for more idiosyncratic information. I’ve been tempted to add addresses to postboxes, because the USPS thinks all postboxes have addresses (and ZIP codes), and I’m not sure they generally match the nearest building’s address. For example, this postbox next to White Plaza is listed at 1 White Plaza, 94305. The county assessor knows nothing about 1 White Plaza; the postbox sits on the parcel for 531 Lasuen Mall. At least we can agree that it has a ZIP code, because it receives a lot of mail. 
Addresses can be weird like that. No one is completely in charge of anything.