Thanks everyone for the thoughtful replies. I’ll give this some more time to cook and get more perspectives. My current rough summary is:
For addr:country tags, for feature that have postal addresses, they are harmless and there doesn’t seem to be an appetite for removing them although some mappers do. I am perfectly happy to stop removing these tags.
For outbuildings and small secondary structures (roofs, sheds, carports etc), the community generally sees tagging these with address tags as incorrect mapping.
Additionally, there seems to be a decent cleanup item from old GNIS imports that should probably be reviewed and cleaned up. I can add that to my long list of mapping tasks but of course would be happy if someone else wanted to tackle that. Definitely a good first project for someone wanting to JOSM a bit!