The longest running arguments about this sort of thing have been based around razed railways. I think this might be the most recent discussion, but I didn’t do much more than observe its existence due to an overwhelming sense of déjà vu. There are probably links to previous discussions in there somewhere.
The threshold for “not existing on the ground” does vary a bit from person to person. Some “long gone” items have residual earthworks still identifiable to people with experience spotting such things, some buried things do “appear” as areas more and less prone to drying out during droughts.
This isn’t to say that nothing should ever be removed, some things should, but it can all get a little heated.