If you look at the picture, the difference is more like 10 meters and sometimes even up to 15/20m.

Would it be a valid approach if I download the area and just adjust every single element?

I tried using JOSM and pressed Ctrl+A and moved them all together. I was focusing only on one point, but the result for all other points was really good as well.