If there is just a single GPS track, trust the aerial image more. If you have enough tracks to average, then this will show the correct position. I am told that there is a possibility to define an offset if you are using JOSM, but I haven’t tried.
If you are mapping in a hilly area, then don’t trust the aerial images too much: they are often not or not well orthonormized: if a location in a valley is at a correct position, then a location on top of the hill might show some offset, because the photo of this point is taken with some angle. If aerial images are designed to be used for mapping, this deviation is corrected with the knowledge of the elevation of this area and the position of the plane. I don’t think Yahoo! did pay for this work as the images were not intended to be used for mapping.

Regards,

Baßtölpel