Echoing what Robert Whittaker says, and revising my position, I think there’s no value in adding the completely destroyed trigs to OSM. I think there is value in adding the present-but-smashed-up trigs, but that seems to already be part of the import.
np. There is no intention to import the OS marked destroyed pillars in this import. I’ll leave them as a layer in the slippy map in case somebody finds them useful at some point.
On the non-duplication of existing potential OSM trig nodes:
The current method (as I think you are aware) is just a distance compare - if there is an OSM node ‘near’ an OS node, then we don’t consider the OS node for the import (it will end up on the edit/merge or review list). Do you have a feel for what you would consider to be near enough distance to be considered a potential duplicate?
I’m currently using 15m, but have show the effects of 25m and 50m - and I’ve no problem increasing the distance to 50m (or beyond) and ‘losing’ 104 (or more) potential trigpoints from the import (104 being the difference between the 15m and 50m filter results). I’m happy to increase the distance check - but where do we stop?
I can then think of a couple of extra checks I can do after the distance check. I could:
check to see if there are any OSM nodes that have a matching flush bracket value (in the ref field in OSM nodes) to the OS nodes, and if one is found, remove the OS node from the import (and move it to the review list). I can also check ref against the old and new style OS unique references - there are a bunch of those scattered through the ref field as well.
check to see if there are any OSM nodes that have a matching name to the OS nodes, and if one is found, remove the OS node from the import (and move it to the review list). I might have to invoke some of the fuzzy matching on these, as the names tend to be a bit of a mess with abbreviations etc.
I can start by coding that up and reporting the results here.
For reference, there seem to be 916 OSM nodes that are not ‘tied’ to any OS nodes in my current data run, with a 15m distance cutoff (and 1644 OSM nodes are matched to OS nodes). 321 of them have names, and 78 of them have refs. Only 5 of them are marked as a ‘pillar’ in their survey_point fields.
I think you need two distances: A smaller one, below which it’s close enough to snap the existing OSM node to the OS position, and a larger one beyond which you can safely ignore the existing OSM node. In between, I think you need manual review prior to the import to determine whether or not the two points are the same, and if so where the more correct location is.
You’ve probably got a better idea from the data than me, but something in the 15m-25m range would probably be good for the smaller value. I’d say something like 100-150m for the larger value.
First, and hopefully this is already clear, but I’d like to make sure it us understood - for this import I’m only planning on importing OS nodes that have no conflation necessary, and avoiding any need for manual review of OSM nodes, snapping etc.
The current model to do that is to only import OS nodes that are not ‘near’ any OSM nodes. I’ll discuss OS nodes that are near OSM nodes and snapping etc. at the end.
To that end, to make that ‘near’ decision we only need one distance - the distance at which we can consider an OSM node to be ‘unrelated’.
Right. Initially looking at the locality graph on github I made the call to use 15m. Maybe that was a little ‘optimistic’ ;-) I’ve grabbed a few more data points for comparison:
OSM node check distance
new OS nodes found
diff from 15m baseline
% left
15m
4437
0
100%
25m
4389
48
98.9%
50m
4333
104
97.7%
100m
4306
131
97.0%
150m
4295
142
96.8%
Looking at the github graph we can see it really tails off after 75m or so. I’m happy to set the cutoff at 150m. I am mildly concerned that we are potentially stalling some good data being imported based on what I suspect is existing bad data - but, if that is what it takes to upload the remaining 4295 good nodes, then so be it.
So, let’s talk about the minimum distance and snapping OSM nodes to the OS node data. This is not part of this import proposal. I don’t think it should affect the import of the new OS nodes.
The code does have a notion of how near an OS and OSM node need to be to be ‘good’. You can see such nodes on the slippy map under the ‘good’ layer. Right now I’ve set that minimum as 1m - that is, if an OSM node is within 1m of an OS node we say ‘good enough’ - for now. My plan is once we have some new OS nodes imported I will remove that relaxed check and it will be a 0m exact check. If an OSM node does not exactly match the OS data then it will be marked as either edit or review. And, why would we not? If we have the data why would we not note that it does not match?
The code currently generates two datasets that could be used, or proposed for new imports, after this import is done:
edit nodes: could also be called ‘auto merge’ nodes. This is where we have found an OS and OSM node that are close to each other and some fields match but others only exist in one or the other node - so nominally the two nodes could be merged - including snapping the OSM position to the OS data. Notionally this could all be done without human intervention, but that would require a deeper discussion of the actual data, which is why I’ve not tried to pull it in as part of this hopefully more simple import.
review nodes: where we have found an OS and OSM node that are near each other, but there is some mismatch in the data that looks like it will need human intervention. I doubt these nodes could be done as an automated bulk import, so would be better aimed at some other infrastructure to add them to a ‘needs review’ list (roulette etc.?).
My goal is to try and import the >4000 OS trigpoint nodes that appear to have no current representation in OSM, which would hopefully more than double the current OSM trigpoint representation and get us ~2/3rds of the OS trigpoints in OSM. After that we can consider what to do with the remaining ~1500 OS nodes not imported and if the ~750 of those or so can be auto-merged, maybe…
Ah, that’s great - sorry that is spelt out on the wiki page, but I’d missed it (or it hadn’t properly sunk in) while I was reading everything else there.
Yes, but we’ll also potentially preventing the addition of duplicate data, and precenting the loss of a convenient motivation to detect and correct the bad data. (It could be argued that a single trig-point 100m off the correct location is better than one in the right place and a duplicate that is 100m off. The second case has a duplicate and a positional error, whereas the first case has only a positional error.)
However if you are at the trig point in the first case you have a 0% chance of being in the right place whereas in the second you have a 50% chance and you can see where the correct place might be.
How densely spaced are OS survey points? That is, if you compare the OS data not against the OSM data, but against itself, how many false duplicates do you get at each distance threshold?
Hi @Carnildo . I think the simple answer is ‘not very’, relative to us trying to find matching OSM nodes anyhow. The closest pair of OS points are 250m apart. The farthest ~63km (I think that’ll be the single lonely trigpoint sat out on StKilda off the West coast of Scotland…). I generated a graph so we can get a visual on it. Peaking around 4km?
I’m not quite sure what you mean by ‘false duplicates’ - I’m guessing you mean how many other OS points are within distance (so, if we use the 150m model, then the answer would be none…). But, I have considered if we should use ‘half the distance to the nearest neighbour’, on the grounds of ‘which OS point is an OSM point nearest’ - but, tbh, for the majority of OS nodes that would mean we were looking at OSM nodes up to 2km away - which I don’t think really makes sense - if an OSM node is that far away I think we should just ignore it.
What I will do is write (finish) the code that does a search for OSM nodes outside the ‘snap distance’ that have matching names or refs, and see if that picks up any stray OSM nodes that look like they match but are way off base. From a first pass I might have to still put some distance cap on the name matching at least (like 10km or something), as it seems some names are common across the UK, and we are picking up some points from far far away… we’ll see.
Looks to me like a threshold of 100 meters (or 125) is good: if there’s an OSM survey point within that distance, it’s probably a duplicate of the point in the OS data. If you wanted to be conservative, a threshold of 250 meters wouldn’t have many points excluded by their neighbors, but if someone misplaced a survey point by 250 meters, something went badly wrong.
If you compare the data against itself, each point had better show up as a duplicate of itself – otherwise, there’s something wrong with your comparison method.
