hi, i need a script to add Wikidata id to OSM entries.
i have a list of Wikidata entities that have a osm id such as a node, way and relation id and i sure they are correct, so i just need add them automatically to OSM.
For context for the benefit of everyone else, an identical question was originally mentioned on Telegram. I replied there (lightly edited):
Your wikidata entries may be entirely accurate and on-message for OSM, but people have added all kinds of stuff in that field before. Unfortunately, if you “just need a script” you may not have a plan to deal with data collation issues.
and also said:
If I was doing that I’d likely do that in a multi-stage process in Josm:
Robustly identify the data to change with an overpass query, check for any unexpected duplication in that data elsewhere in OSM, download data in Josm.
I’d probably use an editor macro to add my data to a .osc file, and then upload via Josm. That’d warn if someone has ninja’d a change to the data while I was working on it.
But before doing all that I’d read the import guidelines and post to the forum with e.g. the sources used and the procedure proposed, so that there are no surprises.
Thanks for posting here; the import guidelines are also worth reading, even if parts won’t apply to you.
8 Likes
how you matched them?
can you upload this dataset so its quality can be checked?
I would highly recommend using a service such as Sophox, which can simultaneously work with OSM and Wikidata. As a minimum, you would want to check that the name in OSM and Wikidata matched.
I would be very suspicious of OSM node and way data in Wikidata, since OSM ids are not stable. Relations are less likely to rot.
Conflation may be non-trivial. Depending on the nature of the objects, there are going to be cases where there is an existing OSM wikidata tag that doesn’t match what is in Wikidata.
Can you elaborate on how you matched these element IDs with QIDs? As mentioned earlier, if you only pulled Wikidata’s statements about OSM element IDs, then the data may need further cleanup. On the other hand, if you matched the IDs in a spreadsheet by hand or relied on a tool like Open Refine to apply heuristics based on various criteria, then it would be helpful to document your process a little more. Then we can offer some advice on getting it into OSM.
By the way, there are streamlined tools for linking OSM and Wikidata, so you don’t have to do as much manually and can take advantage of some built-in heuristics. You’d still have to apply common sense, however.
2 Likes
is that possible to update the osm objects vis sophox ?
i did it manually by my self and now i just want to add QIDs.
those linkers are didn’t get a list to match them by statement that has in wikidata.
i did it manually. this is the list : https://w.wiki/Cakg
If you’re sure these statements from Wikidata are accurate, Sophox can perform a federated query with Wikidata and has a mode and a JOSM plugin for making edits to OSM based on query results, but you still have to review each change manually. A more automated workflow could require exporting a CSV and doing some text processing to load the list of OSM element IDs into Level0, where you can do further text processing to add the wikidata=* tags. But for this relatively small number of features, you might as well edit the data manually, so you can spot any problems in the source data.
i need some more help.
This Sophox query returns the same results but with actual OSM elements instead of their raw IDs. You should be able to adapt this query so that Sophox presents an editor. Unfortunately, the Sophox editor appears to be broken at the moment; I’m unable to log in.
Alternatively, you could try the JOSM plugin or the Level0 approach I described. You’d need to export a CSV and use a text editor or spreadsheet application to turn the first column into something like “n123,w456,r789”, where these numbers are node, way, and relation IDs, respectively. Then you can paste that list into the textbox at the top of Level0 and click “Add to editor”. You’ll get a lot of Level0L code; add the wikidata = … tag to each element, either by hand or automatically using a local script.
Given that it’s only 82 elements, it might be easier to open each element in your favorite editor and add the wikidata=* tag manually. In iD, you can jump to any element by entering the ID into the search box. I recommend taking this approach, because all you did was to ask Wikidata for the element IDs without any due diligence.
if you have just several of matches (82 listed there) it would be faster to edit it manually in online browser than figuring out how to add it automatically, and that would give you a greater chance to spot problems
as a programmer, if I would be adding wikidata ids I would bother with some special setup if I would have 100-150 entries at least, the same would go for learning how to use a new tool
see xkcd: Automation
4 Likes
thank for notice that but yes it is 82 items but i know there is a lot of other item that has the same statement that here is no wikidata and no any other information so, this 82 items is just one project I’m struggling with it and for sure there would be a lot a things that i need a automation. for example I spend 2 hour to add all colleges that this university has around country:
i use osm data and queried it and find 2500 items then i use fuzzy search to find related items then it was about 110 items then i use this tool:
https://quickstatements.toolforge.org/#/
to automate all editions
this tool is grate and it would more greater if osm clone it for it self and use it.
i check the level0. that seems good. after several tries i can do some changes. so this is final questions.
what is the data structure format of level0 editor ?
how can i convert other types of data types to it directly ?
It’s a custom format based on the OSM data model. There are no converters to this format. If you need to convert from other formats without doing much text processing on your own, you probably need to use a combination of QGIS and JOSM with the OpenData plugin installed.
Technically speaking, Level0 is our equivalent to QuickStatements, just with a different format. (Also, Level0 is much faster and more reliable than QuickStatements…) However, there’s a key policy difference between Wikidata and OSM: Wikidata is very open to mass automated or semiautomated batch edits, whereas OSM is much more conservative about automation.
The edits you’re making fall into a gray area: Wikidata is already linking to these OSM elements in one direction, so all you’re doing is making reciprocal links, trusting the original links. However, the OSM community would normally expect you to do more due diligence around batch edits, especially if you touch other tags that describe the feature itself instead of some metadata about it.
The other day, a fellow mapper had to undo over 1,500 incorrect “official name” statements that someone had copied from the Russian Wikipedia to Wikidata using QuickStatements. (The inaccuracies remain on Wikipedia.) Errors of this magnitude would probably get you in trouble in OSM, so it’s a good idea to discuss your ideas beforehand with the Iranian mapping community.
3 Likes
thank you. it helps and for now i just would struggle with level0.