Hi All,
I am currently looking to import MS buildings + address into the Bexar county area. Currently I am trying to get a plan on how to do this so I have not written a wiki page yet. So far this is my plan:
Update existing osm buildings with address points that intersect them/are the closest point to a building that has no tags (if there is a conflict manually review data for conflicting info)
Import all ms buildings that do not intersect any existing osm data (areas/lines) and add addresses to them like above.
Since there is so much data I am currently working with postgis to do the updating/merging. I am using the snapshot schema in my db since it is lossless. I did see user data is missing (possible from datasource?) is this necessary to update objects on the osm side? If so what other data is needed for updating an object?
Any suggestions to how this should be done would be appreciated.
I did see that, the issue is that San Antonio has about 600k addresses. I got the suggestion to add addresses separately from the buildings as different imports as it would probably take quite a bit to get the data where it needs to be for an import
The Microsoft Buildings dataset doesn’t contain any user-contributed data from OSM, so you’ll need to merge the datasets, for example using the conflation plugin mentioned above.
Imports commonly perform these two steps in the opposite order. One advantage is that, if you discover that a building no longer exists in reality, you don’t have to review it twice. Another is that you can use process of elimination when conflating with the existing buildings. On the other hand, your proposed approach would avoid the case where an existing OSM building incorrectly intersects with a building in the Microsoft dataset, preventing it from getting imported at all.
So I was referring to the osm file I downloaded from geofabrik.de (which has all data for Texas). Loading the data into a DB locally in postgis, I saw all the user ids were 0 and the user table is blank.
Also while looking at the data I saw the some of ms buildings intersected. I was thinking of resolving this programmatically where I would either keep the larger of the two objects or try to merge them somehow (haven’t gotten that far). Maybe for conflicting data, something like tasking manger can be used instead of trying to merge things that have conflicts. Maybe keep the larger object but tag it as suspect?
The two most common tools are QGIS (some with custom python) and JOSM (lots of plugins). If you haven’t seen the import catalog, it’s worth reading through some things there to get a sense of what the “state of the art” is. Import/Catalogue - OpenStreetMap Wiki
OK, let’s try this again, without remnants of a very old discarded draft post…
Geofabrik omits any user identifiers from its publicly downloadable files to avoid running into problems under the GDPR. If you really need user names and IDs, it’s available to logged-in OSM users. However, I’m not sure it would be useful to your import. If you intend to verify in step 2 that it isn’t a building you’ve already touched in step 1 by checking the last-changed user ID, it would introduce a reasonably realistic race condition in which someone else edits the building in the meantime.
Oh, intersections between two buildings in the Microsoft dataset? I’ve seen this happen before but didn’t hear back from the RapiD/MapWithAI team when I reported the issue. Hopefully it’s very rare.
PostGIS to convert to OSM and split into “tasks”, and add address information to buildings
Additional processing with custom Python program to deal with multipart buildings. Your input data probably doesn’t have multipart buildings.
Tasks where there were zero existing OSM buildings were uploaded automatically at this point.
Created tasking manager project with one task per file generated in step 1.
Osm mapper would work on one task at a time using the tasking manager and JOSM. If there were duplicate buildings between input data and OSM, they would be handled with the replace geometry function in JOSM. This preserved the history of these objects.
I don’t think that there is a tool to do the conflation programmatically and preserve the history of the object at the same time. But you could go the route you suggested and upload first all buildings that don’t overlap with exiting OSM data and solve the conflicting manually in JOSM.
I personally would do this:
Use this Overpass query inside JOSM to download all existing buildings and save them as GeoJson (You will need OpenData plugin if I am not mistaken).
Load them, as well as your polished import dataset into GeoPandas.
Filter out buildings that overlap between the two datasets.
Use upload script to upload the non-conflicting new buildings. (I’ve never used this script, but worst case you can use JOSM)
So I noticed when I save the file as a geojson, it loses the metadata (osmid, user, timestamp etc). Is this how you edited existing osm buildings or did you import the data into python a different way?
I always modified existing OSM data in JOSM, because I couldn’t find a way how to do a lossless conversion between OSM format and shapely geometry format.
But if you only want to update tags you could make it work with some of the existing packages.
I don’t know how compatible their data formats are so you might need to write your own data transformation between overpass → osm api datastructures.
If there is going to be any modification of existing OSM data I would highly recommend doing so in JOSM as @Mashin suggests. This is what the Denver building import did. @SherbetS had a good suggestion as to how to deal with large amounts of data in JOSM. In addition to losing the necessary metadata, if too much time elapses (e.g. more than an hour) between when the data is extracted from OSM and when you upload changes, you run the risk of conflicts (because someone else has modified the data after it was extracted from OSM).
