Test Drive - AI-assisted road import by Facebook

Communities ประเทศไทย (Thailand)
1

Hello Thailand OSM Community,

I am really glad to hear about the outcome of your meeting. We have the same goals and I’m happy to work with the community to make OSM better with high quality mapping. I acknowledge your feelings about communication and appreciate you understanding my position, but rest assured that we plan to continue openly sharing our work through forums and lists.

For those not on the import wiki where many questions about process have been asked I have added the questions and answers to our wiki page to make it easier to follow :slight_smile:

For the test run we are happy to do the following:

Define an area in Thailand where you plan to import. This area has to be communicated to us and agreed on beforehand.

We plan to start importing from the provinces in the south of Thailand. As mentioned previously in the import list we have created some sample data for Yala to share. To make it easy I have pasted it below.

  • Phase 1 - Generating Road Masks. These are the masks of the initial output from machine learning. These are not processed to a vector format. We have offered to share our tile service to anyone. Sample 1 | Sample 2 |Sample 3 | Sample 4

  • Phase 2 - Creating Road Vectors. This is the processed road vectors merged with current OSM data locally -These .osm files that have NOT been edited or validated by people and would not be uploaded as is. Please download the package here.

  • **Phase 3 - Human Validation. **This is the final output validated by mappers twice in order to be uploaded to OSM. Here are screenshots with the machine generated roads highlighted for human mappers to validate:
    Sample 1 | Sample 2 | Sample 3 | Sample 4

We also ask that you choose an area having high-resolution Bing imagery available. As we don’t get access to the DigitalGlobe imagery you are using we want to have a way to verify the accuracy of your work.

  • Luckily it’s only a matter of weeks before DG refreshes the imagery as they mentioned :slight_smile:

  • But in the mean time we think Songkhla could work since Bing is good. iD-Songkhla, Using JOSM - Bounding box for Songkhla (minlat=“7.0751953” minlon=“100.546875” maxlat=“7.1630859” maxlon=“100.6347656”)

We ask you to restrict the import to higher road classes. So don’t import agricultural and rice-field tracks.

  • We can do this, but I humbly ask you to consider letting us give you everything our machine generates before we add it to OSM? This way you can see exactly what the ML is picking up.

  • This will give you a much more accurate view of what we would like to do going forward. Going through and deleting roads our machine generates would be fairly easy for us, but it is time consuming and counter productive and you might just see the quality is high enough to accept.

  • From my experience with OSM communities agricultural roads are least likely to get drawn so I think this is where Facebook could potentially help. Having the road network was by far one of the most helpful features to give people context of where they are mapping both for remote mapathons and field mapping.

  • If after the sample you feel differently I am happy to re-visit deleting agricultural roads.

We also ask you to only import roads where your algorithm has a high confidence level of the geometry and classification.

  • Yes, I totally agree with you here. We only add roads with a high confidence level. While we do have algorithmic help for tagging, our editors manually check each edit for quality in geometry and classification. We have done a great deal of of research and training, and have discussions for each task to make sure we are uniformly mapping and tagging. In regards to tagging we will ultimately rely on the community to guide us.

  • Using DG’s Vivid+ imagery and have found we can get a 30% increase in road coverage for Thailand. Because imagery in OSM is much older in most areas we are able to pick newer roads with this imagery. Here are some examples of places we have been able to get road masks for in Thailand compared with OSM’s current imagery. Image 1, Image 2, Image 3, Image 4

Here is an example of how we plan divide the tasks for the country. The colors indicates the current density of roads going from blue to red for high density areas. Of the 77 province boundaries we plan to start in the Southern Region in the province of Yala. Image

Please let us know if you agree on the test area in Songkhla so we can prep the data for this area and share it with you before adding it to OSM.

After that I agree to wait for your feedback before proceeding to import directly to OSM and planning out a strategy for scaling up.

If there is another meeting I am happy to join or come in person as I would love the opportunity to meet the community in person :slight_smile:

Best,
Drishtie Patel on behalf of the OSM at Facebook Team

2

Hi,

Being an active contributor in the deep south, I’m very keen for you to get started with Yala, Pattani, Narathiwat and I know there are very few mappers down there.

I’ve glanced at the samples you’ve provided, seems to look ok, waiting to hear people’s ideas on how we move forward on this specific case.

Best regards
Mishari

3

A strong requirement was that we have proper imagery to verify the geometry of the detected roads. This is coming from the AI and is probably the thing the most difficult to fix if it is not up to our standards So this is something we have to carefully evaluate.

Yala does not have Hi-Res Bing imagery. Only Landsat images. Mapbox z17 imagery seems to be there. Compare the area west of the city center to see the difference.

As you are familiar with the area, please suggest a bounding box for Facebook to start with.

4

Hello All,

We have created some sample data for Songkhla where Bing is good. Using iD editor -Songkhla, Using JOSM - Bounding box for Songkhla (minlat=“7.0751953” minlon=“100.546875” maxlat=“7.1630859” maxlon=“100.6347656”)

Please take a look at our sample .osm files here and let us know what you think.

Thanks,
Drishtie Patel on behalf of the OSM at Facebook Team

5

Hello,

I had a look at tile67.

I opened the data in JOSM and moved the new data to a separate layer.

JOSM validator seems to be happy with it, so at least no obvious mistakes in the data.

The areas I have checked seems to have a proper geometry. As this likely depends on the underlying imagery it is up to your validators to spot bad geometry before importing the way into OSM. Here we obviously have easy to process imagery for your AI.

How certain are you regarding the positional accuracy of the imagery you are using?

For example around this coordinates:

7.139488943807362, 100.58336130581272

Google has Digital Globe 2017 imagery, which sounds like the image set you are using.

We can see that the road here is actually drawn south of the position. So was some offset applied to the position or do you have even another image set?

Bing Images quite agree with the digital globe imagery regarding the position of the roads. The north/south way on the very left is a bit off. This might result from differences in the orthorectification applied.
Bing imagery was taken March 2013.

Mapbox images of the same area show a significant offset of roughly 30m in an east/west direction.

The few existing GPS tracks in that area support the alignment of Bing/DG, but does your verification process includes a step to check for positional errors? You should add a verification step for each tile to ensure that the used imagery alignment is plausible.

Some of your highway=service roads could have additional tagging to clarify what kind of service road it is.

This one looks much like parking aisles:
7.136701699999995, 100.57025689999999

Also I am still not convinced how useful it is to have highway=track like these:
7.135251457220321, 100.58416959161377

I understand the “usefulness” is highly subjective, so waiting for feedback from Mishari as he said they want such ways for cycling.

Here you classified a road as “unclassified”:
7.126803812150891, 100.59080393516885

I might have tagged this as a highway=service as it only serves as a way to reach the National Library and maybe another building in the woods.
This is probably a general corner case of tagging and not specific to your import. But we will just see a higher quantity of these.

Similar here:
7.140992100000004, 100.56433730000002

highway=service around something looking like dormitories.

What is the opinion of others on such corner cases? Would it make sense to leave a note in the area saying this is a corner case and needs local survey? This is what a typical mapper probably would do.

Here around the aqua culture I would have used highway=track. If clearly visible adding a surface=paved tag.
7.161198515500401, 100.58023471075015

Here it might be an unpaved residential road. With all those houses and driveways certainly not agricultural.
7.157252703938363, 100.55417505654358

This highway=unclassified does not really interconnect villages or neighborhoods. Could be a residential if it is a housing estate. Otherwise service.
7.147121221968039, 100.56183787951564

Positive thing: You did not connect roads from housing estates with outside roads where there is obviously is a wall and validator warns about a nearly connection.
7.152133193273847, 100.58583235354041

Why is there a new node at the end of a way Johnny created? That one is not connected to the road network:
7.1475017, 100.5727969

In the past you had added surface tags to ways. Your sample data does not contain any. Is it planned to do this in the future?

So a lot of points. Mostly of the “minor” category. Think these edits are as good or bad as the average remote mapper.

Regarding tagging: Do we want to have import=yes tagged on all those ways? You should certainly refer to a wiki page with details of your import in the changeset comment of the upload. This should also clearly explain why you are using an “disallowed” source of DigitalGlobe. Having that source tag seems to be justified as the geometry might differ to the imagery available to the general mappers and avoid confusion.

Stephan

6

Another point: I strongly recommend to add the attribute upload=“never” to the element in your sample data to prevent accidental uploads while the community does the review.

7

Hi DrishT
Thank you for your and your team’s contributions to OSM. I looked at tile80 and was generally happy with the mapping. I did note that some of your edits were higher risk than I would have been happy with, that is they joined up visible road sections with obscured sections of road. For example at 7.0948845 100.6209229 you show a residential road passing between two buildings. My reading of the Bing image is that a footpath at best is there and perhaps no continuous path at all. I am curious how you made your decision. Do you have access to better images, is your AI considering factors that I haven’t thought of or is your process resulting in a higher risk of false positives than I am comfortable with?
Thanks
Tony

8

Hello Stephan and Tony,

Thank you so much for the feedback! We very much appreciate the time you took to provide us with comments and critique. As remote mappers we have taken a more careful approach when it comes to tagging and caution on the side of not adding extra tags to a road if we do not have on-the-ground verification. We hope that local contributors can help with situations that are ambiguous or difficult to map as remote mappers.

How certain are you regarding the positional accuracy of the imagery you are using?
To ensure our contribution is appropriately aligned with pre-existing OSM data we currently follow OSM guidelines; we adjust the position of the imagery by aligning it to pre-existing OSM data, as well as the available GPX tracks in the OSM database.

The debatable usefulness of certain ‘highway=track’ ways.
We agree. We hope that local contributions, and feedback like yours and Mishari’s, will help us to recognize the most useful tracks to include.
In this case (7.135251457220321, 100.58416959161377) we agree that the small tracks, which are not connected to the main roads, are likely not useful.

Some of your highway=service roads could have additional tagging to clarify what kind of service road it is.
We currently don’t apply additional tags to service roads. Since additional tags for service roads usually describe its use (parking_aisle, driveway, drive-through) we hope that the local community can contribute these details not available to us.

I might have tagged this as a highway=service as it only serves as a way to reach the National Library and maybe another building in the woods.
Thank you. We understand it’s better described with ‘highway=service’. For situations as specific as this, local context is difficult to find. We hope that local contributions can better qualify cases like this.

highway=service around something looking like dormitories
This road’s tag was chosen because the surrounding roads are all ‘highway=service’ and contained within a hospital area polygon.

Here around the aqua culture I would have used highway=track. If clearly visible adding a surface=paved tag.
Thank you for finding this! We agree. We would not add the surface tag in this case since it’s difficult to determine paved or unpaved from the imagery.

Here it might be an unpaved residential road. With all those houses and driveways certainly not agricultural.
Thank you for finding this! We agree.

This highway=unclassified does not really interconnect villages or neighborhoods. Could be residential if it is a housing estate. Otherwise service.
Thank you for finding this! We agree.

This feedback is incredibly valuable. We look forward to continuing to learn from your experience and local knowledge.

Best,
Drishtie

9

Hello OSM Thai Community,

Thank-you again for your feedback. After making those changes you suggested on tagging, we plan to start adding those to OSM in the next two weeks for Songkhla.

Best,
Drishtie

10

Hello Drishtie,

I am missing a bit feedback from other mappers, especially from the imports list. Did you cross post the announcement of the test data set there?

11

Hello Stephan,

Other than https://lists.openstreetmap.org/pipermail/imports/2017-April/thread.html. I have not cross posted this.

Thanks,
Drishtie

12

DELETED ORIGINAL MESSAGE.

Sorry, meant to show someone how to use the forum, and hit submit instead of preview.

13

Hello Drishtie,

it’s a bit sad to see such little feedback from others. Based on the example you had given it is reasonable that you started importing.

I ask you to plan for a little pause after you have imported let’s say 25 tiles. This gives us the chance and time to review your work.
As there had been minor issues it sounds reasonable to have a chance to spot them early and correct tagging style if needed.

Don’t forget that your team is mapping way faster than we have the chance to review. So bringing in too many chunks can later cause problems as we did not have the chance to spot systematic problems not observed in the initial samples.

Can you give some details on the planned progress of the import? Which timeline and which areas are planned? Have you worked ahead internally already and have a huge backlog waiting to be submitted at once?

Thank you,

Stephan

14

In reviewing some of the areas being discussed I came up with several questions.

1.)

I don’t see the need to add this sort of way either but more importantly, the track goes right through a couple of buildings. I would never assume something like this was real without a ground check.

2.)

Then, checking this example from Tony’s post and putting aside the question of whether or not it passes between those buildings, I’m wondering why the short way (Way: 491189478) was tagged as a path, a key whose usage is already contentious, when it looks just like other ways in the neighborhood that were tagged residential.

How was the decision to tag that way and the short one south of it (Way: 491189476) as highway=path? I would have made both of them either residential ways or a service ways, even living_street, but never a path unless I could verify my choice on the ground. The Wiki definition of path is:

Although I haven’t been to the area, I’m quite sure that people are driving on that way, at least with motorcycles, which would make the highway=path tag inappropriate.

I’ll try to look at some of the other examples in the next few days and provide feedback.

Best,
Dave

15

Maybe related to the statement of Digital Globe to make imagery for us as well. Not sure about coverage in Thailand. The map linked on the website has areas with only low resolution landsat coverage.

Would be great to know from Facebook if that is the same imagery they are using.

http://blog.digitalglobe.com/news/digitalglobe-satellite-imagery-launch-for-openstreetmap/

16

Hello Stephan and Dave,

Thank you for your feedback. I was hoping for more local feedback as well.

We completely agree with you on being careful to add small amounts of tiles so we can be sure everything is ok before we move on. We have completed 5 tiles last week and can stay below 25 per week for now. Keep in mind we are also running OSMCHA and Osmose for every single tile in addition to JOSM Validator just to be extra careful.

Here is an example of how we plan divide the tasks for the country. The colors indicates the current density of roads going from blue to red for high density areas. Of the 77 province boundaries we plan to start in the Southern Region in the area of Songkhla, same of the sample data we shared.

Here is a zoomed in version of the task we plan to complete over the next 3 weeks of May.

As for tagging, as mentioned before, we caution on the side of using a more generic tag and not adding extra tags to a road if we do not have on-the-ground verification. We hope that local contributors can help with situations that are ambiguous or difficult to map as remote mappers.

In the examples you mentioned Way: 491189478 and Way: 491189476 you can see in DG it does not go through any buildings. For Way: 491189476 because of the tree cover we could not see if it was a residential road, but can see there is definitely a way, so we decided to go with path until it can be verified on the ground that a car can actually pass. We hope that local contributors can help with situations that are ambiguous or difficult to map as remote mappers.



Lastly, the imagery you see now is not the same as what we are using. I have heard Thailand is on the roadmap to be added so you will see the same high resolution imagery we use by the end of the Month :slight_smile:

Thanks,
Drishtie

17

Isn’t this what highway=road is for? For cases where you don’t know what the classification is. “from the smallest footpath to the largest motorway” I suggest you use highway=road for these unsure things, highway=path means it’s a path. highway=road is a good way to highlight to other mappers “Hey, there’s something here you should check out”.

May hang off on any imports until this imagery is released?

18

Helo Rorym,

Thank-you for the feedback. We have only added 9 tiles of data and are moving slow to give the community time to check and hope to see new imagery up soon.

Also great advice on using highway=road for highways we are unsure about.

Best,
Drishtie

19

Thanks a lot for your efforts.
I have not scrutinized your work in detail, but only taken a quick look at the map if I find some strikingly strange items.
Way http://www.openstreetmap.org/way/497749494 looks wrong near its southern end. When you start from the minor road there, I think its first meters should be part of the parallel track on the other side of the irrigation canal, instead of completely parallel. Also: why do these parallel tracks end there? On the imagery available to us “common OSM users”, we can see that they extend farther south. On the other hand, I doubt that the track on the western side of the canal extends such far north.
Closer to the coast, several residential roads can be seen in the imagery which has not yet been mapped. Is that still waiting for import?
I hope I can help optimizing your AI project with these hints.
Have a lot of fun and success!

20

Hello Bernhard,

Thank you very much for looking through our edits. We appreciate the feedback. In the case you mentioned our the imagery seems to show both roads parallel to each other.

Our imagery

Current Imagery

Please note the DG Premium Layer should be updating soon so you will be seeing exactly what we see in a matter of days :slight_smile:

Thanks again for your help Bernhard!

Best,
Drishtie