Poor quality changes from TomTom/MapRoulette challenges

General talk
,
1

There has been discussion in the #oceania community recently around some of TomTom’s MapRoulette challenges largely decreasing the quality of OSM data. While I’m sure TomTom has good intentions, many of the tasks being created are not quick and easy fixes. Their tasks encourage inexperienced mappers to do the bare minimum to fix an issue, which is unhelpful when a problem is part of a larger, more complicated issue or an edge case that is not immediately obvious. Promoting such tasks in a gamified platform that attracts new mappers and encourages them to solve as many as possible, even explicitly stating mappers should not skip tasks that they may not know how to solve, is unfair on the OSM community at large.

image

These challenges are actively damaging OSM data, and TomTom is not fixing issues that have been raised. In the #oceania OSM chat, we’ve asked TomTom to stop publishing MapRoulette challenges without community review, but after their recent Asia Intersecting Buildings challenges somehow included Australia, it’s become clear that there is no one doing basic checks on generated tasks before uploading them for others to “fix”, and many of the mappers attempting to solve these problems are only leaving behind a bigger mess.

There are issues with the tasks themselves - some of the challenges TomTom create are questionable, confusing/poorly explained, or outright incorrect. It’s great that TomTom has these QA tools and are trying to improve OSM with them, but there is no community consultation before challenges go live, and TomTom appears to be doing very little/no due-diligence in checking their work.

For example, the Asia (+Australia?) Intersecting Buildings challenges created yesterday were generated based on out-of-date data, that appears to be up to 3 months old!!

I have previously asked TomTom questions on their processes/tasks, and have not heard back, such as what was wrong with these highways that were incorrectly marked as duplicate, or whether the spiky building tasks they create will remember if a task is marked as not fixable.

Inexperienced mappers do the bare minimum to solve these tasks, often leaving a bigger mess behind - in fairness to TomTom, this arguably a larger issue compared to what is mentioned above. When reviewing changes from MapRoulette tasks, I’ve come across a number of instances where mappers fix flagged issues by doing the bare minimum, overall having a negative impact on the map:

  • mappers separating buildings that are physically touching, but overlapping due to a tiny geometry issue
  • mappers moving buildings apart to try and fix them intersecting, but ignore iD warnings that their changes have created a new overlapping building :upside_down_face:
  • instead of marking a building that physically has a spike with “cannot fix”, mappers incorrectly remove the spike from the building geometry or remove the building all together.
  • instead of removing a building that is duplicated on top of another, they move it so the buildings are now not intersecting

image
image558×569 62.3 KB
image
image571×327 66.1 KB

These issues are common in TomTom’s MapRoulette challenges, and mappers that make such incorrect changes often do so in many small changesets across many parts of the map, making them difficult to review, fix or revert. They also make many changes in such a short amount of time that other mappers don’t get a chance to let them know how to fix their changes before they’ve made a significant amount of them.

What are other mappers supposed to do when a new account has made 100+ changesets that are mostly poor quality/incorrect? Revert all their changes, discouraging them from mapping? Try to fix up their changes, which can easily take up 10x the amount of time it took for them to make theirs?

Why are new users even being introduced to complicated tasks they clearly don’t know how to solve, and why are they then told to avoid skipping tasks they’re unsure of?

When someone gets something wrong with these challenges, it’s time consuming to fix, it’s not constrained to one area, it’s often in quick bursts, and it’s often multiple users all at once. I don’t expect new mappers to get everything right, but it’s unfair on existing mappers that TomTom/MapRoulette encourage these users to practically carry out undiscussed organised edits of such poor quality.

This isn’t benefiting the map or the community, and it’s frustrating that these challenges continue to go on without changes and actively damage the map. I don’t know what the ideal solution is here, but continuing to chase an ever growing number of edits while harming data quality overall is simply unsustainable.

13 Likes
2

While in Nederland it’s not that bad, I still recognize many of your issues with TomTom Maproulette challenges, in Nederland as well as in other countries. The challenges can be helpful, but they need to be reviewed by experienced mappers, then adapted to the comments and suggestions, then published anew.

1 Like
3

TomTom is announcing new challenges here in the Community Forum with the tag tomtom. But I don’t see any announcements for Oceania? Are they announced elsewhere?

Have you tried to report the challenges, e.g. when using outdated source data? There’s supposed to be a process where bad challenges are hidden from MapRoulette users, to have some time for discussions and fixing them.

Where is this “#oceania community”? Is this IRC, Slack, Discord?

4

They are announced both on the forum and Discord, this is the most recently announced post.

Yes, they have taken some down in the past when there were issues. With Australia incorrectly being included in the Asia building challenges, a TomTom employee had acknowledged that a month ago but it was not fixed. That, combined with publishing out of date data, is a significant issue in my opinion.

On Discord (discord.gg/openstreetmap, you need to join with the “All regions” or “Oceania” role) and bridged to Matrix

5

“try do[sic] not use the “Skip” option” is an absolutely crazy thing to write in a challenge description

7 Likes
6

Hi Joel and OSM Oceania Community members,

Thank you for taking the time to write this up so thoroughly. I know it takes effort, and I much rather hear this feedback directly than not hear it at all.

The points raised are fair, and I will not be defensive about them. I respect the OSM community guidelines and the work that community does. Let me go through each point one by one.

On Australia appearing in an Asia challenge

You are right, and here’s what went wrong actua;ly.

We had OSM training planned on 26 September for another Asian country. For that training, we planned to use some existing tasks from an Asia challenge, and those tasks included locations from the Oceania region. Looking back, I understand why that caused concern. It should not have happened. Now I have already stopped Asia challenges. They were not intended as publicly promoted challenges, but reusing existing tasks was my decision, and that’s on me. Thank you for pointing it out.

We are also tightening our review process so that the geographic scope of every challenge is checked properly before anything goes live.

On community consultation

This is the point I think the confusion started and I want to clear the confusion. I would like to clarify that when I post challenge notifications in the Oceania community, the intention is to ask the community to review the challenges and the leads before any mapping starts. This review period is open for 15 days, and we genuinely want feedback during that time so we can make changes based on community input.

The OSM community has asked that challenges be shared before they go live, and that is the process we follow when I share the challenge with you all. Then we wait for 15days for community’s inputs.
Going forward, I’ll make that purpose clearer in my messages so everyone know they are being invited to review and comment on the challenges before any work begins.

I would also like to share upcoming challenges that affect Australia and New Zealand with the community before they go live, so any issues can be identified while they are still easy to fix. If there is a preferred place for that discussion, whether it’s this forum, the Oceania chat, or somewhere else, please let me know and I will surely use it.

On out-of-date data

Again, this happened because I reused existing challenges. That’s my responsibility. I have already shared this internally and I am taking down the Asia challenges, and we will make sure future challenges use fresh data.

On the unanswered questions

I apologise for this. Not responding to direct questions damages trust, and that’s not what I would want.

I’ll come back with specific answers on:

  1. The New Zealand highways that were incorrectly flagged as duplicates, including what triggered the detection and why it was wrong.
  2. Whether “cannot fix” statuses remain in place when a spiky-building challenge is regenerated.

Both the New Zealand and Australia challenges related to Highway have been paused and taken down for now as I mentioned on discord. I want to refresh the leads and make sure everything is up to date before sharing them again with the community. At the moment, they are not public and no work is being carried out on them.

On “please try not to skip”

That wording was intended to discourage people from skipping tasks just by not trying to understand the issue and skip it while it was a solvable case.

However, you are right that it can have the opposite effect and encourage someone to make an edit even when they don’t fully understand the task. That is not what we want.

I will revise that guidance so that skipping a task, or marking it as “can’t complete,” is clearly presented as the correct and responsible action whenever a mapper is unsure.

On task complexity and new mappers

This is an important one. Some of the issues we published are not quick fixes. They are often complex mapping situations that require local knowledge and careful judgement.

Presenting those tasks to someone who is new to mapping, is not fair to either the mapper or the community. We need to be much more selective about which tasks we include, and we need clearer guidance about when the right decision is simply to leave something unchanged.

On the clean-up burden

I have taken this up with validation team. Also MapCup issues from Aug and Sept. I understand it is not reasonable for the cost of our tooling mistakes to fall on volunteers. The examples you shared, such as connected buildings being separated, iD warnings being ignored, or genuine spikes being removed, are very useful because they show us where our process and instructions are failing.

If you can point me to specific changesets, I would like our team to help with the clean up rather than leaving that work entirely to the community.

Where this leaves us

I don’t expect one forum reply to solve everything.

However, I can commit to the following:

  • No new Oceania-focused challenges from us until we have discussed them with the community.
  • The “please try not to skip” wording will be updated.
  • I’ll come back with answers about the duplicate-highway detection and the “cannot fix” persistence question.

You mentioned that you don’t know what the ideal solution looks like. I am in the same position, but I will genuinely like to work through it with you all and the Oceania community rather than around it.

If you or others would be open to a conversation, in whatever format works best for you, I am ready and will be happy to make the time.

Thank you again, Joel. This input was important and helpful for us.

Kind regards, Kiran

5 Likes
7

That wording was intended to discourage people from skipping tasks just by not trying to understand the issue and skip it while it was a solvable case.

What’s bad about that? When I do armchair mapping, I remove myself at the slightest doubt of what’s actually going on. Saved myself from mapping canals in India as highways that way.

No new Oceania-focused challenges from us until we have discussed them with the community.

Are challenges elsewhere not problematic at the moment?

2 Likes
8

Apologies for the long reply from me but I wanted to make sure every point you raised got a proper answer rather than a general one.

1 Like
9

I completely agree and that is why we are taking that wording out.

10

This challenge contained a variety of issues related to road. Some were helpful and valuable to the community, while others were not. I know that similar feedback is being shared in other communities too, and we are discussing it internally, taking actions so we can make improvements going forward.

11

Hi Kiran, thank you for typing out this reply. I can understand that the out-of-date data was a one off accident.

Not to be rude, but I would like to point out that (to my knowledge) this has not happened yet (mainly because there haven’t been new challenges yet since that was discussed). Previous challenges have been created as public/discoverable, which is how these new mappers are finding them. I understand that in future they will be created as undiscoverable to prioritise local mappers reviewing issues first.

I didn’t mean to point this out in my post as a continual problem, but more to highlight that this may be a good idea to introduce this in other regions too as it I don’t believe these quality issues are limited to Australia.

Well, there are at least 3.5k of them created in the past 24-48h. Part of the issue with these challenges is that once created it’s near impossible to review and cleanup without unfairly reverting users changes en-masse.

The ones in Australia are probably easier to review as aerial imagery quality can vary significantly between regions.

12

Absolutely, I can do that from now on. 100%. I will keep the challenges undiscoverable during the review period, we will discuss together, I will address feedbacks, and once everything looks good, I can make them discoverable.

13

I understand what you mean, with so many changesets already created, it is difficult, but let me assure you that, while it may take some time, we will review the cases. I can also check with our internal team to see if they can help identify problematic cases using their expertise.

Having the community review the challenges and leads before they become discoverable should make it much easier to catch these issues early. We will do this going forward Joel.

14

Thank you all for sharing your feedback and thoughts with us. Indeed, this challenge was created about two months ago, so it’s understandable that some tasks may now be outdated, and we do apologize for this! We have paused the challenge and are conducting quality checks on the edits made through it.

We appreciate your patience as we work to address these issues and improve our processes going forward!

Hajar from TomTom

2 Likes