MapRoulette Critique

General talk
, ,
1

I’ve been calling tools like Maproulette “stochastic organized editing”. They tell you in words to pay attention to local mapping and verify and be careful, but in their design they tell you “just quickly check if what we tell you is right (it probably is)”.

As an example, a while back I had to correct some Maproulette “organize brands” users who retagged some Popeye’s in Canada as fast food (example). These were supplements stores and tagged as such. Should the tool’s users have gone in a region where they don’t know the stores and the brands, and accepted the tool’s recommendations more or less blindly? No. Did they? Empirically, yes.

I understand Rapid does basically the same thing. But it’s an editor rather than a tool like Maproulette, so at least we should not have a link to it on the front page of OSM.org.

6 Likes
2

Another difference between MapRoulette and an undirected, standalone editor is that MapRoulette comes with some elements of gamification (the leaderboard, progress meters, etc.). If hypothetically someone were to port StreetComplete to the Web via Kotlin Multiplatform, we might see a similar phenomenon of overzealous retagging because of its gamification features, but without its inherent bias against armchair mapping. Even iD is somewhat gamified because, unfortunately, How Did You Contribute shames mappers based on the rather simplistic metric of their net addition of iD validator warnings. Any validator’s warnings require thoughtfulness.

At least on paper, MapRoulette’s project owners are already subject to community guidelines about organized editing and imports, when applicable. MapRoulette has a responsibility to make sure we can trace an edit back to the project that suggested it and hold the project owner accountable if necessary. By analogy, if Rapid were to add a feature that allows anyone to set up a custom Rapid Assist dataset and share it with other mappers, the community would quickly develop some expectations about how mappers can use it. But for now that’s just as hypothetical as StreetComplete Armchair Edition™.

3

I am the project lead for MapRoulette.

The changeset your are pointing to is 5 years old. MapRoulette has put a lot of safeguards in place since then to make those who create the tasks more a accountable, including but not limited to:

  • requiring Challenge authors to have a valid email address on file with MapRoulette, so that any relevant notifications (users reporting or commenting challenges for example) are much more likely to be seen by them
  • Users can leave feedback on challenges within the app and engage in a conversation with the author
  • Users can flag a challenge (see below) if they strongly feel that a challenge encourages bad edits. This will not immediately remove it from the publicly visible challenges, but it will start a process in which the author will need to engage.
  • Challenge authors have to acknowledge that they understand the OSM automated edits code of conduct and and accept the responsibilities imposed by that code (see below)
  • Every changeset has a backlink to the original task that makes it very easy to trace a changeset you are looking at back to a specific MapRoulette task

Now, none of this is a guarantee that MapRoulette will not allow questionable edits to happen, but we have gone through a lot of effort and discussion with the community to ensure that we do what we can to be a good OSM citizen.

If you have questions, I encourage you to join one of the monthly MapRoulette community meetings, which are posted on OSMCal.

image
image1804×270 47.2 KB

image
image2980×888 145 KB

13 Likes
4

(Sorry for dragging this further offtopic - if a mod wants, maybe Martijn’s and my post can be broken out?)

Hello Martijn,

I’m glad to hear that you take the OpenStreetMap community seriously.

For transparency for others, my past problems with MapRoulette are partially documented in Github issue #791 and other issues linked from there. I created #791 in 2019 and it sat as “to be considered” until March 2023 when you thought that a comment/“feedback” field for challenges was not necessary and that flagging a challenge was sufficient, and believed that “[MapRoulette team] just don’t currently have the personnel to review every issue with a challenge that comes up in the community.”

I want to be clear, I appreciate the changes you’ve made recently. The link to the task in changeset comment is particularly appreciated. I just wish you were also clear that things were not so good in the recent past.

But the problem is also in making those who fulfill the tasks accountable. The example I gave was of a user editing in a country they’re not familiar with. (My other favourite example is that of MapRoulette users “resolving” intersections of roads and ditches or small streams by mapping them as fords, whereas anyone with a passing familiarity with the region knows they’re likely culverts.)

Because the point of MapRoulette is to map rapidly, often by the point we notice something, remote mappers have made 20+ incorrect edits across a region. Then we’d have to either find all the changesets, or try to contact the challenge author (both much more difficult when the backlink to challenge wasn’t there), but then the author isn’t the person who made the mistakes so it’s not really their fault, it’s not MapRoulette’s fault either and the MapRoulette team doesn’t have the personnel to review every issue with a challenge that comes up, and chasing down the individual editors is a lot of work, and a lot of times they don’t respond on changeset comments and…

the end result is you contributed to breaking it, and we have to either fix it manually or it stays incorrect.

You can understand how that makes me feel.

MapRoulette is a way to co-ordinate edits and gamifies doing so in order to fulfill the task as quickly as possible. iD presets are a way to make edits rapidly. Not surprisingly, users end up making edits rapidly.

You can say that OSM is self-governing and it is true that random people can and do rapidly sign up and do damage with any editor - but MapRoulette is one of the tools that encourage it.

As I see it, many MapRoulette challenges ended up being: “we know how this should be mapped - map it for us”. That to me is organized editing. It is good that you are now taking your responsibilities caused by this seriously.

To end on a conciliatory note. Could you assist me in creating and running a challenge to review and correct ford=yes tags on car highways and sidewalks in southern Ontario? Many of them are still wrong, and a co-ordinated effort using MapRoulette might be just the thing to correct them :slight_smile:

4 Likes
5

The above posts have been split off Into this thread at request of a community member. Original thread here.

2 Likes
6

@Jarek I appreciate your thoughtful reflection. Let me add my own :slight_smile:

You home in on the right, and recurring, question of where the responsibility for questionable edits ultimately lies. Is it the sometimes confusing and ever changing taxonomy (“folksonomy”) of OpenStreetMap? Is it the tools? Is it the mapper who made the edit?

In the end they are all sides of the same coin. It’s the same community that together comes up with the taxonomy, the tools, and the map edits. We, as the OSM community, implicitly decide which tools make the most sense for us to make the map, by choosing to use them, or not use them. MapRoulette has, for better or worse, become part of the OSM ecosystem.

Whether it’s JOSM, iD, StreetComplete or any other tool, it is opinionated about how people should map: tag presets, choice of quests / challenges, UX decisions. MapRoulette is no different. I designed it to add a level of focus to mapping, by letting mappers cycle through similar tasks. There was a clear need for a tool like that in 2012, when I released the first version, and there is still a clear need for a tool like MapRoulette today. Around 60k changesets each month are made by MapRoulette users.

Now, mappers can go about making map contributions thoughtfully, or indeed they can go about it quickly and carelessly. Whichever way you design a tool, in the end you cannot prevent individual contributors being careless. MapRoulette nor any other tool can have a surefire way of holding mappers accountable pro-actively. The vast majority of map contributions made by MapRoulette users are beneficial to the quality of the map, but there are bound to be situations where it doesn’t work out well.

Now, if we can accept that MapRoulette is not going away, practically, what can we do? I would love it if more people stepped up and asked me ‘how can I help with challenge arbitrage?’ but no single person has ever asked me that :slight_smile: . I am very open to suggestions for how to improve the challenge reporting system, so if you want to re-open that ticket, or open a new one, and continue that discussion, I would totally welcome that. Ensuring that MapRoulette leads to good map contributions is always a top priority, so if you feel like I have been dismissive of your concerns in the past, please accept my apologies.

12 Likes
7

If the main issue is mapping carelessly (too fast), maybe a timer could be added before subsequent tasks can be fixed?
Such a timer could be a parameter chosen by the challenge creator according to the time estimated necessary to map “properly”.

8

Don’t disagree at all!, but what’s “too fast”? :thinking:

9

Throttling is hard to get right, and would kill part of the gamification. But hey, what’s important, having dedicated and thoughtfull mappers editing the map, isn’t it?
Once a value is chosen, feedback can help refine it.

10

For start: do not use or promote its use for situation where mapping requires local verification or local knowledge. Document that mappers using it are typically armchair mappers quickly going through tasks.

Numerous times I got recommendation to use MR in cases where things required survey or careful case by case checking. MR is unsuitable for such tasks and would be great if MR itself would clearly document this so it will not be recommended and used in such cases.

MR is especially bad when quick fix is involved - when user can blindly press “next, apply edit” then sooner or later you get someone doing this.

Throwing away point and leaderboards may help with mappers who do no checks whatsoever and mechanically go through tasks and blindly click. But it would be quite radical.

Or find another way of limiting bad effects here? Maybe reward editing each day in addition/instead of processed task count?

Maybe introduce way of reporting users who make bad edits with MR and if it repeats remove their points?

There are some users who repeatedly make really poor edits in MR.

Introduce some effect or review of flagged challenges? Currently if challenge is flagged in results in basically nothing if creator ignores it and there is no followup on reported challenges. Hiding/stopping flagged challenge until resolved? Ability to actually report/flag is as a problem?

Note that flagging is supposed to be used after getting in touch with challenge creator that failed/was ignored - but flagging challenge is merely getting in touch with challenge creator again, which can be freely ignored by them.

6 Likes
11

I’m not familiar enough with MR, so sorry if my question sound dumb: is this involve an direct edit via MR without opening any editor?

12

it happens within MR (it shows you tags and location)

you can easily edit several objects within minute

and while for some forms of editing it may make sense, when you combine it “open to anyone” and “you get leaderboard points for each edit” and “anyone can create such challenge”…

13

Agree this feature is a bit to easy to misuse.

14

AFAIK MapRoulette, when the user taps Edit, calls the external editor of choice, and when done the mapper indicates that fact to MR.

15

not in quick fix mode, where challenge can be configured to quickly modify tasks

16

For example in MapRoulette default view does not even show tags to be applied. See

screen03
screen031895×831 278 KB

On pressing yes (or using keyboard shorcut) you get

screen04
screen041196×738 123 KB

without ever seeing what edit will be performed. On pressing submit (or another keyboard shortcut) edits would be saved in OSM. In practice, this is a bot edit as sooner or later someone will go through tasks accepting them blindly. And there is no effective way to ban misbehaving mappers and bad challenges from MR.

Note: some proposed edits in that challenge have quick edit disabled.

6 Likes
17

So I guess next suggestion is to rearrange interface so quick fix mode will at least show performed edit by default?

And maybe add ability to add trap tasks where it adds also tag surface=I am not looking at tag lists and editing blindly or similar - and accepting such edit would automatically block user from using MR for some time rather than perform edit?

4 Likes
18

Doesn’t the "proposed OSM tag changes’ section show the performed edit?

19

The problem is that that requires the actual author to be engaged in the process; many are not. With a DWG hat on I’ve semi-regularly encountered MapRoulette challenges that are basically “change tag X to tag Y”. That’s a clear violation of the way that these things are supposed to be done. Maproulette has taken the “guns don’t kill people” defence and made it difficult to communicate. As an example, to investigate a problem edit by a problem mapper I’m supposed to “discuss” via this challenge, but there’s literally no way for me to do that without signing up for a third-party site.

Edit: To expand a bit more about that particular challenge, “odd” layers are clearly a very valid thing to search for and fix in OSM. What’s not helpful is that this supposedly “easy” challenge gives no guidance beyond “This challenge focuses on layer=* tags that aren’t a simple number between -7 and 7. These values are often wrong and need to be reviewed.”. There’s nothing to say how someone new to this “easy” challenge could tell whether a layer is valid or not, or even what layer tags are for.

Two levels of QA have failed here - firstly @Friendly_Ghost hasn’t looked at problems resulting from the challenge that they have created and secondly @mvexel clearly hasn’t vetted this “easy” challenge before it was added to the list. Thankfully in this case the local community did spot the error and reverted the problem edit.

3 Likes
20

Could you express in some more detail what you mean by “does not even show tags to be applied.”?
(The screenshots show a challenge that I created)
The first screenshot does show the exact edit that will be made, I would say.

1 Like