How can OpenStreetMap better collaborate with businesses seeking to provide information such as opening hours?

woodpeck · 2024-10-08T11:15:23.475Z

I think it all boils down to cost. Of course a big company is allowed to add their data to OSM. Hire a person, train them properly, maybe have them write a small script that compares their own data to OSM’s, update manually and carefully where the discrepancies are. Or use one of the existing conflation tools. Liaise with the OSM community. This requires skilled labour. You can’t just dump a CSV file on OSM and think it will magically work, nor can you expect people in a sweatshop in a low-wage country whom you pay a pittance per edit to produce the quality that OSM is looking for. Enough examples of low-quality data have been mentioned in this thread. If your data is not good enough then that is hardly our problem, and whining on the Internet won’t make your data better.

amapanda_ᚐᚋᚐᚅᚇᚐ · 2024-10-08T11:19:50.161Z

Welcome to OSM @maptheworld2050. That post is … ahem … wildly mistaken. Sorry. That poster claims things that are … just not true.

You have data from Walmart with every Walmart address … OSM tells you that they need the store layout

Sorry, complete codswollop. Today in OSM there are half a million supermarkets (shop=supermarket). 66% of them are simple point features (“nodes”). Forget “store layout”, we don’t even know how big it is. We only have the opening hours for a third of the supermarkets. (7% of our supermarkets don’t even have a name!). And yet we are happy to keep this incomplete data.

amapanda_ᚐᚋᚐᚅᚇᚐ · 2024-10-08T11:28:22.173Z

SomeoneElse:

take the view “never mind the quality feel the width”

In Geomob Podcast #253, Overture’s Executive Director Marc Prioleau claimed that wildly misplaced points are still useful if you’re searching for a result, and that that incorrect data is not intended to be placed on a map.

It’s Facebook, they’d never lie, or retroactively make up excuses.

eisa01 · 2024-10-08T15:43:40.458Z

Thanks for bringing this up!
I sent an email to the HN commenter, just in case there’s more context to provide

POI coverage is probably the weakest spot in OSM, so always good to have a discussion to generate ideas to make it better

Tordanik · 2024-10-08T21:31:29.721Z

arctic-rocinante:

My take on the problem is that OSM just hasn’t reached the “critical mass” needed for it to be worthwhile for businesses to maintain their info in it.

I think the OP indicates that this wouldn’t prevent them from offering data to OSM if they could do so with the same kind of workflow they’re already using for everyone else. But Frederik’s post shows quite well that we ask for a lot more – learn bespoke tools, give special training to your workers, individually respond to concerns of our community members, …

From their point of view, we’re probably both less important and more demanding than our competitors. Which isn’t an ideal position to be in.

amapanda_ᚐᚋᚐᚅᚇᚐ:

Welcome to OSM @maptheworld2050. That post is … ahem … wildly mistaken. Sorry. That poster claims things that are … just not true.

To the contrary, I find that post quite plausible depending on whom in OSM they talked to.

If there’s a building or site with a single address containing multiple shops, many in the community would be unhappy with shops being dumped in some random location in the center or near the address node. Placing the nodes in roughly the correct location is not an uncommon expectation. And this expectation is what I assume is behind the “store layout” comment, especially as it fits well with something they state elsewhere (“Google will take your data and separately figure out how to map out where it is in the mall if it is important”).

Basically, their data only contains addresses, not coordinates, and while other platforms are happy to accept the former, OSM requires the latter.

osmuser63783 · 2024-10-08T22:18:22.371Z

Maybe we can improve the Wiki page on Introduction to OSM for Business Users with information from this thread. It’s one of the top Google results when searching for “OpenStreetMap business how to” or similar, so it could serve as a guide for such requests. The page could use some updating. For example it should probably mentioned the Automated Edits CoC, the Organised Editing Guidelines, and the community forum…

crodthauser · 2024-10-09T02:43:41.503Z

Here is my personal wishful thinking: OpenOpeningHours.org. Provide simple import tools for excel sheets, csv files etc. Importer checks for duplicates, doesn’t require precise geolocations. Each entry gets an ID. OSM can link to that ID, like we link to Wikidata. Opening hours would then not be required on OSM but could be sourced from OOH. Some algorithms can try to match the data on OOH with OSM nodes/areas and suggest links or that a shop may need to be added to OSM. If a link is established, the node/way/relation ID on OSM would be added to OOH. Also links to Google Maps, Wikidata etc. could be added. If a shop closes permanently, OSM QA tools could notify us that the OSM object needs to be updated; perhaps even some minor editing could be done automatically. More features are always possible, in which case the name OOH might not fit anymore. Just like OSM, so that’s OK.

JesseFTW · 2024-10-09T12:05:18.836Z

I think this is AllThePlaces, more or less. In any case, it’s certainly close – and describing the differences would be useful!

eisa01 · 2024-10-09T19:05:15.493Z

He was super kind and provided me a long answer with more context that I could share with you

My takeaway is that it would be helpful with better accessible tooling to match incomplete store information (i.e., not necessarily an accurate location) with existing OSM objects. Especially opening hours is impossible to maintain otherwise

Maybe that already exists, but at least this guy didn’t find it back then

I also think the Walmart data was updated by @whammo via AllThePlaces, so probably relevant for @CjMalone to see this thread too

You can repost this more detailed reply. Maybe there are people involved with OSM now who will realize that this actually is a problem.

I complained about the same thing in Here is my experience. I work for a company that syndicates location data for bu... | Hacker News, back when I still worked at Brandify. I’ve since moved on and the company was sold to someone else, but they still are performing the same job. The issues are exactly what I said they were. After this many years, I do not remember exactly who I dealt with. But I know I tried to make edits, read feedback on data rejections, and we wound up hiring one of the OSM developers on a contract, before we abandoned the project.

Here is some context.

The syndication part of the business was what I described. Here is how it worked. Every business we worked with would come up with their own custom format for spreadsheets to send us. We developed basic imports for each based on their format. We then mapped that information to the various output formats for different mapping companies. Each mapping company had their own format, and often two APIs. One for a bulk upload, one for a per business modification. We worked out all of that logic. After that, each month the business would send up a new spreadsheet, including store openings, closings, holiday hours, and so on. We’d process that, and publish. We literally have no access to any information except what’s on the spreadsheet. Which is usually far less information than OSM wants.

The actual mapping companies all deal with complex data merges. As a verified publisher for a verified business, our data on store hours, phone numbers, and so on would be considered definitive. Actual geographical layouts though? Generally they consider their data more reliable. It usually is - businesses told us our addresses, but not an accurate geolocation. Their merges also generally successfully dealt with some more complex issues. For example streets may have a concurrency, That’s where, for example, State Rt X also has a local name for the same stretch of road. Or a city may go under multiple names. For instance the Post Office will deliver to my house whether you call my city Lake Forest, or El Toro. Mapping companies figure this out.

That was my situation, when I was asked to implement OSM for two clients. A restaurant whose name I forget, and Walmart.

My first attempt was to read up on OSM’s API. After running into issues, we hired an OSM developer as a contractor, and I tried to work with him. Then we decided to abandon the project. Now there is a report that Walmart succeeded. If so, they came up with a custom solution that is very different from the normal syndication process. I can well believe that they did that.

Here are specific problems that I encountered.

Our search for a store of the same name in around the same location often failed, so we tried to create a new store. Only for our data to be deleted as a duplicate. We tried a number of things to improve the matching process to existing data. This improved things, but remained too low quality.

Our data often had state routes, while OSM had a local street name, leading to failure to place the store.

Data for stores in malls was rejected for our inability to place the store correctly within the mall.

Walmart department data was rejected because we were trying to place multiple departments with different phone numbers and hours in the exact same spot. It wasn’t enough to know that the Walmart had a pharmacy with specific hours and phone number, we were supposed to know where in the Walmart that pharmacy is.

Our business hour data for newly changed business hours was reverted because it was considered less trustworthy than previous in person reports from some time earlier.

I do not at this remove remember what fraction of this discussion was on forums, with that developer, and so on. I just remember the overall picture of the feedback. Which was:

It was our fault for having such crappy data. (Sorry, but it is all we have. Do you want accurate store hours, or not?)

You should just download our easy mobile app and use that! (I’m working in Anaheim, CA. How am I going to walk around a mall in Las Vegas?)

Oh, just fly someone to every store and have them do it. (Our budget is limited by our contract with the company. It doesn’t cover airline travel.)

Just get an employee at each store to download the app, and map the store. (We di not have the authority to order Walmart employees around.)

That was where we abandoned the contract and took the penalty.

In the thread they say that Walmart has now worked with OSM. My strong guess is that Walmart itself took solution 3 or 4. Which they have the power to do, even though a syndication company can’t. But very few companies are willing to put that level of effort out just for OSM. And so most companies will just outsource the work to syndication companies, who will syndicate it to companies who have a workflow that they can use. Which means Apple Maps, Google, Facebook, Foursquare, and so on. And OSM will continue to be able to complain about crappy companies that can’t be bothered to update office hours on OSM.

I understand why OSM does this. Merging crappy data is a lot of work, especially for volunteers. A map interface that is usable by people walking around requires far more detailed data than syndication companies have. If you don’t insist on quality in some way at some point, you’ll never get it.

But it comes with a cost. Right now OSM says, “You have to be this tall to work with us.” So you get to both complain how crappy the data from businesses is, and complain that you don’t have business data. Seriously? Pick a lane!

Xvtn · 2024-10-09T20:39:44.394Z

Perhaps this has already been suggested. But it sounds like there is a need for an application of some sort that would do the following:

Accept tabular data for a given feature (eg walmart stores) and translate that into OSM tags
Work around the well-known “there are no static ids for POIs in OSM” problem, by reading and writing ref tags, by geocoding addresses and searching nearby, by using other approaches, or a combination of these.
Provide an easy way for data providers to manually review edge cases and follow established “automated edit” guidelines. Without wasting the provider’s time.
Reject data that is not up to OSM’s quality standard.

These requirements sound to me like they would satisfy both btilly’s gripes and OSM’s import requirements.

I know this is a tall order. I don’t have time right now to spearhead this, so I’m just throwing out ideas for a possible technical solution.

CjMalone · 2024-10-09T21:07:26.397Z

Looks like an interesting thread, thank you

Google does isn’t because Google crowdsources it. It is because Google is straightforward to work with for the companies who provide that data, and OSM is impossible.

Kinda a mix right? Depending on the country/region, but basically everywhere branded shops aren’t dominant, most shops are independent, and they are infinitely harder to get data for. Crowdsourcing is really the only way, yes there are a few companies selling this data, but it’s either a mix of Open Data sources (including OSM :/), or crowdsourced via apps (eg Foursquare) or crowdsourced via “not an employee” employee (eg LDC or Uber).

Payment transaction data could blow this whole world open…

Yes, there are companies whose job it is to get opening hours to various sites. Someone like Walmart will write a contract, and send a regular spreadsheet. The company then processes the spreadsheet, and pushes data out to Google, Facebook, Apple, and so on.

I know this because I was the lead developer at one of those companies for a while. Everyone except OSM is happy to accept an address, maybe a suite number, and then information about phone numbers, opening hours, holiday hours, and so on. OSM insists on detailed geolocation data. If you don’t have it, then they won’t take your data. Period.

As they point out those spreadsheets aren’t always golden. And it’s the same for data on these brands websites, having a feedback loop to inform the brand they have incorrect data is vital and you’d hope these SEO companies have it. We don’t and it’s painful.

You have data from Walmart with every Walmart address, and every department’s phone number, opening hours, holiday hours, and so on? Apple, Yelp, and so on will love you for it. OSM tells you that they need the store layout.

It’s not about “store layout” but accurate coords, yeh these brands and their SEO companies may have “authoritative data”, but the coordinates are often trash. OSM has golden coords. This has changed over time, originally you’d be lucky if a camp site was on the right side of town. Nowadays OSM is the only map with perfect POI coords (literally 99.9%). Other maps doesn’t even have POIs on the right side of the road. OSM shouldn’t go backwards, but “OSM” should have a place for data without coords. Hello All the Places. Hello Chain Reaction.

With the Walmart references they have doxxed them selves as working for SOCi/Brandify. Another SEO company with the exact same story will be Uberall/Navads.

I suspect a lot of the interest from these companies “get us on Maps.me” not “get us on OSM”, in Maps.mes credit. They tried.

maptheworld2050 · 2024-10-09T21:13:27.914Z

Thanks, that’s very helpful. I wish I could pin that or edit it into the OP but it doesn’t seem like I have permission for either.

CjMalone · 2024-10-09T22:22:01.784Z

SomeoneElse:

I can’t think of an occasion where a company was told “you are not allowed to add data to OSM” where that data was (a) accurate and (b) not unverifiable spam (e.g. “The best Tire Company in Springfield” or worse).

(c) “You’re doing it wrong” eg Distriktstandvården

SomeoneElse:

I’d suggest, as a first step, that anyone who has data that they’d like to add to OSM that they post about it here - not just “I have a spreadsheet” but “my spreadsheet has fields X, Y, Z and example data would be A, B, C, and here’s how I plan to combine my spreadsheet with what is already in OSM and here is how I plan to upload it”.

Completely agree, individual companies should give us data and talk to us. SEO companies the same, but I’d expect them to do some of the work with us, they are getting paid to do it.

maptheworld2050:

Satellite imagery won’t help since these are often indoor locations that are part of a larger building

Someone solved this 33.9484223, -118.333149

But honestly, Mapillary, Bing Streetside, in a lot of cases we can geocode via street level imagery. It takes a human, but it’s possible.

Peter_Elderson:

The way this should work is that all businesses publish their opening hours in a common open format, all the same way, so that applications can tap into that.

Many do, independent and brands. Google spearheaded schema.org. Ironically opening hours used to be defined as “OSM opening hours”, now a subset, or a format that easier to parse and generate.

dieterdreist:

Apple Maps is OpenStreetMap-powered, although it’s not their only source, it seems it’s the most important one, judging by the prominence of attribution.

I don’t think they take any of our POIs, at least not in most countries or categories. They take our roads in most (all?) countries.

amapanda_ᚐᚋᚐᚅᚇᚐ:

In Geomob Podcast #253, Overture’s Executive Director Marc Prioleau claimed that wildly misplaced points are still useful if you’re searching for a result, and that that incorrect data is not intended to be placed on a map.

I do have some empathy here, but the coords and Overture Places in general… Isn’t great.

The Facebook data was probably never meant to be rendered on a map, maybe it was meant for a list view, but in reality it wasn’t meant for either. And in Marcs defence, we live in a world with vector tiles and search. Basically none of Overture Places should be rendered on a map, unless directly searched for. Google Maps has an astonishing amount of bad data but you’ll never see, if you search hard enough it’ll pop up, they honestly do a good job hiding it.

The actual mapping companies all deal with complex data merges. As a verified publisher for a verified business, our data on store hours, phone numbers, and so on would be considered definitive. Actual geographical layouts though? Generally they consider their data more reliable. It usually is - businesses told us our addresses, but not an accurate geolocation.

I don’t like “definitive data”, all data, including OSM, has errors. The difference is, we allow anyone to improve it.

But this is absolutely right, nobody has more accurate opening hours than the first parties. And nobody has better coords than OSM. ATP and Chain Reaction are OSMs glue here, but we still don’t fully utilise the data available to us.

Walmart department data was rejected because we were trying to place multiple departments with different phone numbers and hours in the exact same spot. It wasn’t enough to know that the Walmart had a pharmacy with specific hours and phone number, we were supposed to know where in the Walmart that pharmacy is.

OSM doesn’t really map departments of a store, but we often do map pharmacies separately, for exactly this reason, different opening hours and contact details.

Years ago I added all the Asda pharmacies, within the Asda, with it I added fixme=Location not accurate, to this day, most haven’t been given good coords…

Our business hour data for newly changed business hours was reverted because it was considered less trustworthy than previous in person reports from some time earlier.

As previously mentioned, “authoritative” data doesn’t mean correct data. But a lot of OSMs opening_hours are decades old. In the vast majority of cases it’s OSM that is wrong. (Although see the previous link.)

You should just download our easy mobile app and use that! (I’m working in Anaheim, CA. How am I going to walk around a mall in Las Vegas?)

Oh, just fly someone to every store and have them do it. (Our budget is limited by our contract with the company. It doesn’t cover airline travel.)

Just get an employee at each store to download the app, and map the store. (We di not have the authority to order Walmart employees around.)

I feel for them, they obviously can’t get better coords like this. And the interaction with an OSM contributor isn’t going to change anything. But I have often thought ~“A brand would only need to train a handful of delivery drivers to record accurate coords” or “An email to every manager with a map of there store and mandate for them to reply with yes or no” would get them perfect coords relatively quickly.

Also note, some brands have perfect coords already. Hello McDonalds. It’s just some companies care about the data, some don’t.

eisa01:

In the thread they say that Walmart has now worked with OSM. My strong guess is that Walmart itself took solution 3 or 4. Which they have the power to do, even though a syndication company can’t. But very few companies are willing to put that level of effort out just for OSM. And so most companies will just outsource the work to syndication companies, who will syndicate it to companies who have a workflow that they can use. Which means Apple Maps, Google, Facebook, Foursquare, and so on. And OSM will continue to be able to complain about crappy companies that can’t be bothered to update office hours on OSM.

I don’t know the history of how (basically) every Walmart got mapped in the US. But ironically they block All the Places from collecting up to date opening hours. OSM and Walmart will suffer because of this.

Xvtn:

Work around the well-known “there are no static ids for POIs in OSM” problem, by reading and writing ref tags, by geocoding addresses and searching nearby, by using other approaches, or a combination of these.

URLs can be refs too. If the brand cares enough to have stable urls or redirects.

overpass-turbo.eu

overpass turbo

A web based data mining tool for OpenStreetMap which runs any kind of Overpass API query and shows the results on an interactive map.

Peter_Elderson · 2024-10-10T07:41:07.249Z

eisa01:

Here are specific problems that I encountered.

Our search for a store of the same name in around the same location often failed, so we tried to create a new store. Only for our data to be deleted as a duplicate. We tried a number of things to improve the matching process to existing data. This improved things, but remained too low quality.

The general conflation problem: matching without a unique and reliable matching key, and knowing what to keep, what to replace, what to add and what to change, on both the object level and the atrribute level. I have yet to enounter a case where this is handled fully automatically and reliably.
If all the stores and their additional data were fed to OSM from an outside source, this could be done: you would have a unique and persistent matching key and you could simply add missing data, remove obsolete data, and replace old information with new. OSM-wide, this is not the case, and it’s not going to happen any time soon. In a bubble, such as one company or brand with all its stores, or every store within a small area, it probably could be arranged.

Our data often had state routes, while OSM had a local street name, leading to failure to place the store.

That, I would say, is for the data supplier or syndicator to solve. It sounds like they were trying to match on address, which is an unreliable string key. The key must be unique and reliable on both sides before you can expect reliable matching.

Data for stores in malls was rejected for our inability to place the store correctly within the mall.

Sure. You can’t just dump stores in the vicinity, just because you want to store opening times. You can put pins with data on an OSM-based map, but they would not be stored in OSM unless you link it to a mapped object, which requires the actual and verifiable location.
If the mall is an object, I could think of a solution to specify the stores in the mall as tags, without adding them individually as objects of their own, and link opening times in the same way. But the data is too much for the OSM data model, so it would involve a key to an outside system where the opening time details are stored. And, I think this would be an awkward, error prone solution that will satisfy no-one, and the data will get stale very fast.

Walmart department data was rejected because we were trying to place multiple departments with different phone numbers and hours in the exact same spot. It wasn’t enough to know that the Walmart had a pharmacy with specific hours and phone number, we were supposed to know where in the Walmart that pharmacy is.

Same thing really. If you map different departments, they have diffferent locations; if you don’t know where they are you can’t put them in the map database. OSM is not a pin database; it’s a map of verifyable objects where they really are. This sort of data should be kept in a separate system with a unique and persistent identifier key; then it is possible to record the key in relevant OSM objects so applications can link the external system for the details.

Our business hour data for newly changed business hours was reverted because it was considered less trustworthy than previous in person reports from some time earlier.

If your data was truly and verifiably trustworthy, it should not be reverted. Could it be that the trustworthyness was not obvious or easily verifiable, and the mapper was pissed off because his/her precious survey data was replaced, without survey, with other data?

SomeoneElse · 2024-10-10T09:28:27.720Z

Peter_Elderson:

You can’t just dump stores in the vicinity, just because you want to store opening times. You can put pins with data on an OSM-based map, but they would not be stored in OSM unless you link it to a mapped object, which requires the actual and verifiable location.

Yes - that’s a key point. OSM is not a business directory, it is a geographical database.

Peter_Elderson:

If your data was truly and verifiably trustworthy, it should not be reverted.

Indeed, but I can think of examples where “data reported to head office” does not match “what actually happens”.

Peter_Elderson · 2024-10-10T10:17:27.997Z

I know only one example of a non-commercial enterprise (Rustpunt.nl) in Nederland where we managed succesful matching and conflation, aligning data at both sides, followed by a periodocal maintenance process.
The corrections and matchings work and the data is high quality because one man running this enterprise drives all over Nederland to visit his ‘Rustpunten’, adding new ones, updating his own system, deleting the ones leaving and kicking out the ones no longer adhering to his high standards.

(it’s actually a very nice initiative: a Rustpunt is shielded but outside seating, plus selfservice coffee, tea, beverage and snacks, access to a toilet, and bicycle stands, air pump for the tubes, electricity for e-bikes. They cannot provide table service or provide alcohol or smoking products. They are all over Nederland, recognizable by an official logo, and are very popular with recreational cyclists and hikers. I have seen comparable locations in England announced as “honesty bar” and “honesty cafe”, but these looked like completely separate private initiatives.)

The problem of matching was solved by proximity matching, followed by manual checkup and correction. Once mapped, the id of the POI in the external database is stored as the ref. The source site can, and actually does, grab the exact location in OSM which is much more accurate than the “pin place” location of the source database. This actually happens, because we can see that in the data in the maintenance rounds.

In principle, maintenance has been setup as: we have access to the source site, can grab a file with all the data of all the POIs, one mapper uses QGIS to process this. New locations (new id’s) are automatically added and postchecked manually, mainly to correct the location. Id’s in OSM which are no longer in the source file, are manually processed to see if the can be removed bluntly, or maybe just altered to a different type of POI (e.g. they lose the Rustpunt-status, but continue as a cafe with service, which we also map).

This works, but it is flimsy, because it hinges on two or three persons personally taking the time to do this once in a while. If one person quits, it’s gone. If the technical tools and sytems change, it’s gone unless someone start all over. I am one of the few persons doing this, and as I take on other things, it tends to slip to the background.

All this to say: sometimes a lot can be made to work technically, but organisational setup for the long run is maybe even more important to ensure a working service.

Mateusz_Konieczny · 2024-10-10T16:16:17.514Z

Improving OpenStreetMap shop coverage with AllThePlaces is relevant here (thread has more about ATP and specific attempt to process it in way that is supposed to be useful for OSM mappers)

Fizzie41 · 2024-10-10T23:23:09.627Z

Peter_Elderson:

If your data was truly and verifiably trustworthy, it should not be reverted

I’ve been wondering if the posted complaint actually followed the OEG & Import rules, or just started dropping “slightly” off data on the map?

That would be enough for any to either revert their work themselves, or a DWG complaint, who would do the same & possibly also block their account.

SomeoneElse · 2024-10-11T10:05:17.704Z

With a DWG hat on I can search through some very old DWG tickets, and it looks like we had a couple of related complaints (both about one problematic mapper for that organisation):

Not replying to numerous changeset comments, and ignoring DWG messages saying “please reply to changeset comments”.
Adding duplicate data - actually in this case duplicate data with their own reference duplicated it it, so the problem wasn’t related to OSM search, more their own processes.

In both cases other OSM mappers were actively trying to help them add their data, but came to us when messages of help were ignored. I’m sure that there were other interactions with the community (perhaps on OSMUS Slack?) that we weren’t aware of too.

ArgyleZombie · 2024-10-11T19:39:07.052Z

You are correct. I only meant to point out that Walmart may not have been the best example to use in the original post. Sorry, I see my wording was unclear.