Foursquare releases 100M+ POI dataset under Apache 2.0

RedAuburn · 2024-11-19T17:43:54.607Z

This is pretty huge!

Foursquare Open Source Places: A new foundational dataset for the geospatial...

Stay up to date with the latest from Foursquare! Learn more about Foursquare Open Source Places: A new foundational dataset for the geospatial community

Est. reading time: 5 minutes

link to data:

Open Source

OS Places

Est. reading time: 2 minutes

chris66 · 2024-11-19T17:54:57.645Z

I don’t see that apache 2.0 is compatible with odbc.

RedAuburn · 2024-11-19T18:04:42.331Z

Interoperable Europe Portal

Compatibility between the Apache License, Version 2.0 (inbound licence) and...

Seems to suggest it is compatible, but i’m no lawyer. Here’s hoping
it’s definitely an odd licence to choose, though.

silversurfer83 · 2024-11-19T21:28:47.883Z

Another Database filled with very (!) varying quality

eisa01 · 2024-11-19T21:48:03.224Z

Similar to Overture this does not contain opening hours, you have to pay to get access to those fields

Minh_Nguyen · 2024-11-20T00:33:01.898Z

Between this release and Overture places, it must be an interesting time to be in the business of selling basic POI data on the basis of big splashy numbers. Not sure how big of a deal it’ll be, but the related Placemaker program says it’ll gather a mix of the objective facts we care about as well as more subjective attributes.

SimonPoole · 2024-11-20T15:20:06.642Z

I’ve had a look at a bit of the data in country and it seems a -lot- better than the Linux Foundation Overture Maps data, but then that isn’t exactly hard.

Differences seems to be mainly a lot of offices that we don’t have, but obviously that will depend on where you look.

License: using Apache 2.0 as a data license is more than weird and leaves many many questions open (the text from the EU site is machine generated nonsense).

SimonPoole · 2024-11-23T09:12:52.293Z

Seems as if either the quality varies hilariously widely or I was just very lucky, see ▪️Harel Dan on LinkedIn: #opensource | 18 comments

Minh_Nguyen · 2024-11-23T15:30:04.772Z

Yes, Foursquare and Overture places are like many geolocation-centric datasets: users aren’t supposed to ever see the raw data, either in a list or on the map. You have to filter by a confidence score. Otherwise, you’ll get tons of user-generated junk – pranks, mistakes, etc. It’s probably even messier than Overture in some places, because Foursquare was essentially a mobile location game for many years.

This kind of data is more readily consumable in a geocoder than on a rendered map. In the past, Foursquare would charge big bucks for the confidence scores as an upsell. If these scores aren’t part of the dataset, then no wonder the company feels comfortable releasing the data.

Here’s a marketing idea for OSM: publish a dataset of all the POIs that have ever been added to OSM in any form, including deleted ones, and tout the relatively huge total number of them with an asterisk that refers to an OSM confidence score and the URL to a forum thread where we bikeshed about what that confidence metric should be.

Minh_Nguyen · 2024-11-23T22:39:26.295Z

Minh_Nguyen:

pranks

Many of my schoolmates were into Foursquare back in the day (specifically, the City Guide app that they’re discontinuing on December 15). They became “mayors” of various places around town by checking in frequently. It doesn’t take very much of value to juice youth engagement with an app. Similar to the gymgoers of Pokémon Go, there were some pranksters who created fake venues near them specifically so they could be mayors.

Anyhow, here’s a viewer you can use to get a sense of what the data is like if you don’t purchase Foursquare’s confidence scores:

wipfli.github.io

Foursquare OS Places PMTiles

Happily, my favorite tells of Foursquare-sourced data are still around to “amplify the Digital Echo Chamber effect”, as they put it.

SomeoneElse · 2024-11-23T23:06:01.890Z

… and here’s my traditional link to “businesses unfeasibly allegedly located inside York Minster”

SimonPoole · 2024-11-24T07:58:48.480Z

So gave my surroundings another look thanks to Olivers map and my initial impression is kind of confirmed, less outright inventions than the Linux Foundation, more offices than OSM.

The offices have the same issue that office POI data from google has (typically from trade registry data), a lot of it is just peoples single proprietor companies home address with very very very limited usefulness outside of propping up the numbers.

What is disappointing is that the date_refreshed doesn’t seem to indicate anything QA related, there are two POIs nearby (a bank and a post office) that have been gone for more than 5 years (so did actually exist at one point in time) that have a date_refreshed from this year.

I kind of have the suspicion that given 4sq was never popular here (contrary to Facebook) the motivation to add outright nonsense was less and that is the reason that I’m getting a more positive impression than others.

PS: just in case people have forgotten this isn’t the first open data from 4sq, GitHub - foursquare/quattroshapes has been around for a long time, and at least its “locality” data has always been quite amusing.

eisa01 · 2024-11-24T10:12:51.792Z

Someone made a blog post looking into Foursquare dataset, currently being discussed on Hacker News

alan_gr · 2024-11-25T13:13:05.347Z

SimonPoole:

the date_refreshed doesn’t seem to indicate anything QA related, there are two POIs nearby (a bank and a post office) that have been gone for more than 5 years (so did actually exist at one point in time) that have a date_refreshed from this year.

I see there are even POIs where date_closed indicates correctly that they closed many years in the past, but date_refreshed is in 2024 (although closed POIs without any indication they are closed seem far more common).

They define date_refreshed as “The date the POI last had any single reference refreshed from crawl, Listing Syndicators, users or human validation”, which I guess could mean almost anything really.

Geonick · 2024-11-25T13:31:30.007Z

I have not yet had time to do a systematic, quantitative analysis - but when I look at Rapperswil (SG) [1] nearby for example, I see so many POIs that are literally miles away or just rubbish, that I can confirm what many people like Simon have already said: This data seems to be of very questionable quality at least in European regions.
For those interested, Oliver Wipfli added an example of how to directly extract a region as GeoJSON using Duckdb under Linux [2] .

[1] Foursquare OS Places PMTiles
[2] GitHub - wipfli/foursquare-os-places-pmtiles: All 100M+ open source places of Foursquare in a single PMTiles file

wcedmisten · 2024-11-29T15:59:00.317Z

I’m curious, do you think trying to cross reference this Foursquare dataset with Overture would create a more accurate dataset, or do you think both are subject to the same systemic bias that would defeat the purpose of using both?

Geonick · 2024-11-29T20:12:30.994Z

I took a quick look at the FSQ data of the couple of hundreds of McDonald’s and Burger King restaurants in Switzerland. I applied a filter like this (the ‘date_refreshed’ attribute is unusable because it contains recent dates like 2024, even though it’s not checked…) and compared them with the OSM POI:

"date_closed" IS NULL AND "fsq_category_labels" NOT LIKE '[Event%' AND "fsq_category_labels" NOT LIKE '[Arts%' .

At least ~30% of the FSQ POIs were unusable. Mostly because they were hundreds of metres - even kilometres(!) - off, and that makes any automated matching almost impossible (please correct me if someone has a matching algo that is smart enough).

It may be that some of the nearly 12,000(!) FSQ categories are of better quality. But without the ‘confidence’ attribute - which is withheld in the free distribution - the FSQ data seems to be practically unusable for many main categories, according to my and other assessments - at least unusable for OSM IMHO.

Thibaultmol · 2024-12-01T19:53:28.158Z

small tip: you can run a filter on both layers to filter them out.

if you want to check the map but only see current things (so exclude all items that are ‘closed’):
open devtools in your browser by pressing F12 and add this text to the console and press enter

map.setFilter(‘foursquare’, [‘!’, [‘has’, ‘date_closed’]])

and do it again with

map.setFilter(‘foursquare-10M’, [‘!’, [‘has’, ‘date_closed’]])

github.com/wipfli/foursquare-os-places-pmtiles

Ability to filter out closed items

opened 02:26PM - 01 Dec 24 UTC

thibaultmol

it would be nice to see what the 'current' map looks like. So being able to fil…ter out all items with a 'date_closed' entry would be nice

lipuma · 2024-12-02T08:28:55.889Z

Looked a bit around my hometown (in the Netherlands) and most points are very out of date (and missing date_closed), not in the right location or not in the dataset to begin with…

0xb7767000 · 2024-12-02T13:51:47.313Z

I had a look around in the areas I’m familiar with and the data there is… interessting to say the least.

POIs and even a bridge that are not even within the correct ZIP code. Name of the operator instead of the POIs name. Places that closed 10+ years ago. Many MANY POIs that were off by one building, usually on the wrong side of the street. Streetnames that are obviously from Monopoly. Temporary stuff like ships that had been moored at that place at that point in time. Various areas and stages from the Wacken Open Air festival.

But I also found one buisness that is still missing OSM and within the correct building in Foursquare. I only know this because I know the guy running it. There is nothing signed outside as far as I know, making it unverifieable even if you sand right in front of it.