Best way to consume OSM notes modifications

AngocA · 2022-10-26T00:23:53.961Z

I would like to get/poll periodically the latest OSM Notes modifications (creation, comments, close, reopen) for the whole world. What are the possibilities?

I only see the notes/search option from the API where I can specify the “from” parameter.

Is there any other option? Websub (PubSubHubbub). Something for real time but without impacting the API with periodic requests? Or even with a Messaging System (IRC, Matrix, Telegram)?

Kovoschiz · 2022-10-26T02:48:07.069Z

https://resultmaps.neis-one.org/osm-notes has RSS feed for each country. You set up in your application, or service.

AngocA · 2022-10-26T03:40:36.853Z

I want to have other kinds of filtering to do analysis on notes. That’s the reason I want something directly from OSM.

Also, the precision of the country boundaries in that page is not very accurate: https://twitter.com/angoca/status/1475298343203090433

flohoff · 2022-10-26T08:09:55.771Z

AngocA:

I would like to get/poll periodically the latest OSM Notes modifications (creation, comments, close, reopen) for the whole world. What are the possibilities?

I am using a standard rss feed reader with a manual URL construction from a bounding box like this:

http://api.openstreetmap.org/api/0.6/notes/feed?bbox=8.1706155,51.8144663,8.5269845,51.9737924

A feature missing for me in Notes would be “subscribe”. Without otherwise interacting with a Note i would like to subscribe, to get future updates. Same for Changesets. Sometimes changeset comments go back and forth and i’d like to be updated on comments.

Commenting a “subscribe” to a Note/Changeset is a workaround.

Flo

mmd · 2022-10-26T08:19:02.998Z

flohoff:

Same for Changesets. Sometimes changeset comments go back and forth and i’d like to be updated on comments.

Nope. You can simply hit the “subscribe” button on changesets even if they’re not yet closed, and you will receive a notification for every comment.

Matija_Nalis · 2022-10-26T12:37:28.948Z

If once-a-day updates are good enough for your case, you can periodically grab https://planet.openstreetmap.org/notes/planet-notes-latest.osn.bz2 (~230 MB) and parse that XML to extract whatever information you need (filtering by comment timestamp, and any other rules you might have, should be trivial).
Additionally to that maximum customizeability, it does not create load on other servers, you do not depend on their acceptable usage policy (or speed / availability), etc.

AngocA · 2022-10-26T13:16:02.716Z

In fact, I would like something like “real time”. To be notified once a change in a note is done.

Parsing the planet is what I have already done, but this process is very slow (processing the whole XML takes a time) and this daily periodicity is not what I am looking for.

(Pull notification) I can create a RSS for the whole world, and polling it each minute, but this impact the API, because many of the feeds will not contain any new value. This operation is started form the requestor. My view is that if everyone is polling the API, the server has to respond these requests. I want something real time without huge impact, or unnecessary CPU cycles. I would like something different that: “Hey API, is there anything new in this area?”

(Push notification) I would like to “receive” the notification from the API, once I have subscribe to it. To receive from the server when there is something new. This operation will be started from the server.

Probably, the current API status does not provide subscription like this, and I will have to use the API by polling it periodically. It could eventually be part of the API 0.7.

mmd · 2022-10-26T13:24:52.883Z

AngocA:

I want something real time without huge impact, or unnecessary CPU cycles. I would like something different that: “Hey API, is there anything new in this area?”

I think we’ve talked about this already. The API 0.6 notes search endpoint provides start and end timestamps. Why can’t you use those? If there are too many notes, simply make the time interval smaller, and query again.

I know that the wiki pages states that the “q=” paramter is mandatory. I don’t find this in the source code (openstreetmap-website/app/controllers/api/notes_controller.rb at master · openstreetmap/openstreetmap-website · GitHub), and the endpoint also works without it: https://api.openstreetmap.org/api/0.6/notes/search

AngocA:

To receive from the server when there is something new. This operation will be started from the server.

I’d say, this is very very unlikely to happen for a number of reasons.

Matija_Nalis · 2022-10-26T13:44:01.222Z

The API 0.6 notes search endpoint provides start and end timestamps

True, e.g. this seems to successfully return 20-minute range of Note updates earlier today:
https://api.openstreetmap.org/api/0.6/notes/search?from=2022-10-26T13:20:00Z&to=2022-10-26T13:40:00Z

One should take care of following API Usage policy which states (among other things):

The editing API is provided in order to edit the map data, not for read-only purposes or projects

However other AUP suggestions (like using dumps + diffs) in AUP do not work for Notes if one need better than 24-hour granularity (as seems to be the case here), because I don’t think we have minutely diffs for Notes (only for map data).

Maybe if one calls it reasonably rare (e.g. once every 5 minutes instead of every second) it would be often enough for @AngocA usecase, and one might ask ops if that would be considered acceptable usage? (due to alternatives being non-existent)

Adamant1 · 2022-10-27T01:06:28.724Z

I’m not sure if it’s exactly what your looking for, but in my opinion ENT8R’s NotesReview is pretty good for reviewing notes, regardless of if they were recently opened or not. I think you can search for notes by specific users to.

Matija_Nalis · 2022-10-27T03:06:36.540Z

Adamant1:

ENT8R’s NotesReview is pretty good

It is a nice tool. However, unless I’m very wrong, it is simply calling API 0.6 mentioned above and displaying it on the map. So, it has all the limitations of the API (usage policy, max 100 notes, etc)

Adamant1 · 2022-10-27T03:27:28.962Z

Matija_Nalis:

It is a nice tool. However, unless I’m very wrong, it is simply calling API 0.6 mentioned above and displaying it on the map.

Oh, bummer. I wasn’t aware that’s what it was doing. Your probably right though

kmpoppe · 2022-10-27T06:37:00.887Z

Matija_Nalis:

So, it has all the limitations of the API (usage policy, max 100 notes, etc)

You can set the Notes API limit to 10000 in every call, but I doubt that’s gonna solve OP’s request.

Yesterday I checked discussions-latest (5.4 GB, DON’T DOWNLOAD UNLESS YOU WANT TO), but there are no Notes in there as well, only changeset discussions.

So … I guess without asking someone like @Firefishy if a downloadable XML of all Notes (and a nightly Diff or something) could be provided on planet.openstreetmap.org (which I would be more than happy to see!), this might not be solvable. Removed for reasons of stupidity. Thanks for pointing out the obvious @mmd.

K