OpenStreetMap France is hosting the largest Panoramax instance so far, and since the start we have accepted pictures from all over the world FOR TESTING PURPOSES.
This means testing how Panoramax works, on the upload side, how its viewer works, how the API work without requiring everybody to setup a server.
As our storage reached 80% of its total capacity, I’ve checked how much pictures outside of France have been uploaded on it.
23 millions on a total of 50, that’s 46%, almost half !
Some contributors have more than 1 million pictures, the highest one has almost 4.5 millions.
This is not sustainable and setting up local instances should really be considered.
OSM France board will soon have to decide what to do and here are the options (and/or) :
do not accept pictures outside of France anymore
delete pictures from some accounts who did not follow the “testing only” rule (or move them to new local instances)
For example, there are at least some imagery from ex-Yugoslavian area which we’d gladly move to our OSM Croatia Panoramax instance to help relieve the French OSM instance; but not if it will break all the panoramax=* and related OSM tags and other links/references made to those UUIDs elsewhere (who knows where).
And I’d wager at least some of the other location-based instances would be happy to do so too, when that downside of breaking all the references after move can be avoided.
I’d say this is best short-term measure if things are tight. But not as a long term solution - more instances are needed (as I’ve tried to emphasize in my Panoramax talk in SotM-HR just few days ago).
You could ask for donations for that purpose specifically (like here in this thread!), and I’m sure some people would be willing to pitch it (myself included; even if I have very few pictures outside Croatia).
Perhaps also contact contributors (at least top ones) of imagery outside France directly, and make them aware of the problem? It is likely that many are not aware of the issue, and will consider other alternatives thus reducing further (unsustainable) load on OSM-FR servers.
do not accept pictures outside of France anymore
Understandable, but it would be quite a blow to the project to lose the main fallback Global instance IMHO. But it the extra storage does not help enough, then it is only acceptable remaining choice.
delete pictures from some accounts who did not follow the “testing only” rule (or move them to new local instances)
I would plead not to do this one.
Firstly, that “testing only” was not always clearly communicated (e.g. there were plenty Fediverse posts asking people to contribute pictures and improve growth of users, pictures, kilometers; without any mentions that stuff outside of France will be deleted in few months and efforts wasted unless non-OSM-FR instance was chosen)
Secondly, destroying existing data that users have uploaded in good faith would be a public-relations-suicide move for the project as a whole IMHO. (remember what the much smaller CC-BY-SA vs. ODbL license-schism and resulting data removal almost did to the OSM project?)
I’d instead recommend existing OSM-FR uploads be grandfathered in, even if new contributions outside France are forbidden. Perhaps directly contact those contributors which use most disk space outside of France and help them to find a solution?
What are the steps to making a local instance and moving all of my photos there? And what will happen to the photos in Ecuador, since those probably can’t be put in a US local instance.
Also, perhaps it might be possible to enforce “testing-only” somehow instead of completely geo-locking ?
E.g. allow 10k or 1k pictures (or max 10 sequences, or whatever) per user outside of France, and block further uploads with specific error message like OSM-FR allows uploads outside of France only for testing purposes, and that test is limited to only 10k pictures / 10 sequnces. See <link> for your options)
That should stop most of problematic uses (heavy contributors), while still allowing for testing and not creating support boom like blocking all contributors outside France would.
But it would also be more technically challenging to implement, of course.
Another idea is to have a really dedicated “test instance”, with limited global storage and have a bot that will delete the oldest uploaded pictures. That’s something I was thinking for the foundation.
No change in Panoramax code, just a script to run with admin rights when storage is short.
at least it gave us a relatively stable version of 2012 data to go there occasionally and look how much progress we achieved in the meantime (including work on the style): fosm.org - earth map
some hardware and hosting… full cloud hosting is possible but will be expensive
some people to take care of it, install then maintain / update the software/hardware
This is best done with a local community, like we did in France or like what OSM Croatia did who started its instance recently.
Currently the panoramax command line tool allows to download and upload pictures between two instances, but this should be improved.
We plan to implement this at the server level, so that you will be able in the future to ask a direct migration of pictures between 2 servers. It will also keep the picture UUID so that reference to them will not be broken.
This is more difficult to answer, but even if I mentionned deleting pictures, this is really a very last option we will consider.
I feel somewhat responsible as well, since over the past weeks I’ve uploaded several hundred thousand images to Panoramax. The workflow there is simply much faster: images are available immediately, and I don’t have to wait 2–3 days for processing. Updating positions in JOSM is also significantly quicker compared to using the Mapillary plugin.
I do miss the automatic detection of exakt traffic signs, but the better performance makes it easy to move quickly through pictures and identify things manually.
Without wanting to offend anyone, I’m not sure I’ll still be around by the time a Germany- or Austrian-based instance becomes a reality. In what ways could I support the project? Financial contributions, hardware, or is it generally not desired to regularly upload images from outside France to the main OSM-FR-instance?
I’ll let OSM-FR fellas speak for themselves (although beside main solution of more instances, I guess financial contributions are usually likely going to be most versatile to help some instance specifically), but I’d like to point out that there is NO such thing as the “main” Panoramax instance.
There are:
instances which are younger, and those which are older,
instances which are georestricting uploads, and those which are not,
instances with more users, and those with less users,
instances which are faster, and those which are slower
instances with more (total or currently free) disk space, and those will less
etc.
But just like with Fediverse, there is no“main instance”, in the same vein as on the Internet there is no “main payment site” or “main Internet website”.
So this thread is specifically about OSM-FR instance (i.e. https://panoramax.openstreetmap.fr/) which is currently running out space; but the issues mentioned apply to other instances as well - none can take on all the pictures for the whole world, which is why the whole idea of panoramax is to be distributed – i.e. to have many independent instances, each taking a part of the load, with none of them being “the main one”.
P.S. There exist something called “meta catalog” in Panoramax (i.e. https://api.panoramax.xyz/) which might be considered something like “main instance” but it is not an instance really[1] (i.e. you cannot upload any pictures there, as it cannot host any pictures - it is not its purpose).
well it’s complicated, but for purposes of this thread here it is not an “Panoramax instance” at all ↩︎
I didn’t explain myself clearly. By “main instance,” I meant the instance that, at least in my circle, is being promoted the most and most visible. I understand how decentralization works. Thank you.
For many people (including me!) the bottleneck is time, not necessarily cost.
There’s lots of things I’d love to self-host, and wouldn’t mind buying the hardware for (whether directly myself or from a cloud provider) but that I simply cannot dedicate the time to maintaining.
As I asked here - is there an official “TL;DR: Acquire this hardware, install these things” guide for setting up an instance yet, something that takes less than a day to get running and is mostly hands-off?
@Kai_Johnson has done this for self-hosting Overpass, and I’d love to see something similar for Panoramax or be pointed to it, if it exists!
So, something like “most active” or “most popular” might be better wording in English I think, but even that still needs qualifier lest people mistake it for e.g. IGN instance which is very close runner-by (by some stats). Something “most popular non-geolocked instance” would be pretty identifying (at least today) – but I’d still recommend just saying directly “OSM-FR instance” as it is both clearer, non-ambiguous, and shorter.
I understand how decentralization works.
No worries, I did not mean to imply otherwise (I apologize if it sounded that way!), but to explain to others who may not be so familiar with the concept, and might leave with the impression that there really is “main” Panoramax instance (which in turn might end up discouraging setting up of new “non-main” instances, as they’d be automatically considered lower caste or something - which is my main reason for jumping in with explanation)
There is 🖥️ Host an instance - Panoramax - Docs which explains how to install the software (Assuming you already have the hardware). Docker-compose method is rather easy to get up and running, with few caveats.
I plan to detail my experience (at Matija Nalis's Diary | OpenStreetMap) setting up the OSM-HR instance in a week or two, so may want to subscribe there if it sounds interesting. Update: I did it, here.
As for the hardware, main thing is enough disks.
The big thing is also knowing what your local law says (but that was a case already with other instances too).
Over the past decade, we have been used by Big Tech companies to have centralized platforms (and their walled gardens).
We have also been used to have a lot of services for free… until enshitification hits them because hosting is never free.
With Panoramax we took a very different path but this path is not as confortable as the above.
Panoramax has matured over the pasts year, showing its potential, showing that decentralization can work (there is now 10 instances listed in the meta-catalog). I’m quite sure this will accelerate.
The goal of my first post was really to remind that OSM-FR instance cannot host much more than pictures covering France and put some more light on decentralization and the need of more local instances.
We’re working on creating a Panoramax foundation, and my point of view is that the foundation should not host a “main instance”, only a test one (for example with self deleting sequences) and the meta-catalog.
(I apologize if it sounded that way!), but to explain to others who may not be so familiar with the concept, and might leave with the impression that there really is “main” Panoramax instance (which in turn might end up discouraging setting up of new “non-main” instances, as they’d be automatically considered lower caste or something - which is my main reason for jumping in with explanation)