One of the immediate conversation points that came up when discussing creating new instances of Panormax is how to sustain an instances. I’m keen to get instances going for my home country (New Zealand ) but also for England/the UK and Spain where I now live.
Building on the basis that the Panoramax foundation will have different classes of membership it maybe me think that there will be some different approaches to this for each type of organisation and these are not mutually exclusive.
The second one could be making instances easily self hostable by individuals - through PaaS tools - to internalise costs for individuals in terms of their time and effort to run an instance for themselves and their social groups. At this stage I’m not clear if there are incentives for doing this (obviously beyond the fact that people like me do enjoy self hosting. The comparison here would be running an ActivityPub server where you do get control of additional features - probably mainly around moderation of content and federation.
A third might be exploring how Government agencies might engage as server operators for their own use in collecting and using data. I know in the UK there are transport and data related Government bodies within places like Innovate UK that are designed as ‘accelerators’ for things like this.
A forth type is commercial organisations, I’ve already started listing and mapping out a few types of organisations but one thing that I have become aware of is that there might be quite an imbalance between organisations that are primarily providers and users of imagery and we may need to work out an equitable way of recovering costs base on the cost of services that they consume and being able to incentivise the provision of imagery (perhaps through incentivising the provision of services that make those images available)
I’m interested in getting people’s views. I’ve already got some interest from people in this last group who want to know more and talk about how they might be involved and I’ve been talking to them to see if I can build them up to get involved.
Still need to test out the backup plan and add in the instructions on configuring the S3 buckets (e.g. to set CORS rules) but the basics of it work pretty well so far.
It really depends. It pays to shop around, though some providers have multiple components to their pricing (e.g. data, ingress, egress, “operations”) which makes it a bit tricky. Cloudflare and Backblaze is ~USD$15/TB/mo; OVHCloud or Hetzner seem to have slightly cheaper options than that.
S3 is cheaper and scales more linearly than paying for additional SSD volume space to go with a VPS (usually seems to be in the range of USD $50-60/TB/mo). But if you’re running a really small server, you could look at what the Fulda instance did with running their storage off of a Hetzner storage box product, which is in the range of USD$4/TB/mo. That approach seems to have some technical limitations, but it does seem to work pretty well as long as the instance is small.
I believe the OSM-FR instance is very cost-efficient because they buy their own hard drives. My current understanding is that doing this is harder at a smaller scale unless you’re willing to homelab; otherwise you need to be hosting enough data where paying for colocation works out or to have some other place that will give you server rack space? Curious if anyone has ideas about how to make it easy to host with your own hardware at a smaller scale (e.g. an instance that has in the range of 1 to 5 TB of data).
For very small instances, it could be a solution, but the cost per TB in the cloud does not allow medium or large instances.
Event if hard disk prices have increased a lot during the past year, it is still way cheaper to self-host or have on premise or colocated hardware.
OSM France instance currently have 300+ TB and could not pay 3000$ monthly.
Yes, and used ones !
We currently have 35 x 8TB and 12 x 16TB HDD + 2 x 4TB nvme SSD (for the databases). The total cost since the setup 3 years ago is around 10k€, and no monthly fees (bandwidth is donated by a non-for-profit, colocation bay is also donated).
Other cheaper than full cloud option is renting a bare-metal server (OVH SYS-STOR ?), like the server covering Argentina.
Yep, we run Panoramax AR on a OVH KS-STOR, with 4x HDD SATA 6TB Enterprise Class (+ 1x SSD NVMe 500GB for the DB), which we run in a software raid for single drive redundancy giving us 16 TB of usable storage at USD 44 / month (excluding backup costs as well).
When you compare that to the S3 calculation, you quite quickly reach the point where the bare metal rental becomes cheaper
What’s the lifetime of those drives? - Do you factor in the cost of replacing them periodically or is that just rental of hardware? Does your calculation take into account redundancy/backup that comes with S3 and implementing anything equivalent?
That’s very cool - so in our case, I guess it probably makes sense to start looking for a dedicated server if we hit about 3-4TB of storage? Is the amount of set-up/admin any different for a dedicated server or is it basically the same as a VPS?
I think it should be virtually the same: When ordering the server you pick a barebones Linux Distro and then install stuff from there (I have not tried the OVH VPS, so I don’t know how much extra managing they are doing on their end, I assume with those you still are in charge of hardening the virtual machine to some extend?).
The only ‘difference’ will be that you’ll have to setup the drive configuration for the RAID yourself, I’ve written up the config we used, based on @Matija_Nalis’s example of OSM HR. But also feel free to ping me if you ever switch and have questions
When installed in servers in a datacenter bay with steady temperature, they can last easily more than 10 years without trouble. The 8TB we have on our server are from 2017 and have around 40000-60000 hours. I have some drives around 100 000 hours.
I’ve been selfhosting a lot of things during the past decades. 3.5" hard drive failures do happen but they are really not that frequent and the global lifetime has greatly increased over time. I would not say the same for 2.5" and SSD where failures are more frequent.
On our on premise server, we replaced the first set of 3TB drives with 8TB mostly for the additional storage, not because of their age. When renting a server, it is always possible to rent a new one to replace an aging one, but this is usually also because you need more space or CPU power… and because the monthly rental of old server stays the same where new models are not much more expensive while providing more resources.
As drives can fail, I’m using ZFS to take care of redundancy. It provides a very safe global storage system, with different levels of RAID, checksum on everything (to detect bad data), but also allows to replicate data remotely with differential snapshots.
OSM-FR has a backup for the original pictures and the databases, all done using ZFS. This backup server uses older drive (3TB/4TB) and is powered up only during daily backup time.
S3 backup is also too expensive, it would cost more than the main server !
It is very similar. When your rent a baremetal server, the hoster provides a web interface to setup the server and install the OS on it. You do that once at setup, then log on your server like you would on a Virtual Machine.
Thank you for reminding me about this post! I made a small edit to add it to my write-up. So it seems that ~2-3TB, dedicated server is more cost-effective (if the right configuration is available), or you can just start with a dedicated server if you want room to grow. And then maybe colocation + donated/second-hand hardware once you outgrow the dedicated server option?
I would also love to see a Canadian municipality run their own instance. Montreal already has municipal staff uploading imagery to Mapillary when they patrol the bike network, so it’s not such a huge leap to imagine them being interested.
That’s at least the thinking we had for the Argentine instance, if/when we grow so much that renting bare metal is starting to become an issue, we hopefully can demonstrate enough impact that hardware donations & finding co-location spots is easier (compared to just showing up at potential donors with nothing but an idea).
