Why does OSMF Budget €25,000 on Amazon

Foundation
1

Cloud hosting, as explained here, is only used for S3 storage, which is a fancy way of saying “hard drives”. That’s a 25% of estimated 2023 yearly low-budget for OWG, 25,000€ to be exact. This sounds insane to me.

I wanted to better understand, what’s exactly being stored on those S3. This information is quite difficult to find, and it accounts for a substantial amount of money. I have only been able to find this post, which indicates that at some point, we stored 86TB in Fastly logs - insane. I know it’s no longer the case but this does not boast well for the remaining kinds of data stored there.

It also sounds like the backups we produce are complete - not incremental. I cannot say for sure, but if that’s the case, I would be worried about the competence of some people. I am really struggling here to justify the 25k€ yearly spending, sorry if that’s not the case and I offended someone.

The biggest reasonable data store for OSM I can think of, is the full history planet file, which weighs about 200GB. Multiple it a few times and I think OSM should not need more than 100TB in storage - ever. This is easily achievable with own hardware. It’s worth noting that after this huge 25k€ yearly spending, OSM still does not own the S3 hardware, it’s just renting it out. It could have easily been a one time spending (+small yearly maintenance).

I am also concerned as to how much dependency does OSM have on Amazon, a big tech. We have all these servers. Why do we ruin this now for some S3. We could as well host it on Google Cloud and it would be no differ, I thought OSM wants to fight with big tech and stay independent.

I know there will be an argument about how sending data also costs, not just storing it. My answer to that is: use the fricking CDN for that. That’s the whole point of a CDN - to send out data, so we don’t have to. When we use a sponsored/free CDN, we only have to pay for storing the data pretty much.

Literally OSM in 2022, 2023:
image

I am personally tired of people who want to use the cloud for everything. Such individuals really seem to love living under indefinite subscriptions and not owning anything. I don’t and nor should OSM.
image

2 Likes
2

PS. please stop this as soon as possible.

3

Asked some people responsible:

Update: it has been closed down by @TomH just after 1 response, without the issue being actually resolved. The behavior of OWG moderators is quite concerning to me. It’s a very important issue which requires explanations and it feels like it’s being (intentionally?) downplayed.

1 Like
4

I am on holiday at the moment. We have spent $0 on AWS for the last 6 months. We use AWS for backups and distributed storage. I believe we have over 70TB stored.

Yes, we do have future plans to host the planet data on AWS. The expected cost is $0/month due to our hosting for this being under AWS public data program.

I can give a more extensive response once I am back from holiday later this week.

10 Likes
5

Let’s quote the rest of what you’re citing

There is considerable uncertainty as to what our costs will be next year as we have cut our cloud hosting bills in half, meaning data from earlier in the year cannot be used to forecast future costs. The estimate used is from the high end of possible costs.

Budgeting is not the same as spending, and the budget explicitly says the budget for this item is from the high range of possible costs.

The OSMF budget is not the operations budget. The highest case AWS in the budget is 3% of the OSMF budget.

That we’ve cleaned up log storage usage is bad? Reducing costs is bad?

3 Likes
6

Why does it need 25k€ yearly then? 0€ * 12 != 25,000€
You say one thing here, but the report says something completely opposite.
If none of this goes to AWS, where does it go?

You also say “last 6 months” okay sure, but what about before that? Could you provide more context? How much was spent in total? If AWS ever cancels free storage in the future, what will be the cost? I haven’t seen that being discussed in budget risks.

The last OWG report is from December 22’. I always assumed that if such a significant change were to occur, we would be notified about it. I suppose that’s not the case. Is it intentional?

7

Expected costs: 0€.
Budget: 25k€
Makes total sense to me.

8

Sorry, I misunderstood that! Thank you for correcting me. I updated the post.

1 Like
9

No, letting this to happen is bad and is not a good sign in general. I work with very limited information, since most of the S3 details are kept private for some reason.

10

General note: I would not start from assuming some drastic incompetence even if some things appear to be confusing/weird/unwelcome.

Both in cases where what happened makes sense and in cases where some serious mistakes happened it is more helpful to be a bit less emotional.

17 Likes
11

As I see it from the information available publicly, it’s a mess.

The operations group don’t seem to reveal exact costs or usage which makes it so much harder to analyze. They also say that it’s actually 0€ which makes me much more confused. Where does this 25k€ actually go to?

Sorry for my lack of respect, but I am really frustrated as to what I learnt today.

12

There is still “25% of my donation money went straight to Amazon” which is definitely mistaken

3 Likes
13

A budget is different than an income and expenses statement or balance sheet. In a budget you allow for contingencies, while in an income and expenses statement you’re reporting actual costs and deprecation. Given how you’re engaging on this and attacking the volunteers, I find it difficult to respond to this.

6 Likes
14

Which part has offended you? That’s not my intention. Your argument is quite peculiar, and you haven’t provided any valuable context that would help in resolving this matter. Why are OWG operations kept secret? Is me figuring out an issue with that, attacking you?

15

Er, what? Please leave us (the Data Working Group) out of whatever point it is you are trying to make :slight_smile:

More seriously, I don’t think that it’s fair to describe anything that you need to “better understand” as “insane”. It’s certainly judgemental to include stuff like the “Literally OSM in 2022, 2023” picture when you have yourself said you don’t know the whole story.

I suggest that you revisit your posts here, ask yourself which things you’ve said you’re sure about because you definitely have the full story about and which you’re not, and reword accordingly.

6 Likes
16

That’s the point, what’s exactly being stored on those S3, that we need 25,000€ of them per year. I was not able to find enough public information which made me very concerned. That’s a quarter of the OWG budget and it’s completely private.

@SomeoneElse

17

Are you really, really sure about DWG (Data Working Group) being involved at all?

Note that your understanding of situation may have other similar gaps.

+1

18

Fixed, typo, I meant OWG and seeing the @SomeoneElse made me type “D” instead :stuck_out_tongue:

19

From 3 months ago, Operations/Minutes/2023-05-04 - OpenStreetMap Foundation, OWG talks about the AWS cost for GPX traces and user images being 400-500 USD. Why not zero? 3 months ago is surely is within the last 6 months range.

20

Key word “expected”, I am asking about today, actual, current cost. As seen above, I found conflicting information to your statement. I found a talk about potential AWS sponsorship but I couldn’t find any information about it being actually realized.

  • A big part of the costs for the general account is the logging for the CDN and the planet backups.

Right now I understand that we pay €25000 yearly on storing primarily CDN logs and backups? Is that correct? This can’t be right.