Item — HN Reader

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage.

Compute is an active resource, when you exceed your budget it can be automatically shutdown.

Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action.

Providing fine-grained cost limits help some, as passive resources usually don't have massive cost spikes while active resources do, so you can better "protect" your passive resources by setting more aggressive cost limits on the active resources.

This quickly gets more complicated. Another example is most monitoring services are a combination of active (actual metric monitoring) and passive (metric history) resources. A cost limit on that monitoring service likely won't provide sub-service granularity, mostly depending on whether the service even has different charges for monitoring vs history.

Oh, also, even for a passive resource like storage, you also have active resource charges whenever you upload/download your data.

Ugh, what a mess. The best thing to do is pay attention to your spending, just like you do with your personal & corporate budget.

5 comments

But we've had disk quotas before that mostly worked?

If anything it seems an easier problem than processor time.

I recall disk quotas on shared systems at university back in 1998 and I'm sure they existed before that.

Two thresholds IIRC, one at which you get a warning, second at which you can't write any further and the disk write operation fails.

I don't think they deleted files, it was just you couldn't write more than [quota] bytes to your disk.

Is there something particular about cloud based systems that prevent this from working?

ie. is this a specific problem with distributed storage?

edit:tone

S3 costs money to keep your files in, even if you're not touching them, so just preventing further uploads wouldn't do much to prevent your AWS bill from increasing.

It would let you set an upper limit on the price you pay though. Better than accidentally misconfiguring a logging service and writing gigabytes of unneeded data.

>>> But we've had disk quotas before that mostly worked?

AWS has quotas on everything, including quotas on EBS storage per region.

You will realize that after you spin up some instances with disks and it's failing because you've hit 10 TB of EBS storage. Have to raise a ticket to raise the limit.

The IaaS services are easy. Other services are more difficult. Something like BigQuery ML could generate massive bills pretty easily.

> Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action.

A better option would be to automatically reduce the budget by the amount it would cost to keep the storage forever. If doing that would reduce the budget to zero, do not allow increasing the amount of storage. That is: assume the storage will not be deleted, and budget according to that.

How does this actually work? It clearly can't be forever, since any non-zero dollar amount * infinity months is infinity dollars, which is going to reduce the budget below zero since any non-infinite number minus infinity is less than zero... thus locking it immediately.

Even if we say "you get N months of storage before we delete it" and subtract N * current storage cost/month, what happens after you're locked out of all actions because you added an extra GB? Storage APIs cost money to use, so you would get locked out of those too (note that if you're not, people would set arbitrarily low limits and get storage access for free) and couldn't retrieve anything. The only remaining actions are delete (which is free) or raise the quota and do the whole rodeo over again.

Abuse is impossible to ignore at public cloud scale, so "free storage forever" (or even, storage at a one time fixed price) as the fallback isn't a viable option.

Lastly, from an optics perspective, which blog post would you rather see on the front page of HN: "I did something dumb and spent too much money on Cloud" or "Google is holding our data hostage" (or "Google deleted all my data")?

Source: I launched Firebase Storage, which has a GCS bucket that has a hard limit.

For it to work, obviously the budget has to be per month (for instance, $100/month), instead of an absolute limit. Most of the time, that's what you'd want: if you calculated that what you use will cost $50 each month, setting a budget of $100 per month would give some room for growth while preventing billing disasters (and you can always increase it a bit if necessary).

Off the top of my head I'd say that if you're budgeting for storage, the max you can afford for the time period that you'd need to recover data in the event of a budget overrun taking into considerations the delay time for notifications would be a way to calculate that. And that sounds like something that is reasonable to put on the customer to calculate.

You've explained why it's hard for Google to not give me resources I can't pay for, but that's not what I care about, or what I'm asking for. What I'm asking for is a feature where I set a hard limit of $100 and that's the most I get billed - if my account accidently uses $5000 of resources before Google reconciles the usage with my budget then Google automatically waives the additional $4900 and then limits my account in some way until the problem is rectified.

Practically every time these blog posts come up they end with the provider refunding the costs. I just want that refund to be a feature.

So...you're saying that Google should give away $4900 of usage?

How will Google automatically differentiate between an "honest mistake" and someone taking advantage of this feature?

So...you're saying that Google should give away $4900 of usage?

Yes. But they should also develop mechanisms to warn users that they've made a mistake before it happens, and improve the speed they can detect mistakes to lower the cost, and invent some way to detect someone intentionally abusing the feature.

But mostly they should make the fact they do give away $4900 when a mistake happens explicit. That isn't actually a change. They just need to make it clear that's what happens.

It's almost like you could make it configurable so users can choose what happens if they go over, and to what extent.

It's not really that complex. All compute should shut down. All API calls should fail. Storage should be (optionally) preserved at normal cost.

Your examples are simple given this framework. Uploading/downloading data to storage is an API call. Monitoring is compute. Metric history storage is storage.

But storage costs continue to add up even when you're not accessing them - there's a cost to storage existing which continues to accrue with time.

When there's no budget left, what do you do with those accruing costs for existing storage?

Storage costs are predictable and slow to accumulate. They are rarely the problem people are trying to address when they set a budget. As I said, storage would optionally continue to be charged at the normal rate, the other option being immediate deletion if you really need a super hard budget cap.

Once you get the alert that your budget is tripped you can go and see what's in storage via the console and delete it, only paying for a few hours of storage for things you don't want.

If the amount of storage that you can use is limited by quota (say 50GB) the problem becomes relatively easier.

You set a quota for 50GB of storage and no more. The server then restricts you by disk quota to that amount of storage.

The cost is then calculated as 1.15USD per month.

So you don't pay more than 1.15 per month.

Compute and transfer (and other things) could be covered by separate similar quotas with a single maximum spend figure at the bottom of the table.

Moreover, once API calls are locked, what next? You can't delete files, and even if you can delete them, you aren't able to retrieve them before deletion... If a platform allows you to do those actions, then it's rife for abuse, and at public cloud scale that ends up being a far, far bigger problem than the occasional blog post that ends up as a refund (because the other blog post is "I got free storage forever with this one weird trick").

It's really not a simple problem because the next action depends on the choice the developer wants to make: do they increase the budget or decrease usage, and no cloud provider wants to make this choice because no matter what the choice is it will be viewed as wrong. The best they can do is provide developers the best insight and tooling to make this choice themselves.

Once API calls are locked you can open the console, disable all the things that caused you to hit your budget, and then raise the budget a bit to get access to the storage APIs again and manage your storage. Or, the console's storage browser should let you browse and delete files as well. And again, there should be an option to delete all storage immediately for a hard cap on your budget if you really want that.

You need separate costed quotas for each type of activity with a combined total at the bottom.

You could also have a setting in the admin panel as to what the system should do:

[ ] I want to keep going beyond my quotas (but email me)

[ ] Please shutdown my site

If the answer is "you have a dollar limit set of GCS GETs, GCS PUTs, etc." I guess I could see this working, but hot damn that'll be a horrific interface.

The other issue is that many large customers pay different prices, so billing and quota aren't really tied to each other, and it wouldn't be easy to reconcile this.

As for the button... having been on the product side of building this button, there is no right answer: people will say they never got the email (or it went to the wrong inbox, or their dog ate their phone...) or that they never checked the box to "shut down the site" ("I didn't think it would do X that made my app not work").

I'd probably want it grouped by category with a drill down interface for the specifics.

Probably arranged so you can type in a figure at the bottom for monthly expenditure and it would balance out the requirements based on typical use cases.

So enter $50 in the monthly cap figure and it allocates, say, $20 to compute, $20 to transfer operations and API calls, $10 to storage

which you could then fiddle with of course.

I can't offer much on the second point other than to say that unexpected bills annoy me much more than services that stop working.

I've also never worked anywhere with unlimited budgets. (alas)

I can see that there are probably cases where uptime is more important so they would be more annoyed the other way around.