Does anyone know how this even happens inside the walls of google? Is it an automated process? How is such a (presumably) high revenue account just magically blocked without human intervention? I'm quite perplexed.
back
2 comments
There would have been efforts to contact them, but it would have been via their contact method, aka the email they set it up with.
Common ways this happens? They are using a credit card to run their business with no backup payment method. Then the company's contact person is on vacation.
Sign up for terms. It will get you payment terms!
Yeah, I'm not sure what to think here. We know Google is not the best at customer service and has automated account suspensions. But, what I'm curious about here is why this happened.
Railway hosts applications for customers. An uneducated guess for some possible reasons: 1) one of those customers hosted something they shouldn't have 2) railway had something spawn that took up too many resources 3) Or their account balance was too high 4) Or something...
But all of this probably culminates in someone needed to read an email that was missed.
Scaling a customer infrastructure setup like Railway is hard. This is one of the non-technical hard parts - how to make sure your account with your primary vendor is safe. But, I'm willing to wait to pass judgement here until more information is available. I'm sure the post-mortem will have lessons. I'd like to know more.
> via their contact method, aka the email they set it up with
If it's anything like AWS, that may be just one of hundreds of emails they send every day, most of which are just noise.
Honestly still insane to nuke a high-volume client's business after a single payment issue. There would be no reason for Google to believe that a single hiccup like that is evidence that they won't get paid and have to cut account access immediately.
It's not a single payment failure, it would be multiple days, possibly even a week to 10days.
This is why businesses should put in the effort and sign up for credit terms. Then it's an invoice, and you reduce this risk substantially.
Credit cards are _not_ reliable at this scale. Banks are offline all the time, cards are marked stolen, protocols change, all sorts of things that will cause flags indicating "the money can't move down that path".
Businesses that pay for AWS/Cloud/etc via credit cards are trying to buy reliability but put it behind a single point of failure.
Credit cards are not how you should be paying for business services with uptime requirements!
It is insane, but my past experience with GCP is they suspended all service only days after a failed payment, after years of paying on time. It's a major factor in why I don't use them anymore. I'm not waking up to angry customers again because the CC is expired and I missed an email.
I'd be curious to know why Railway's account was suspended. Was it a similar payment issue or something else?
Railway might not be even in the realm of high-volume clients for Google. For all we know they might be efficient in utilizing Google infrastructure.
But most likely, it's just automations in place without an appropriate human override coupled with gross negligence.
Yeah, compared to the AWS experience:
I had a toy Free Tier account that managed to overstep a limit one month and rack up $0.0038 in charges.
AWS hounded me about it for an entire year before finally putting the account on hold. Then kept at it for months more before finally deleting it.
It’s pike the paperboy from Better off Dead, if he were to continue delivering newspapers while hounding you for his two dollars.