▲ 461 points
back
134 comments
> As for fly.io, well, I work there, so, they pay me.
Well, that’s nice and all, but if a fly.io customer were attacked with 3.1GB/s throughput, according to the lowest outbound bandwidth price of $0.02/GB [1] they’d be burning at least $3.72/min. 6 times that if attacked from India. That would be a lot less fun.
[1] https://fly.io/docs/about/pricing/
Edit: They mentioned they waive charges as a result of attacks: https://community.fly.io/t/about-rate-limiting/156/4
We do waive attack related charges. We're not interested in making money on bandwidth. We have a blog post with a few more details here: https://fly.io/blog/we-cut-bandwidth-prices-go-nuts/
That said, bandwidth does cost money. There are three ways we could handle it:
1. Charge as little as we can get away with, be transparent about it, eat the cost when someone has a negative experience.
2. Charge for bandwidth capacity, but don't meter it (ie: give VMs unmetered 100mb interfaces).
2. Don't charge for it, call it unlimited, put a hidden cap in place, and restrict what kinds of apps can run on the platform.
We opted for #1. I could give you a lot of post hoc reasoning, but the reality is that it's what I'd prefer as a customer. Companies that promise "unlimited bandwidth" feel a little slimy to me. Amos wouldn't be able to run his video hosting on one of those platforms.
Unmetered interfaces with restricted throughput do seem pretty nice. I've used a bunch of services like that. The cost to get started is high, though. And in my experience, the quality is poor. I've never had worse network performance than when I was paying for an unmetered network connection. Which makes sense, because these people attract all the users who want cheap, unmetered bandwidth. And they can't really afford to build enough upstream network capacity to handle all of them.
I don't love surprise expenses any more than you do. I do think we picked the least bad option, though, even though it puts some people off.
Indeed. fly.io encouraged me to move projects there, but since I budget my side projects as if they were an actual business, parts of the pricing page turned me off.
For the time being, I've decided that getting insights into "how well the platform worked for me" was more valuable for me, so my video platform is staying there, but I've initiated multiple discussions about pricing and I intend to keep doing so until I'm happy with the answer.
OpenResty (Nginx + LuaJIT) can help you limit the damage of unsophisticated DDoS attacks like these. I keep a count of the requests-per-second I'm getting in each nginx worker. I also set a special cookie for every response from the upstream (it could literally be foo=bar). When the RPS goes approve a certain threshold, if the special cookie is not present, I serve a static HTML page (bypassing the upstream) that sets the cookie and reloads the page (Nginx can do 20K+ RPS without breaking a sweat). In my experience, these fly by DDoS attacks never use cookies, so legitimate users can get through, but the bots are blocked.
Of course, if you get hit with something slightly more targeted, this defense is worthless.
Either it's the cost/benefit ratio that been keeping them from handling this, or it simply hadn't crossed their minds until now in which case you might soon need to come up with a new mitigation strategy ;)
Mirror in case the attack successfully picks up again: https://web.archive.org/web/20220502013024/https://fastertha...
As a treat, this is a testament to a logic error I made in the caching code (inserted uncachable versions of pages into the cache for a little while). Enjoy!
You have mentioned AS8075 (Microsoft), but with a comment of "Corp MSN"?. It's definitely using Azure (whether compromised or rented is something that I can't ascertain). The MSN here is due to legacy: it stands for MicroSoft Network (back in the days that MS is also an American ISP, although it seems that they're still a dial-up ISP: https://en.wikipedia.org/wiki/MSN_Dial-up).
Also, Contabo Asia (AS141995) is misclassified as in Germany. Although they are German, that AS is exclusively used for their Singaporean operations.
I'm spending a lot of time in Rust but I always learn so much new when reading your articles. Particularly, how do you manage find all these interesting crates that you're using. Like arc-swap, or Moka, or color-eyre. Do you actively search for these things or do you have a habit of reading the what's new crates.io feed in the morning with a cup of coffee?
Regarding the issue of people sending requests directly to your server, bypassing Cloudflare: While you can use IP allowlisting for this, Cloudflare can also be configured to send requests to your server exclusively over mTLS, with a client certificate chaining up to a root CA which they publish. If I were worried about this kind of attack, I'd probably turn that feature on (and then reject requests that don't present a valid client cert); I don't fully trust IP allowlisting. (On the other hand, I can't say for sure that the performance overhead wouldn't be a problem in a DDoS scenario. But it doesn't seem super likely?)
This is a good alternative when security is the priority. Wouldn't verifying an mTLS certs validity be significantly more expensive than whitelisting IP blocks though? Especially when a server is potentially under duress? Can you cache the verification so it wouldn't have to be done each time? All legitimate questions! I don't have the answers but am curious.
Or you use Cloudflare Tunnel and don't open Ports 80/443 at all.
Fun!
I tend to be paranoid about exposing things to the Internet, so just put my raw servers behind Envoy. I have tuned that to do rate limiting, circuit breaking (stop sending requests to an upstream when it returns too many errors), idle connection termination, and to shed load when a certain amount of memory is in use, so without any additional configuration for a new service behind the proxy it's somewhat difficult to get the proxy and other services to not respond at all.
I'm guessing that in a real attack, the rate limiting service is a weak link. I use a custom rate limit service to aggregate rate limits across a /24 (and hacked that together in an evening), and that is likely the first thing to blow up and erroneously deny service to legitimate users. (I'm sure I have it set up to fail closed, which will be annoying.)
I had a hard time ever generating enough load to test any of this for the static serving path. I just set up a mirror of my production environment on my workstation, limited the critical services (Envoy + nginx + rate limit + Redis) to some low amount of CPUs, and then had 31 workers generate synthetic load. I was able to get circuit breakers to open to at least prove that that code works, but I somehow think that I'll run out of network bandwidth before I run out of memory to keep track of open streams. Difficult to load test when the upstream can respond to most requests out of memory.
Would be interesting to dig into it more. But for those of you reading this and thinking "I'm going to launch an attack right now", I will just turn off the site if I go over my bandwidth quota. Clone the config repo, host everything locally, run your tests, and send me the results ;)
This inspired me to actually look at my CloudFlare stats, and I realized it's not caching some of my HTML pages even though I had page rules setup to cache everything. I use a static site generator, and it's never been an issue at this point (nobody has any reason to DDoS me). But this is a good motivator to fix that up. I've been procrastinating on fixing a few things on my site, so I'm adding this to the list when I update it. I might also setup the CloudFlare Tunnel at the same time.
Hello, friendly Cloudflare employee here... there's a much simpler alternative to managing cache policy if your whole website is static. We built a tool for deploying static websites to Cloudflare [1]. It watches your Git repo for changes, then rebuilds your site and serves _all_ the assets from our edge servers. No need to configure page rules or cache policy, because there's no more origin. The free tier covers 500 builds per month (at most, you'll do 1 build per commit, so you should be covered just fine).
I helped develop Tunnel over the last ~3 years, and I love it, but it's definitely overkill if you just want to serve some static files on the edge.
(Please delete if this goes against HN policy, I'm trying not to be a shill here, just help tristor avoid spending an hour configuring page rules and cache policies)
It's more fun in my head if "CloudFlare Attack Mode" allows you to wield CloudFlare as a Black-ICE weapon
https://en.wikipedia.org/wiki/Intrusion_Countermeasures_Elec...
I had never considered service back-pressure or circuit breakers until it became a buzz in the java world a few years ago. The concepts are universal though and genuinely interesting building blocks for architecture.
Site Reliability Engineering is a fascinating problem space.
Probably doesn't matter much with only a few networks, but this is using the wrong data structure:
if let Some(net) = ip_nets.load()
.iter()
.find(|net| net.contains(&addr.ip()))
Something like https://lib.rs/crates/iprange
The `iprange` library contains two bits of `unsafe` code [0].
I looked for that kind of data structure for 30 seconds in the ipnet crate itself, didn't find it, noticed there were only 23 IP ranges and decided it was fine.
(Keep in mind this happened during the attack, so compromises)
why
Out of curiosity, I googled how much a DDoS attack goes for these days. Apparently they can cost as low as $10/hour. I don't know if the shady people will deliver, but that's what the internet says. So apparently it's pretty easy to DDoS anyone and make it difficult to trace back to you.
This is only if you have _no_ idea how to use very basic open-source tools to wreak havoc via some open proxies. The real cost of launching small-scale attacks like this is $5/month on your favorite VPS provider.
> if you want to skip it, search for "After the storm".
> Yes, yes, I know, I should add anchor links for headers.
This is a rare case where blind people using screen readers have it (a little) easier. Every serious screen reader I know of has a command to skip to the next heading. It's too bad most sighted web users don't have a similar feature handy.
An attempt was made: https://news.ycombinator.com/item?id=30463632
That was fun! Showed some caveats to "I have cloudflare so I'm fine."
What a great post-mortem write-up! Thans for sharing
For sites that are mostly read only from non-logged in users Cloudflare can be great. I've got a WordPress content site configured with Cloudflare's APO. Almost 100% of actual human requests for HTML pages or static resources are cached. And there are also a few caching layers at the server level too (setup automatically by Cloudways). The site generates around $15k per month and growing fast on an overprovisioned $50/mo server. I don't think I'll need to spend much more on hosting at even 10x the current traffic due to the caching by Cloudflare. Maybe a bit more for disk space since we store large images but DigitalOcean's Block Storage also makes that incredibly cheap
I switched my site to static and it's become hassle-free to deploy ever since. Just copy the files anywhere and you're done. I assume it can't be DDoSed either, though I don't think nobody has ever tried (or they did and I didn't notice).
> That lets me answer questions like "what RSS readers (that aren't browsers) is my audience using?"
Very interesting... I also use NNW and FreshRSS
I'm working on two projects that could let a stand-alone Rust web server weather a moderate DDoS attack like the author had:
Beatrice [0] - A web server with built-in connection limits and thread limits. It's async but supports non-async request handlers.
fair-rate-limiter [1] - In theory, one could use this to shed most of the load from DDoS attacking nodes.
> The traffic doesn't look like something like headless Chrome was used
Setting aside the fact that headless chrome or other browser testbeds do a good job at hiding their presence, what could be the vector for a botnet infection if this were true? Extensions?
> Setting aside the fact that headless chrome or other browser testbeds do a good job at hiding their presence
It doesn't look like headless chrome, or headed chrome, or any kind of chrome... because there's just requests for one file and no other resources. Chrome would do a lot of other stuff.
This is a wget loop or some other minimal agent with a changed user agent string.
Could be anything.
Could even be just a user with a few beefy machines and a lot of proxies.
Most botnets I've encountered seen seem to consist of enterprise routers and the like, random IoT-stuff. Most of it probably too old and low-powered to even run headless chrome.
> Minutes after I posted this article, the attack resumed. Same shit, different AS.
Noob question: was that caused by the article getting posted to HN? Or was it really an attack?
This guy has some great energy.
And humor:
>> Because it doesn't return an AddrStream but instead a Pin<Box<TimeoutWriter<TimeoutReader<TcpStream>>>>...
>> Gesundheit.
I'm interested in what the fly.io would cost if you didn't work there.
I got a fly app and can't proxy it through cloudflare cause that doesn't work.
I have proxied a fly.io app through cloudflare. The last time I tried, it worked.
A security audit of my small 6 person office found a Digital Ocean machine trying to brute force one of our Windows machines. I'm becoming less and less impressed with Digital Ocean as time goes on.
does it make you a freeloader xD
Not to criticize, but how the hell do you write this long posts? Why don't others do it too?
I was very surprised to see that your article about golang got flagged here on HN. Never realized that go was such a sensitive topic.
I assume this is related? Can anyone in the golang community give some context?
Why was the site breaking under such a little stress? For requests per second, that is really not that much.
The code in this article is great advertisement for Golang.
Pin<Box<TimeoutWriter<TimeoutReader<TcpStream>>>>? Gesundheit indeed.
Great writing as always.
34 million requests really isn't many.
Bigger sites might handle that number of requests every second. Hand coded and highly optimized services can handle that number of cached small requests every minute on one machine. After all, that's only an egress rate of a few GBits. And your homepage certainly ought to be both cached and small.