This sort of Monday morning quarterbacking is pointless and only serves as a way for random bloggers to try to grab credit without actually doing or creating any value.
back
5 comments
I disagree. I learnt good stuff from this article and it’s enough.
> I disagree. I learnt good stuff from this article and it’s enough.
That's perfectly fine. It's also besides the point though. You can learn without reading random people online cynically shit talking others as a self promotion strategy. This is junior dev energy manifesting junior level understanding of the whole problem domain.
There's not a lot to learn from claims that boil down to "don't have bugs".
I laughed out loud when he said Cloudflare should have formally verified its systems.
Not to single you out in particular, but I see this sentiment among programmers a lot and to me it's akin to a structural engineer saying "I laughed out loud when he said they should analyze the forces in the bridge".
You can't formally verify anything that uses consensus, which is the backbone of the entire web. It's a complete non-starter.
Care to elaborate? Perhaps the tools to do this in practice aren't there (which just shows how young the field of software "engineering" really is), but what consensus are you talking about and how is it an obstacle to verifying code? Most of the web follows standards and protocols, which actually sort of a prerequisite for communications across different systems...
Basically the modern web uses orchestration, for pretty much everything. Usually Kubernetes is doing that. Theoretically protocols like RAFT are formally verifiable, but their implementations in orchestration tools like etcd have not been, and I would go so far as to say that that is an impossible task. Therefore, the entire exercise is kind of silly.
A bridge failing is a high likelihood of death or serious injury. How many people died or were seriously injured in the latest Cloudflare outage?
For life or death systems, I agree that we should be looking to implement analogous processes/systems to a structural engineer or doctor, etc. Cloudflare is not a life or death system. If you operate a life or death system and you have Cloudflare as a single point of failure, for some reason, that should not be Cloudflare's problem.
> How many people died or were seriously injured in the latest Cloudflare outage?
I would not be surprised if the answer is "several". The average impact per human is obviously pretty small, but across billions of humans, there will be outliers.
Maybe a fire department uses a coordination system that relies on cloudflare, and with cloudflare down they have to resort to their backup system, and their backup system works but is slightly worse and causes one engine to be delayed in their response, and because they're 3 minutes late, they just miss being able to save someone from the fire.
Maybe someone's running a scientific study on nutrition, and the cloudflare outage means their data collection system is goes down for a bit, so their data flawed, and they end up just barely not passing a some necessary threshold, and they have to rerun their study, and that takes an extra week, and then they miss that quarter's deadline, and then the resulting adjustment to a product/procedure is delayed, and that 3 month delay causes 100,000 people to be slightly more malnourished than they would be otherwise, and one of those people ends up just barely too unhealthy to survive an unrelated deadly illness.
Sure, these scenarios are far-fetched. The chance of if it happening is one-in-a-million.
There are 10000 one-in-a-million people on the earth.
Sure, but this sentiment is why software "engineering" isn't really. You can justify it by not being important enough for actual engineering practices I guess, but to me it's a lack of pride in and care of your product.
more like "I laughed out loud when he said they should FEM the whole structure, down to the last bolt and strand of cable".
(More seriously, 'formal verification' is not a single thing, more a class of techniques which allow you to statically guarantee some properties of the system you are working with. When you propose it, you should have a clear idea of what properties you care about and how you intend to prove them, as well as a strong concern about whether those properties are actually going to capture enough of what you care about for it to be worthwhile)
It's very similar to LinkedIn posts, where everybody seems to know better than the people actually running the platforms.
This article actually explains how this bug in particular could have been avoided. Sure you may not consider his approach realistic, but it's not at all saying "don't have bugs". In fact, not having formal verification or similar tooling in place, would be more like saying "just don't write buggy code".
> This article actually explains how this bug in particular could have been avoided.
Not really. The article is a textbook example of hindsight bias. It's a simplistic analysis of a far more complex problem that goes over the blogger's head, and results in a string of simplistic assertions that fail to address any of the issues. Read up on the definition of monday morning quarterback.
Read up on the value of snarky and dismissive comments spouting simplistic cliches.
> You can learn without reading random people online
Somebody has to write something in the first place for one to learn from it, even if the writing is disagreeable.
You failed to cite the comment you were replying to.
The comment is:
> You can learn without reading random people online cynically shit talking others as a self promotion strategy.
Not commenting on the quality of this post but occasional writing that responds to an event provides a good opportunity to share thoughts that wouldn’t otherwise reach an audience. If you post advice without a concrete scenario you’re responding to, it’s both less tangible for your audience and less likely to find an audience when it’s easier to shrug off (or put off).
What did you learn? The suggestions in the post seem pretty shallow and non-actionable.
Like your comment? j/k :)
I'm using this incident to draw attention to Rust's panic behavior.
Rust could use additional language features to help us write mostly panic-free* code and statically catch even transitive dependencies that might subject us to unnecessary panics.
We've been talking about it on our team and to other Rust folks, and I think it's worth building a proposal around. Rust should have a way to statically guarantee this never happens. Opt-in at first, but eventually the default.
* with the exception of malloc failures, etc.
It's already in the box... there's a bunch of options from unwrap_or, etc... to actually checking the error result and dealing with it cleanly... that's not what happened.
Not to mention the possibility of just bumping up through Result<> chaining with an app specific error model. The author chose neither... likely because they want the app to crash/reload from an external service. This is often the best approach to an indeterminate or unusable state/configuration.
> This is often the best approach to an indeterminate or unusable state/configuration.
The engineers had more semantic tools at their disposal for this than a bare `unwrap()`.
This was a systems failure. A better set of tools in Rust would have helped mitigate some of the blow.
`unwrap()` is from pre-1.0 Rust, before many of the type system-enabled error safety features existed. And certainly before many of the idiomatic syntactic sugars were put into place.
I posted in another thread that Rust should grow annotation features to allow us to statically rid or minimize our codebase of panic behavior. Outside of malloc failures, we should be able to constrain or rid large classes of them with something like this:
panic fn my_panicky_function() {
None.unwrap(); // NB: `unwrap()` is also marked `panic` in stdlib
}
fn my_safe_function() {
// with a certain compiler or Crates flag, this would fail to compile
// as my_safe_function isn't annotated as `panic`
my_panicky_function()
}
In any case, until something is done, this is not the last time we'll hear unwrap() horror stories.
What you're suggesting is perfectly reasonable, I wouldn't object to labeling methods that can panic via bare unwrap...
I'm just saying that having a program immediately exit (via panic or not) could very well be the appropriate behavior.
pointing out that the basics matter is a valuable insight
if this comes with the side-effect of some random blogger getting "easy" credit then so be it
the same thing happens in economics, where the bitter lessons have to be learned again and again, and nowadays in politics (and even in ethics) too
and of course here people very correctly immediately started talking about the trade-offs involved in making sure that these aforementioned basics are really taken care of, and whether are there low-hanging fruits (does having a better DB schema provide enough benefit for it to worth it? or do you need to couple it with this or that compiler too? but isn't that simply too much? when the critical infrastructure provider should transition to the mindset that most people expect from a critical infrastructure provider? and so on...)
whereas in ethics (and intersectional politics) for example there's an unfortunate bitter denial of trade offs
Backdooring the internet is certainly a productive venture!
You may have missed the point of HN.