▲ 1179 points
back
418 comments
> The biggest effect was that it gave our tiny engineering team the productivity of a team 50x its size.
I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out.
But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to be possible, which then gives amazing leverage to deliver working software in a fraction of the time.
It seems like the industry would get a lot more 10x behavior if it was recognized and rewarded more often than it currently does. Too often, management will focus more on the guy who works 12 hour days to accomplish 8 hours of real work than the guy who gets the same thing accomplished in an 8 hour day.
Also, deviations from 'normal' are frowned upon. Taking time to improve the process isn't built into the schedule; so taking time to build a wheelbarrow is discouraged when they think you could be hauling buckets faster instead.
When I was in college, I've met a few people that coded _a lot_ faster than me. Typically, they started since they were 12 instead of 21 (like me). That's how 10x engineers exist, by the time they are 30, they have roughly 20 years of programming experience behind their belt instead of 10.
Also, their professional experience is much greater. Sure, their initial jobs at 15 are the occassional weird gig for the uncle/aunt or cousin/nephew but they get picked up by professional firms at 18 and do a job next to their CS studies.
At least, that's how it used to be. Not sure if this is still happening due to the new job environment, but this was the reality from around 2004 to 2018.
For 10x engineers to exist, all it takes is a few examples. To me, everyone is in agreement that they seem to be rare. I point to a public 10x engineer. He'd never say it himself, but my guess is that this person is a 10x engineer [1].
If you disagree, I'm curious how you'd disagree. I'm just a blind man touching a part of the elephant [2]. I do not claim to see the whole picture.
[1] https://bellard.org/ (the person who created JSLinux)
[2] https://en.wikipedia.org/wiki/Blind_men_and_an_elephant - if you don't know the parable, it's a fun one!
I'm tired of hearing about 10x engineers. I just want to be a good 1x engineer. Or good at anything in life realy.
Your definition is also vague. Someone still needs to do the legwork. One man armies who can do everything themselves don't really fit in standardized teams where everything is compartmentalized and work divided and spread out.
They work best on their own projects with nobody else in their way, no colleagues, no managers, but that's not most jobs. Once you're part of a team, you can't do too much work yourself no matter how good you are, as inevitably the other slower/weaker team members will slow you down as you'll fight dealing with the issues they introduce into the project or the issues from management, so every team moves at the speed of the lowest common denominator no matter their rockstars.
On my team, one of the main multipliers is understanding the need behind the requested implementation, and proposing alternative solutions - minimizing or avoiding code changes altogether. It helps that we work on internal tooling and are very close to the process and stakeholders.
"Hmmm, there's another way to accomplish this" being the 10x. Doing things faster is not it.
I've always thought a x10 is one who sits back and sees a simpler way - like some math problems have an easy solution, if you can see it. Also: change the question; change the context (Alan Kay)
(And absolutely not brute-force grinding themselves away)
What makes a car go fast? The brakes:
https://manuel.kiessling.net/2011/04/07/why-developing-witho...
Was it 50x productivity due to 10x engineers, or 50x productivity due to optimized company structure? (edit: obviously, these do not need to be mutually exclusive - it's a sum of all the different parts)
It's easy to bog down even the best Nx engineers if you keep them occupied with endless bullshit tasks, meetings, (ever) changing timelines, and all that.
Kind of like having a professional driver drive a sportscar through a racetrack, versus the streets of Boston.
This is a perceptive observation. In my experience, so called "10x" engineers are as productive as they are because they have a process by which they practice the development on software that anticipates future problems. As a result when they check something in, they spend very little time "debugging" or "fixing bugs" with code that does what they already need it to do.
It is always very useful as an engineer to log your time, what are you working on "right now" and is it "new work" , "maintenance work", or "fixing work." Then for each log entry that isn't "new work" thinking about what you could have done that would have caught that problem before it was committed to the code base.
I find it is much better to evaluate engineers based on how often they are solving the same problem that they had before vs creating new stuff. That ratio, for me, is the essence of the Nx engineer (for 0.1 < N < 10)
The point that Wilson makes that having infrastructure/tools that push that ratio further from "repair" work to "new work" is hugely empowering to an organization.
This might be the best introduction post I've read.
Lays the foundation (get it?) for who the people are and what they've built.
Then explains how the current thing they are building is a result of the previous thing. It feels that they actually want this problem solved for everyone because they have experienced how good the solution feels.
Then tells us about the teams (pretty big names with complex systems) that have already used it.
All of these wrapped in good writing that appeals to developers/founders. Landing page is great too!
This is a great pitch, and I don't want to come across as negative, but I feel like a statement like "we found all bugs" can only be true with a very narrow definition of bug.
The most pernicious, hard-to-find bugs that I've come across have all been around the business logic of an application, rather than it hitting into an error state. I'm thinking of the category where you have something like "a database is currently reporting a completed transaction against a customer, but no completed purchase item, how should it be displayed on the customer recent transactions page?". Implementing something where "a thing will appear and not crash" in those cases is one thing, but making sure that it actually makes sense as a choice given all the context of everyone elses choices everywhere else in the stack is a lot harder.
Or to take a database, something along the lines of "our query planner produces a really suboptimal plan in this edge-case".
Neither of those types of problems could ever be automatically detected, because they aren't issues of the programming reaching an error state- the issue is figuring out in the first place what "correct" actually is for you application.
Maybe I'm setting the bar too high for what a "bug" is, but I guess my point is, its one thing to fantasize about having zero bugs, its another to build software in the real world. I probably still settle for 0 run time errors though to be fair. . .
I've been super interested in this field since finding out about it from the `sled` simulation guide [0] (which outlines how FoundationDB does what they do).
Currently bringing a similar kind of testing in to our workplace by writing our services to run on top of `madsim` [1]. This lets us continue writing async/await-style services in tokio but then (in tests) replace them with a deterministic executor that patches all sources of non-determinism (including dependencies that call out to the OS). It's pretty seamless.
The author of this article isn't joking when they say that the startup cost of this effort is monumental. Dealing with every possible source of non-determinism, re-writing services to be testable/sans-IO [2], etc. takes a lot of engineering effort.
Once the system is in place though, it's hard to describe just how confident you feel in your code. Combined with tools like quickcheck [3], you can test hundreds of thousands of subtle failure cases in I/O, event ordering, timeouts, dropped packets, filesystem failures, etc.
This kind of testing is an incredibly powerful tool to have in your toolbelt, if you have the patience and fortitude to invest in it.
As for Antithesis itself, it looks very very cool. Bringing the deterministic testing down the stack to below the OS is awesome. Should make it possible to test entire systems without wiring up a harness manually every time. Can’t wait to try it out!
[0]: https://sled.rs/simulation.html
[1]: https://github.com/madsim-rs/madsim?tab=readme-ov-file#madsi...
[2]: https://sans-io.readthedocs.io/
[3]: https://github.com/BurntSushi/quickcheck?tab=readme-ov-file#...
The writing is really enjoyable.
> Programming in this state is like living life surrounded by a force field that protects you from all harm. [...] We deleted all of our dependencies (including Zookeeper) because they had bugs, and wrote our own Paxos implementation in very little time and it _had no bugs_.
Being able to make that statement and back it by evidence must be indeed a cool thing.
Three thoughts:
1. It's a brilliant idea that came at the right time. It feels like people are finally losing patience with flaky software, see developer sentiment on: fuzzers, static typing, memory safety, standardized protocols, containers, etc.
2. It's meant to be niche. $2 per hour per CPU (or $7000 per year per CPU if reserved), no free tier for hobby or FOSS, and the only way to try/buy is to contact them. Ouch. It's a valid business model, I'm just sad it's not going for maximum positive impact.
3. Kudos for the high quality writing and documentation, and I absolutely love that the docs include things like (emphasis in original):
> If a bug is found in production, or by your customers, you should demand an explanation from us.
That's exactly how you buy developer goodwill. Reminds me of Mullvad, who I still recommend to people even after they dropped the ball on me.
I met Antithesis at Strangeloop this year and got to talk to employees about the state of the art of automated fault injection that I was following when I worked at Amazon, and I cannot overstate how their product is a huge leap forward compared to many of the formal verification systems being used today.
I actually got to follow their bug tracking process on an issue they identified in Apache Spark streaming - going off of the docs, they managed to identify a subtle and insidious correctness error in a common operation that would've caused headaches in low visibility edge case for years at that point. In the end the docs were incorrect, but after that showing I cannot imagine how critical tools like Antithesis will be inside companies building distributed systems.
I hope we get some blog posts that dig into the technical weeds soon, I'd love to hear what brought them to their current approach.
I'm trying to avoid diving into the hype cycle about this immediately - but this sounds like the holy grail right? Use your existing application as-is (assuming it's containerized), and simply check properties on it?
The blocker in doing that has always been the foundations of our machines: non-deterministic CPUs and operating systems. Re-building an entire vertical computing stack is practically impossible, so they just _avoid_ it by building a high-fidelity deterministic simulator.
I do wonder how they are checking for equivalence between the simulator and existing OS's, as that sounds like a non-trivial task. But, even still, I'm really bought in to this idea.
> a platform that takes your software and hunts for bugs in it
Ok but, what actually IS it?
It seems like it is a cloud service that will run integration tests. I have to figure out how to deploy to this special environment and I still have to write those integration tests using special libraries.
But even after all that integration refactoring, how is this supposed to help me find actual bugs that I wouldn't already have found in my own environment with my own integration tests?
I got really excited about this, and I spent a little time looking through the documentation, but I can't figure out how this is different than randomizing unit tests? It seems if I have a unit test suite already, then that's 99% of the work? Am I misunderstanding? I am drawing my conclusions from reading the Getting Started series of the docs, especially the Workloads section: https://antithesis.com/docs/getting_started/workload.html
Great read. Great product.
I've been an early user of Antithesis. My background is dependability and formal distributed systems.
This thing is magic (or rather, it's indistinguishable from magic ;-)).
If they told me I could test any distributed system without a single line of code change, do things like step-by-step debugging, even rollback time at will, I would not believe it. But Antithesis works as advertised.
It's a game-changer for distributed systems that truly care about dependability.
I don’t want to sound silly, but there are 24 open and 37 closed bugs on the FoundationDB Github page. Could it perhaps be that bug-free is somewhat exaggerated?
Antithesis looks very promising by the way :-)
Edit: perhaps Apple didn’t continue the rigorous testing while evolving the FoundationDB codebase.
FoundationDB is an impressive achievement, quite possibly the only distributed database out there that lives up to its strict serializability claims (see https://jepsen.io/consistency/models/strict-serializable for a good definition). The way they wrote it is indeed very interesting and a tool that does this for other systems is immediately worth looking at.
I am really intrigued by organizations that build effective test cultures. I am not interested in people who have testing teams (ala how it was done before, say 2004) or teams that simply do unit tests and integration tests. I am interested in people who realized that building the right testing culture is key to their success. Before reading this article sqlite would probably be my top reference. I don't have the article handy, but the sqlite developers spend like a year building a test framework to make incredibly bulletproof software. I wasn't aware of foundationDB before but the idea of the simulation engine - that's exactly what most distributed systems folks need.
disclaimer - I work at AWS. And we have a combination of TLA+, fuzz, and simulation testing. When I first started it was obvious my team had a huge testing gap. It pains me to say this but for a big part of AWS testing is sort of an after thought. It comes from the "we don't hire test engineers" mentality I suppose - but this likely differs wildly by team. Over the years we've tried to backfill the gap with simulators. But it is really hard to do this culturally. because it is really hard while you are trying to build new stuff, fix bugs, etc. And your entire team (and leaders) have to be bought into the foundational value of this infrastructure. And because we don't have it, we write COEs, we fix the obvious bugs, but do we take the 1 year it would take to avoid all problems in the future - no. So yeah I am super jealous of your "fully-deterministic event-based network simulation".
Congratulations to the Antithesis team!
I actually interviewed with them when they were just starting, and outside of being very technically proficient, they are also a great group of folks. They flew my wife and I out to DC on what happened to be the coldest day of the year that year (we are from California) so we didn’t end up following through but I’d like to think there is an alternative me out there in the multiverse hacking away on this stuff.
I highly recommend Will’s talks (which I believe he links in the blog post):
Checking their bug report which should contain "detailed information about a particular bug" I am not sure I can fully understand those claims: https://public.antithesis.com/report/ZsfkRkU58VYYW1yRVF8zsvU...
To my untrained eye I get: Logs, a graph of when in time the bug happened over multiple runs and a statistical analysis which part of the application code could be invovled. The statistical analysis is nice but it is completely flat, without any hierarchical relationships making it quite hard to parse mentally.
I kind of expected more context to be provided about inputs, steps and systems that lead to the bug. Is it expected to then start adding all the logging/debugging that might be missing from the logs and re-run it to track it down? I hoped that given the deterministic systems and inputs there could be more initial hints provided.
On mobile, the "Let's talk" button in the top right corner is cut off by the carousel menu overlay. Seems like CSS is still out of scope of the bug fixing magic for now.
On a more serious note, it's an interesting blog post, but it comes off as veeery confident about what is clearly an incredibly broad and complex topic. Curious to see how it will work in production.
Not directly related to this post, but clicking around the webpage I chuckled seeing Palantir's case study/testimonial:
I got similar productivity boosts after learning TLA+ and Alloy.
Simulation is an interesting approach but I am curious if they ever implemented the simulation wrong would it report errors that don't happen on the target platform or fail to find errors that the target platform reports? How wide the gap is will matter... and how many possible platforms and configurations will the hypervisor cover?
> We thought about this and decided to just go all out and write a hypervisor which emulates a deterministic computer.
Huh. Yes, that would work. It's in the category of obvious in hindsight. That is a very convincing sales pitch.
To me this is very reminiscent of time travel debugging tools like the one used for Firefox’s C++ code, rr / Pernosco: https://pernos.co/
I really like antithesis' approach: it's non-intrusive as all the changes are on a VM so one can run deterministic simulation without changing their code. It's also technically challenging, as making a VM suitable for deterministic simulation is not an easy feat.
On a side, I was wondering how this approach compares to Meta's Hermit(https://github.com/facebookexperimental/hermit), which is a deterministic Linux instead of a VM.
> The biggest effect was that it gave our tiny engineering team the productivity of a team 50x its size.
49 years ago, a man named Fred Brooks published a book, wherein he postulated that adding people to a late software project makes it later. It's staggering that 49 years later, people are still discovering that having a larger engineering team does not make your work more productive (or better). So what does make work more productive?
Productivity requires efficiency. Efficiency is expensive, complicated, nuanced, curt. You can't just start out from day 1 with an efficient team or company. It has to be grown, intentionally, continuously, like a garden of fragile flowers in a harsh environment.
Is the soil's pH right? Good. Is it getting enough sun? Good. Wait, is that leaf a little yellow? Might need to shade it. Hmm, are we watering it too much? Let's change some things and see. Ok, doing better now. Ah, it's growing fast now. Let's trim some of those lower leaves. Hmm, it's looking a little tall, is it growing too fast? Maybe it does need more sun after all.
If you really pay attention, and continue to make changes towards the goal of efficiency, you'll get there. No need for a 10x developer or 3 billion dollars. You just have to listen, look, change, measure, repeat. Eventually you'll feel the magic of zooming along productively. But you have to keep your eye on it until it blooms. And then keep it blooming...
I was mentally hijacked into clicking the jobs link (despite recently deciding I wasn’t going to go down that rabbit hole again!) but fortunately/unfortunately it is in-person and daily so, so flying out from Chicago a week out of the month won’t work and I don’t even have to ask!
More to the point of the story (though I do think the actual point was indeed a hiring or contracting pitch), this reminds me a lot of the internal tests the SQLite team has. I would love to hear from someone with access to those if they feel the same way.
"At FoundationDB, once we hit the point of having ~zero bugs and confidence that any new ones would be found immediately, we entered into this blessed condition and we flew. Programming in this state is like living life surrounded by a force field that protects you from all harm. Suddenly, you feel like you can take risks"
When this state hits it really is a thing to behold. Its very empowering to trust your system to this extent, and to know if you introduce a bug a test will save you.
the palantir testimonial on the landing page is funny
We've done something similar for our medical device; totally deterministic simulations that cover all sorts of real world scenarios and help us improve our product. When you have determinism, you can make changes and just rerun the whole thing to make sure you actually addressed the problems you found.
Another nice side effect is that if you hang on to the specification for the simulation, you only have to hang on to core metrics from the simulation, since the entire program state can be reproduced in a debugger by just using the same specification on the same code version.
Reminds me of my work in the blockchain space. I built a Decentralized Exchange which gets all of its inputs from blockchain data. Since all the inputs are deterministic (and it's all based in block time instead of real time), I can replay any scenario any number of times... But it goes one step further than what is mentioned in the article because I can replay any bug which ever happens, exactly as it happened, IN PRODUCTION.
The output is also deterministic so all operations are idempotent; they can be replayed any number of times, going as far back in the past as you want and end up with exactly the same state.
But, ironically, the DEX hasn't had any bug since it started operating 3 years ago... It makes debugging so easy, I was able to resolve every possible issue I could imagine before the launch.
I think I haven't updated the code in a year but I've been running it this whole time and it's been processing trades.
Another unusual feature is that you could technically trade between 2 blockchains while the DEX is offline... Even if the DEX is only restarted 1 year or decades later, it will resolve the trades as though the DEX had been online the entire time. Only difference is that the disbursement of funds will be in the future after the DEX is restarted.
In my career I learned two powerful tools to get bug free code. Design by Contract and Randomized testing.
I had to roll this by myself for each project I did. Antithesis seems to systematize it and created great tooling around it. That's Great!!!
However, looking at their docs they rely on assertion failures to find bugs. I believe Antithesis has a missed opportunity here by not properly pushing for Design by Contract instead of generic use of assertions. They don't even mention Design by Contract. I suspect the vast majority of people here on HN have never heard of it.
They should create a Design by Contract SDK for languages that don't have one (think most languages) that interacts nicely with tooling and only fallback to generic assertions when their SDK is not available. A Design by Contract SDK would provide better error messages over generic assertions, further helping users solve bugs. In fact, their testing framework is useless without contracts being diligently used. It requires a different training and mindset from engineers. Teaching them Design by Contract puts them in that frame of mind.
They have an opportunity to teach Design by Contract to a new generation of engineers. I'm surprised they don't even mention it.
I wonder if they are working on a time travel debugger. If it is truly deterministic presumably you could visit any point in time after a record is made and replay it.
I can see how determinism can be achieved (not easy, but possible), and I can see how describing a few important system invariants can match 100's or 1000's of hand rolled tests, but I'm having a hard time understanding how it's possible to intelligently explore the inputs to generate.
e.g. if I wrote a compiler, how would Antithesis generate mostly valid source code for it? Simply fuzzing utf8 inputs wouldn't get very far.
There’s a straightforward way to reach this testing state for optimization problems. Write 2 implementations of the code, one that is simple/slow and one that is optimized. Generate random inputs and assert outputs match correctly.
I’ve used this for leetcode-style problems and have never failed on correctness.
It is liberating to code in systems that test like this for the exact reasons mentioned in the article.
Hmmm, so this excited me a great deal since I spent some time building a property-based testing system for a trading execution engine. (Which really can't have any issues). It's an agent-based system with many possible race conditions.
Now, having built such a system (and being incredibly happy with it). I do not understand where antithesis comes in.
Broadly there were 2 components:
1. A random strategy that gives random commands that are distributed in a very specific way.
2. Properties to test, observing the system (by listening to the messages) and checking them for some properties (like positions always being correctly protected etc..)
Now, there is not much else (besides random trading data which is very easy to do or you can use actual live data). And all of them depend to 99% on domain knowledge. Now, where is there space for a company/generic framework to come in? I'm genuinely curious.
There are situations where no bugs is an important requirement, if it means no bugs that cause a noticeable failure. Things such as planes, submarines, nuclear reactors. For those there is provably correct code. That takes a long time to write, and I mean a really long time. Applying that to all software doesn't make sense from a commercial perspective. There are areas where improvements can have a big impact though, such as language safety improvements (Rust) and cybersecurity requirements regarding private data protection. I see those as being the biggest win.
I don't see no bugs in a distributed database as important enough to delay shipping for 5 years, but (a) it's not my baby; (b) I don't know what industries/use cases they are targeting. For me it's much more important to ship something with no critical bugs early, get user feedback, iterate, then rinse and repeat continually.
I love this idea. In the early days of computing computers were claimed to always be deterministic. Give it the same inputs and you get the same outputs. Little by little that disappeared, with interrupts, with multithreading, with human derived inputs, with multitasking, with distributed processing, until today computers and applications are often not deterministic at all, and it does indeed make them very difficult to test. Bringing back the determinism may not only be good for testing, it seems likely to improve reliability. While I see how this is great for distributed databases I wonder if it has application when inputs are inherently non-deterministic (e.g., human input, sensor derived inputs).
Was an eaaaaaaaarly tester for this. Pretty neat stuff.
I appreciated this post. Separately from what they are talking about, I found this bit insightful:
// This limits the value of testing, because if you had the foresight to write a test for a particular case, then you probably had the foresight to make the code handle that case too.
I often felt this way when I saw developers feel a sense of doing good work and creating safe software because they wrote unit tests like expect add(2,2) = 4. There is basically a 1-1 correlation between cases you thought to test and that you coded for, which is really no better off in terms of unexplored scenarios.
I get that this has some incremental value in catching blatant miscoding and regressions down the road so it's helpful, it's just not getting at the main thing that will kill you.
I felt similarly about human QA back in my finance days that asked developers for a test plan. If the dev writes a test plan, it also only covers what the dev already thought about. So I asked my team to write the vaguest/highest level test plan possible (eg, "it should now be possible to trade a Singaporean bond" rather than "type the Singaporean bond ticker into the field, type the amount, type the yield, click buy or sell") - the vagueness made more room for the QA person to do something different (even things like tabbing vs clicking, or filling the fields out of sequence, or misreading the labels) than how the dev saw it, which is the whole point.
This is how I have always developed hard-to-make-bug-free software. Write a simulator first that will deterministically give your software the worst possible environment to run in with failures happening all the time. I have delivered complicated software into production using this method with zero bugs ever reported. It works brilliantly.
Reminds me of the clever hack of playing back TCP dump logs from prod on a test network, but dialed up. Neat.
Naturally I’d prefer professional programmers learn the cognitive tools for manageably reasoning about nondeterminism, but they’ve been around over half a century and it hasn’t happened yet.
What’s really interesting to me is that the simulation adequately replicates the real network. One of the more popular criticisms of analytical approaches is sone variant of: yeah, but the real network isn’t going to behave like your model. Which by the way is an entirely plausible concern for anyone who has messed with that layer.
This is really exciting.
I am an absolute beginner at TLA+ but I really like this possible design space.
I have an idea for a package manager that combines type system with this style of deterministic testing and state space exploration.
Imagine knowing that your invocation of
package-manager install <tool name>
or an next gen Helm with type system and state space exploration is tested:
kubectl apply <yaml>