I am a big fan of CAST for software teams, and of MIT Prof. Nancy Leveson who leads CAST.
My CAST summary notes for tech teams:
https://github.com/joelparkerhenderson/causal-analysis-based...
MIT CAST Handbook:
There are two problems with automated testing. 1) tests take too long to run 2) difficult to root cause breakages.
Most devs solve this with making unit tests ever more granular with heavy use of mocks/fakes. This “solves” both problems in a narrow sense: the tests run faster and are obvious to root cause breakages.
But you didn’t actually solve the problem. Since the entire point of writing tests in the first place was to answer the question: “does my system work”? Granular and mocked unit tests don’t help much.
However, going back to the original question, we can actually reframe the problems as: 1) a work scheduling problem and 2) a signal processing problem.
Those are pretty well understood problems with good solutions. It’s just that this is a somewhat novel way of thinking of tests, so it hasn’t really been integrated into the open source tool chain.
You could imagine integration tests automatically be correlated to a micro service release. Some CI automation constantly running expensive tests over a range of commits and automatically bisecting on failure. Etc.
Put another way, automated tests don’t go far enough. We need yet another higher layer of abstraction. Computers are better at deciding what tests to run and when, and are also better at interpreting the results.
Sounds like you might be interested in https://antithesis.com/ (no affiliation).
I have gotten a ton of value out of targetting these problems specifically. Some low-cost-high-value test infra changes:
1. Passing tests are not allowed to emit error level logs without expecting them; if the expected error does not occur, the test fails. This is good in general, but also makes root causing test failures easier as if you encounter an unexpected error log you can attach it to the failure message, and often that pinpoints the root cause.
2. Have lots of off-by-default instrumentation. For example, if a test fails auto-rerun it with sanitizers on.
3. In tests, gather backtraces when moving tasks between threads. This makes identifying what triggered an error straightforward.
4. Collect data during test runs, like line coverage, in a sequential order that can be diffed between passing and failing runs.
As for making tests fast, the solution is aggressive optimization just like prod code - profile the test, and fix what makes it slow. Cut corners where it makes sense - should you ever fsync in a test? VM snapshotting can be very useful in this area for things that are slow to start up.
Here's a fast, easy, practical way to think about CAST:
1. Causal: Novices may believe accidents are due to one "root cause" or a few "probable causes", but it turns out that accidents are actually due to many interacting causes.
2. Analysis: Novices may blame people, but it's smarter to do blame-free examination of why the loss occurred, and how it occurred i.e. "ask why and how, not who".
3. Systems: Novices may fix just one thing that broke, but it turns out it's better to discover multiple causes, then plan multiple ways to improve the whole system.