back
126 comments
I find it interesting that while logic bugs (e.g. off by one errors, switched if conditions, etc) make up less than 1/3 of bugs I encounter day to day, it seems to comprise of about 80% of all discussion around testing.

Bugs caused by misunderstood or unexpected interactions between subsystems and specification bugs seem to predominate these days IME but discussion about sophisticated ways of dealing with these things seems fairly scant.

The roots of the problem you described are many. One is that there are very good unit test libraries out there, which encourages writing unit tests. That sounds good, and is a good starting point, but it often become the be-all end-all of testing.

In my experience working on many very large software, I have encountered one that was head and shoulder above the others in quality. This is not just a hunch. There were metrics tracked, thing like MTBF, and percent of sessions with clean exits, etc and this software was ridiculously better than the others.

The difference lied entirely on the fact that the better software had almost no unit test, but had 21K+ integration tests. I've worked on other software that also had more integration tests than average, and they too were of better quality.

The problem is making integration testing easy needs to be designed in from the start and requires not just one simple trick but multiple inter-locked design decisions that accumulate into making integration testing easy. In this particular case, one could write a new test just by using the software and a new test would be generated automatically. That's how you achieve writing 21k integration tests.

There are many unit test framework, but I've yet to see a generic integration framework. The reason is that integration tests are run with the whole software, so the whole software needs to be designed around them.

I guess it depends on a system you build and its maturity - in my area (automation for finance and revenue management) 80% of bugs are related to business logic and misinterpretation of user requirements.
That's because these are knowable errors; you can't talk about a bug if you don't know it might occur.

Testing is not a panacea, by design you can only test for what you know and predict. In addition to testing, your setup needs to be resilient enough that unpredictable bugs can be solved quickly and won't lead to data loss.

This sounds like the problem that despite all of the effort at preventing simple programming errors, 1/3rd of all the bugs you encounter are "simple" programming errors.

Programming errors are of special interest to programmers, as we tend to judge each other a bit harshly when these errors emerge as the problem is not on the business side or other teams making requirements.

I am both accountant (actually educated) and developer. I will not rant (today) on how I disagree with unit testing and there are better ways to get the result.

I just wanted to point out double bookkeeping analogy is bad.

Double bookkeeping is there to make certain types of errors very visible.

Double bookkeeping comes from idea that things do not just show up or disappear, but they change state -- value appearing on one account must have come from another. And so double bookkeeping is basically saying -- we want to have a system where it is immediately visible or even impossible to write something down that does not preserve this invariant.

It has nothing to do with whether the operation is accounted for correctly. You can still mess up accounts very easily.

Double bookkeeping does not mean double checking. You just select two accounts and put the same value on opposite sides of those accounts. This is a single operation. Everything about it can be wrong -- the value can be wrong, the accounts can be wrong and nothing is going to catch that error.

Actually, using invariants is one technique I use to ensure my code works correctly. Has nothing to do with unit tests. I just write code, whenever it is possible, in a way that makes it impossible to put it in a wrong state.

Regardless of the relative value to ascribe to unit testing, the confidence value is obvious from the intentional and functional double checking, which can be done piecemeal. There is still a problem with added side effects, which unit tests dont pick up on. This will probably be addressed with fn signatures...which doesnt do much but alert for changes. Eventually unit testing can start to approach formalism with monitored functions, wherein side effects must be asserted for a unit test to succeed.
Yes, tests are good.

At the same time, my measuring stick is always, "Is the customer happy?" Software will never be perfect, so a zero bug rate is not a reasonable goal. Happy customers is quite a reasonable goal, and when they are not, their concerns direct where you need more quality.

Tell that to NASA or your local nuclear powerstation provider.
Software will never be perfect, but most people aren't actually arguing for perfect. I've seen complicated web apps shit the bed with single-digit requests per second, and those developers also said it doesn't need to be perfect. Forget perfect, how about "doesn't immediately fall over under load"?
As far as I'm aware, unless your software is Hello World simple, the only way to make sure it's correct is to not write it in the first place.

After that point, everything is a compromise.

Well, my software is always correct.

Sometimes the spec is a bit wrong, like forgetting to be explicit in precisely how the system should crash on a certain unexpected input.

I write UI and communication software.

It is pretty much impossible to write meaningful tests that cover everything.

This is especially true, when you mix complex, asynchronous server interactions, with random-access UI.

Today, I am fixing a couple of bugs in an app that interacts with three different servers (all asynchronously, but some semi-synchronously. The timing diagram is a bit ... intense), and also allows the user to do things like switch contexts, while some part of the server interaction is still unresolved.

I write a bit about my approach to testing, here: https://littlegreenviper.com/miscellany/testing-harness-vs-u...

You can try, though - and demonstrate that you did. I've seen bugs that would have been trivially caught if the software had been run even one time - so it was clear that the software had never been run even one time.
> ... Hello World simple

End even then only if you stick to English strig hello world.

I have had hello worlds fail, in other languages :)

Formalized verification would say otherwise.
Among other, they demand/receive better requirements. Underspecified and underconstrained systems leave much space for variations of conditions “they may find themselves working in”. I put that in quotes because software is not sentient, and it’s really a programmer who encounters that enormous space at the implementation (or ignores it for some reason). When that happens, they often can’t ask anyone what to do, because nobody except them could understand the problem deep in the structure of a program/process. It leads to defensive approaches which may or may not trigger in the wild, producing cryptic messages, because it was hard to formulate it in the first place. Unrestricted underspecifity puts roots deep into the code, when they could be cut much earlier. Programmer’s clients and employers are rarely aware of that, and it’s a programmer’s responsibility to specify the details as deep as client is able to grasp.

It’s another (imo most important) level of correctness, unrelated to rigid math-like correctness of a result of programming of completely coherent requirements, which almost none of us ever receive.

Very well explained. Looking back I remember the projects where you had one of those unicorn colleagues/customers that truly could tie business and technical complexities together. It's often those projects that end up the best, because you actually CAN ask some about "uh I can see that this weird state could appear in the code, is this something we should work around in a special way or is the state not representable?"
I find odd that there was no discussion about figuring out edge cases and testing specifically for those instead of random values. Especially for functions with large domains, having few hand-picked test values could give much more confidence than thousands of random values.

Additionally with generated test values figuring out the validation logic becomes complex, and in complexity lies potential for bugs. If your testing code more complex than the code being tested, do you really gain that much confidence about the correctness? In comparison with hard-coded test values you can also hard-code the expected results, which can be gotten from some reference or just manually checked for correctness.

Of course ideally you'd have both random fuzzing and more manually defined tests, but that is setting quite high bar.

That's par for the course if you use an actual framework for Property-based Testing. All the ones I've tried certainly do mix in the usual suspects (-1, 0, 1, MAX_INT, empty list, etc.) into the randomly sampled data.

The author of this piece probably chose to make the post framework-agnostic by rolling their own random generation, but left that nuance out.

"Our most important goal in writing software is that it be correct."

Because I'm a particular kind of idiot, it took me about 10 years of working in and around software engineering to understand that this is just not true. In many, many cases the most important goal for software is for it to take up some space in the market (either the external market or the internal corporate reputation market). In order for software to meet this goal, it must more or less function, but it doesn't need to be _correct_. It can crash, make mistakes, lose data, whatever.

In many situations this goal can be understood as "gathering capital." You write software to impress investors or to attract users or to convince other people in the corporate hierarchy that you are a going concern. Thus the valuable part of the software is not its correctness but its _surface_. How shiny is the software? Does it impress and/or please users? Does it impress investors?

If you've read Guy Debord you could have predicted this is how software would work. Software is spectacle.

How shiny is the software? Does it impress and/or please users? Does it impress investors?

Yup. And in many cases (2) doesn't matter at all - just heaping portions of (1)+(3).

You know the kind of "software" I'm talking about -- the kind that exists (and does all kinds of things) on pitch decks, or in conversations with gullible industry press types. But in no other sense.

If we all wrote perfect code we'd be putting our QA departments out of work. Do we want that on our consciences?
I suppose there's a kind of hierarchy:

Can I get paid to work on this software?

Will customers buy this software?

Is this software fit for some purpose?

Is the software correct according to what the customer expects?

Writing correct software is just a straightforward way of meeting customer expectations, which is a way to ensure that the software is fit for some purpose, which is a way to convince customers to buy the software, which is a way to ensure that you'll continue to be employed working on the software. It's not the only way to get paid, though.

(Sometimes software is developed by end users, in which case you can skip over the "will customers buy this?" question.)

I hope that my life or preservation of the environment does not depend on the software you are working on being correct.

In fields where that is the case, methods for ensuring correctness are important.

I really liked it, and I will probably use it as documentation for newcomers in my team. However there is one point that I don't really agree:

"it can be difficult to add tests to a large application that has been developed entirely without testing. And that is why testing should be first on your mind when building serious software."

I've being doing it all my life. Join a new project, without any test, and start to add test code, while getting used with the code base. Actually, I would say that "adding tests" for a code base is the easiest and safest way to get used with the code base itself.

.. they don't!
I'm not sure why the example used for formal verification is Z3, and not proof assistants like Coq or Agda. SMT solvers have made verification a lot more accessible, but Coq is strictly more expressive than Z3.
Coq requires thinking like a functional programmer, and most programmers are resistant to that paradigm for some reason.
Right now I'm reading "PSP, A Self-Improvement Process for Software Engineers" by Watts Humphrey (2005): https://resources.sei.cmu.edu/library/asset-view.cfm?assetID...

PSP = "Personal Software Process", the step before TSP, "Team Software Process".

Personal software process (PSP): https://en.wikipedia.org/wiki/Personal_software_process

Team software process (TSP): https://en.wikipedia.org/wiki/Team_software_process

It's there for anyone who is interested. Right now I'm skimming it to see what I can learn. Can be quite involved but can reduce errors and improve quality by orders of magnitude. (Proven.)

Given that, Humphrey repeatedly emphasizes that communication and understanding about what needs to be built are the most important parts, and that trying to test quality into larger systems can literally take years (of full-time work, by teams of dozens of developers).

Since I'm retired and reviewing all the ways that I wasted my life, then for me it's only an intellectual exercise, but this could help someone still in the thick of it. Lots of food for thought.

I have recently realized that I am not progressing my skills in the way I wanted, partly because I cannot seem to get off the ground.

I read through the first chapter of PSP now, and this is exactly what I need! Thank you so much for sharing this.

> There are generally two main categories of tests. There are unit tests [and] Integration tests

Later on there is a mention of fuzzing. However, there is something better called property tests. Where the on failure there is automatic "shrinking" of input complexity to find a minimally failing test scenario.

For Erlang and Elixir I recommend https://propertesting.com/ book and the PropEr test framework https://github.com/proper-testing/proper.

As an example, here is the use of an advanced feature called targeted property based testing: https://proper-testing.github.io/tutorials/PropEr_testing_wi... used to test a labyrinth.

EDIT: as spockz mentioned below, the original idea was developed by QuviQ as QuickCheck http://www.quviq.com/products/erlang-quickcheck/. It's a commercial offering but there is a QuickCheck mini offered as a free download as well.

My understanding was that property tests and fuzzing are orthogonal in roughly the same way unit and integration tests are: you do property test to ensure well-delimited pieces of your code satisfy well-defined constraints, and you fuzz to ensure the whole thing just doesn’t go completely nuts when faced with something unusual. But I could be wrong because I never could figure out how to set up fuzzing for some sort of moderately-involved realistic thing (a CSV parser? something like that).
The original is quickcheck for erlang which also has an Haskell flavour. It has inspired scalacheck for scala.

I test pure logic with property based testing and simple integration testing to ensure all the steps are aligned properly.

Those are separate classifications, right? A unit test can be property-based and an integration test can be accomplished theough fuzzing, and versa.
+1
The article doesn't mention the additional problems with stateful software, which account for a lot of the worst problems in practice.

Consider an application with persistent state. Version 1 has a bug that introduces some slight inconsistency in the state file. Version 7 reads the state file, assumes that the state file is consistent, and ends up producing crazy results. Version 7 is "correct" but still fails.

One answer is to always validate the state file, but that may be impractical due to size or complexity.

A better answer is to use a database that offers declarative constraints that help prevent inconsistencies. This is such a good solution that something written in PHP could be very robust in practice if it uses a good database, whereas something written in haskell that uses a database without good constraints might fail miserably.

You don't need a full database, you just need well-specified serialization/deserialization. Typically this includes explicit versioning mechanisms, so the app can be aware of what version it is dealing with and convert the format accordingly.
I think a lot of the common memes around programmers like these is due to the wrong-think gatekeeping during the interview process that everybody is copying from FAANG.

I have 5-10 years programming experience and find an unlimited amount of bugs in other developer's work of any and all skill levels.

On occasion, something like a billing calculation will have a function doing the opposite of what they intend to do but somehow nobody has caught it in 5 years. I've also seen ancient security reports that just have a chunk of code that doesn't work like it sounds it should, causing 25% of results to be missed. These are just a couple of actual examples.

I focus on generally not screwing up my work like that as well as finding these problems in my team's code. It's mind blowing to me this has absolutely 0 value though. I and others like me are gatekept out of a lot of work since we process things differently. Developers can't seem to imagine a world where somebody thinks differently than them and memorizing leetcode is just as easy for everybody as it is them. "If you can't talk me through what you're doing while writing your memorized perfectly compilable code on a whiteboard or google doc, I guess you're either just lazy and not really invested, or a bad developer. Probably both."

The only thing that technically matters to get into the dev roles is memorizing leetcode problems in order to pass the interview so that A) You have a job at all (even small companies are doing these now)and B) You can get your actual salary corrected every couple years.

If it's not memorizing leetcode, it's how many (inefficient) lines of code did you contribute? (Thankfully this is finally starting to be phased out as a measurement AND bragging point.)

The people who make it through all this then go on to store their critical world-infrastructure Exchange version number in a 32 bit integer because "It worked and compiled when I tested it, so it's probably good enough." Nobody dares test it due to the toxic hostility back, or because a senior dev wrote it, or "I ran the code and it compiled for me, so it's good."

There's just so much talking down to others and gatekeeping about how leetcode solvers are superior and that it's the only possible starting block for a "real dev". I can't believe the frequency of simple yet major mistakes these people go on to actually do after they spend so much time bragging about it.

Sorely lacking from this article is the topic of static type systems and immutable data structures.

One guarantees the absence of certain classes of bugs; the other reduce the number of moving parts in a system.

I'm reminded of what I'm going to call a poem, but which isn't a poem, but more of a long saying. I can't remember the exact wording (if someone remembers and has the link, please link it!) but roughly:

If you tested your program, congratulations. Your program is now tested, incorrectly.

If you tested your program, and verified that the tests are correct, congratulations. Your program is now verified to do the wrong thing, correctly.

It has at least two other lines which I can't even think of the paraphrasing for.

But from a scientific standpoint, do you think a software can be 100% correct? Bugs will eventually be found.

The best bet, when writing (100%) correct code, is through thorough testing, and hoping for the best, because when you are handling a million lines of code, errors, will definitely creep up.

Personally, I feel achieving a 100% correct code is impossible, since developers are always updating, and adding new features to the code.

Hence why a team, should be put in place to create patches quickly, when bugs are found.

> But from a scientific standpoint, do you think a software can be 100% correct?

Sure it can. I've written 100% correct code using Coq. For example, I wrote a relatively simple program (~1.7k LoC) to interpret a simple programming language, but it was definitely correct (certified by a machine-checked proof).

Of course, for larger programs it's much harder to do that. But it's just a matter of how much time you're willing to invest.

On some level, that's sort of like asking whether we can be 100% sure that the solution to a math problem is correct. I mean, I'm pretty sure that 2+3=5, but what if I made a mistake somewhere? What if I'm falling prey to some cognitive bias that makes me blind to certain math errors? What if all of humanity is wrong about this math problem?

If we can be convinced that we're 100% sure that 2+3=5, then what about a more complicated problem? What's the most complex math problem we can be sure about? What if we have to use a computer to check its correctness?

(In general, I think it's usually good enough to say we're pretty sure something is correct if we have some solid basis for believing that it is. I'm okay with being 99.99% sure that 2+3=5, but in the world of software "I'm 50% sure this software is correct" is a standard that most software I use on a daily basis fails to meet.)

CompCert is an example of 100% proven correct software. There are other examples out there. Verification tools are getting better every day, making it less costly to prove code correct.
In hardware (chip) design code verification is done by seperarate engineers. Design engineers have some test code to kick the tyres, but sign off is done independently and rigoirlsy, with functional coverage linked to specification documentation. This is a cautious approach, but hardware bugs can be multi million dollars, so it pays to have this level of testing.
On a side note I always find it interesting to see the static type and algorithm fans taut how correct code is. Great we now got typescript everywhere… it’s still amazingly difficult to test and validate user interfaces, styles and layouts.
Interesting that the 2 books referenced at the end are from 2008 and 2012 (I read them when I was learning more about testing) - nothing new since then?
I'm surprised at the lack of stack overflow jokes in this thread.