Entries from June 2015 ↓

Testing can’t even show the presence of bugs

In the 1968 NATO conference on Software Engineering, Dijkstra criticized testing as a “very inefficient way of convincing oneself of the correctness of programs”. In the next year’s follow-up conference, his objection turned into the famous judgment that “testing shows the presence, not the absence of bugs” (SoC, p.123).

Dijkstra’s statement was analogous to falsificationism in the philosophy of science: an experiment can disprove a theory, but never prove it to be correct.

But although Dijkstra was strong in his conviction, he did not take his argument far enough. In the philosophy of science, the Duhem-Quine thesis notes a weakness in falsificationism: when an experiment yields anomalous results, one can never be sure what caused those anomalous results. It’s not always the case that the theory was wrong: The anomalies could have resulted from anything in the test situation, including the theory, apparatus, theory of how the apparatus should work, or any of the auxiliary hypotheses and theories.

Similar, when program testing yields erroneous results, those erroneous results could result from a bug in the program, a compiler bug, a hardware bug, an operating system bug, a problem with the whole system’s interoperability, a cosmic ray interfering with the circuitry, and so forth. That is, running a perfectly correct program can give erroneous results regardless of the program correctness. Similar, a compiler bug may cancel out a program bug, letting a buggy program pass the test. In other words, testing can’t even show the presence of bugs in a single program. Testing can only show that “something” went wrong.