The Fallibility of Tests


Recently I posted about Test Flow and Method Contracts. The key takeaway was that we can use tests to prove the contracts our system lives up to.

We all create a logical representation of our software in our minds in order to reason about it. Tests (either manual, or automatic) allow us to prove the rules of that representation to be true.

Today I’ll talk about what it means when there are mistakes in those tests. For the purposes of this discussion, “tests” may be manual or automatic, but examples are given in code for clarity.

Test Fallacy vs Test Absence

The first distinction I want to make is between incorrect tests, and missing tests.

Test Fallacy is when a test exists, but creates a logical fallacy within our reasoning system.

For example:

// {new Cache} r = cache.has(x) {r == false}
public void has_shouldReturnFalse_whenItemHasNotBeenAddedToCache() {
    Cache c = new Cache();
    boolean b = c.has("random");

Clearly the code disagrees with the test name. Since the name of the test is more likely to correspond to the rule I have in my head that I believe the system follows, this test could be extremely destructive to my system.
Test Absence is when a test is absent, but we assume behaviour anyways. This is a logical mistake on our part, but it leads to bugs in the system.

I gave an example of this in my post Test Flow and Method Contracts where we “forgot” to define the behaviour of “has” when the cache is empty.

The problem with test absence is that we often assume the behaviour exists, and is “reasonable” by our own subjective measure. That creates a problem – what is “reasonable”?  

The Danger of Test Fallibility

While tests are fallible, either through a mistake or through absence, it’s useful to keep in mind why that is dangerous.

A professor for my logic class said something one class that stuck with me:

Logical fallacies don’t just create problems in logical systems, they destroy them. If you consider any false statement to be true, anything can be proven.

Let’s say, for example, that 2 = 3. I’ll prove I’m the Queen of England.

How? Well 3 – 2 = 1, but 3 – 3 = 0. Since 2 and 3 are equal, that implies 1 and 0 are equal.

Furthermore, 2 – 1 = 1, but 2 – 0 = 2. Since 1 and 0 are equal, so too are 1 and 2.

Now the Queen and I are, in fact, two distinct people. But 2 = 1, which means the Queen and I are one. So you see, I am in fact the Queen of England.
(Paraphrased from memory)

Honestly, I probably would have chosen to prove I was Metallica (the entire band), but if one admits that 2 = 3, the rest of the logic is difficult to argue against. With one false “fact”, we destroyed our entire reasoning system.

I’m sure many of us can relate this back to software. Try to recall a time when you received a bug complaint from a user, and after tracking down the bug you stared in awe at your computer screen. “That shouldn’t even be possible…”

The real problem? These “glitches” in our reasoning systems are inevitable. We’re all humans, we will make a mistake. One way or another, you will eventually have a bug in a test, or assume behaviour exists that really doesn’t.

How do we Address Test Fallibility?

This brings me back around to the original topic – how do we avoid this fallibility?

We automate our test suite.

It seems strange to say automation will “cure” all of our problems reasoning about systems, and that’s because it won’t. It’s simply the best way to combat the problem.

Automation addresses Test Absence by:

  • being cheaper (time) than manual testing
  • eliminating any reasons to skip tests
  • growing as a regression suite over time as we think of new test cases

If you forget a test, simply add it and it will never be forgotten again.

Automation addresses Test Fallacies by:

  • being repeatable
  • being correctable (you can’t prevent humans from re-making mistakes)
  • being easy to read, examine, and reason about

If you make a mistake in a test, simply fix it and the mistake will not reoccur.

In summary, automating your tests doesn’t guarantee that your tests will be good, but it allows you to confidently improve your tests over time. Good test allow you to reason correctly about your system.



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s