According to the great American philosopher C. S. Pierce, to attain a high level of clearness of apprehension, one must first “consider what effects, which might conceivably have practical bearings, we conceive the object of our conception to have. Then, our conception of these effects is the whole of our conception of the object.” Link
The question is what for. Why am I testing this? What am I expecting to get? What is the practical consequence of these steps?
This kind of questions allows us to design our test cases better. We are used to the execution order of test cases: preconditions – steps – expected result. But this is not necessarily the order in which we design them, and from a pragmatic mindset, it should not be. In a similar fashion, mathematicians often discover their proofs in a very different way than the one they logically lay them out.
When we think of a system, the most basic aspect to understand what it “is” is knowing what it does. That means that when the system is so complex that we find it difficult to narrow down the relevant elements for a test case, then we should ask ourselves: what should this be doing? Only then we ask ourselves how to accomplish that, and what we need to do it.
If we want to build a house, first we think of the final result. Then we think how we are going to build it and what materials we need. Or when we code a function, first we think of the return value, then of the algorithm to accomplish it and then what are the possible parameters we might need.
In other words, the way we discover test cases is the exact opposite of the way we document them: expected result – steps – preconditions.
This is directly related to ambiguity and vagueness, since defining test cases from the expected result necessarily entails adding precision to our definition of the output. Here I would like to outline a few points to take into account to make better test cases.
The general structure of a test case should contain preconditions, steps and “an” expected result. The word “a” is key here. The expected result should be ideally just one. Although it is true that we often do not have enough time to create all the test cases we would like to, but as a general rule there should ideally be a 1:1 ratio between test case and expected result.
The following example is considered a bad practice:
Preconditions: having Windows 10 and the latest version of Google Chrome installed.
Steps:
- Open Chrome and go to https://www.google.com/
- Search for “Lenio blog”
Expected Result:
- https://www.leniolabs.com/blog/ should be the first result.
- The first result should contain the most widely read post of the blog.
Why is this a bad practice? Because if the test fails, it creates ambiguity regarding the expected result that failed. Which one is it, 1 or 2?
Besides, there is another bad practice I would like to address, this time related to vagueness. Let me show you an example first and then I will do the explanation:
Preconditions: being logged in as a System Administrator.
Steps:
- Open “Settings” tab and click on “Update database”
Expected Result: the database should be updated accordingly.
Do you see what I did here? The problem here is not ambiguity, but vagueness, that little thing that happens when things are ill-defined. What does “accordingly” even mean? That is not even testable, because you cannot simply test something as generic as “updated accordingly”. According to what anyway?
This test case needs to be re-written into something awfully more specific, for example:
Preconditions: being logged in as a System Administrator.
Steps:
- Open “Settings” tab and click on “Update database”
Expected Result: the database should be updated to include the latest user IDs.
Interestingly, this notion of ‘specificity’ of test cases has a strong parallel in epistemology, the part of philosophy that deals with scientific knowledge. As Daniel Steel reminds us:
In The Logic of Scientific Discovery, Popper characterized a testable or falsifiable theory as one that is capable of being “refuted by experience”.
As is well known, Popper insisted that only falsifiable theories were genuinely scientific, but not because they were more likely to be true. To the contrary, Popper emphasized that the more falsifiable a theory is, the more it sticks its neck out, and hence the more improbable it is.
In other words, saying that the database should be updated “accordingly” is improbable, meaning that it is not likely to be verified by testing. This is essentially because there cannot be an experience of “accordingly”. “Accordingly” is just a general overview of what the expected result should do, but not a concrete description of what it should be.
In testing you need to be as specific as possible in the same way that you cannot just order a “soda” in a restaurant. You have to specify whether you want a Coke or a Pepsi. Only if you ordered a Coke and the waiter brought you a Pepsi you can complain, or its testing equivalent, report a bug.
In conclusion, the process of designing effective test cases is not just a technical endeavor, but a philosophical one as well. So, the next time you design a test case, remember to think about what you expect to achieve and how to achieve it, just as you would when building a house or coding a function. By doing so, you will not only improve the quality of your test cases but also contribute to a more robust and reliable software testing process.
