I like collecting these little adages about testing. I didn’t come up with this one, but was reminded of it by a tweet yesterday:
Lots of people (including myself) chimed in about having done this themselves in the replies. She eventually attributed the idea to Paul Grizzaffi. It’s not explicitly saying “tests encode knowledge”, but that’s the truth behind this.
In my case, this was a good strategy because behaviours that weren’t explicitly defined or expected tended to be discovered and used by people as if they were features. What might be brushed off as “nobody will do that” can become a useful little lifehack for someone else. When it does, “fixing a bug” might easily be interpreted by someone else as “breaking a feature”. That’s the kind of stuff you want to know about.
I don’t know who I might have heard the exact phrase “tests encode knowledge” from, and a quick search turns up only medical references about memory and recall. Apt, but not helpful.
Nonetheless, I find it a useful little adage. It affords several possible heuristics for whether a test should be automated or not.
- Is this something I want to enshrine as true in a test?
- Must this be true or does it merely happen to be true?
- Is this something I want other people to know?
- Is this behaviour worth documenting?
- Is it important to know if this changes?
How to answers those in turn may require their own heuristics. Usually some understanding of the business impact of any given behaviour.
The last comes up not just in writing assertions, but also in how tests are implemented. I see it come up all the time that a test inadvertently assumes something unrelated to the point of the test, making it prone to brittleness. This is advice I give to developers writing unit tests all the time: If the underlying implementation changes, will this fail? Should it? If it’s not important to know about the implementation change, the test shouldn’t be written to be sensitive to it.
In that sense, the direct connection between tests and monitoring becomes pretty clear. Developers sometimes complain that too-thorough unit tests become a useless change detection system. In early stages of a product especially, when things are changing rapidly, tests can become a hindrance because they can effectively lock down the way the product behaves. Writing a test—at any level—encodes knowledge about how the product behaves now in a way that incurs a cost to change. The skill of automation in testing, then, is in knowing what behaviours should be encoded into a test and when.
Automated tests are often intended to be a change detection system, but to be well written means to detect only the important changes. (And your definition of “important” may vary from mine.) Automated tests can then be a useful type of monitoring system by comparing your knowledge about how things should work (as you’ve encoded it) to how things actually work.
What knowledge are you trying to encode into your tests?
Note added 2019-07-31 – I found in some old notes that I originally wrote down as “tests codify knowledge”. That phrase does turn up some google hits, e.g. this article by Johan Hoberg and this blog post from John Stevenson that frames tests as converting tacit knowledge into explicit knowledge. I still don’t know where I heard it first, but it is certainly out there.