skip to main content

Blog | About me

Lumpers and Splitters in Testing

I find myself getting frustrated a lot with the online discourse around testing. Often, it comes down to what language we choose to use to describe what we do as testers. I have a running list of “debates I no longer care about” in my notes app, but this post isn’t about airing those grievances. Instead, I want to highlight one of the most useful things I’ve ever learned to help understand these kinds of debates.

Intro to Human Evolution

Oddly enough, this insight doesn’t come from any computer science background or some testing guru. I learned it in Anthropology 101, an elective I took in my first year of undergrad, almost 17 years ago. The context was studying the human evolutionary tree, including labs where we learned to identify different humanoid species from their skeletons. (This one’s an Australopithecus, that one is Paranthropus, etc.) We learned about how entire species could be defined based on just a few teeth, or the curvature of a skull fragment. The inevitable question, our professor told us, is this: where do you draw the line between each species? Especially once you start looking at the variety of shapes and sizes of modern human skeletons, how do you know this jaw sample comes from a entirely different species than that one? On what basis does an anthropologist decide whether to write a paper describing a new species, or a new specimen of an already discovered one?

There isn’t actually a clean answer to this. Nature is messy, and defining species is hard enough among living animals, let alone fossils. Our professor himself couldn’t tell us with absolute certainty whether species actually should be separate or whether they might be the same. You can’t tell for sure if two skulls could have successfully reproduced. What actually happens, he told us, can be thought of in terms a simple observation:

Some of you will be lumpers, and some of you will be splitters.

If there’s a disagreement about whether two samples come from different species or not, it is likely going to come down to a personal worldview about what is significant enough to split species apart. Some people will tend to think there are only a few distinct hominid species and lump all fossils into those few groups. Others will tend towards saying each variation in bone structure is significant enough to split it into its own species.

Often, the prof admitted, the person who discovered the new fossil will be more inclined to split. Finding a new species is a more significant discovery than finding yet another sample of an existing one, and people can be motivated to split just to help make a name for themselves. But it isn’t always that. It’s isn’t just vanity or personal whims, and it isn’t about finding an objective truth. It’s about what’s important to you. It’s about what’s useful to others.

Splitting in software testing

How this helps me today is that I find a lot of disagreements in testing, especially debates about what words to apply to what things, come down to this. The distinctions we choose to make say a lot about how we think about the underlying things we’re differentiating. But those distinctions are usually imposed, not intrinsic to the things themselves.

To start with an uncontroversial example: Can you compare apples and oranges? Popular wisdom says no. You can’t say whether one is better than the other because they’re too different from each other. But then again, they’re both fruit. One is red, one you have to peel, one breaks into individual mouthfuls, one is crunchy, one you can bake into pies, both can be turned into juice… I have both on my kitchen table and when I want a snack this afternoon you better believe I’m going to compare them!

In the world of testing, one example a lot of us will be familiar with is whether you think or care that “testing” and “checking” should be thought of as two distinct activities. I don’t necessarily think the difference is only about lumping or splitting, but I do think that making the distinction has a very particular purpose that may or may not be important to some people.

Closer to home, I recently had an exchange on twitter about whether Cypress and Selenium are comparable, and I think whether you say yes or no — assuming you have a good understanding of what those two things are to begin with — is probably a reflection of whether you’re a lumper of test tools or a splitter. Some would object with that sentence characterizing Selenium as a “test tool” at all, in fact.

Yes, of course, people might lump things together because they don’t have the right words to describe them, or may even not understand some fundamental difference about them. But they might also be choosing to lump things into a category because they do understand that difference and don’t think it’s relevant. Is Selenium a test tool because it’s used for testing, or is it not a test tool because it wasn’t designed for testing? The answer says more about your definition of what a “testing tool” is than your understanding of what Selenium is.

A Lumper’s perspective

I’ve probably tipped my hand by this point, but I consider myself a Lumper. I tend to think of broad categories and things existing on a spectrum. My first instinct is usually to look for what makes things the same, rather than what makes them different. When someone says “A and B are different things because A does X and B does Y,” I tend to think “But they both do Z!”

So how does this help me think about these kinds of debates?

If there’s appetite on both sides, I’ll usually try to press into why a particular distinction is being made. That has two parts:

  1. Find out what the distinction actually is in a person’s mind — what puts something in one category versus another — rather than what they call it. Try not to debate whether category A and category B are different, but ask what makes something an A versus a B.
  2. What value is this distinction giving them? Why is it useful? What insight might I be missing from lumping them together?

But that can be exhausting, especially via a medium like Twitter where I tend to stew on even the most benign tweets for an hour or more before figuring out how to respond. So quite often, when I recognize that someone is Splitting something that I would Lump, it goes one of two ways:

  1. Put it on my list of “debates I no longer care about” and move on.
  2. Try to improve the language used.

The second case is obviously the more productive one. It might mean using a different umbrella term: apples are apples, oranges are oranges, and they’re both fruit. It might also mean being more specific about what I’m grouping together for a specific conversation: “tools that can be used to drive a browser” instead of “testing tools”. If you’re dealing with someone who insists that tomato is a fruit, you might instead talk about “culinary vegetables” or “plants typically used in savoury cooking”. It can be annoying, and sometimes limiting, but might help you get past arguments about labels.

The point

The point of this post isn’t to say either way whether Lumping or Splitting is a better way of thinking. We certainly shouldn’t just label each other as one or the other so we can stop caring about semantic disagreements. As a Lumper, I think Lumping and Splitting exists on a spectrum, after all! Despite being on the Lumper side of the spectrum, there are plenty of times where I will recognize the importance of drawing a distinction where others might not.

Rather, the point of this post is mostly so I can refer back to it when I do want to speak about one of these debates in Lumper vs Splitter terms. It feels at times like 80% of chatter from testing “thought leaders” falls into this category. But maybe that’s just me lumping again! I really do think that recognizing these two thought patterns has done a lot for my sanity and being able to understand where people are coming from.

I hope it might be a useful lens for other people to view these debates through as well.

About this article

Leave a Reply