skip to main content

gerg.dev

Blog | About me

Zendo and the art of testing

Controversial opinion: I don’t like The Dice Game. The good news: I have a much better alternative that brings a very similar challenge in a less adversarial package.

The basic game

The concept of the game is basically the same as The Dice Game. One person has a rule known only to them and the rest of the players use some equipment to make guesses and test hypotheses until someone is able to correctly guess the secret rule.

In The Dice Game, popular among testers, the rule typically takes a roll of several dice and generates a single number. You roll the dice, make a guess for what number that roll makes, and then I tell you what the real number is. Repeat until you’ve figured it out.

In Zendo, the players use a set of blocks in different colours and shapes to build a small structure, and are told whether it does or doesn’t fit a secret rule. The official rules have details for when you’re allowed to make a guess, different ways of guessing, and what makes a reasonable rule, but the core is simply that.

There’s a lot of chatter on the BoardGameGeek entry for Zendo about using it to teach the scientific method, and this statement from the makers of the game makes it pretty clear that this is really a game about testing (and applies just as well to the Dice Game):

“We have found that the most successful players are those who are able to constantly be seeking patterns on a global level while simultaneously creating very focused experiments on individual structures that help lead them in the right direction.”

Zendo instruction book

Adding a testing theme

When I introduced Zendo to my local testing community of practice, I decided to add a bit more of a testing theme, and make explicit some of the concepts that I wanted people to watch out for.

Zendo box art, with "For Testers" added as a subtitle
Zendo box art, with my own subtitle added

For our group of testers, I changed the premise a bit: You’re testing a product and getting some tests that fail and some that pass. Find the bug by determining what all the failing tests have in common.

Out of the box, Zendo comes with white and black tokens to mark structures that do or don’t meet the chosen rule, respectively. For this challenge, I changed that to green tokens for a passing “test case”, and red ones for a failing case. As the person running the game, I chose the rule that described my “bug”, and built two structures to provide a starting point: one “failing” (red) case that fit the rule and one “passing” (green) case that didn’t. (Sidenote: while red and green are nicely thematic, you do still need to take care that the tokens still work for our colourblind friends.)

After that setup, we went around the room taking turns, in a slightly simplified version of the game:

  • Build a structure using any of the blocks you want.
  • “Run the test”: ask me whether it’s a pass or fail, and I’ll put either a green or red token down.
  • Optionally, try to identify the bug by stating the pattern that explains all the failing tests.

The fun part for me was that when someone did try to identify the bug—“the structure has to have exactly two yellow pieces”, for example—my job wasn’t just to say yes or no. I had to build a counter example: a structure with exactly two yellow pieces but still passed, or a structure that had a different number of yellow pieces but still “had the bug”. The person guessing is only declared the winner when the guess can’t be disproved, even if the guess isn’t exactly the same as the rule chosen. Disproving a hypothesis in that way was sometimes as challenging as guessing the rule. We were able to play three rounds of the game in 1 hour and rotated who played the role of the rule-keeper each time just so others could experience that as well.

This aspect is actually one of the reasons I like Zendo better than the Dice Game: it’s less antagonistic. You could certainly run a version of the Dice Game in a similar way, but with Zendo it’s built into the rules of the game. Both sides challenge each other, and there’s a less of a feeling of “I know something you don’t know”.

During the game, to help keep the game flowing smoothly given that most had never played before, I left a slide up with the steps and some of the basic terminology and attributes that we’d focus on. The attribute possibilities here are reproduced from the instruction manual, but I redacted the more complicated ones to show that we were going to keep it simple. I had planned to test in teams of two if I had a large group, but didn’t end up needing to.

A summary of the rules of the game and important terminology, contained to a single slide.
Summary slide kept up during the game, with the premise and turn order on the left and the possible attributes reproduced from the Zendo instructions on the right.

Testing concepts highlighted

Before running this activity, I got some advice from Chris (kinofrost) over on The Ministry of Testing forum about workshops he’s run using this game. Based on his suggestions and my own experiences in playing the game a few times on my own, I highlighted each of these concepts before the game started so people had an idea what to look for:

  • Focus. Try small perturbations on a structure that you already know about. Change just one thing at a time. Simplify around one element and thinking just about what variables it has. Look for emotional triggers like the feeling that “It must have something to do with X!”
  • Defocus. Take a step back. Don’t think about that last test. Think big picture and look at the whole table. What patterns are developing? What variables haven’t been considered yet? If you feel lost, consider trying something totally new.
  • Falsification. In you have a hunch, build a test that would disprove it. For the rule-keeper, can you build a test that has the suggested bug but still passes? Can you build a structure that doesn’t have the suggested bug but fails anyway? Is a suggested bug equivalent to what you’ve chosen even though it’s stated differently?
  • Precision. Are we using common terminology? Clarify terms when unsure. Avoid ambiguity. Get agreement about what a test contains (“are those two pieces touching? do you consider this piece ‘on top of’ that one?”) and what a proposed bug means (“according to your guess, would this test pass?”).

We had several cases of someone finding out the hard way that too much or too little precision made a suggested bug easily falsifiable. Having to be very precise about the statement you’re making—“exactly one or at least one?—can be annoying, but should be familiar to anybody who has spent time trying to reproduce a finicky bug. Further, being able to spot ambiguity is often how testers are able to identify bugs before they ever make it into code.These are all muscles worth exercising.

The second reason that I tend to like Zendo more the Dice Game is that Zendo becomes about literally seeing patterns in all the structures on the table, rather than the math exercise that pips on dice creates. In fact, the first edition of the game used pyramids with 1, 2, or 3 pips on their sides, and tended towards numerical manipulation just like dice. Early play testers for the 2nd edition highlighted how the new pieces nicely avoided the math problems that the first edition was prone to. I wasn’t able to find out whether this was the reason for the change, but it was definitely a benefit. Some people may have a good mind for numerical problems and enjoy that kind of challenge, but it’s less accessible.

The one thing missing

After a few rounds of the game, we had a short debrief with two focus questions:

  • Did the game feel similar to software testing?
  • How was it different?

The most interesting insight was that missing from this experience was any sense of intuition. Whereas we are all have experience with how even unfamiliar software should behave, none of us had any idea what these blocks on the table should be doing. We had no domain knowledge about whether blue and yellow blocks were supposed to work together or if stacking blocks was important to functionality. This wasn’t hugely detrimental, but it did change the experience.

Let’s play it again

Overall, Zendo was a big hit with the testers in the room. Several asked if we could play it again. We all enjoyed practicing those skills around inference, deduction and induction, poking at hypotheses, debating ambiguity, exploring the variable space, and focusing in on identifying what really mattered. Unsurprisingly, a bunch of developers I played this with later enjoyed all of that too!

If you can get your hands on a copy of this game, I highly recommend playing it with your local group of testers.

About this article

Leave a Reply