Showcasing failure (and what Claire Saffitz taught me about testing)

What if we had a YouTube series of someone trying to create “gourmet” versions of ideal automated test suites that are described by “thought leaders” and experts in the field online? Let people see how hard it is, how long it can take, and how important seeing the failures are, while at the same time showing that the challenges are surmountable and how much can be learned.

This idea came to mind in part because of something Angie Jones tweeted awhile ago (before I learned that bookmarks were a thing, so I’ve lost the link; I’ll come back to that) but also from watching Claire Saffitz try to make a Kit Kat bar. I realized that the reason I love Claire’s videos so much is that we get to see the whole process of whatever she’s making, including the failures and missteps. In her series of gourmet junk food videos, she doesn’t just present a video that says “this is how you can make Oreos at home”. She says, “these are Oreos and I’m going to try to make them from scratch.”

Of course, she’s a pro and ends up doing an amazing job, but that’s part of what makes seeing the rest of the process so satisfying. Even she’s going to try some things that don’t go well. She’s going to decide to compromise on some things because she can get just as good a result without. Sometimes you don’t actually need to temper the chocolate!

What does this teach us about testing? And especially about automation?

The obvious answer is that there’s a lot of trial and error, thinking about the variables at play, tweaking things to see if you get the result you want, and generally experimenting. Incremental development, small refactor steps, all that jazz. See how to make perfect pizza dough. But there’s something else about how we actually make things in the real world.

I find a lot of material online is just about successes and silver bullets. Purportedly simple ways to do things that end up being quite complicated when you start trying to apply toy architecture models to real systems.

Compounding the issue, very few of us get to start from scratch when building up testing infrastructure and frameworks for our products. Even those of us who do won’t have infinite time and resources. So we all end up compromising. We can watch the “How It’s Made” video for a frame of reference, but the tools in our own kitchen are going to be different. We try to do the best we have with the tools available and we need to respect that.

If I recall correctly, this is the sort of sentiment Angie Jones was tweeting about a while back. It’s not an excuse to become complacent and write everything off as “good enough”, but recognition that we shouldn’t expect absolute perfection. Of ourselves it leads to disappointment, of others it leads to being condescending. (This is also why I get frustrated when people jump to criticizing motives when asked a question.)

In a social context, it’s well known that seeing our friends present only their perfect selves online takes a real psychological toll. It was somewhat surprising when I started to feel the same sensation from watching webinars and using Twitter professionally. I would love to see a lot more of people’s journeys and struggles, what compromises they had to make and why, what other approaches might work just as well, and where otherwise great concepts might fall flat.

In academia, I was told once that a good talk will spend 90% of the time on background and only the last 10% on the new content. That may seem extreme, but I would wager the best talks will spend just as much time, if not more, on the motivation, history, and problems as on the solution. I learn a lot more from that.

Thanks, Claire.