Reconciling twitter threads and manual testing

This weekend on twitter I saw this thread from Alan Page in response to this article about why manual testing is still important. I read both several times, and was left very confused because most of what Alan wrote (as I understood it) didn’t seem to have much relationship to the article (as I understood it).

I jumped on the thread to ask for some clarification, but it wasn’t very productive. Without getting too much into that, I decided during the exchange that I wanted to tackle each point and counterpoint one by one and look at how I was interpreting each. Whether or not Alan or the original (unnamed) author agrees with any of this, I have no idea. I just want to do the exercise to sort out my own understanding of the situation. I’m only going to paraphrase my interpretation of what each was saying. This is more about my thoughts than theirs. Feel free to follow the original article and thread in the links above.

With that in mind, let’s go.

1. Cycle time

• Point: Setting up automation takes time, so at the start of something new, manual tests might give info earlier than automation will.
• Counterpoint: Cycle times are faster if automation is written along with code.

In what will be a theme: these aren’t incompatible. Very early in a process, where things are new, experimental, and changing frequently, I think there are cases where automation isn’t useful. Devs, myself included, constantly check things manually when writing new code. I do this when doing something I’m not familiar with, asking myself “does this even work?” I could do that by writing some tests, sure. But if it’s likely that it’s something I’m going to throw away or change again in 3 seconds it may not be worth it.

So yes, I want developers to write automation along with code. That also makes cycles faster. Depending on what it is, and especially if there’s not going to be a second or third cycle, then investing time in automation might not be the right call. The details of when that is true might differ, but Alan also admits as much. No controversy here.

2. Automation has errors

• Point: Manually testing core functionality is a good check that automated tests didn’t return an incorrect result.
• Counterpoint: Errors in automation is avoidable.

First, a clarification I want to make is that I treat flakiness and bugs in automation as two different things. The original point here is that automation can have bugs in it. I think that should be obvious. Test automation is just software, and all software can have bugs in it.

What do we do to reduce that risk? Personally, I spent a fair amount of time making tests fail while writing test code and paying attention to what the app is actually doing as I write in each step. It’s actually very similar to how I write the application code. I’ll also do sanity checks around deployments depending on what’s being deployed and where I think the risks are.

Sure, that’s probably not the kind of manual testing either one of these points was talking about. But are you shit if your automation might have bugs in it? That was probably the statement that triggered me most in my back and forth with Alan. It’s quite a leap to go from not having perfect code to being a shitty team. I don’t accept that.

3. UI Validation

• Point: Automated tests can’t tell you if things look right.
• Counterpoint: A dedicated testing specialist isn’t required for that.

Both true. Moving on.

4. Scenarios that can’t be automated

• Point: They exist.
• Counterpoint: They can often be fixed with refactoring and testability improvements, so should be rare.

Both still true! I think there’s probably more subtly there in terms of where these scenarios come from and what risks you’re trying to test for. Recent cases that come to mind for me were helped by introducing more seams so as to isolate the tricky bits from the bulk of testing. The hard-to-test bits were still there, though, just smaller and well-contained. Maybe that’s the same as making them rare. If you need to see the full end-to-end thing working, maybe that still has to be done manually. How rare you can make the hard scenarios probably depends on context, especially when it comes to third-party interfaces.

5. Short term cost

• Point: Automation is cheaper in the long run, manual tests are cheaper in the short term.
• Counterpoint: Citation needed on cost, and most frameworks have testing built in now.

I admit I’m not savvy about the costs of either, I only know the costs of the tools I use and my own salary. There was a tangent thread that justified the cost difference with the fact that manual testers are paid less than an automation specialist. Maybe that’s true, maybe it’s not. Let’s not miss the fact, though, that the original article did not claim “manual testing is cheaper, full stop.” The “short-term” bit is right in the heading!

What was more interesting to me was that one is probably talking about a Selenium suite independent of the app, whereas tests built into frameworks like “go test” are typically focused on unit testing. Fancy new UI test frameworks are easier to set up than their older cousins, but it felt to me like we’re just talking about two different things here. UI vs unit. Long term vs short term. Initial cost vs ongoing cost.

Reading between the lines, the better counterpoint could be to focus on unit tests and minimize UI tests. Not that that really changes the setup time for either.

6. Exploratory testing

• Point: It’s great, and having automation gives you more time to do it.
• Counterpoint: You don’t need a dedicated test specialist for that.

Also both true! To a degree, at least. There’s a lot more depth to this too, especially when talking to people who have thought much longer about testing as a craft than I have. Other threads talked about that in response to this idea.

7. Automation is hard

• Point: Automating is hard and it’s hard to hire someone who can do it well, whereas manual test cases are easy to follow.
• Counterpoint: Not worth responding to?

Until now I’ve been glossing over the article’s use of “test cases”, but here it’s unavoidable. Yeah, anybody can run through scripted manual test cases. It’s certainly not something I want to do though, and nobody should build a career on that.

It’s also true, judging from talking to people trying to do it, that hiring people with specialized skills in test automation is hard. I think Alan’s point here, if he had made one, might be that developers already have those skills and are perfectly capable of writing those tests themselves. Probably true as far as programming goes, but not as simple as that. At two extremes there are skilled testers with no coding chops and skilled programmers with no skill or interest in devising good test scenarios. There’s an ideal balance somewhere (summed up over multiple people on a team) and people that do both well are (anecdotally) rare. They can all manually test when appropriate, but none of them should spend their days blindly following scripts.

8. Agile is hard

• Point: You can manually test stuff that you didn’t have time to automate in the sprint.
• Counterpoint: You’re doing agile poorly in that case.

On this one I agree with Alan. If something is otherwise ready to release and you haven’t had time to automate tests for a feature somebody else wrote, that could be a sign of sprint-length waterfall projects. Crunch times do happen where business value of delivery trumps that of further testing, but playing catch-up shouldn’t be the status quo.

So what’s the crux?

I think you can probably boil all of this down to that Alan read an article that said “dedicated manual test specialists are essential” and I read an article that said “manual testing is important sometimes”.

I might be giving too much benefit of the doubt to the article as written, as I tend to do. I am not a man of extremes. I suppose it’s equally possible that I don’t understand Alan’s position either, even though I’ve been pretty much on board with everything I’ve followed around Modern Testing. I was somewhat reassured when Alan eventually confirmed that the manual test specialist is what he was actually arguing against. Maybe, then, I’m not complete off-base with my understanding regardless of what the original author meant.