“We got a good run from the tests” the tester stated.
“So what’s the story?” the scrum master asked.
“85% Pass” comes the reply, meekly.
“OK, just need to fix that 5% then.” The scrum master announces before striding off to announce that the team is only a couple of % away from success.
Our tester takes a moment to try and process the exchange…
Firstly, their own words:
“We got a good run”
Why had they said that? Well - in a sense - it was true. They had executed the tests before, and they had returned a much higher failure rate. But the code being checked was the same...
OK, so there were at least 3 obvious ways to interpret the data.
- The app code meets the criteria checked by the tests. ( Based on test run 2 )
- The app code does not meet the criteria checked by the tests. ( Based on test run 1 )
- The tests are as reliable a the toss of the coin. ( Based on both test runs )
Its surprising how unlikely people are to choose (3).
Secondly, the scrum master’s words:
“just need to fix that 5%”
Our tester assumes this relates to the de-facto “threshold” that is usually considered as good enough to release. As if the results were a linear scale, such as height or weight. If your code gets over 90% then it gets to pass the gate and get on the release roller-coaster.
The threshold tends to be arbitrary, I worked with a client that thought 86% was good but 83% was just not fit for purpose! Their use tends to indicate a problem. Why are we caring about a number rather than a possibly broken feature? What features or risks do the failing 10% represent? Why do we have so many routine failures?
Do you hear these sort of conversations in your team? If so, then your team might need some coaching.