Skip to main content

A Good Run!


“We got a good run from the tests” the tester stated.
“So what’s the story?” the scrum master asked.
“85% Pass” comes the reply, meekly.
“OK, just need to fix that 5% then.” The scrum master announces before striding off to announce that the team is only a couple of % away from success.

Our tester takes a moment to try and process the exchange…


Firstly, their own words:
“We got a good run”
Why had they said that? Well - in a sense - it was true. They had executed the tests before, and they had returned a much higher failure rate. But the code being checked was the same...

OK, so there were at least 3 obvious ways to interpret the data.
  1. The app code meets the criteria checked by the tests. ( Based on test run 2 )
  2. The app code does not meet the criteria checked by the tests. ( Based on test run 1 )
  3. The tests are as reliable a the toss of the coin. ( Based on both test runs )

Its surprising how unlikely people are to choose (3).


Secondly, the scrum master’s words:
“just need to fix that 5%”
Our tester assumes this relates to the de-facto “threshold” that is usually considered as good enough to release. As if the results were a linear scale, such as height or weight. If your code gets over 90% then it gets to pass the gate and get on the release roller-coaster.

The threshold tends to be arbitrary, I worked with a client that thought 86% was good but 83% was just not fit for purpose! Their use tends to indicate a problem. Why are we caring about a number rather than a possibly broken feature? What features or risks do the failing 10% represent? Why do we have so many routine failures?

Do you hear these sort of conversations in your team? If so, then your team might need some coaching.
Labels:

Comments

Post a Comment

Popular posts from this blog

Can Gen-AI understand Payments?

When it comes to rolling out updates to large complex banking systems, things can get messy quickly. Of course, the holy grail is to have each subsystem work well independently and to do some form of Pact or contract testing – reducing the complex and painful integration work. But nonetheless – at some point you are going to need to see if the dog and the pony can do their show together – and its generally better to do that in a way that doesn’t make millions of pounds of transactions fail – in a highly public manner, in production.  (This post is based on my recent lightning talk at  PyData London ) For the last few years, I’ve worked in the world of high value, real time and cross border payments, And one of the sticking points in bank [software] integration is message generation. A lot of time is spent dreaming up and creating those messages, then maintaining what you have just built. The world of payments runs on messages, these days they are often XML messages – and they ...
Post a Comment
Read more

What possible use could Gen AI be to me? (Part 1)

There’s a great scene in the Simpsons where the Monorail salesman comes to town and everyone (except Lisa of course) is quickly entranced by Monorail fever… He has an answer for every question and guess what? The Monorail will solve all the problems… somehow. The hype around Generative AI can seem a bit like that, and like Monorail-guy the sales-guy’s assure you Gen AI will solve all your problems - but can be pretty vague on the “how” part of the answer. So I’m going to provide a few short guides into how Generative (& other forms of AI) Artificial Intelligence can help you and your team. I’ll pitch the technical level differently for each one, and we’ll start with something fairly not technical: Custom Chatbots. ChatBots these days have evolved from the crude web sales tools of ten years ago, designed to hoover up leads for the sales team. They can now provide informative answers to questions based on documents or websites. If we take the most famous: Chat GPT 4. If we ignore the...
Post a Comment
Read more

Text to SWIFT - making data from prose (What possible use could Gen AI be to me? - Part 2)

 As I write this, my dog is grumpily moving around the room pausing intermittently to give me disappointed looks - looks that only my elderly mother could compete with. She (my dog) is annoyed by the robot vacuum cleaner. Its not been run for a while in that room - and its making a noisy foray into dark corners in a valiant effort to cleanse the mess. Its grinding gears and the cloud of dust in its wake is not helping to ease the dogs nerves. The dog's pleading puppy dog eyes & emotions have of course been anthropomorphised - at least a bit - by me (My dog is 7 years old and weighs over 20kg - so has little to fear). That is - I've taken human feelings and mapped them onto my dog. I know she has emotions - but she lacks language - or at least a language that (1) we humans understand, (2) maps to the same phrases or concepts I'm using. But I'm human, That's how I think and how I interact with people and sometimes - machines. Deciphering the problem and representi...
1 comment
Read more