Investigating Software

Investigating Software

Tuesday, 11 April 2017

Why you might need testers

I remember teaching my son to ride his bike. No, Strike that, Helping him to learn to ride his bike. It’s that way round – if we are honest – he was changing his brain so it could adapt to the mechanism and behaviour of the bike. I was just holding the bike, pushing and showering him with praise and tips.

If he fell, I didn’t and couldn’t change the way he was riding the bike. I suggested things, rubbed his sore knee and pointed out that he had just cycled more in that last attempt – than he had ever managed before - Son this is working, you’re getting it.

I had help of course, Gravity being one. When he lost balance, it hurt. Not a lot, but enough for his brain to get the feedback it needed to rewire a few neurons. If the mistakes were subtler, advice might help – try going faster – that will make the bike less wobbly. The excitement of going faster and better helped rewire a few more neurons.

When we have this sort of immediate feedback we learn quicker, we improve our game. When the feedback is about problems we don’t even know might exist, that expands our knowledge letting us adapt before its painful to do so.

My son had his own plan, skill and will. He could have learned on his own. It probably would have taken longer and maybe been more painful or even dangerous. Working as a team, we achieved something, more than we could alone.

In software development that’s how successful and sustainable teams work. We don’t have gravity, so we need other sources of feedback. Some of those might be ‘automated tests’, but much like my sons own ‘plan’ they only covered predicted behaviour. “So I just peddle and avoid the trees - Dad, it won’t be hard”.  We need another mind and experience to help us see outside our view of the problems. “Stay off the grass as well, it’s harder and more unstable there”.

That’s where skilled testers come in. Some of us are highly technical and will highlight technical issues or opportunities you haven’t noticed. Some of us are skilled at seeing the app from the customers view point and highlighting problems that will cost you customers. Many of us do the above and much more. This could be letting you know if we are breaking the law. We often write tools that enable us to test more things – better. Testers can be useful for just pointing out the simple stuff like – we are developing for Red Hat Linux, but 60% of our users are on Windows, and 25% are on Apple Macs.

“Our analytics will tell us what the customers are doing! We can tell if a feature or bug matters from the metrics!” Great idea. Do the analytics work well enough? Do you know if a user experiences an error? Do your analytics double-report or under-report? A second pair of eyes will probably find the analytics probably don’t work as well as we’d hoped.

Monday, 20 February 2017

Thank you for finding the bug I missed.

Thank you to the colleague/customer/product owner, who found the bug I missed. That oversight, was (at least in part) my mistake. I've been thinking about what happened and what that means to me and my team.

I'm happy you told me about the issue you found, because you...

1) Opened my eyes to a situation I'd never have thought to investigate.

2) Gave me another item for my checklist of things to check in future.

3) Made me remember, that we are never done testing.

4) Are never sure if the application 'works' well enough.

5) Reminded me to explore more and build less.

6) To request that we may wish to assign more time to finding these issues.

7) Let me experience the hindsight bias, so that the edge-case now seems obvious!

Monday, 16 January 2017

Google Maps Queue Jumps.

Google Maps directs me to and from my client sites. I've saved the location of the client's car parks, when I start the app in the morning - it knows where I want to go. When I start it at the end of the day, Google knows where I want to go.

This is great! It guides me around traffic jams, adjusts when I miss a turn and even offers faster routes en-route as they become available.

But sometimes Google Maps does something wrong. I don't mean incorrect, like how it sometimes gets a street name wrong (typically in a rural area). I don't mean how its GPS fix might put me in a neighbouring street (10m to my left - when there are trees overhead).

I mean wrong - As in something unfair and socially unacceptable. An action, that if a person did it, would be frowned upon.


Let’s assume a road has a traffic jam, so instead of the cars doing around 60 mph, we are crawling at <10 mph.

In the middle of this traffic jam, the road has a junction, an example is shown here:

Click to enlarge.

Google Maps, using its algorithm/AI, directs me off at exit (A), but rather than finding an alternative route it directs back down on to the road at point (B).

Google Maps has queue-jumped. Google's decision was reasonable and met its goals. Goals, That I assume include reducing journey time (for me). It has bypassed approx 1/4 mile of queuing cars.

It’s an intriguing issue, for a number of reasons:
  •  In cultures where queues are not expected, it might be OK (it’s a clever optimisation!)
  •  Conversely, some cultures may consider it a bug.
  •  Could we even alter/educate the algorithm to reliably distinguish between some routes (short-cuts) and others (queue-jumps)?
  •  What else might the AI deem OK, that I would consider wrong?
  •  What are Google Maps goals? Are they all in my interest and safety?
  •  These seem very far from pass/fail scenarios

For example, What if Google noticed that users used google, YouTube or its adverts more after a certain route, than if they had taken another?

What if that advert-hungry route was slower? Or more dangerous? (E.g. people use google more after that route, as they need to get their car repaired.)

Are these issues being tested for? Are the right questions being asked?