Skip to main content

How did you find that bug? Are we sitting comfortably, then I'll begin.


How did you find that bug? - They asked with a sort of puzzled "he dun't thunk like uz" look on their faces. An expression that suggested they were unsure whether to commend the discovery or gather their pitchforks and organise a well overdue witch burning.

Likewise, I now knew why they needed me. The team members were genuinely hard working people trying to build something new and exciting. But they lacked one thing, someone exploring & asking questions - trying to find out new things about their application. Exploring is literally a step into the unknown, and that can be uncomfortable for those not experienced in how to do it well.

Related image
Exploring is literally a step into the unknown.
So how did I find that bug? It's easy to tell a story of how I tried that particular input value because... Paragraph 3 of v4.6 of the requirements document stated that the user shall indeed on occasion X given input Y in Chrome v62 do... Or spout some other overly verbose explanation of why that broken 'scenario' came to be valid. 

But I wouldn't be answering their question, and I wouldn't be telling the truth. The truth of how I (and other testers) do this is harder to remember. But having a better idea of how you got there is going to help you get there again.

Let's take a trivial example. A few years ago, working on a new editorial system for a financial news organisation: I discovered that entering a double quote character into the content search feature caused it to crash. Even worse it returned no results (But it didn't fail to provide the users with a nasty technical error message). This had not been noticed before, yet many tests had been written and run. They had more acceptance tests than they could run in a night.

I had not examined requirements documentation for that feature. I had not perused the Acceptance Criteria for the Search feature. I had not reviewed the unreliable test automation. What I did was speak to the team and then type some quoted text into the search form. I had thought that the search should behave like Google, whereby if you include text in quotes - it will search for an exact match to that string of characters. I was interested to see what happened.

I did not know whether the search should support this sort of exact phrase matching. But I was exploring. I knew 'searches' [in my experience] worked like this - so I decided to check if this one did. This was the first step in a series of related tests that allowed me and my colleagues to uncover many more issues with the search feature.

When a team is focused on short-term delivery goals they can lose sight of the features not listed right in front of them. You need someone to question, someone to explore someone to point out that crashing when an editor uses a quote is sub-optimal. That's what I do at Investigating Software.

Comments

Popular posts from this blog

The gamification of Software Testing

A while back, I sat in on a planning meeting. Many planning meetings slide awkwardly into a sort of ad-hoc technical analysis discussion, and this was no exception. With a little prompting, the team started to draw up what they wanted to build on a whiteboard.

The picture spoke its thousand words, and I could feel that the team now understood what needed to be done. The right questions were being asked, and initial development guesstimates were approaching common sense levels.

The discussion came around to testing, skipping over how they might test the feature, the team focused immediately on how long testing would take.

When probed as to how the testing would be performed? How we might find out what the team did wrong? Confused faces stared back at me. During our ensuing chat, I realised that they had been using BDD scenarios [only] as a metric of what testing needs to be done and when they are ready to ship. (Now I knew why I was hired to help)



There is nothing wrong with checking t…

Manumation, the worst best practice.

There is a pattern I see with many clients, often enough that I sought out a word to describe it: Manumation, A sort of well-meaning automation that usually requires frequent, extensive and expensive intervention to keep it 'working'.

You have probably seen it, the build server that needs a prod and a restart 'when things get a bit busy'. Or a deployment tool that, 'gets confused' and a 'test suite' that just needs another run or three.

The cause can be any number of the usual suspects - a corporate standard tool warped 5 ways to make it fit what your team needs. A one-off script 'that manager' decided was an investment and needed to be re-used... A well-intended attempt to 'automate all the things' that achieved the opposite.

They result in a manually intensive - automated process, where your team is like a character in the movie Metropolis, fighting with levers all day, just to keep the lights on upstairs. Manual-automation, manumatio…

Scatter guns and muskets.

Many, Many years ago I worked at a startup called Lastminute.com (a European online travel company, back when a travel company didn't have to be online). For a while, I worked in what would now be described as a 'DevOps' team. A group of technical people with both programming and operational skills.

I was in a hybrid development/operations role, where I spent my time investigating and remedying production issues using my development, investigative and still nascent testing skills. It was a hectic job working long hours away from home. Finding myself overloaded with work, I quickly learned to be a little ruthless with my time when trying to figure out what was broken and what needed to be fixed.
One skill I picked up, was being able to distinguish whether I was researching a bug or trying to find a new bug. When researching, I would be changing one thing or removing something (etc) and seeing if that made the issue better or worse. When looking for bugs, I'd be casting…