Skip to main content

Controlling software development

Do you ever feel like we do all this work and maybe we needn't of bothered? Things might have worked-out without our intervention. Or we are actually worse off, now, after the work? You're not alone. This is a common problem in any role where you need to investigate the effects of changes. What you're feeling is a lack of control.

A control is a view of the world, without your work. It's an alternate view of the world where everything is the same except for your fix/hack/intervention. They behave like 3D TV, they let your mind's-eye 'see' the effects, by making them standout from the background.

They are commonly used in scientific and especially pharmaceutical research studies. They let the researchers know how effective a treatment was, compared with similar patients who received placebo (or  older established medicine) pills rather than the new treatment. The researchers can tell whether, for example, a new flu remedy actually helped the patients. Or whether [like the control group] the patients got better/worse at the usual rate, suggesting the medicine had little or no affect.

In testing you probably already use controls, and sometimes without knowing it. For example, if a programmer claims to have fixed a bug - you probably check that the old buggy code before you check the new [allegedly] fixed code (you do don't you?). This lets you see the difference, and discern whether the fix is really -the treatment- the system needed and did it actually fix-the-problem? Controls can even help highlight that which was thought of as a bug, was in fact useful albeit buggy behaviour. For example what if that ugly error in the data-entry application is 'fixed', so that users are no-longer bothered by that error. Great! the new code shows no error message, thats good right? Well maybe not, if I go back to the old code, I might see that the ugly error was stopping users from typing in bad data into the computer. Bad data that might cause other important systems to fail. The real fix to the ugly error might have been to better report the problem to the user, so they can see the data is corrupt and fix it at source.

Often you might find that there is resistance to your wish to examine and test the older pre-fix code. This can be for several reasons, including limited test-system resources or a poorly set up build and deployment system that doesn't let testers deploy old 'builds' of the system.

Performance and load testing is another area whether controls are often needed and used. Though here many projects proceed blind, not using controls or even a baseline. Not knowing how your latest changes affect the system, compared with unchanged code running on the same systems at a similar time. Often teams use a baseline, established long ago. This of course is better than no comparison at all, but makes accurate measurement difficult. Who knows what other changes have occurred during the time-period since the baseline was taken. Network upgrades, code-library changes, database growth and other peoples changes are just a few possible things that can influence your results. Unless you have results from before and after your changes [and -only- your changes], you are guessing at the effect of your fix.

Being aware of the usefulness of controls is an essential part of software testing. When asked if the new build is better, we can answer "Better than what?" When we suspect the new feature is more smoke and mirrors than fix, we can ensure we have access to the unchanged system for comparison. If you find resistance to your requests for access to controls, well, you have there some interesting information to put in the test report.


  1. In IT, the word "control" is most often used as a synonym for management. It's great to see the word used in this important meaning (which by the way is the more common use of the word in Danish).

    Controls are needed where you can't accept the risk of not controlling. But quite often the problem is that people don't thinkg aobut the consequences of not controlling. It's easier to make something if it doesn't need to work.

    This is where testing should be controlled too: On the static level before testing takes place e.g. doing a SWAT to assess what's in focus, and on the dynamic level by controlling the test results when the product is released and one can start learning how it performs in real life.



Post a Comment

Popular posts from this blog

The gamification of Software Testing

A while back, I sat in on a planning meeting. Many planning meetings slide awkwardly into a sort of ad-hoc technical analysis discussion, and this was no exception. With a little prompting, the team started to draw up what they wanted to build on a whiteboard.

The picture spoke its thousand words, and I could feel that the team now understood what needed to be done. The right questions were being asked, and initial development guesstimates were approaching common sense levels.

The discussion came around to testing, skipping over how they might test the feature, the team focused immediately on how long testing would take.

When probed as to how the testing would be performed? How we might find out what the team did wrong? Confused faces stared back at me. During our ensuing chat, I realised that they had been using BDD scenarios [only] as a metric of what testing needs to be done and when they are ready to ship. (Now I knew why I was hired to help)

There is nothing wrong with checking t…

Why you might need testers

I remember teaching my son to ride his bike. No, Strike that, Helping him to learn to ride his bike. It’s that way round – if we are honest – he was changing his brain so it could adapt to the mechanism and behaviour of the bike. I was just holding the bike, pushing and showering him with praise and tips.

If he fell, I didn’t and couldn’t change the way he was riding the bike. I suggested things, rubbed his sore knee and pointed out that he had just cycled more in that last attempt – than he had ever managed before - Son this is working, you’re getting it.

I had help of course, Gravity being one. When he lost balance, it hurt. Not a lot, but enough for his brain to get the feedback it needed to rewire a few neurons. If the mistakes were subtler, advice might help – try going faster – that will make the bike less wobbly. The excitement of going faster and better helped rewire a few more neurons.

When we have this sort of immediate feedback we learn quicker, we improve our game. When the f…

Thank you for finding the bug I missed.

Thank you to the colleague/customer/product owner, who found the bug I missed. That oversight, was (at least in part) my mistake. I've been thinking about what happened and what that means to me and my team.

I'm happy you told me about the issue you found, because you...

1) Opened my eyes to a situation I'd never have thought to investigate.

2) Gave me another item for my checklist of things to check in future.

3) Made me remember, that we are never done testing.

4) Are never sure if the application 'works' well enough.

5) Reminded me to explore more and build less.

6) To request that we may wish to assign more time to finding these issues.

7) Let me experience the hindsight bias, so that the edge-case now seems obvious!