investigating software

In 2008 I attended GTAC in Seattle , a conference devoted to the use of automation in software testing. Since their first in London in 2006 , Google have been running about one a year, in various locations around the world. This post isn't really about the conference, its about a realisation that I had the day after the conference. After the conference I went sight-seeing in Seattle. I rode the short Simpsons -like Monorail and took a lift up the Jetsons -like space needle. I enjoyed my time there, and found the people very friendly. The conference had been very technology focused, many (but granted, not all) speakers focused on tools and how to use tools. While useful, the tools are only part of testing - and even then they typically just support testing rather than "do" testing. The Seattle Monorail. While I was at the top of the Space needle, I took out my phone and like a good tourist started taking pictures. I'd typically take a couple of pictures then

We are fighting an invincible opponent. The legions of bugs in our software far outnumber our attempts to find them all. Even the simplest of software releases, inevitably contains a 5th column of hidden pre-existing bugs or quirks that combined with our changes could strike at any time. The question we need to understand as testers is, how can we win? or at least: not lose this battle? Military examples and analogies can be useful in software testing, and not just those in reconnaissance . For example: the Millennium Challenge . This pre-gulf war 2 military exercise pitted two forces against one another, in the middle-east. In summary the modern US military was fighting a rogue element in a smaller country. The vast resources of the western power should of have faced few problems. But in fact the former US general  playing the role of the 'Rogue nation' trounced the western forces in a devastating blow that saw several warships sunk. How did the 'rogue' general do

Testing doesn't complete, it might end, it might finish, but it doesn't complete. There's too much to test. If you ever need confirmation of this, test something, something that's been tested already. Better still test a piece of software, you know has been tested by someone you think is a brilliant tester. A good tester like you, will still find new issues, ambiguities and bugs. That's because the complexity of modern software is huge: as well as all the potential code paths of your code, there's all the other underlying code's paths and the near infinite domain of data it might process. Thats part of the beauty of testing, you have to be able to get a handle on this vast test space. That is, review a near infinite test-space in a [very] finite time-frame. We are unable to give a complete picture of the product to our clients. But we are also free to find out new issues, that have so far eluded others. In fact the consequences are potentially more drama

In my last post I discussed how test automation could be used to do things that I couldn't easily do unaided. In that example, execute thousands of news 'content searches' and help me sort through them. With the help of some simple test automation I found some potential issues with the results returned by the REST API. In that case, I started out with the aim of implementing a tool. But your testing might not lead you that way, often your own hands-on investigation can find an issue. But you don't know how widespread it is, is it a one-off curiosity? or a sign of something more widespread.? Again, this is where test automation can help, and if done well, without being an implementation or maintenance burden. Many test automation efforts are blind to the very Agile idea of YAGNI or You Ain't Gonna Need it . They often presume to know all that needs to be tested in advance, deciding to invest most of their time writing 'tests' blindly against a specific

Have you ever had to test an API that's accessible over the internet? or even one thats available internally within your organisation? They often take the form of a REST service (or similar) through which other software can easily access information in a machine readable form. Even if you are not familiar with these APIs, you've probably heard-of or seen the results of them. Some examples of APIs are the Twitter API ,   Flickr  and the Guardian's Open Platform . Some examples of what people have built using the Flickr API are published on the flickr site. Despite being 'machine-readable' they are often human readable, greatly helping you test and debug them. Companies use these APIs to ease the distribution of their content, encourage community and commercial development around their content or to simply provide a clear and documentable line between their role as data-provider and where the consumer's role begins. When testing an API like the above, many

I recently blogged about some of the tools I use , and how some are so useful I keep using them. As I mentioned, randomness is pretty useful, and I have tools to help me generate random text. A few of my readers requested a copy of my simple random text generating script, so I've decided to open it up for everyone to use and test. It will have bugs, like all software, please send details and I'll try and fix them. If you are interested in what UTF-8 is and what all that Unicode stuff is about, there is a great article by Joel Spolsky that explains all, and the wikipedia page is ok . To use it... First download the script, its on GitHub . The script is fairly short and is all in one file. You don't have to 'install it', its not a GEM. Second, make sure you have Ruby version 1.9 or greater. You need version 1.9, because Ruby didn't handle UTF-8 well in older versions. Thirdly run the script like this: ruby fuzzutf8.rb That will give you some usag

Consistency, is one thing I test for in software. For example, if software refers to something by a particular name, then [usually] it should always refer to it by that name. Furthermore, when it uses that name e.g. 'London Tube Map' I would expect to see such a map, when I click to view it, and not another kind of map e.g.: a street map. Conventions, These are also an important part of software. People will [usually] expect your software to use conventions that are appropriate for the field. For example, The traditional London Tube map is a schematic diagram, designed to show the relative positions of the stations rather than their geographic location. Though, sometimes it's actually useful to have geographic information, e.g.: is Queensway (Central line) station very close to Bayswater (Circle line)? So if a map isn't using the schematic form, then the geographic form also has it uses. I would be surprised if I received a London Tube map that was neither schemati