Skip to main content


Showing posts with the label agile

What possible use could Gen AI be to me? (Part 1)

There’s a great scene in the Simpsons where the Monorail salesman comes to town and everyone (except Lisa of course) is quickly entranced by Monorail fever… He has an answer for every question and guess what? The Monorail will solve all the problems… somehow. The hype around Generative AI can seem a bit like that, and like Monorail-guy the sales-guy’s assure you Gen AI will solve all your problems - but can be pretty vague on the “how” part of the answer. So I’m going to provide a few short guides into how Generative (& other forms of AI) Artificial Intelligence can help you and your team. I’ll pitch the technical level differently for each one, and we’ll start with something fairly not technical: Custom Chatbots. ChatBots these days have evolved from the crude web sales tools of ten years ago, designed to hoover up leads for the sales team. They can now provide informative answers to questions based on documents or websites. If we take the most famous: Chat GPT 4. If we ignore the

Development and test environments - on demand at the press of a button (That actually work!)

“Works on my machine!” “Fails most epicly on my test system!” “Oh, wait… it works on CI but fails in Test env 3.” Sound familiar? These sorts of conversations are thankfully a thing of the past.  Wait, hold on - are you still having these sorts of conversations? That's probably because you are working somewhere where the development, test, production & CI servers are being created by people, painfully, once. Alexander the Great cutting through the Gordian knot of a particularly gnarly micro-service deployment. You set up your laptop, you pray to the god of operating system patches and upgrades and hope that nothing ever changes (ever). You're gonna be the last person in the team to take that new Mac OS upgrade - let the rest of the team run through those mine fields first. And the test systems? Last time you asked for a new one of those your programme manager ended up on new & stronger heart meds. Luckily, there are tools that can help.  Gitpod , for example, allows you

Test Engineers, counsel for... all of the above!

Sometimes people discuss test engineers and QA as if they were a sort of police force, patrolling the streets of code looking for offences and offenders. While I can see the parallels, the investigation, checking the veracity of claims and a belief that we are making things safer. The simile soon falls down. But testers are not on the other side of the problem, we work alongside core developers, we often write code and follow all the same procedures (pull requests, planning, requirements analysis etc) they do. We also have the same goals, the delivery of working software that fulfills the team’s/company's goals and avoids harm. "A few good men" a great courtroom drama, all about finding the truth. Software quality, whatever that means for you and your company is helped by Test Engineers. Test Engineers approach the problem from another vantage point. We are the lawyers (& their investigators) in the court-room, sifting the evidence, questioning the facts and viewing t

Avoiding Wild Goose Chases While Debugging.

When I’m debugging a complex system, I’m constantly looking for patterns. I just ran this test code... What did I see in the log? I just processed a metric $^&*-load of data, did our memory footprint blip? I’m probably using every freedom-unit of screen space to tail logs, run a memory usage tool, run an IDE & debugger, watch a trace of API calls, run test code… And I’m doing this over and over. Then I see it. Bingo, that spike in API calls hits only when that process over there jumps to 20% processor usage when the app also throws that error. Unfortunately, I may have been mistaken. On a sufficiently complex system, the emergent behaviour can approach the appearance of randomness. Combinatorial explosions are for real, and they are happening constantly in your shiny new MacBook. My bug isn’t what I think it is. I’m examining so many variables in a system with dozens of subsystems at play, it's inevitable I will see a correlation. We know this more formally as

Convexity in Predictive Value & Why Your Tests Are Flaky.

A long time ago, in a country far away, a cunning politician suggested a way to reduce crime. He stated that a simple test that could be used to catch all the criminals. When tested, all the criminals would fail the test and be locked up. There’d be no need for expensive courts, crooked lawyers or long drawn out trials. The politician failed to give details of the test when pressed by journalists, stating that the test was very sensitive and they wouldn’t understand it. His supporters soon had their way and the politician was elected to office. On his first day in office, he deployed his national program of criminality-testing. Inevitably the details of the test leaked out. The test was simple and was indeed capable of ensuring 100% of criminals were detected. The test was: If the person is alive, find them guilty and lock them up. The test had a sensitivity of 100%, every single actual... real... bonafide criminal would fail the test and find themselves in prison. Unfo

DevOps and Software Testing.

Most of my recent work has been with DevOps teams. While in one sense DevOps is another evolution in software development. It also introduces some new skill requirements and responsibilities into the daily routine of a tester. These diagrams tend to confuse people, hence the video... I've created a short video to highlight some of these changes and the opportunities they bring. It's not an exhaustive view of DevOps but it gives a highlight of what you could be working with. While DevOps isn't a panacea to our software development problems, I have found that empowering teams with the ability to build and use the tools they need, can rapidly improve team morale and productivity.

Software development is in the Doldrums

"Don't get off the boat." "Seriously, never get off the boat," The instructor said, leaning forward and looking at each of us in turn. "But surely if it's sinking..." We reply, somewhat confused and slightly incredulous. We've seen Titanic, we think to ourselves, we know how this sea survival stuff works... "OK" He concedes, If things get really bad, "Get on the life raft if you can step- up from the boat to the life raft". "But, But... the yacht is like 37ft long, Do we want to wait until that whole boat is lower than the life-raft? When less than 1ft of the yacht is above the surface? Meanwhile all the time the life raft is just there... floating happily alongside." "Pretty much, yes," he said nodding. The movie Apocalypse Now speaks the truth. That was about 15 years ago. Not much has changed since. The reasons are manifold. Firstly, the yacht is a decent shelter. The thin plastic

A h̶i̶t̶c̶h̶h̶i̶k̶e̶r̶'s̶ software tester's guide to randomised testing - Part 2

How would test a water sac? (Wow there, calm that tester brain... I know what you are thinking, Whats it used for? Who / what uses it? how long does it need to last? Does the temperature of the water matter?  Is it single use? etc. But let's assume a generic hiking or camping water sac for now) I'm guessing one of your suggestions includes filling it with water, shaking it a bit and checking for leaks. Seems kind of obvious right? but when it comes to software, we often do away with old-fashioned techniques such as filling something up and looking at it. Where's the machine learning test algorithm? Call this a BDD scenario? Can Selenium check for H₂0? I have to run this past the B.A... This is your software. We can treat randomly generated test data and inputs in much the same way as water . Data files or other inputs like user interactions are the ever-moving parts of our applications. Think about it, the code is entirely static - it's the state or data that i

How did you find that bug? Are we sitting comfortably, then I'll begin.

How did you find that bug? - They asked with a sort of puzzled " he dun't thunk like uz " look on their faces. An expression that suggested they were unsure whether to commend the discovery or gather their pitchforks and organise a well overdue witch burning. Likewise, I now knew why they needed me. The team members were genuinely hard working people trying to build something new and exciting. But they lacked one thing, someone exploring & asking questions - trying to find out new things about their application. Exploring is literally a step into the unknown, and that can be uncomfortable for those not experienced in how to do it well. Exploring is literally a step into the unknown. So how did I find that bug? It's easy to tell a story of how I tried that particular input value because... Paragraph 3 of v4.6 of the requirements document stated that the user shall indeed on occasion X given input Y in Chrome v62 do... Or spout some other overly verb

Was there a test for that? No, and there shouldn't be.

The release shipped. For a while, the team felt good. The work was done, the team had achieved something, and that was rewarding.  Unfortunately for the team, it wasn't long before a problem was found. The Product Owner wasn't happy and had asked was going on down there in the galley, do we need new coders? Better ones? Hipster coders? The release shipped... After an investigation, some blushes, raised eyebrows and a couple of "Oh... Yeeeah's" they found the cause. A confusion had collided with a bodge, and the result was a mess. Should they write an automated - test for this problem?  An embarrassing mistake or a misstep can make us feel we have to do something . An action greater than a fix is needed. A penitance needs to be performed, to redeem ourselves, to make us right again. Sometimes the penitence is best spent adding a test for that issue. Especially if writing that test has a low cost, the frequency of the problem occurring is high or

Pick a card...

Take a pack of cards, shuffle them well, and place them on the desk in front of you. Could you accurately tell me what the order of the cards would be, without looking?   By Rosapicci - Own work, CC BY-SA 4.0 Now spend 2 weeks in a software development team, writing code & using services, and deploy that code to your cloud server environments. Could you tell me where the bugs would be, before looking? In both cases we have a rough idea of what’s in the product at the end. But the detail, how its actually going to play out? we have no realistic idea. Skeptical? Take the playing cards… If we lay the cards out one card at a time. Then the order in which they are laid out, has probably never been seen before. Ever.The number of permutations of the well-known 52 playing card pack is 80,658,175,170,943,878,571,660,636,856,403,766,975,289,505,440,883,277,824,000,000,000,000. OK, now let’s get back to our code. Even trivial apps, include dozens of code libr

Why you might need testers

I remember teaching my son to ride his bike. No, Strike that, Helping him to learn to ride his bike. It’s that way round – if we are honest – he was changing his brain so it could adapt to the mechanism and behaviour of the bike. I was just holding the bike, pushing and showering him with praise and tips. By 齐健 from Peking, People's Republic of China (Down the Hutong) If he fell, I didn’t and couldn’t change the way he was riding the bike. I suggested things, rubbed his sore knee and pointed out that he had just cycled more in that last attempt – than he had ever managed before - Son this is working, you’re getting it. I had help of course, Gravity being one. When he lost balance, it hurt. Not a lot, but enough for his brain to get the feedback it needed to rewire a few neurons. If the mistakes were subtler, advice might help – try going faster – that will make the bike less wobbly. The excitement of going faster and better helped rewire a few more neurons. Whe

Build, Test, Ship, Learn, Rinse & repeat.

Ever feel like your team is in a deadlock? The product owner wants Gizmo+ to be shipped, your senior engineers are split between grokking Gizmo+ and fixing Widget++ . Meanwhile the SDETs are franticly updating automated checks/BDD scripts and exploratory testers are uncovering that Widget+ and Gizmo+ should have been named ...+10 given the number of surprise bonus features they are finding. As a consequence feature delivery can start to slow and quality is inevitably hit as difficult decisions are made on what to fix. The typical reactions to such a situation can depend on your project's context, but to highlight a few common ones: Ramp up team size. Push back on deadlines. Push back on new features. Delay releases until 'it all gets sorted' ... I don't have to break it to you that these options are 'far from optimal'. In summary they all revolve around costing more and delivering less (from my time as a programme manager I can tell you - thats wha

Cincinnati Test Store

Monday 3rd September 1827, A man steps off the road at the corner of Fifth and Elm, and walks into a store. He's frequented the store a few times since it opened, and he's starting to get to know the owner and his range of merchandise. In fact, like many of people in town he's becoming a regular customer. He steps up to the counter, both he and the store owner glance at the large clock hanging on the wall and nod in unison. The shop-keeper makes a note of the time, the two then begin a rapid discussion of requirements and how the shop keeper might be able to help. When they've agreed what's needed, the shop keeper prepares the various items, bringing them to the counter, weighed, measured and packaged ready for transport to the customers nearby holding. The store keeper then presents the bill to the customer, who glances at the clock again, and the prices listed on each of the items arranged around the store's shelves and then pays. The customer smiles as he l

Using test automation to help me test, a Google Elevation API example

Someone once asked me if "Testing a login-process was a good thing to 'automate'?". We discussed the actual testing and checking they were concerned with. Their real concern was that their product's 'login' feature was a fundamental requirement, if that was 'broken' they wanted the team to know quick and to get it fixed quicker. A failure to login was probably going to be a show-stopping defect in the product. Another hope was that they could 'liberate' the testers from testing this functionality laboriously in every build/release etc. At this point the context becomes relevant, the answers can change depending the team, company and application involved. We have an idea of what the team are thinking - we need to think about why they have those ideas. For example, do we host or own the login/authentication service? if not, how much value is their in testing the actual login-process? Would a mock of that service suffice for our automated c

A simple test of time.

Last week I was performing another of my 5 minute testing exercises. As posted before, if I get a spare few minutes I pick something and investigate. This time, I'd picked Google Calendar. One thing people use calendars for is logging what they have done. That is, they function as both schedulers and record keepers. You add what you planned to do, and they also serve as a record of what you did - useful for invoicing clients or just reviewing how you used your time. Calendars and software based on them are inherently difficult to program and as such are often a rich source of bugs. People make a lot of assumptions about time and dates. For example that something ends after it starts. That may sound like something that 'just is true', but there are a number of reasons why that might not be the case. Some examples are: You type in the dates the wrong way round (or mix up your ISO and US dates etc) You're working with times around a DST switch, when 30min after 0

How to avoid testing in circles.

I once had an interesting conversation with a colleague who worked in a company selling hotel room bookings. The problem was interesting. Their profits depended on many factors. Firstly, fluctuating demand e.g.: Holidays, Weekends, Local events etc. Secondly, varying types of demand e.g. Business customers, Tourists, Single night bookings or e.g.: 11 day holidays. They also had multiple types of contracts on the rooms. For some, they might have had the exclusive right to sell [as they had pre-paid], for others they had an option to sell [at a lower profit] etc. My naive view had been they priced the room bookings at a suitable mark up, upping that markup for known busy times etc. For example a tourist hotel hotel near the Olympics would be a high mark up, the tourist hotel room in winter would have incurred less of a markup. Better to get some money than none at all). He smiled and said some places do that, but he didn't. He had realised his team had a bias towards making a hea