investigating software

There’s a great scene in the Simpsons where the Monorail salesman comes to town and everyone (except Lisa of course) is quickly entranced by Monorail fever… He has an answer for every question and guess what? The Monorail will solve all the problems… somehow. The hype around Generative AI can seem a bit like that, and like Monorail-guy the sales-guy’s assure you Gen AI will solve all your problems - but can be pretty vague on the “how” part of the answer. So I’m going to provide a few short guides into how Generative (& other forms of AI) Artificial Intelligence can help you and your team. I’ll pitch the technical level differently for each one, and we’ll start with something fairly not technical: Custom Chatbots. ChatBots these days have evolved from the crude web sales tools of ten years ago, designed to hoover up leads for the sales team. They can now provide informative answers to questions based on documents or websites. If we take the most famous: Chat GPT 4. If we ignore the

 In 316 AD Emperor Constantine issued a new coin,  there's nothing too unique about that in itself. But this coin is significant due to its pagan/roman religious symbols. Why is this odd? Constantine had converted himself, and probably with little consultation -  his empire to Christianity, years before. Yet the coin shows the emperor and the (pagan) sun god Sol.  Looks Legit! While this seems out of place, to us (1700 years later), it's not entirely surprising. Constantine and his people had followed different, older gods for centuries. The people would have been raised and taught the old pagan stories, and when presented with a new narrative it's not surprising they borrowed from and felt comfortable with both. I've seen much the same behaviour with Large Language Models (LLMs) like ChatGPT. You can provide them with fresh new data, from your own documents, but what's to stop it from listening to its old training instead?  You could spend a lot of time collating,

I play a card game called cribbage. I often play it with my son . One interesting part of the game is the muggins rule. This means that you can claim points from other players turns, if they miscount the score.  The scoring is slightly nerve racking, with each of us double and triple checking our scores, to avoid falling foul of ‘muggins’, that’s part of the fun.  But my son and I also find ourselves discussing other hands of cards, in a sort of alternate history version of the game. “So if I had a 7 instead of a 2 of hearts, then I’d get a double run and score at least 8 more points”.   “Yes Dad, if you had different cards then you would likely have a different score, but you don’t” he says while rolling his eyes.  This sort of bitter-sweet history rewriting is a convenient tool for us to swallow the awkward truth of the real world. We often create alternate things to object to.  Take Chat GPT 4 and tools like Copilot X. These are powerful tools, capable of doing useful tasks quicker

 I was lucky enough to have started my career in a small company and then in a start-up. Both provided me with an environment perfect for learning. I sat with experts who took time out of their day to help answer my questions. From them, I learned the basics of what I still use today.  I’ve built on those foundations, but things would have been much harder if I didn’t have those foundational moments of my career. I’m not just talking about technical skills, the mentoring on how companies work, consulting and how to be better generally.  But those technical skills were also a big part of it – and a part many people miss out on in their careers. The rise of Large Language Models like ChatGPT4 is rapidly helping to fill that gap – where people don’t have a technical mentor who can explain and help work through those technical problems.  I’m no longer that junior team member – asking the dumb questions (OK, well usually I’m not) but even I find Chat GPT excellent at consolidating a broad s

I recently open sourced a simple Micropython library for LoRaWAN on the Raspberry Pi Pico.  (If you are interested, You can find it on GitHub .) If you are unsure what that all means, let me unpack it for you... Micropython is a slimmed down version of Python 3.x that works on microcontrollers like the Raspberry Pi Pico, and a host of other microcontroller boards .  LoRaWAN is a wireless communication standard that is ideal for long range, low power & low band width data transmission. Its based on a clever technique for making signals work well over distance, called LoRa. The library I've shared is a wrapper around the existing LoRaWAN support provided by the RAK Wireless 4200 board. The RAK4200  (affiliate link) essentially provides a modem, that can establish a connection to the network and relay messages. It uses the traditional AT command syntax (used by the modems of yore!) The Pico and RAK4200 Evaluation board (there is also a UPS under the Pico there - that's optio

“Works on my machine!” “Fails most epicly on my test system!” “Oh, wait… it works on CI but fails in Test env 3.” Sound familiar? These sorts of conversations are thankfully a thing of the past.  Wait, hold on - are you still having these sorts of conversations? That's probably because you are working somewhere where the development, test, production & CI servers are being created by people, painfully, once. Alexander the Great cutting through the Gordian knot of a particularly gnarly micro-service deployment. You set up your laptop, you pray to the god of operating system patches and upgrades and hope that nothing ever changes (ever). You're gonna be the last person in the team to take that new Mac OS upgrade - let the rest of the team run through those mine fields first. And the test systems? Last time you asked for a new one of those your programme manager ended up on new & stronger heart meds. Luckily, there are tools that can help.  Gitpod , for example, allows you

A long time ago, in a country far away, a cunning politician suggested a way to reduce crime. He stated that a simple test that could be used to catch all the criminals. When tested, all the criminals would fail the test and be locked up. There’d be no need for expensive courts, crooked lawyers or long drawn out trials. The politician failed to give details of the test when pressed by journalists, stating that the test was very sensitive and they wouldn’t understand it. His supporters soon had their way and the politician was elected to office. On his first day in office, he deployed his national program of criminality-testing. Inevitably the details of the test leaked out. The test was simple and was indeed capable of ensuring 100% of criminals were detected. The test was: If the person is alive, find them guilty and lock them up. The test had a sensitivity of 100%, every single actual... real... bonafide criminal would fail the test and find themselves in prison. Unfo

“Muggins! 2 for 15. My points!” My son shouts in my face. I frown and re-examine my cards, sure enough, I’ve missed a card combination. I scramble for a response… “Err... ACC Rule 10 sub-section 1 part (b) states I have to be informed prior to the game that the rule is in operation!” He’s not impressed, But sounding confident is my only hope here. I think the inclusion of the rule details has made him think twice about arguing. (It's the only rule I know off by heart, and it's a lifesaver. I might even get it tattooed on my arm.) A typical cribbage board and pegs, to help with scoring. For the uninitiated, we were playing Cribbage. An old English card game, now popular around the world. Cribbage has an interesting system of scoring, that you progressively learn the more you play. The Muggins rule my son referred to is an optional rule whereby if you underscore your hand the opponent can claim those points. As such it's become a small obsession of mine to lea

It's 1982 and there's a bull market in the western stock exchanges. After being in the doldrums for 6 years the Dow Jones Industrial Average index is climbing steeply. In London, the FTSE 100  index is also witnessing a steady climb, despite the ongoing war in the South Atlantic. The rise in share prices also leads to an increase in the popularity and prominence of these stock 'indices', the algorithmically derived snapshots of leading stock prices, frequently, used as an indicator of overall market health. At that time a relatively small North American exchange decides to institute its own new index, allowing investors to discern, at a glance, the state of the market. The Vancouver Stock Exchange creates its index at an arbitrary 1000.0 starting value. The index value is then recalculated thousands of times a day as transactions are processed through the exchange. Fast forward to late 1983, western markets have continued to boom, stimulating sustained increa

Mostly Harmless, I've talked and written about randomisation as a technique in software testing several times over the last few years. It's great to see people's eyes light up when they grok the concept and its potential.  The idea that they can create random test data on the fly and pour this into the app step back and see what happens is exciting to people looking to find new blockers on their apps path to reliability. But it's not long before a cloud appears in their sunny demeanour and they start to conceive of the possible pitfalls. Here are a few tips on how to avert the common apparent blockers. (Part 1) A good motto for software testing, as well as pan-galactic hitchhiking. Problem: I've created loads of random numbers as input data, but how will I know the answer the software returns, is correct? - Do I have to re-implement the whole app logic in my test code? Do you remember going to the fun-fair as a kid? Or maybe you recall tak

The release shipped. For a while, the team felt good. The work was done, the team had achieved something, and that was rewarding.  Unfortunately for the team, it wasn't long before a problem was found. The Product Owner wasn't happy and had asked was going on down there in the galley, do we need new coders? Better ones? Hipster coders? The release shipped... After an investigation, some blushes, raised eyebrows and a couple of "Oh... Yeeeah's" they found the cause. A confusion had collided with a bodge, and the result was a mess. Should they write an automated - test for this problem?  An embarrassing mistake or a misstep can make us feel we have to do something . An action greater than a fix is needed. A penitance needs to be performed, to redeem ourselves, to make us right again. Sometimes the penitence is best spent adding a test for that issue. Especially if writing that test has a low cost, the frequency of the problem occurring is high or

There is a pattern I see with many clients, often enough that I sought out a word to describe it: Manumation, A sort of well-meaning automation that usually requires frequent, extensive and expensive intervention to keep it 'working'. You have probably seen it, the build server that needs a prod and a restart 'when things get a bit busy'. Or a deployment tool that, 'gets confused' and a 'test suite' that just needs another run or three. The cause can be any number of the usual suspects - a corporate standard tool warped 5 ways to make it fit what your team needs. A one-off script 'that manager' decided was an investment and needed to be re-used... A well-intended attempt to 'automate all the things' that achieved the opposite. They result in a manually intensive - automated process, where your team is like a character in the movie Metropolis, fighting with levers all day, just to keep the lights on upstairs. Manual-automation, manu

Shutter sync is an interesting artefact generated when we video moving objects. Take a look at this video of a Helicopter taking off: Notice how the boats are moving as normal, but the rotors appear to be barely moving at all. This isn’t a ‘Photoshop’. It’s an effect of video camera’s frame rate matching the speed/position of the rotors. Each time the camera takes a picture or ‘frame’ the rotors happen to be in approximately the same relative position. The regular and deterministic behaviour of both machines enables the helicopter to appear to be both broken and flying. The rotors don’t appear to be working, while other evidence suggests its rotors are providing all the lift required. What's so exciting is that this tells us something useful, as well as apparently being a flaw or fail. We could both assume the rotors move with a constant rotation, and estimate a series of possible values for the speed of the rotors, given this video. Your automated checks/tests can

Google Maps directs me to and from my client sites. I've saved the location of the client's car parks, when I start the app in the morning - it knows where I want to go. When I start it at the end of the day, Google knows where I want to go. This is great! It guides me around traffic jams, adjusts when I miss a turn and even offers faster routes en-route as they become available. But sometimes Google Maps does something wrong. I don't mean incorrect, like how it sometimes gets a street name wrong (typically in a rural area). I don't mean how its GPS fix might put me in a neighbouring street (10m to my left - when there are trees overhead). I mean wrong - As in something unfair and socially unacceptable. An action, that if a person did it, would be frowned upon. Example: Let’s assume a road has a traffic jam, so instead of the cars doing around 60 mph, we are crawling at <10 mph. In the middle of this traffic jam, the road has a junction, an

After a weary few hours sorting through, re-running and manually double checking the "automated test" results, the team decide they need to "run the tests again!", that's a problem to the team. Why? because they are too slow. The 'test' runs take too long and they won't have the results until tomorrow. How does our team intend to fix the problem? ... make the tests run faster. Maybe use a new framework, get better hardware or some other cool trick. The team get busy, update the test tools and soon find them selves in a similar position. Now of course they need to rewrite them in language X or using a new [A-Z]+DD methodology. I can't believe you are still using technology Z , Luddites! Updating your tooling, and using a methodology appropriate to your context makes sense and should be factored into your workflow and estimates. But the above approach to solving the problem, starts with the wrong problem. As such, its not likely to find

“We got a good run from the tests” the tester stated. “So what’s the story?” the scrum master asked. “85% Pass” comes the reply, meekly. “OK, just need to fix that 5% then.” The scrum master announces before striding off to announce that the team is only a couple of % away from success. Our tester takes a moment to try and process the exchange… Firstly , their own words: “We got a good run” Why had they said that? Well - in a sense - it was true. They had executed the tests before, and they had returned a much higher failure rate. But the code being checked was the same... OK, so there were at least 3 obvious ways to interpret the data. The app code meets the criteria checked by the tests. ( Based on test run 2 ) The app code does not meet the criteria checked by the tests. ( Based on test run 1 ) The tests are as reliable a the toss of the coin. ( Based on both test runs ) Its surprising how unlikely people are to choose (3). Secondly , the scrum

In many of my clients, more effort is spent on 'test automation' than on other forms of testing or quality assurance. That can be the right choice, for example, I worked on a Data Warehousing project where we needed to write some test automation before we could test the data and its processing. Many other projects in different technology areas also spend a lot of time on their test automation. To be precise, they spend an increasing amount of time fixing & maintaining old 'tests' and 'frameworks'. There are great tools around to help us write these automated checks quickly. But as with many software systems: maintenance, in the long term, is where the time and money goes. That is why I'm surprised we don't use short term automation more. We have the skills. One good example of short term automation is Bug Automation . A simple script / executable that recreates or demonstrates a bug. This isn't a new idea, I've been doing it for year