Skip to main content

Posts

Convexity in Predictive Value & Why Your Tests Are Flaky.

A long time ago, in a country far away, a cunning politician suggested a way to reduce crime. He stated that a simple test that could be used to catch all the criminals. When tested, all the criminals would fail the test and be locked up. There’d be no need for expensive courts, crooked lawyers or long drawn out trials. The politician failed to give details of the test when pressed by journalists, stating that the test was very sensitive and they wouldn’t understand it. His supporters soon had their way and the politician was elected to office. On his first day in office, he deployed his national program of criminality-testing. Inevitably the details of the test leaked out. The test was simple and was indeed capable of ensuring 100% of criminals were detected. The test was: If the person is alive, find them guilty and lock them up. The test had a sensitivity of 100%, every single actual... real... bonafide criminal would fail the test and find themselves in prison. Unfo

Fire Tower Tests & The GRIM Test.

Sometimes finding out why something is broken is a long and painful process. You might have to trawl through a tonne of data, logs or equipment. Filtering out what looks OK from what looks, suspect. These laborious investigative tasks are in the back of our mind when we’re asked to do a code review, test some new code or a pull request. What people often forget is that finding out if something is broken is completely different from finding out why it's broken. Being quick and efficient at finding broken stuff often takes a different approach to the task of clarifying the causes. A failure to test efficiently is therefore often a failure in imagination, a difficulty in creating these new techniques. Kettlefoot Fire Lookout Tower atop Doe Mountain, Johnson County, Tennessee Take fire towers, for example, large forests in places like Canada and the US used to have large networks of Fire towers. These steel structures literally towered over the neighbouring landscape

Dealing With Bugs Using Impossible Tests.

“Muggins! 2 for 15. My points!” My son shouts in my face. I frown and re-examine my cards, sure enough, I’ve missed a card combination. I scramble for a response… “Err... ACC Rule 10 sub-section 1 part (b) states I have to be informed prior to the game that the rule is in operation!” He’s not impressed, But sounding confident is my only hope here. I think the inclusion of the rule details has made him think twice about arguing. (It's the only rule I know off by heart, and it's a lifesaver. I might even get it tattooed on my arm.) A typical cribbage board and pegs, to help with scoring. For the uninitiated, we were playing Cribbage. An old English card game, now popular around the world. Cribbage has an interesting system of scoring, that you progressively learn the more you play. The Muggins rule my son referred to is an optional rule whereby if you underscore your hand the opponent can claim those points. As such it's become a small obsession of mine to lea

DevOps and Software Testing.

Most of my recent work has been with DevOps teams. While in one sense DevOps is another evolution in software development. It also introduces some new skill requirements and responsibilities into the daily routine of a tester. These diagrams tend to confuse people, hence the video... I've created a short video to highlight some of these changes and the opportunities they bring. It's not an exhaustive view of DevOps but it gives a highlight of what you could be working with. While DevOps isn't a panacea to our software development problems, I have found that empowering teams with the ability to build and use the tools they need, can rapidly improve team morale and productivity.

Unicode Babel

I've written about the joys of Unicode and software development before . Using unexpected data in your testing is usually a good way to test for text encoding issues. Finding and fixing these those bugs early could save your team from a host of other related issues and hackery. Even if you don't expect to have unusual text content, this type of testing can help indicate if all your systems are configured consistently. Failure to do so can result in users seeing the dreaded Mojibake . Mojibake, when encoding goes bad I've recently created a python package for generating random Unicode codepoints so they can be incorporated easily into your automated tests and tools. It's called Unicode Babel , and can be used to create a simple iterator for supplying 'international' text to your app: from unicode_babel import tools, filters genny = tools.CodePointGenerator() for point in genny.random_codepoints( 10 , filters.filter_out_if_no_name) pr

As near as damn it.

It's 1982 and there's a bull market in the western stock exchanges. After being in the doldrums for 6 years the Dow Jones Industrial Average index is climbing steeply. In London, the FTSE 100  index is also witnessing a steady climb, despite the ongoing war in the South Atlantic. The rise in share prices also leads to an increase in the popularity and prominence of these stock 'indices', the algorithmically derived snapshots of leading stock prices, frequently, used as an indicator of overall market health. At that time a relatively small North American exchange decides to institute its own new index, allowing investors to discern, at a glance, the state of the market. The Vancouver Stock Exchange creates its index at an arbitrary 1000.0 starting value. The index value is then recalculated thousands of times a day as transactions are processed through the exchange. Fast forward to late 1983, western markets have continued to boom, stimulating sustained increa

Testing a maybe with machine learning.

“I figured it was just a jumbo jet.” My son and I shake our heads & then adopt blank stares as if a non-body-snatcher has been exposed in our midst. “Twin engine,” I utter, as I glance skyward again. “Single decker” My son adds as an explanation. “It’s a plane”, she retorts, rolling her eyes. My wife, (who is far smarter than myself) lacks my son and I’s ability to recognise aircraft. She has the typical persons ability to recognise aeroplanes. I grew up around airforce bases. I had a father who was an aircraft engineer. Years of exposure and explanations regarding aeroplanes, their mechanics and features. We took this to the next level... My son is an avid flight sim game player and has consumed many hours of relevant youtube material on the subject. He also had the luck/misfortune of me discussing the planes that frequent the skies, above us here, near London. Given our combined experience & expertise, we probably have a reasonable ability to reco