One of the reasons that software testing is challenging, both intellectually and practically, is that the information about the state of the system under test is partial. It's part of the testing role to formulate a model (or, more usually, a cloud of overlapping, incomplete and contradictory models) that represent our best view of the system at any given time and we've developed a collection of monochrome boxes that reflect the idea that access to source code can help make sense of it. But even that doesn't equate to an understanding of the model that the software has when it operates. For example:

• The tester may not follow the source code (completely).
• External libraries may implement a substantial part of the functionality but appear minimally in the source.
• Interactions with other layers, such as the operating system for file operations, will form part of the model without being part of the codebase.
• If the source code is compiled, it may be optimised in ways that contradict the tester's understanding.

And the tester isn't the only person without a clear idea of the system's take on the world. Some end users would find value in understanding the software states and how they are transitioned. Even developers would welcome a way to see the bigger picture easily when they're in code they don't modify often.

An aside. A few weeks ago, during heavy rain, I heard a rapid and repetitive thudding on our flat kitchen roof. I assumed was a drip and when the rain had stopped I got up and had a look. There were two obvious candidates: a join in the guttering between us and next door and a TV aerial pointing slightly below the horizontal. The weather was dry but I know about soak testing, so I poured a bucket of water over the aerial and another into the guttering which prompted water droplets forming on the joint and falling in a rhythmic way.

I'm no guttering expert (although as a student I once got mistaken for a tramp; that's a different kind of gutter)  but I could see that a clip on a plastic band that applied pressure to the two pipes had cracked, opening up the seal. I squirted some sealant into the joint and forced the clip shut.

It broke.

After cursing for a while, I drilled through the band and the guttering, put a bolt through the hole and tightened a nut onto it. Pouring more water in showed no leak so I put some grease on the nut and bolt to waterproof them for the future me revisting the cheap and cheerful repair and made myself a nice cup of tea.


And the point of this DIY yarn? While I was on the roof it occurred to me that my model of the system I was testing and working with was very close to being the system itself. I can touch or visualise the entire thing easily. Sure, there are levels beyond my comprehension - I don't understand the chemical or physical properties of the materials used to manufacture the guttering, the nut and bolt or the clip but I have general experience of plastics, metals and so on that covers enough of that to give me what I need.

Even considering the wider systems in which this is a small component, I could initially see that there were multiple candidates for the source of the drip and latterly recognise that when it gets wet the bolt might rust which would make further maintenance more difficult.

That's not to suggest that all software can be reduced to the complexity of a joint between two half-pipes or that all physical things can be analysed simply by looking and interacting - I wouldn't have a chance with the engine in my car, for example. But, it is the case that the more of the underlying thing that can be inspected, the less effort is required to create the initial models and the more time can be spent on refining and testing them.

So I'm going to be giving myself some time to think what we can do to make the model the software I'm testing has of its state - or, more realistically, the sub-models it has of the bits of state of interest at any given time - more available and useful to the testers and other users.

For the record, I noted down my initial thoughts while I was writing this:

• when reporting derived metrics the raw data should be available too,
• logging should be as complete as possible or (to some sensible level) complete logging should be available,
• log time stamps from different components should be in step,
• error and warning messages should be precise, clear and informative,
• similar operations on the model should be similar operations in the view,
• similar structures (semantically and/or physically) should have similar realisations in the product,
• naming conventions should be consistent and transparent from the UI through the variables in the code to the model itself,
• any extra reporting must be trustworthy, and the trust should be economic to establish, or else we'll have an additional test burden.

These seem to be concerned with consistency and testability. Where else should I be looking?
Image: http://flic.kr/p/bpTUr 

As the tickets pour in and your will to live seeps out you need something to cling on to, something to keep you afloat, a software development life raft, if you will (with patches, natch).

Unfortunately, this is not it.

Instead, this is a bunch of those drug, chug, smug and shrug report definitions you've seen all over the place that the Dev manager and me have come up with in triage meetings over the years. We tried using them as buoyancy aids but, frankly, it takes more than Archimedes to lift this mood.

debug report: A note from the developer to his future self as insurance: "Well, I did submit a report about that issue back when I committed the code". Yes, instead of fixing it.

doodlebug report: Everyone knows it's coming and hopes it doesn't land on them.

Doug report: The premise is fair, all the points made are apparently plausible, each step in the logic seems reasonable and internally the report appears to be consistent but you know the conclusion is pure fantasy.

dug report: Not to be confused with the Doug report, this type of ticket reveals a hole somewhere in the product. Sometimes it looks like a shallow grave. Nothing to do with Star Wars but feel free to repurpose it.

earplug report: Yes, we understand that you feel strongly about this one but give it a rest for now, eh?

fug report: Never in the field of software development was so much different misbehaviour reported by so many lines of impenetrable text to so few readers willing or able to even attempt to comprehend it.

hug report: A thinly veiled enhancement request for something your mate in Dev has been angling to do for ages.

humbug report: A ticket deficient in every way. As the kids used to have it, it sucks.

jug report: Usually a one-liner. Usually asking for something that's hard to argue against. Usually very generic. Usually references an ambiguous customer need. Usually includes a buzzword. Usually hiding enormous complexity. Usually so appealing to BizDev that it'll get immediate traction.  Usually needs a huge pitcher of cold, cold water poured onto it, right away.

lug report: A large report that you've been carrying around from release to release.

mug report: You'll recognise this one by the way you slap yourself on the forehead when it arrives in your inbox. You forgot to re-read the report before you submitted it; you didn't check your repro; you didn't search to find out whether it was a known issue; you didn't read the manual; you didn't read the spec; you didn't talk to the developer on the next desk. You didn't even get to mark it invalid before the gleeful assignee did.

plug report: Thinly disguised praise for a favourite person, technology, or feature. Often contained in a hug report.

rug report: A cover-all ticket obscuring a set of product deficiencies.

slug report: Completely valid, with all the necessary explanation and even clear, reliable repro which identifies a problem in an area of the codebase so repellent that none of the developers want to touch it.

snug report: The perfect ticket, no excess flab, no missing details. You get a warm feeling when you read one of these. Unless you're the assignee.

thug report:  A heavyweight user whales on some (generally non-critical) aspect of the software in a completely over the top fashion.

tug report: This ticket has been hijacked by various parties to try and make it reflect their current issue. Over-enthusiastic reporters might tug their own.

ugh report: Those times when you want to push a dirty nail deep into your eye with disgust at the reporter. Why would they submit that? In that component? Assigned to that developer? At P1, critical? Twice? This week?

With thanks to Jason Trenouth and apologies to everyone else, especially those who've received or had to verify the bugs of the above types that I've submitted myself. Count yourself lucky we didn't get on to Heisenbugs, Hindenbugs, bugword bingo or buggerangs.
Image: Salvatore Vuono / FreeDigitalPhotos.net

So long ago it feels like another geological age - the Pre-Childrian, anyone? - I used to be in bands and I even made a few records. I don't have any formal schooling in music, and I didn't make much effort to learn by myself. Playing guitar was a creative endeavour and I took the view that I could do it however I wanted and when it didn't sound too good I just turned up the distortion because that always sounded good.

When I started using trackers to compose on the computer I unlearned what little I'd found out about music and the skill required to make it. Now I could focus on the selection as much as production; which samples to employ, how to arrange them, ways to process them to generate something new. The machine gave me space to improvise (read: bash the keyboard randomly) and then pick the bits that sounded good afterwards without ever having to be able to play them again. I embraced this freedom and began to collaborate with other people using the same simple technology.

But what I found was that the people I was working with actually knew what they were doing. They had RTFM. They had spent time understanding the limitations and capabilities of the tracker software. They had put in the effort to trying to exploit every last bit of functionality and pushed it to do things it wasn't intended to do.

They understood music too. Yes, they might arse around sometimes, but if they had a sound in their heads they had a good chance of getting into a track. If I had a sound in my head the best I could hope for was that it would go away before I wanted to sleep.

Away from what I'll loosely call art, on a practical level I was and continue to be reluctant to throw stuff away. I'll always try to fix something rather than chuck it. And if that fails I'll probably keep the bits in my shed for a while, just in case they come in useful.

Sometimes they do. It's great when you've got a spare part for something or when your daughters need pirate swords right now, Dad, and you can knock something up from an old bit of pipe lagging and some bamboo while they stand there and watch anxiously, paper hats all askew and eye patches slipping down over their cheeks and then run off happily batting each other over the head. But a lot of the time it just means I've got a shed full of crap. And there's not much space in the loft either.

When I started in employment it took me a long time to realise that, although I've got a lot of attributes that are relevant, and a real benefit, to my work, some tendencies need to be toned down or kept out of the workplace altogether.

When I started writing code I would often keep tinkering with something that already existed to try to bend it to what was required rather than start again. I can make it work, I would think. I can obscure the original intention, destroy any elegance it may have had, and chop it into minced meat, I would find. Still, the mince went with the spaghetti.

It's also perfectly valid to pick up a tool and use it for what you need to use it for and put it down again. We all do that all the time - just the other day I needed to monitor HTTP traffic on the client side to debug a web service interaction. I installed HTTPFox, which I've never used before, got what I needed and forgot about it. It's less valid to be using tools day-to-day, investing in them and accreting inertia to move to something better, without exploring the options they give you. Perhaps there's a major efficiency gain or a new technique just around the corner, but you won't know if you don't look.

And if you're using a tool to do something with a particular data format, well perhaps you should learn something about that format too. Who knows what you might notice if you have some idea  what you're looking at?

You're a tester: monitor yourself. Be selective about the aspects of yourself you choose to deploy at work and the contexts in which you apply them. Be honest about how well they're working. Keep the ones you don't need out of the office. Johnny Cash said it: "Don't take your guns to town, son", especially when you risk shooting yourself in the foot.

P.S. To my colleagues reading this: your suspicions are right, I have other character flaws traits in my private self and they still creep into work sometimes, although I try, try, try.
Image: Boss

The focus/defocus distinction is important in Rapid Software Testing. It encourages the tester to consciously employ both logical reasoning and lateral thinking and to switch between them particularly at those times when one or the other is not productive.

But while reasoning can often be ground out, it's hard to make intuitive leaps just because you want to, to simply hop out of the rut when you find you're bogged down, to free your mind from its box, especially when you haven't realised it's in a box yet, let alone that it's shaped like a straightjacket.

So it's gladdening to find that we'll soon be able to pop on an electro hat and all our problems will be solved, literally.  I foresee a future where the brainly-enhanced QA staff trample all over the latest build seconds after it's delivered, booting it back to the Dev team for reworking and gleefully following it with an eruption of bug reports like Mount Etna on Ash Wednesday, all the while casting admiring glances at each other's millinery.

Hang on a minute... I fear Tomorrow's World may have arrived already, without the funky headgear.
Image: http://flic.kr/p/bntRk8

The Dev manager scooted his chair over to my desk yesterday with that particular smile on his face that means he knows something I don't know.

Dev: Got a question for you. [Grin]

QA: One of those questions with implications? [Grimace]

Dev: Hidden implications. [Toothpaste advert]

QA: Lots of hidden implications?  [Frown]

Dev: I'm sitting on the tip of an iceberg of implications. [Cheshire cat]

This is not a novel phrase  for not a novel idea but it reminds me to keep myself in check when I'm trying to think through whatever the issue turns out to be.

Dev: We need to audit these components against specific criteria for a stakeholder. They're used everywhere in the product. [Richard Branson]

Hmm. I can see what's above the water for free; I can get an idea of what's just below well enough; I could dive deep down but I have to get my gear ready and the currents and the winds and new snowfall will have shifted things by the time I resurface.

Dev: Now.  [Rictus]

When I can't see the scale of the berg yet and I don't have time I'll look for a cheap way to model or approximate it instead, then give a qualified answer. The kinds of things I might consider include

• a previous testing task in the same area
• an existing product with the same functionality
• gut feeling estimates from several people
• asking colleagues who've worked on similar things elsewhere
• web search for known solutions in the area
• estimated amount of code change compared to earlier projects with similar amounts
• the complexity of the spec against other features
• any existing estimate for this project and their accuracy to date
• some notional per-day figure for e.g. proof reading or code review vs number of pages to be read
• stability and code health of the product components being affected
• the experience of the developer and tester in this area of the product, this toolkit, this domain etc

We kick it around for a few minutes and decide to base our response on the documented behaviour of the platform we're using for those components, with caveats according to the differences we can easily identify. We'll postpone the need for more investigation until later, or  not at all.

Dev: Thanks. [Problem solved]

He wheels away. I turn back to what I was doing and let my face unwind into its natural growler position.
Image: http://flic.kr/p/5bq6XE

Sometimes it's hard to resist the temptation to just go for it, to run at the fresh build and leave long trails of filthy QA bootprints all over its virgin white snow. But hold on, Captain Scott! Sometimes you want to know it's not ice before you start stamping.

Sometimes includes projects with significant risk outside of the software itself, like running over budget or headed for late delivery or even non-delivery. When that sometimes is now it's reasonable (although, as always, not always) to try to shorten each iteration somehow, to limit effort until the likelihood of it being well spent is higher. That kind of somehow should look to fail a build early and at low cost and, to do that, somebody needs to find cheap and discriminatory tests.

Of course, the somebody who'll have to do that is you and you want tests which, while they might not have the power to explain what the issue is, can say that there's an important issue someplace. It's more than a smoke test. You're after sanity check oracles who'll work cash-in-hand. Remember that they'll be heuristic: a pass doesn't mean that everything's hunky dory but it does mean that it's more likely to be worth continuing testing.

Some examples might be:

•  the application can run in multiple modes and, regardless of mode, some values in or features of its output must be identical (or derivable from each other.)
•  a grep of config files for inconsistencies or perhaps comparing values with the installed application under test.
•  you have downstream processes that you could feed output to as a cheap validation test.
•  there are performance constraints that can be easily monitored and tested for on some dummy runs.

And once you've got these you can start to automate them, and give them back to the developers to use, and start to build up your test suite. And maybe then you can pull the project back on track somewhat, somewise. Some chance...
Image: J Fry / FreeDigitalPhotos.net

You can waste a lot of time on formatting. In The Power of Fancy Plain Text I said that I prefer plain text when possible and for a while I've been using Ascii Flow, a web-based tool, for making quick and dirty flowcharts and simple block diagrams that I can store in text documents too.

Here's a bit of a chart  I made recently to plot out some test scenarios for a timing-related issue I was investigating and wanted to share on our wiki. The blocks represent actions occurring over time and their relationship to a some significant times and events.

      t1                            t2
       +                             +
   +---|-----------------------------|------>  time
       |                             |
       |   +----+      +----+        |   +----+
E1     |   | A  |      | B  |        |   | C  |
       |   +----+      +----+        |   +----+
       |                             |
    +-----------------------------------------+
E9  |        V                                |
    +-----------------------------------------+
       |                             |
       | +----+                      | +----+  +  +----+      +----+
       | | W  |                      | | X  |  |  | Y  |      | Z  |
       | +----+                      | +----+  |  +----+      +----+
       |                             |         |
                                          external event

It took a couple of minutes to put together and now anybody can edit it easily without needing to fire up some other application and then save out an image that in turn has to be uploaded and can't be changed easily by the next person.
Image: http://flic.kr/p/8UkSQ3

When you're testing you're observing, mapping, inspecting. Sometimes it's non-specific - you're exploring, trying to understand the size, shape, performance envelope, functionality, scope, stability, usability or other characteristics of an application. Sometimes it's very specific - you are implementing an attack vector that you think you can exploit to expose a bug. Sometimes it's neither of those things or a combination of those things.

Whatever it is, it always involves looking and when I'm looking at a system I try to apply what I think of as peripheral testing by analogy to peripheral vision, the ability to see objects and movement outside of the direct line of vision. I have my main focus on the thing I'm interested in, but I permit and encourage part of my focus to wander briefly, just a little, but often, while I'm there.

Let's say we're working in a GUI. I'll take a few seconds to hover over all the components in a dialog to see whether tooltips are present and correct. If there are multiple tabs I'll click through them all quickly - a kind of blink test - to see whether there's empty fields where I'd expect values, enabled fields which should be unavailable given the context, odd colouring, inconsistent component size or naming and so on.

In any chooser I might try clicking on a few options that I know I don't want, just to see what happens, before choosing the one that I do want. I might browse down a few levels in a file chooser, or open all of the sub-folders in one folder wondering whether there is inconsistent presentation or missing data, and then move on. I don't give this my full attention. Often I'll do it while I'm thinking about what I actually want to do next.

If I'm working in a configuration file at the command line, I'll scroll through the whole thing to get to the point I want to edit rather than searching through the file. I'll be skimming, looking for typos, missing comments, unexpected comments, obviously incorrect definitions or documentation, out of date names, incorrect copyrights or other attributions etc.

When I'm in the bin directory of an installation on a Linux machine, I might just type 'ls -l' to check the date stamps of all the files in there and quickly eyeball them to see that all the files, which should have come from the same installer at the same time, have the same date stamp. Or I might type 'file *' and scan for any files that have an unexpected type or word size, given the build I'm working on.

These kinds of things take hardly any time. They are done idly, they are not systematic or exhaustive, they give your unconscious the chance to operate. They are not the main focus of a mission and they shouldn't detract from it but the technique can and does find bugs - often the kinds of bugs you wouldn't have found otherwise.
Image http://flic.kr/p/3yt5HU