Skip to content

Hiccupps - James Thomas
Syndicate content
James Thomas
Updated: 9 hours 18 min ago

Cambridge Lean Coffee

Thu, 11/26/2015 - 09:07
Yesterday's Lean Coffee was hosted by Jagex.  Here's a brief note on the topics that made it to discussion in the group that I was in.

Automated testing.
  • A big topic but mostly restricted this time to the question of screenshot comparison for web testing.
  • Experience reports say it's fragile.
  • Understanding what you want to achieve with it is crucial because maintenance costs will likely be high.
  • Looking to test at the lowest level possible, for the smallest testable element possible, can probably reduce the number of screenshots you will want to take.
  • For example, to check that a background image is visible for a page, you might check at a lower level that the image is served and assume that browsers are reliable enough to render it rather than taking a screenshot of the whole page which includes much more than the simple background image.
Why go to a testing conference?
  • It builds your confidence as a tester to find that other people think similar things, make similar decisions, have similar solutions.
  • It also reassures you when other people have similar problems.
  • You are exposed in a short space of time, in a sympathetic environment, to new ideas or new perspectives on old ideas.
  • You can meet people that you've only previously followed or tweeted at, and deepen the connection with them. "Testing royalty" is accessible!
  • When you come back, sharing what you found can clarify it for you and hopefully make positive changes to the way you work.
Strategies for compatibility testing.
  • Experience reports say that there's reasonable success with online services - to avoid having a stack of devices in-house - although not when high data throughput is required.
  • Reduce the permutations with a risk analysis.
  • Reduce the permutations by taking guidance from the business. What is important to your context, your customers?
How do you know which automated tests to remove?
  • Some tests have been running for years and never failed. This is wasting time and resource. 
  • Perhaps you shouldn't remove them if the impact of them failing is considered too serious.
  • Perhaps there's other ways to save time and resource. Do you even need to save this time and resource?
  • Can you run them differently? e.g. prioritise each test and run higher priority tests with greater frequency?
  • Can you run them differently? e.g. run only those that could be affected by a code change?
  • Can you run them differently? e.g. use randomisation to run subsets and build coverage over time?
  • Can you run them differently? e.g. run every suite frequently, but some configurations less frequently?
  • Chris George has a good talk on legacy tests.
Why isn't testing easier?
  • We've been testing software for decades now. Why hasn't it got easy?
  • It's bespoke to each solution.
  • Teams often want to reinvent the wheel (and all the mistakes that go into invention.)
  • You can't test everything.
  • Complexity of the inputs and deployment contexts increases at least as fast as advances in testing.
  • Systems are so interconnected these days, and pull in dependencies from all over the place.
  • People don't like to change and so get stuck with out of date ideas about testing that don't fit the current context.
Categories: Blogs

Means Testing

Sun, 11/22/2015 - 07:40

I've found it interesting to read the recent flurry of thought on the testing/checking question, thoughts referring back to Testing and Checking Refined, by James Bach and Michael Bolton, in which the following definitions for the terms are offered:

  • Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modelling, observation, inference, etc.
  • Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

The definitions are accompanied by a glossary which attempts to clarify some of the other terms used. These chaps really do sweat the semantics but if I could be so presumptuous as to gloss the distinction I might say: testing is an activity which requires thought; checking is simply a rigid comparison of expectation to observation.

There has been much misunderstanding about the relationship between the two terms, not least that they are often seen in opposition to one another while at the same time testing is said to include checking, as in Why the testing/checking debate is so messy – a fruit salad analogy,  where Joep Schuurkes laments:
What we’re left with are only two concepts, "testing" and "checking", and the non-checking part of testing is gone.In Exploring, Testing, Checking, and the mental model, Patrick Prill proposes that
When a human is performing a check, she is able to evaluate many assertions, that often are not encoded in the explicit check.For me, by the definitions above, this means that the human is not simply performing a check. The check is narrow while the human perspective can be broad (and deep, and lateral, ...) which, for Bolton and Bach, as I interpret it, means that this is testing. I think that this is also what Anders Dinsen is getting at in Why the dichotomy of testing versus checking is the core of our craft:
As a tester, I carry out checks when I test, but when I do, the checks I am doing are elements in the testing and the whole activity is testing, not checking.One thing that I find missing from the conversation that I've seen and heard is the notion of intent. I'm wondering whether it's useful to think about some action as a check or test depending on the way that the result is expected to be, or actually, used in the context of the person who is doing the interpretation.

Here's an example: in Migrate Idea I talked about some scripts that I designed to help me to gain confidence in the migration and upgrade of a service across multiple servers. The scripts consisted of a series of API requests followed by assertions, each of which had a very narrow scope. Essentially they were looking to see that some particular question gave particular answers against the pre- and post-migrated service on a particular server.

At face value, they appear to be checks, and indeed I used them like that when evaluating the migration server-by-server. After putting in the (testing) effort to craft the code, I expended little effort interpreting the results beyond accepting the result they gave (all clear vs some problem).

However, by aggregating the results across servers, and across runs, I found additional value, and this is something that I would happily call testing by the definitions at the top. So perhaps these scripts are not intrinsically a check or a test; they simply gather data.

Could it be that what is done with that data can help to classify the actions as (part of) a check or test? At different times, I conjecture that the same instance of some action can be checking, testing, both or neither.  If I don't exercise thought, that (at best) means checking. If I do, that means testing.
Categories: Blogs

Telling Remarks

Fri, 11/13/2015 - 00:04
The last Cambridge Tester Meetup was a Show and Tell: "Bring whatever you found valuable or interesting recently in your testing." I imagine Karo will blog about it in detail, so I'll just briefly mention the couple of things I took with me.

I showed my Kindle and that it's currently got Jerry Weinberg's Tester's Library and Laurent Bossavit's Leprechauns of Software Engineering on it. I described how much I enjoy Weinberg's writing and how much I get from it. I talked about how Bossavit's book is both a deconstruction of some factoids around software development and a guide to critical thinking. I said that I now read more software-related material than I do anything else, and I enjoy it for the most part. But I stop reading anything I'm not enjoying. I read to learn, to make connections, to expose myself to new ideas. I concentrate on the content because it's research and I want to try to understand and retain it.

I showed my phone and pointed out the bunch of podcasts on it, including Analysis, Inside ScienceMore or Less, Radiolab, TED Radio Hour, The Comedian's Comedian (from which came Oh Kay!) and Testing in the Pub. I choose some podcasts just because they'll cover material I wouldn't encounter any other way - magazine shows are great for that. I noted that I consume these podcasts very differently to the way I consume the books on my Kindle. I listen to them when I walk to and from work and when I'm strolling round the block at dinner time and when I'm washing up after tea. But I don't concentrate on the content much at all.

I let the podcasts wash over me. I mind-wander, I half-listen, I drop in and out, I leave myself open for information but don't seek it. I miss things, I kinda-hear things, I misunderstand things, I don't catch all of things and I rewind things ... from time to time. And yet I still learn, I still make connections and I still expose myself to new ideas. Different stuff, differently.

I showed both because I like both and I get great value from both. And so I wanted to tell you about both.
Categories: Blogs

Can Dour

Mon, 11/09/2015 - 22:56

My face rests just north of miserable. (In a previous blog post I once called it the growler.) Sadly for me, from that starting point just a little puzzlement, thought, irritation or even too much milk in my tea can turn it into an apparent mask of rage. I usually remain blissfully unaware at the time, despite the fact that friends, family and colleagues have all told me about it.

Given this, you might think it ironic that my EuroSTAR 2015 talk was about making people laugh, about the analogy I perceive between joke-making and testing and how I exercise my testing muscles when I'm not testing by exploratory joking. Or perhaps that simply makes you chuckle.

Here's the slides:

I enjoyed presenting, was delighted with the reaction, and then a little surprised to find myself instinctively asking everyone who spoke to me afterwards whether they felt there was something practical that they could take from it.

Dour? For sure, but also a doer.
Image: Twitter
Categories: Blogs

The Antics in Semantics

Sun, 11/08/2015 - 12:11
In his EuroSTAR 2015 talk, Iain McCowatt described an ongoing project which involves establishing testing principles for his teams, across projects. In a large organisation in a heavily-regulated environment such as his, this is a really significant challenge, but he described an interesting, pragmatic approach based on Simon Sinek's Golden Circle.

While discussing the principles that he and his team arrived at, he said something slowly and with great emphasis:
We really sweated the semantics.The principles are important. They are directly tied to and defined to support the purpose of testing for his company. They are the foundation for all decisions that will be made about testing in the company. As you might expect given this, he was careful to also define what he means by a principle. For the record, this is it:
Principles are heuristics and shared values that guide our thinking about how to test and organize our testing. I enjoyed Michael Bolton's talk, No More Exploratory Testing, as a story about the evolution of the semantics of some terms in our field (terms like checking, testing, ad hoc testing, exploratory testing) and how this evolution is intimately tied up with other factors such as changing context, new thinking, the recognition that the terms don't suit their intended purpose or are inconsistent with the use to which they are being put.

Bolton is big enough to note times at which he has helped (in retrospect) to make things worse (although in a well-intentioned quest to make things better) for instance by supporting the term 'sapient testing' which was widely misinterpreted to oppose 'non-sapient' or 'stupid' testing. And he arrives at an interesting conclusion: defining the terminology he wants to use in the way he wants to use it within a Rapid Software Testing namespace.

These two talks recognise a problem that Humpty Dumpty saw a long time ago: words can mean whatever you want them to mean. They both resolve the problem by (a) being careful to define the terms that matter to them as well as they can given their current understanding and context and (b) making a safe place to use that terminology in that way.

This approach is reasonable, sensible and practical and I've enjoyed mulling it over since. Here's a few thoughts:

Neither speaker refuses the right of others to use other terminology or the same terminology in different ways. Indeed, I perceive that both are prepared to make efforts to understand the terminology of others, even when they find strong reasons to object to it or the motivation for it. Evidence for this was seen in their questions at the session Stuart Reid and Anne Mette Hass lead on ISO 29119 at the conference.

There are risks to (being seen to be) setting up a walled garden. For example, that insiders will feel smug and become less inquisitive about the outside or that others will feel excluded, will ignore or reject it in its entirety, will begin some other process designed to be different for the sake of being different.

Neither speaker claims, as I see it, that their terminologies are correct, complete, static. I imagine they'd be happy to describe them as something like working definitions - the best they have given the context, experience, evidence and so on.

You might think that this stuff is theoretical nonsense; intellectual chin-stroking; deckchairs on the Titanic; fiddling while Rome burns. You might just want to get on and test. And that's your prerogative - and sometimes it's also the right thing to do - but I find that meaning matters massively and dealing with that is one of the hardest problems in testing because it is present to some extent in every single interaction you have with any person via any medium.

And, for me, that includes myself and my own mind.

Edit: Iain has now blogged about his EuroSTAR talk.
Categories: Blogs

Weinberg Woz Ere

Sat, 11/07/2015 - 08:48

Jerry Weinberg oozed out of many of the talks at EuroSTAR 2015. He was most visible as the hero at the heart of Kristoffer Nordstrom's keynote but numerous other talks that I attended and conversations that I had directly or indirectly, knowingly or unknowingly, as quote or in spirit offered some variant of this famous line:
No matter what the problem is, it's always a people problem. The observation that in testing we need to remember that we deal with people was a seam running through the the conference. A seam from which value can be extracted. A seam containing testing ore. And probably also testing lore, shared around by these speakers, these self-declared conversationalists, coffee cup evangelists, shoe leatherers.

These kinds of testers do not view testing as just being heads-down in the product trying to break things (as valuable as that can be). They are out and talking to people, they are recognising (actual or potential) problems outside of the software that impact adversely on the software and taking action - person-to-person action - to reduce or remove that impact.

Weinberg wasn't here. But also he woz.
Image: Twitter

Categories: Blogs

Testing is Simple (and Complicated)

Fri, 11/06/2015 - 08:05
In his keynote at EuroSTAR 2015 Rikard Edgren said many things that resonated with me. This was the one that rang out loudest:
Testing is simple: you understand what is important and then you test it.Followed almost immediately by
Testing is complicated. Testing as recursion. A simple statement hiding deep complexity. An elegant surface belying the turbulence underneath.

This is so beautiful.

It put me in mind of fractals such as the Mandelbrot set where a benign-looking equation, if exercised, generates never-ending, self-similar, ever-finer detail.

Searching for related insight, I see that Adam Knight has arrived in a similar place from a different direction. (And be sure to read the comments there for a salutary caution against shallow analogy.)

Edit: In the comments in How Models Change, Michael Bolton describes a model of testing as fractal and Adam later revisited his ideas.
Categories: Blogs

Euro Stars Oft Ware Testing

Sat, 10/31/2015 - 07:15
If you give yourself some space you can usually find another perspective.

I'm British, and so European, but wouldn't call myself a star and I rarely look at warez these dayz.  I am speaking at EuroSTAR next week, though:
Categories: Blogs

Practitioner Makes Imperfect

Tue, 10/27/2015 - 08:39

I love listening to people who know and care about something and can communicate both that knowledge and emotion in an accessible way, without patronising and without compromising the message.

In the tweetstream GeePaw Hill Riffs on MVCDave Rooney captures Michael D. Hill talking about the Model-View-Controller pattern for system architecture but also about the dangers of rigid adherence to such things and how expertise and experience will build on and blur and bend any theory to suit the context.

One extract:
[Experienced developers] have ... seen enough solution-variety to realize 3 key insights. first, "there's no one way". or if there is we don't have it. that's key because it frees the mind. second, "code tells you". that is, the code is the best & most direct expression of the design, and as such reveals its merit or lack. third, "it never ends." code is never done, always in progress, always *changing*. we build code knowing this, working in a certain way. that certain way involves a whole huge variety of techniques & concepts, but it starts with "prevent hard shots" & "know it's working". so when i say this might be MVC-ish, all i really am talking about is keeping an eye out for certain things.Image:
Categories: Blogs

Own Goals

Sun, 10/18/2015 - 07:43
We're used to talking about delivering what the stakeholders want (or perhaps what they need, to the extent that we or they understand either). You'd generally hope to find at least one stakeholder engaged directly in a project - if not, it's likely doomed - and sometimes more. These stakeholders are known, their role is clear, perhaps a Product Owner and domain expert, and they are given the opportunity to state their requirements in some way. And if they aren't, the project is probably still doomed.

Some stakeholders exert little influence even when directly engaged. They are present and visible but quiet, or even silent. (Often until quite late on when they reveal that they really wanted something else and, at this point, the project is doomed.) The reasons for this are many and varied and include that they are shy, they are out of their depth, they suffer from impostor syndrome, they don't get on with someone else on the team, they doubt the value of the project, they have no faith in the approach taken in the project, they think the team is ill-equipped to deliver the project, or the budget is unrealistic, or they are protecting their own time for another project in which they feel they have a bigger stake, they are assigned to the project against their will, they are moving to a new position and won't see the end of the project anyway or they just plain think the project is doomed.

There are frequently more stakeholders outside of the project team who are engaging with it indirectly via the Product Owner or their own staff on the project or by direct manipulation of the project goals, such as when a strategic change is made in the company. These stakeholders are by-and-large visible too. The lines of communication will be clear - often it will be outreach from the project to such stakeholders that is the primary direction - and indeed it's an important part of the project's remit to ensure that such stakeholders' views are taken into account. Projects do not exist in a vacuum and bringing feedback from the outside into the project can be crucial. Without the perspective of these kinds of stakeholders, the project may be doomed.

Then there are also those stakeholders who exercise influence in less-visible ways, such as by reviewing or debriefing aspects of the projects with team members informally at tea breaks, by asking individuals for reports or product changes directly, by selling a product vision different to the one the team as a whole is chasing. These need not be intended or interpreted as malicious acts but they may serve to bring uncertainty, lack of clarity or confusion about motives to the project and hence to delay or divert effort. If enough of this happens, doom looms large over the project's prospects.

Another set of stakeholders on a project are often not engaged and have little or no influence. "The User" is one such, a mythical being of inestimable importance and sadly often with involvement inversely proportional to that importance. Some projects may be fortunate to have users involved, or perhaps internal proxy users but even then there can be a noisy line of communication to them. You may instead or additionally use cardboard cut-out personas - useful, but usually not forthcoming when asked for an opinion. While this may not hamper the success of the project in delivering its objectives, the objectives themselves are doomed to be at best best guesses.

All of these stakeholders are interested in the project outcome, but likely on different axes to different extents. Some of the desires will overlap, and part of the skill of running a project is to arrange the Venn diagram of overlapping requirements to provide an intersection that's sufficiently acceptable to make the project tractable. Where there is not such an intersection, the project is likely doomed to tread water as it gets tugged one way and then another in a series of attempts to  move the intersection more in one person's favour than another's.

This is how projects work: stakeholder influence is more or less visible; stronger or weaker; direct or indirect; consensus-based or imposed; with motivations that are clear, and motivations that are not. Regardless of this, as a member of the team, you work on the project to service these stakeholders as best you can within the constraints that always exist: time, resource, scope ... and the potential for doom.

But there's another class of stakeholder in the project too: the team members. Which includes you. And while you might have little stake in the outcome - although it can be personally profitable to be on teams that deliver projects well - you definitely have a stake in your own work and in the work of your colleagues and in the spirit and productivity and running of the team. There's no reason why you can't have personal goals within the scope of the project. In fact, you should have personal goals within the scope of the project.

Generally speaking, your goals should not contradict, inhibit or compromise the project goals, or lead to its doom-laden demise. But you can be looking for opportunities that align with it and with your own interests and needs and those of the company outside of the project, such as:

  • Is there new some area I can learn about as I work?
  • Are there new technologies or techniques can I try out?
  • Is there a skill that I have already that can I consolidate?
  • Are there people on the team I can develop a relationship with?
  • Is there a role on the project can I get some experience of?
  • Is there someone that I'd like to observe working, to learn from them?
  • Are there are any opportunities to share something that I know on this project?
  • Does the project have tools that I'd like to try out?
  • Can I find parallels between this and other projects?
  • If so, what suggestions can I make that would exploit those parallels? (Or break them, if they are negative?)
  • Is there some infrastructure that we need generally that I can implement on this project and share?
  • Can I find new ways to do my work efficiently and ethically?
  • Can I find some ways to increase team collaboration, morale or enjoyment? 

There are many stakeholders on a project, with many goals and motivations for them. Just because you are not necessarily instrumental in deciding the scope of the project, it doesn't mean that you aren't a stakeholder in your own work. Creating personal targets enables you to get value from a project however it's running, whatever its outcome. It may not reverse the doom, but it can lift your mood.
Categories: Blogs

What Did You Expect?

Mon, 10/12/2015 - 21:58

On a whim, with hardly any forethought and with even less expectation that it'd turn up some gold, at the last Cambridge Lean Coffee I asked whether it was possible to quantify user experience and whether any of the testers there had tried.

Some of the things you'd expect were suggested, including A/B testing, wireframes, and putting the thing in front of a handful of tame users. Of those only A/B testing quantifies in any meaningful, statistical sense (the other approaches described were essentially qualitative) but has a significant flaw in that possessing the data about user behaviour without understanding the intent behind it is only half the story.

To the extent that I'd given it any consideration as I walked to the meeting that day, what I'd been wondering about was the possibility of rating a design before it gets in front of users. There are many toolkits that provide sets of components to be used in building a user interface and it might be reasonable to think that these have been crafted in such a way as to be suitable for the kind of use they're expected to be put to, and for them to be consistent across the set so that the same kind of gestures have similar functions, that accessibility concerns have been taken into account and conventions about, say, when to display tool tips and so on are common.

But are there guidelines about ways to put them together?

Questions. One of the things I get out of this kind of event are questions. Here's a few more:
  • Can a design be scored for usability based on some general principles? 
  • Are there - or could there be - design principles which give a good "average design"? 
  • If so, how do they differ across applications? (Usability considerations in the design of a paper cup are probably quite different from those in the design of an airport.)
  • Like quality, is the concept of good design so context-dependent that there's, in general, little chance of evaluating it objectively? 
  • Even if so, are there areas where there is a chance of getting such a measure?
  • There are trends in design and - just like trends elsewhere in the world - their visual appeal changes over time. But I'm less bothered about visual appeal and more about usability. How related are the two concepts of visual appeal and usability?
  • Is there something like the golden rectangle  for designs? (Side question: is there in fact even much evidence for a golden rectangle with appealing aesthetic properties?)
One of the other things I get out of events like this is new references, new perspectives, new vistas. And that's what turned up while talking afterwards. We were trying to get to the bottom of a disagreement we'd had over  analogies between software development and civil engineering or construction industries that turned into a conversation about design patterns and eventually resulted in David suggesting that I might be interested in Steve Krug's book, Rocket Surgery Made Easy.

I borrowed it from Roger  at work and read it in two sittings this weekend.

The book doesn't answer the kinds of questions I'm asking above. It doesn't even pretend to try. In fact, it's not a book about usability or design considerations at all. It's a book about how to test for usability in a cheap, convenient way and then how to decide what to fix and when and how to get buy-in for the whole process. And it is very explicitly not about quantification.

It's a book written by a practitioner for practitioners, written by someone who's been there and done it. And done it over. And seen it done well. And seen it done badly. And paid very careful attention to what actions tend to result in what outcomes. And then boiled that down. And experimented with what he found. And repeated that. And then boiled that down. And then looked for only the essential stuff in what's left. And applied that pragmatically, with a keen eye to the need for context to be at the fore. And turned that into a book.

So: an idle thought lead to some interesting thoughts and then to much food for thought. Not what I expected, but very, very welcome. File this one next to Lessons Learned in Software Testing.
Categories: Blogs

Food for Thought

Sat, 10/10/2015 - 07:18
I've had a nice reaction to Only Kidding, a post about how I try to help my kids experience wonder, take control and look for scope in the world around them. So here's something else that we've been doing recently which I like to think is fun, thought-provoking and stimulates creative problem solving.

I'll say something like this to them:
I've got a problem; I'm on one side of the river and all my toys are on the other. I want to get to my toys ... but the only thing I've got to help me is food.It's always food.

And then it's on them to come up with suggestions. Answers to that one included:

  • make a raft out of breadsticks glued together with Nutella
  • throw bread into the river until it absorbs all of the water
  • make a dam out of sausages
  • use half an Easter egg as a ladle to empty the river
  • throw piles of chocolate into the river and use them as stepping stones

We've established a few conventions as we've played, for example it's not allowed to use containers in the solutions - so a chair made out of tins of baked beans was rejected in one game. We've also built up some solution libraries - so if a sheet of material is required it's taken as read that it can be knitted out of spaghetti using, say, celery as needles.

Plausibility is important but not fundamental - so, a breadstick raft seems to be acceptable because it could in principle float even if in practice using chocolate spread as adhesive is impractical. Out-and-out fantasy - such as making a helicopter out of butter - is always shouted down.

Sometimes I'll provoke things by offering a new perspective such as getting a cow to swim across the river and riding on its back. And then we're onto questions: Is a cow food? Is a live cow food? Is a cow a container of food?

I really love this stuff. (And thankfully, at the moment, so do they.)
Categories: Blogs