Here are some small, free, nifty tools I use now and then:

FreeMind – to model and communicate
WinMerge – to diff or merge files or folders
Process Hacker – to monitor resource usage
Process Monitor – to monitor registry and disk activities
InCtrl5 – for installation testing (what happended to Install Analyzer??)
Fiddler/Wireshark – to see network traffic details
Firebug – to see and edit details in Firefox
Xenu’s Link Sleuth – to find broken links
Zed Attack Proxy – lightweight security testing
NetLimiter – to simulate low bandwidth
Om/Perlclip – put testdata in Clipboard
Rapid Reporter – to document your testing
Color Oracle – to simulate color blindness
Console – a better command prompt

Additions are welcome!

Sometimes when I read status reports or hear project managers talk about testing, I hear that “testing is blocked”. What do they mean by that? When I delve deeper in what they are talking about I sometimes see that the progress on workpackages for the testing team or testers in a team have been combined with the result of a test case such as pass/fail/blocked. But that is hardly the truth? As a tester we do a lot of things, not just running test cases (if we even do that form). If we say that one test might be blocked in the sense that it does not provide value in testing further there. That does not really mean we are blocked? Is a blocked test when you cannot get the answer to a certain question you have about a system? Does it mean that a specific test could not be done, but perhaps a similar one be done that would provide as much value?

If everything around you is blocked and there is nothing you can do as a tester, then you can perhaps say that “testing is blocked”. But what about preparing for coming tests? What about automating some of the things that you have found? Did we then mean that we were blocked in the sense that we did not come up with any ideas on things we could do? Well, perhaps you can look at various test idea triggers that can get you going?

Progress in performed test related workpackages and the result of testing are two different things. Combining them does not really give a good picture of what is happening. So, before stating that “testing is blocked” consider if you actually mean that you have no progress or that you are unable to test in a certain area. Based on what you inform and how you communicate, stakeholders will make decisions based on that.

I usually tell my stakeholders that they should not worry about me or my test team being blocked in progress, there is usually something valuable to do. Still, if I say that some tests cannot be performed or that they are blocked, where the result of those tests are important they need to assist with removing those obstacles.

During a daily stand-up meeting in a test team I was part of, nearly all testers reported that they were blocked and that they had no progress in their test cases. We had a long list scripted tests that needed to be executed according to the test project manager. I investigated what they actually meant when they said that. Interestingly, most of them had a test cases that wanted them to go in a certain direction through the system. What was blocking them was bugs. They found lots of bugs, but did not feel they had time to report them because they really needed to gain progress in those intended test cases. At that time we had to work overtime if we did not have enough progress in our test cases. I asked them to instead of trying to run the test cases they use the test case as a test mission or charter to guide them where to go. When they saw fire or smoke, they followed that trail and documented (session notes) what they saw and what happened, but also log all bugs they found. From then on not one reported being blocked in the daily stand-up meetings. The flow of reported bugs increased and the progresss was fine (in the sense that test cases were executed as far as project management thought). Stating that you are blocked might in fact be how you work that does not enable freedom and creativity.

If we were to measure something as a manager of testing, would we then see a decline in testers reporting being blocked when using methods such as SBTM instead of scripted tests that report pass/fail/blocked?

I have been using the Scripted – Exploratory Testing Continuum (For one source of this, see page 56 in http://www.ryber.se/wp-content/EssentialTestDesign.pdf ) in classes to explain how scripted testing and exploratory testing intertwines; and to explain that most of our testing is somewhere in the middle of the continuum.

But I have also had some issues with this description.

If this continuum exist, when can you say that you are doing more exploratory than scripted (and vice versa)?
How do you recognize when your testing is more exploratory than scripted (and vice versa)?

I do not believe that these questions can be answered because I think it has to do with the mindset of the person who sets the approach; and how far this particular mindset can be stretched.

The continuum is a simplified (yet useful) description of how your testing approach is a mix of the two approaches. And the simplification lies in that it can be seen as if our testing approach can be found in some point on the continuum. But I think that someone who has an exploratory testing approach is doing more or less exploratory testing; and someone who has a scripted testing approach is doing more or less scripted testing. Maybe this is how many of you already think of this?
However, I would like to visualize the differences in a slightly different way in order for me to understand this better.

Testing can be viewed from different point of views; and if I as a tester look at my testing approach I can have good use of the continuum. But the decision on which approach we should focus on is often driven by the management of the project or test team. So, from a managerial point of view, the testing approach is driven from a core dimension (interest, factor) which is more or less strict. I.e., I think that each approach can be seen as a set of continuums where each continuum consist of a certain dimension; and you will only modify these individual dimensions, not the entire approach.

Scripted Testing Continuums (some examples)

Exploratory Testing Continuums (some examples)

Let’s say that my main interest is Control, and I believe that a scripted testing approach is the best option in order to fulfill my interest. From this, if I now gradually become less stricter when it comes to my control demand, I won’t transit into an exploratory testing approach. My approach would instead be gradually less scripted.

In the same fashion, if my main interest is trust. My approach would gradually become less exploratory, but never transit into a scripted approach.

I think that this has to do with that the approach is a mindset; or rather an ideology. And it is hard for many people to change their mindset/ideology because your approach is derived from your school of thought.
If you have a scripted approach to testing, it is because of something you strongly believe in. If you have an exploratory approach to testing, it is because of something else you strongly believe in.

When I explain pair testing I usually say that you pair up two persons. One drives the testing while the other documents, comes with suggestions and asks for clarifications.

In this context, if we consider the use of testing and checking. What if the person driving focus on testing (that is usually the case) and the other focus on his other tasks but also that of managing checks? Try to avoid disrupting the creative, exploratory mind of the tester driving the testing. Still, at some stages asking if it is possible to check certain things. I am sure a lot of persons work this way and it is nothing new, still perhaps they have not expressed it this way.

If you set it up this way you might avoid the various biases and blindness that the driver might experience. Instead you let the passive player focus on those things. By having a dialog you will naturally affect the other, but perhaps not as much as when you focus on it yourself.

What if you in a pair testing session, switch place every now and then. Each time you switch place you also switch from Test Driven to Check Driven.

What if you did pair testing but both of you tested in the same area at the same time with collaborative note taking. One had focus on Test Driven Checking while the other had Check Driven Testing. Would you get any interesting result based on that?

Introduction

In 2009 Michael Bolton had a talk at Agile 2009 from which he later on wrote about a distinction between testing and checking [1]. He has since then elaborated more in the area and digged deeper into it [2], and continue to do so [3]. By clarifying what we are doing at a certain time it makes it easier to determine what tools and heuristics to use among other things. By making this distinction I think it helps us as testers to do a better job and to better be able to explain to other non-testers why we need to do both testing and checking, but also what the difference is.

The idea that you can (and should) automate all testing is not interesting. It is like discussing automation of all decision making. In a discussion like this we can instead offer that checks are usually good to automate, but testing is not as easy since there is no true pass/fail result when it comes to open-ended information, that might have different value to different persons. James Bach makes an example how this is handled in BDD [4] or rather making it obvious that some checks might be easy to confirm while others are close to impossible.

When I talk about testing vs. checking and touch the subject about sapience [5], there are many who are known to promote the checking part as their main activity as a tester, who then feel uncomfortable. What is really important is to point out that they do exploratory testing as well [6], but perhaps not framing [7] or structuring it well. I’ve heard arguments that exploratory testing is less structured and formal, well perhaps for you it is. Why not change that?

Simon Morley makes some interesting observations in his post on Sapient checking [8] and I agree with him that it might confuse more if we talk about testing and checking when we are delivering information to inform decisions. Should the focus in our information be on if it was checking or testing being made? For most cases, I believe we should not.

Test Driven Checking

If we assume that testing and checking is intertwined, you might still consider that in some cases one thing drives the other or that at least one activity is in focus while the other is in the background.

Depending on how well I have organised my charters and missions for testing during a project I test differently. Sometimes when I run a test session (following principles from SBTM [9])  I do note taking on what tests and checks I have on the system, mixing the two activities. Afterwards I might go through a checklist and mark items off as done. This might be coverage related, requirements related or whatever was suitable to have in a checklist. I might have stored most of the information in my session notes, but I might still go into other areas to mark it there as well, if that was easily done. A college did exploratory testing, without note taking, and then went into an excel sheet to check off coverage areas. He then reported progress and coverage based on that.

To avoid missing some checks you might consider that after you have tested in an area and you are ready to move on, you take a peek at things to check in that area. By doing so you might avoid the confirmation bias [10] and unintentional blindness. You might also get new ideas for testing when seeing things that you should check.

When you do regression testing you might consider to first test around the area of the bug, then as a final step check if the reported bug is in fact fixed.

Check Driven Testing

This way of testing is probably the most common one by those who are unfamiliar with concepts of exploratory testing. As I see it, you might have a list of checks (or for some scripted tests) that you check, but you might test around them as well. In this case the checks are driving the testing. When you have one-liner test ideas as your main focus, I think it is most common that you let them drive you. So depending on how you define them you will have different focus. Simplifying it, are they “What if”-focused or “Verify that”-focused?

When you retest a fixed bug you might select the methd to check that the bug as it was reported is actually fixed, then you continue with what valuable testing around to see that nothing else has broken in the vicinity, as far as you know.

If you do a testing tour you might set out some designated targets (check points) and along the way do your testing reporting what you actually find out.

If you consider how kids might act when they walk, on their own, on one side of the street in one direction and all of a sudden they see their parents on the other side. The kid forgets everything they had learnt about traffic rules and risks, then just runs to the parents by crossing the street. I believe we can see this with testing and checks as well. When you get to the point where you can do the check, you forget all other things that you should really be testing, thus a case of unintentional blindness.

Conclusion

This is just one way of looking at it, this is no real conclusion. For me it helps to make a distinction between doing Test Driven Checking and Check Driven Testing. By being aware of the distinction you might plan your testing differently, you might consider a bit more what where you put your checks and where you do your testing. They will affect each other and they will be drawn to each other.

References [1] http://www.developsense.com/blog/2009/08/testing-vs-checking/ [2] http://www.developsense.com/blog/2009/09/transpection-and-three-elements-of/ [3] http://www.developsense.com/blog/category/testing-vs-checking/ [4] http://www.satisfice.com/blog/archives/638 [5] http://www.satisfice.com/blog/archives/99 [6] http://www.developsense.com/blog/2011/05/exploratory-testing-is-all-around-you/ [7] http://www.developsense.com/blog/category/test-framing/ [8] http://testers-headache.blogspot.com/2009/09/sapient-checking.html [9] http://www.satisfice.com/sbtm/index.shtml [10] http://www.workroom-productions.com/papers/The%20Irrational%20Tester%20v1-06.pdf

I see humility as a very good virtue.
It is something I have failed miserably at, partly because it is easy to think something is bad just because there are many problems.
I think it’s a common fallacy for many ambitious testers – you are last in line, maybe with lower status, you want to get heard and become too persuasive…

Humility tend to give better collaboration, and thereby better results. A humble tester more often changes her mind, which is necessary.
The complexity of reality demands a humble tester.

Here is my list of humbling experiences; things I think have made me a better tester.

* write requirements
* programming something bigger than an exercise
* manage projects
* customer report of an important bug you should have caught
* experiencing too little time to test the very important stuff
* failing to explain why your testing is good
* reading something you wrote some time ago, and realizing it’s very shallow
* understanding your lack of humility
* realizing you were wrong

Which ones do I have left?

Status reporting of testing activities is extremely project-dependent. The needs of when and how and what will differ every time. Maybe that’s why there’s so little good writing about status communication; you have to make it up every time.

Templates are out of the question, and I believe examples will mislead you as well.
You’re better off by thinking around these questions (inspired by final section in Lessons Learned)

Why are you reporting?

Are you giving quality assessments, trying to prove why testing is taking place?
Is focus on the product story or the testing story?
What kind of information can change decisions that are based on your report?

What is most important?

The everlasting and most difficult question for all testing efforts.
This is what you should start your report with.

To whom are you reporting?

What kind of information is the audience interested in?
Are developers the only ones reading bug reports?
Do you have a grounded quality model?

Are you adjusting the language for the audience?

Communication includes making sure it is understood.

In which ways are you reporting?

Just one way is rarely enough; use talking, writing, long, short, formal, informal.

Are there certain times when things need to be communicated?

Project milestones and meetings might need your information.
Some things must be communicated continuously and timely; bringing up bad news on the last day is a very bad idea (but sometimes inevitable.)

Are you using status reporting as a means for new test ideas?

I often start writing an important report early, to identify things that needs more testing.
You should start thinking about the reporting before the testing starts.

Are you reporting just because you are supposed to?

Then you should read these questions again…

 

If status reporting is difficult, there might be something wrong with your testing.

Generic test report:

 DateTime
 ID
 Author
 Time for writing report
 Time maintaining scripts for generating report
 Application Under Test
 Areas tested

For each area:
 Blockers
 Testers
 Tester mood
 Tests planned
 Tests executed
 Tests passed
 Tests failed
 Tests remaining (untested+fail)
 Bugs found, per priority
 Old bugs status (priority, severity, tester, assignee, days active)
 Bug resolution aggregation

Coverage (overall, or for each area)
 Requirements coverage
 Specification coverage
 Code coverage (statement/decision/condition)
 Data coverage
 Models' coverage
 Model of models coverage
 Serendipity
 For each quality characteristic:
 Importance
 Measurement scale
 Points
 Violations
 Gut feeling

Enhancements
 Noteworthy information
 Other artifacts
 Obstacles
 New risks
 Old risk status
 Learnings
 Test plan change requests

Quality assessment
 Information objectives fulfillment
 Go/No-go recommendation
 upcoming activities

 

This is of course practically useless.
You should report what is important.