It is a long tradition in the software world to ‘borrow’ things from Japan and apply them in North America. Lean? Kanban? Shu Ha Ri? All come from the other side of the Pacific. And in most cases I would say the ‘experts’ espousing them don’t really have the right to. Sure, they know the ‘concept’ and the ‘actions’ but do they really, really understand the cultural underpinnings that lead to them?

So along these lines, I’m going to start stealing from the Japanese. Here are my credentials…

• I did Karate for eight weeks
• My friends in highschool were ninjas
• I did Kendo for a year
• I have seen hundreds (and hundreds) of hours of Anime
• My son now takes Karate so I’m hanging out at a dojo again (if only in the lobby)

Infinitely qualified, see?

First, I think we should steal the Tea Ceremony for its ceremony and presentation. Specifically designed rooms, lots of steps in specific order, repeatability and lots of history make it a fantastic specimen.

Oh.

Wait.

The Factory School already got to it.

Next then let’s steal Anime, or more specifically its Tropes. But let’s steal in a sensible way. Rather than apply things directly, each of these should be considered heuristic. And some of these are almost designed to baffle management. Imagine seeing ‘Quivering Eyes’ or ‘Snot Bubble’ or ‘Idiot Crows’ in a status update on the techniques used on a project. Priceless!

When I first thought of this post, I’m sure I had something on Japanese Gardens but can’t remember what, so that is left to the reader.

The final thing to steal is the notion of Shinbutsu shugo which is the syncretism of Shinto and Buddhism. In a testing context, there is Agile, Context-Driven, ATDD/BDD and Artistic camps. Each has its own priests and followers which get involved in religious debates. But the reconciliation and idea sharing is important regardless of religious differences. And now we can give a Japanese name for it.

And if we’re really good, we’ll mispronounce it.

So previously I posted that factors outside our own control make accurately estimating the total test schedule impossible. In addition, I posted that even estimatingf simple, known, static things by breaking them into tasks, then adding up the tasks is much less accurate than you might think.

Example Three:

Imagine your typical workday that ends at 5:00 think about it. Sure, I'll likely be home from my drive at 5:30, but I might hit traffic, my boss might ask me a question at 4:55PM, someone in the parking lot might need a jump start. So if you want a commitment from me to predict when to invite the neighbors of when to pull the turkey out of th oven, I should likely say 6:00PM.

That's a 100% timing padding.

One hundred per cent.

On a task I've done hundreds, if not thousands of times with known physical 'resources' and very few dependencies.

Which reinforces the conclusion that test accurate estimation is impossible. At least in the sense of estimating a single number for total time without context.

Yet we are tasked with doing it anyway -- better yet, creating that context. So where do we start?



About a year ago my good friend Ben Simo pointed out to me that even if you know nothing at all about a project, you can always estimate it. Here's an example:

Q: "How long will testing take?"
A: "Two Weeks."

See that? I know absolutely nothing at all about the software. Nothing at all. But I made up a number.

For any given project, we likely know more than nothing. So we can certainly do a better estimation than "two weeks."

Next: I'll start talking about how I do that.

Unit Testing in C++ – it’s even harder than Inidian Pole Gymnastics

I mean seriously - CPPUnit is the devil. What kind of test framework do you use in C++?

In this session we review the top 10 ways you can cause unit testing to fail to work for your project. From readability problems and lack of trust to mock object abuse and semi-integration testing – these gotchas can save you lots of time, sweat, and tears on your current and next projects. Download video in [...]

In this interactive exercise, you will learn what makes a good acceptance test and how to solve the most common problems with such tests. The participants will work in groups on restructuring several tests and then discussing the results. Gojko Adzic will run around facilitating the exercise and helping you if you get stuck. http://skillsmatter.com/podcast/agile-testing/anatomy-of-a-good-acceptance-test

[Part of a series of posts on Java Selenium RC patterns I find useful.]

What we Want: Expressive, Succinct Tests

Let’s say I want to test that I can log into an app. I want my Selenium RC Java test code to look something like this, because I want it to read, aloud, the way I would describe actually logging in and checking that everything went mostly OK:

@Test
public void canLoginToFatFreeCRM() throws Exception {
  LoginPage loginPage = new LoginPage();
  DashBoard homePage = loginPage.login(BrowserDriver.DEFAULT_USERNAME,
                          BrowserDriver.DEFAULT_PASSWORD);
 
  assertTrue(homePage.isLoaded());
}

Several techniques make this possible, but the one we will focus on in this blog post is the notion of a self-verifying PageObject, which is essentially a “testable representation” of an actual web app page we wish to traverse, manipulate, and test. So my LoginPage class, above, allows me succinct manipulation of the corresponding login page in the system under test.

Note that this test knows nothing about Selenium. Indeed, neither does the LoginPage class (below). More on that later. Note that this test does not include any waitUntilPageIsFrickinLoaded() calls, nor (worse) Thread.sleep() code, and very little direct instantiation of PageObjects. The page flow mechanics is encapsulated elsewhere. Note also that as I go from page to page, I don’t explicitly assert that I got there. I shouldn’t have to. My test should be able to take that for granted. It turns out that assertions are being made under the covers about having arrived successfully on intermediate pages, but that code is abstracted away. That’s what this post covers.

The only useful assertion in the test proper is that, once I have logged in with proper credentials, I arrive safely at the homepage (and even this assertion in the test is redundant, and included here only to reveal intention; the DashBoard page is also self-verifying).

Glance-Readability

A test as short and expressive as the one above passes the “glance readability” test. Anyone familiar with automated testing semantics, and the biz language of this CRM domain, ought to be able to grasp what this test does at a single glance.

Suffice to say, we want our Selenium RC tests, as much as they can, to be this succinct. And, again, that requires that they stick to expressing page traversal and manipulation in the local biz-specific language of the web app’s business domain. So, finally, let’s see the under-the-covers page self-verification mechanics.

PageObjects as I Use Them

The selenium-rc-patterns sample Java codebase that illustrates all of the patterns in this series of posts talks to a localhost copy of a Rails project called Fat Free CRM. If you want to learn a bit about this app, you can play with a hosted sample version here (you have to sign up for an account first, which is easy).

This CRM system has discrete web pages like a Dashboard (home page, essentially), and pages for Tasks, Leads, Campaigns, Contacts, Accounts, and Opportunities. For each of these actual, production pages, my selenium-rc-patterns Eclipse project contains a matching PageObject with methods and fields that provide access to the services on that page.

Above, when you ask an instance of a LoginPage to login(username, password), then the TextFields and PageLink type know how to return us a PageObject that should then be cast to a DashboardPage and returned. (More on that in another post.)

Selenium RC developers have been using a PageObject pattern for awhile. The pattern dates back, at least, to the original HTMLUnit, whose API rather strongly encourages you to represent your pages under test as extensions of what it calls an HtmlPage. I don’t use Se 2 yet (might someday, might not), so I don’t use its PageFactory pattern.

Instead, I use my own PageObjects that extend a BasePage (or BasePane), partly so that I can control how and when PageObjects are instantiated, and verify automatically and ambiently on instantiation that Selenium is indeed on that page. Again, that’s much of what keeps the above test code so simple: I’m not explicitly waiting for a page to load, and I’m not asserting all over the place that I’ve successfully arrived on a given page.

So, my LoginPage class has a login() method that accepts a username and password (we’ll cover the return types of these fields and methods in another post that describes how page-flow works):

package bizdomain.pages;
 
import util.BasePane;
import util.elements.PageLink;
import util.elements.TextField;
 
public class LoginPage extends BasePane {
  public static final String PAGE_IS_LOADED_CSS
                                           = "input[id=authentication_username]";
  private TextField userNameField;
  private TextField passwordField;
  private PageLink loginButton;
 
  public LoginPage() {
    super();
    userNameField = new TextField("input[id=authentication_username]");
    passwordField = new TextField("input[id=authentication_password]");
    loginButton = new PageLink("input[id=authentication_submit]",
                                                                                DashBoard.class);
  }
 
  @Override
  public String getPageLoadedCssLocator() {
    return PAGE_IS_LOADED_CSS;
  }
 
  public DashBoard login(String userName, String password) {
    userNameField.enter(userName);
    passwordField.enter(password);
    return (DashBoard) loginButton.clickToNewPage();
 
  }
}

Verifying That the Production Page has Been Loaded

The real point of my flavor of PageObject pattern is the self-verifying bit. Let’s dive into that — not later, but now.

Note that each of these PageObjects extends BasePane, and has a PAGE_IS_LOADED_CSS constant:

package bizdomain.pages;
public class LoginPage extends BasePane {
  public static final String PAGE_IS_LOADED_CSS = "input[id=authentication_username]";

In the LoginPage constructor way above up there, you can see we first explicitly call super() on BasePane. Here is BasePane:

public abstract class BasePane {
  public BasePane() {
    waitUntilLoaded();
  }
 
  public final boolean isLoaded() {
    return BrowserDriver.isElementPresent(getPageLoadedCssLocator());
  }
 
  public final boolean isVisible() {
    return BrowserDriver.isElementVisible(getPageLoadedCssLocator());
  }
 
  public final void waitUntilLoaded()  {
    BrowserDriver.waitForElementVisible(getPageLoadedCssLocator());
  }
 
  public abstract String getPageLoadedCssLocator();
}

You can see that this constructor calls waitUntilLoaded(), which makes a static call to a method on our BrowserDriver (the Facade / Decorator that handles all of the actual Selenium calls) in order to loop until our LoginPage (in this case) actually is loaded. The argument supplied is the result of calling getPageLoadedCssLocator().

But wait! That’s an abstract method! So Yes, we have something very like a template method pattern here in waitUntilLoaded(): the concrete implementation of getPageLoadedCssLocator() on LoginPage returns that PAGE_IS_LOADED_CSS constant String.

Deep, deep under the covers, the BrowserDriver.waitForElementIsVisible() method looks like this:

public static void waitForElementVisible(String cssLocator)  {
  for (int second = 0;; second++) {
    if (second >=
 Integer.valueOf(BrowserDriver.STANDARD_DHTML_LOAD_WAIT_  TIME) / MS_PER_SECOND)
 
  fail("Timeout waiting for element " + cssLocator + " to become visible.");
    if (isElementVisible(cssLocator)) break;
    sleepForASecond();
  }
}
 
public static boolean isElementVisible(String cssLocator) {
  injectJqueryIfAbsent();
  return executeJavascript(JqueryCodeFactory.getVisibilityCode(cssLocator)).equals(TRUE);
}

We’ll discuss the BrowerDriver class at length elsewhere. And we’ll also discuss the injectJqueryIfAbsent() method, which we hope an upcoming Selenium RC release will obviate.

The upshot of the waitForElementVisible() method and the LoginPage and BasePane code above is that the LoginPage object will not successfully finish instantiating until Selenium can successfully verify that a unique element on that page has been loaded. In other words, once a LoginPage instance is loaded in memory, we actually are on the LoginPage, by definition, as long as that CSS element selector syntax is correct. Voila, automatic, ambient page flow assertion.

This is much of how we get our test methods so succinct. The PageObjects take care of verifying for us, at instantiation, that we have safely arrived on their corresponding production app web pages.

Caveat Lector: there is a flaw in my code I have yet to squeeze out:  duplication between a BasePane, used above, which I usually use for dynamic changes in the HTML, and a BasePage, which presumes that a real HTTP Request/Response cycle has occurred. I will collapse those two together shortly.

Next: we’ll talk about reusable HTML Element objects, and how the linkish ones know, in my code, how to return the PageObject we wish to traverse to next.

There’s a mildly rollicking little discussion going on the in the Software Testing Club at the moment, in which Rob Lambert observes, “I’ve seen a couple of conversations recently where people are talking about red, green and yellow box testing.” Rob then asks “There’s the obvious black and white. How many more are there?” (For [...]

By James Whittaker

Possessing a bill of materials means that we understand the overall size of the testing problem. Unfortunately, the size of most testing problems far outstrips any reasonable level of effort to solve them. And not all of the testing surface is equally important. There are certain features that simple require more testing than others. Some prioritization must take place. What components must get tested? What features simply cannot fail? What features make up the user scenarios that simply must work?

In our experience it is the unfortunate case that no one really agrees on the answers to these questions. Talk to product planners and you may get a different assessment than if you talk to developers, sales people or executive visionaries. Even users may differ among themselves. It falls on testers to act as the user advocates and find out how to take into account all these concerns to prioritize how testing resources will be distributed across the entire testing surface.

The term commonly used for this practice is risk analysis and at Google we take information from all the projects stakeholders to come up with overall numerical risk scores for each feature. How do we get all the stakeholders involved? That's actually the easy part. All you need to do is assign numbers and then step back and have everyone tell you how wrong you are. We've found being visibly wrong is the best way to get people involved in the hopes they can influence getting the numbers right! Right now we are collecting this information in spreadsheets. By the time GTAC rolls around the tool we are using for this should be in a demonstrable form.

You may or may not have heard of SWELL (Swedish Verification & Validation Excellence) Institute. I was accepted to attend this seminar of testing, to be honest, the 1st one for me. Unfortunately the number of participants is limited, but I expect a lot from the presenters. They seem to have a lot of experience [...]

One thing that I get occasionally complemented on is my writing. People write and let me know how they appreciate my writing style. Initially this suprised me, because I do not consider myself a...

A foreign key constraint violation can happen if you delete a row that is referenced in another table. When the constraint is violated the error message in SQL Server is "INSERT statement conflicted with COLUMN FOREIGN KEY constraint".

table named Person with an identity column PersonID
table named PersonPhones references Person.PersonID
the constraint is on the Person table

ALTER TABLE [dbo].[Person]  WITH CHECK ADD  CONSTRAINT [FK_PersonID] FOREIGN KEY([PersonID])
REFERENCES [dbo].[PersonPhones] ([PersonID])

The row cannot be deleted from Person if the PersonID exists in PersonPhones. The concept is the same in any RDBMS. The constraint could also be set up to automatically update and delete rows from the child table. It would look like this:

ALTER TABLE [dbo].[Person]  WITH CHECK ADD  CONSTRAINT [FK_PersonID] FOREIGN KEY([PersonID])
REFERENCES [dbo].[PersonPhones] ([PersonID])
ON UPDATE CASCADE
ON DELETE CASCADE

It's just something the program should check. If you add a new person all fields should contain a value before you can say CRUD was successful. (CRUD=Create, Read, Update, Delete) If nothing else the user should be given a more friendly error message.

Another post I made to the Agile-Testing Group recently:

Here's a simple estimation excercise. My honest advice is don't just read it; actually try it. It takes about two minutes.

To start, think about the space between the bottom of your feet and your knees. Write it down. Then think about the space between your knees and your middle. Write that down.

Then estimate the and write down the size of your torso, then your neck, then your head.

Next, add up all five numbers.

Now compare that to how tall you /actually/ are.

That difference - between how you imagine things and how they actually are - is a picutre difference between task estimates and how long things will actually take.

Except of course, you can see and touch your body and it's been about the same height for decades, whereas code is new and fresh and symbolic and 'tests' are an even-more abstract concept.

When you think about it, tests are a first-order derivative of the code itself. Also, most testing is exploratory in nature, EG early predictions are not the best predictions.

So would I be reluctant to make task estimates on a testing task, given the typical American shorthand that estimate==commitment? Certainly.
I like to think of this as the test estimation rabbit hole. First, we have to have the bad news that test estimation is conceptually impossible.

Then we figure out how to do it anyway.

More to come.

I’ll be traveling and training this fall. Take note of where I’ll be, because if I come near where you are and want some relatively free consulting, all you have to do is take me to dinner.

I’ll do almost anything for free food.

September

• Reston, Virginia
• San Diego, California

October

• Stockholm, Sweden
• Tartu, Estonia
• Cluj-Napoca, Romania
• Bucharest, Romania

(The event in Cluj-Napoca is a one-day public seminar that introduces my Rapid Testing methodology through a series of puzzle challenges and lecture. I will also take any and all questions about testing. Click here to sign up for that.)

November

• Tallinn, Estonia

December

• England
• (tentatively) Singapore

(I have one class to teach in England. I’m interested in doing something more, if there is any interest. Email me.)

My full schedule is published here.

If all you knew about testing you learned from the interview, what would you assume after watching interviews for testers at the top 10 software companies?

• Testing is an activity where you write code on a whiteboard.
• All tests are automated at this point in time.
• There is no important work in testing except for coding mainly on whiteboards.
• Fewer lines of code are always better so we don't waste pens.
• It really isn't possible to be too technical.
• It is just a matter of time before we are all replaced by robots, and thank goodness. How efficient will that be?
• Solving for palindromes and list order is really vital in testing.
• Knowing the maximum programming languages makes you the best tester.
• Having experience writing automation in one of the languages the company is looking for makes you the best candidate.
• Testing is mostly a programming craft.
• Hiring the best tester is mostly about matching a skill checklist and verifying the programming you want
• Test planning means designing a good program to categorize things.
• Because you CAN do the job, you WILL do the job.
• Programmers know that testers are just like them, only focused on a different problem.
• We think just like programmers because we are programmers and that is best for the user because that is best for the programmers because we let them define what they wanted in testers because they know what is best for the product is what serves them best, so it must be best for users.

If a young person asked me how to give themselves the maximum chance of being hired by learning software testing today, I would advise them not to bother. Learn 1 programming language well and work on building out a longer list  (I'd suggest C# right now since that is what the jobs around here tend to be looking for most often, maybe Java since I've been asked for that a few times, certainly not Python because people don't seem to take it that seriously.) Then call yourself an SDET and practice solving problems on a whiteboard.

The above process might make some sense if it wasn't paired with the http://agilemanifesto.org/. How are you possibly putting individuals over tools if you are ignoring every aspect of what makes an individual a good long term investment or a tragic one?

These interviewing and hiring practices only match with the http://www.halfarsedagilemanifesto.org/ and should come with a certificate that says, "We are too terrified to try the real agile manifesto. We can't put individuals and people first. It is too scary and radical for us. Agile for real would make us feel powerless, and we can't give up that power. We are perverting Agile, and proud of it." It will never be stated that way. Instead, just say "We're customizing Agile to suit our needs." The mercenaries will comfort them, "There there, at least you got started and had the stand up meeting. You can be in the Agile club! Each company is different. So what if you've danced around the important part and implemented an empty shell. You'll still get your on time under budget bonus! And the users? Well, we've got a product owner who speaks for them.

As Lisa Crispin pointed out, the perversion of agile is harmful to everyone, not just to testing. There is no possible way for me to put a positive spin on what I see is the greatest danger to increased craftsmanship in testing. For that reason, let me share instead an alternative I'm seeing that fills me with great hope.

I'm seeing some interviews where people have a chance to audition. Find and write real bugs. Write code in their language on their system to show someone else. To have an interview as more of an entire person. To join with the team for a trial period. I see a future where you earn a chance to work with a team short term, say 6 weeks, and then they decide to keep you or you move on. This would be a better way to hire. What if you could audition 10 of your best candidates for 6 weeks and keep the best overall fit? Can you imagine how great this would be for your team? What about for the employees? At 6 weeks you are going to see who is learning fast, and who just started with a slight advantage. Who just interviews well, and who really works hard? Who asks the team for help and who stays lost for hours?

I posted this yesterday to the Agile-Testing List, thought I would share it here as well:

--- In agile-testing@yahoogroups.com, "daswartz@..." wrote:
>
>
> Can you help us understand why the QA people care whether
> you estimate in hours or points? I'm sure they have a reason, which
> should help us better answer the context for your question.
>

I'm not the Original Poster, but consider you are testing feature X. You break it down into tasks and say it will take 5 hours to "test."

The first build is total garbage. You can't even click the submit button.

The next day, you get a new build. You find five bugs. You get a new build
late in the day - four of the five bugs are fixed, and you find three new ones.

You get the fixes in the morning on day three. You find another bug. At noon,
your boss comes up: "You said this would take five hours to test and you are on
DAY THREE of testing? Wassup with that?"

---> Bottom line, there are elements in how long it takes to do testing beyond
the testers control. It's generally possible to estimate a test /cycle/ with
some accuracy, but estimating the entire test /process/(*) is rarely possible
unless you know the devs very well and have had some stability in delivered
software quality for some time.

Estimating in terms of points 'smooths' those gaps.

That's one possible explanation, anyway ...

--heusser
(*) - Yes, this pre-supposes that testing is a separate and distinct activity, I
know, we should be involved up front, whole team, etc. I'm with you. But
you gotta walk before you can run. Let's have that discussion on a separate
thread, ok?

At Agile 2010, there were about 20 of us at the AA-FTT (Agile Alliance Functional Test Tools) workshop. The emphasis was on “the state of the practice” of Acceptance Test Driven Development (ATDD). In one of the breakout sessions, we had an interesting discussion on what we actually meant by ATDD, which made me think a lot about the practice itself. Although we all knew that acceptance tests defined by the customer / business expert is what drives the TDD (test driven development) process, there were discrepancies in how each of us thought of full process.

JB Rainsberger had considered ATDD as two concentric circles with TDD, the developer practice of Test Driven Development, to be the center, with ATDD surrounding it. I felt his view of the practice was from a developer perspective – give me an example, and I will start coding. Many of us at the table thought it was more than that; it included the collaboration necessary to get the examples. Based on that discussion, the next question asked by JB was, “Then, what’s the difference between ATDD and BDD?”

Part of the problem is that we don’t all use the same vocabulary relating to the practice. I started by writing down all the words I know are used in conjunction with the practice. The following words and phrases are some of the ones I could think of:

• Examples, scenarios, acceptance tests, customer tests, behaviour, business-facing tests, story tests, functional tests, acceptance criteria, conditions of satisfaction, business rules, executable specification

There have been many explanations about ATDD and I’ve included a couple here to help explain.

Jennitta Andrea wrote a short article for Iterations about what ATDD is, and why teams should be doing it. She described ATDD as:

“
the practice of expressing functional story requirements as concrete examples or expectations prior to story development. During story development, a collaborative workflow occurs in which: examples written and then automated; granular automated unit tests are developed; and the system code is written and integrated with the rest of the running, tested software. The story is "done"—deemed ready for exploratory and other testing—when these scope-defining automated checks pass”
The full article is at: http://www.stickyminds.com/Media/eNewsLetters/Iterations

A couple of years ago, Elisabeth Hendrickson used an example to describe ATDD in her blog: http://testobsessed.com/2008/12/08/acceptance-test-driven-development-atdd-an-overview/

As a result of the AA-FTT workshop, Declan Whelan, Gojko Adzic and a few others came up with a diagram trying to get a common understanding around Specification by Example. It is on Google Docs - http://tinyurl.com/3x52ap9

Dan North’s description of Behaviour Driven Development (BDD) http://blog.dannorth.net/introducing-bdd/ is a bit more complicated, but I interpret it as using the word ‘behaviour’ rather than ‘test’ , and use the “Given, When, Then” guidelines to describe the precondition, trigger and post-condition. In a recent agile testing yahoo post, Liz Keogh posted this:

“BDD's focus is on the discovery of stuff we didn't know about, particularly around the contexts in which scenarios or examples take place. This is where using words like "should" and "behaviour" comes in, rather than "test" - because for most people "test" presupposes that we know what the behaviour ought to be. "Should" lets us ask, "Should it? Really? Is there a context which we're missing in which it behaves differently?"
Where we choose to call it BDD or ATDD or Specification by Example, we want the same result – a shared common understanding of what is to be built to try to build the ‘thing’ right the first time. We know it never will be, but the less rework, the better.

Using business rules, use cases, business expertise (customers, Business Analysts, domain experts, etc), the development team’s (programmers, testers, DB guys, etc.) expertise, and any other supporting information, we can describe what we expect the software to do in the format of examples which are just a specialized test. In workflow type applications, describing behaviour may be applicable. If it is calculation heavy, then a spreadsheet type test format with specific examples might be more appropriate.

Do these tests have to be automated first? I encourage new teams that are struggling to find the right tool, to focus on the purpose of ATDD which is to drive development to deliver the right ‘thing’. If we can do that with a sentence, let’s do that first. We do have to automate them at sometime during the iteration, but until a team has their rhythm or cadence, the automation may not come first. Once you have the automation, you then have documentation that is up to date and describes the system behaviour.

The picture I use to describe ATDD for new teams is:

I think the story isn’t ‘Done’ until the customer has actually validated the actual completed story which includes pairing with the developer after he’s completed coding for a first look, and the exploratory testing.
I will continue to use the phrase Acceptance Test Driven Development (ATDD) unless the industry decides on a common vocabulary, because I find that business stakeholders not only understand how to give examples, but also understand when I talk about acceptance tests that prove the intent of the story or feature. Once we have that, the team knows enough about the scope to start coding and testing. That's what's important.

So, I am using Google Voice as my voicemail these days. It's neat, and I like it, mostly. They offer, for free, a transcription service that sends me an email/txt message of what the caller said. Over the last 3 days, I've gotten calls form my son's middle school about him being absent (he is actually withdrawn!).
The beauty of this setup is that the tester in me LOVES certain characteristics of this test set. The call is an automated one -- in this case, I know that the caller is saying the *same exact* thing every time. The tone is the same, the inflections are the same, it's the same voice source data. However, over 3 days, I've gotten 3 different transcriptions from Google Voice!

The voice mail *actually* says this:
Hello. This is the attendance office at Durant Road Middle school, calling to inform you that your child, Steven Cannan, was marked absent today. Please send a signed note upon returning to school, explaining the reason for the absence. Thank you.

And following are THREE Google Voice transcriptions ... they crack me UP!

Day 1:
Hello, this is the attendance office ed to Ron corrode middle school calling to inform you that your child. Hey, it's Dawn Cannan was marked absent today. Please send a signed note on returning to school, explaining the reason for the absence. Thank you.


Day 2:
Hello, this is the attendance office and to Ron corrode middle school calling to inform you that your child. Jeanne cannon. You're a smart ass in today. Please send a signed note up on returning to school, explaining the reason for the absence. Thank you.


Day 3:
Hello, this is the attendance office at Deron corrode middle school calling to inform you that your child G intended You're a smart ass IN today. Please send a signed note up. I'm returning to school, explaining the reason for the absence. Thank you.

I *kind of* think they did the best job on the first day :)
But this brings up an interesting point for me, and it's one that I have encountered several times in my career as a tester.
Roughly, I can ask this question as something like this: How can you test something that is not feasible to prove, logically or mathematically, is *correct* every time?
I can think of so many examples where testing isn't cut and dry. I started working as a tester in a company that did software that processed genetics algorithms. How did we know that the genetics algorithms were correct? At that time, I started off by having all of my testers know how to do the algorithms by hand, so that results that weren't right would just stick out to them.
At one point, I was testing a search engine. I was lucky in this case to have knowledge of the source data in a database. One approach I took here was to find objects in the source data that exhibited certain special (and easy to identify) characteristics. Then, I could perform searches that I *knew* would return those objects and look for them early in the test results. But even then, I didn't really know that the search engine was performing correctly.
Recently, I read a set of slides by Harry Robinson, talking about testing Google Maps. Here's another good example: How do you test something that has multiple right answers, in such a way that you feel confident releasing it to customers?
I've given two approaches that I have used, but I'd like to know how others have solved similar issues. Please, let's talk, I'd love to hear more from you.

I have four or five blog posts in the hopper, each almost ready to go. I’m working on a whole book and a chapter of another one, and I’m on a deadline that I’m about to blow. The kids are still out of school, and I really should be cooking dinner right now. And yet… [...]

Sometimes analysing what needs to be tested can be overwhelming, but I don't see this as such a big deal. I always start with what I see as the 3 key test building blocks Add, Edit & Delete and then expand and build on them. Too often I've seen some really complex test matrix created only for the most obvious tests to be overlooked.

By following the Add, Edit & Delete principle, you cover many bases such as data loss when saving and usability.

Once you've got these basics covered, then you think about other tests to widen the coverage such as: validation (positives & negatives) and boundaries. See my checklists for other tests.


This approach is particularly useful in agile where full functional requirements might not be available, However, this is just as effective for waterfall development.

Ray

Many months ago at the Writing about Testing conference I did a short point/counterpoint with Jon Bach about certification. I explained why I don't care about the topic. The reason I gave was that certification is annoying, like a small ankle biting yippie dog latched onto the ankle of testing. I didn't want to spend time on it when there is a giant pitbull attached to the jugular of testing as a craft & profession. Jon felt that anyone profiting from harming the profession of testing and preying on the ignorance of the uniformed should be called out and held responsible for their behavior, but at the end it turned out not to be a point counterpoint afterall. He has also seen the damaged caused by the ferral pitbull. In his talented thinking testers and the requests from his clients time after time.

For months now I've been thinking about how I can express what I see as the biggest threat to testing, and still I've not thought of safer words or a better way to express it, so I'm going to just get started.

Without much detail, the biggest issue facing testing as a profession is the perversion of Agile.