It is a long tradition in the software world to ‘borrow’ things from Japan and apply them in North America. Lean? Kanban? Shu Ha Ri? All come from the other side of the Pacific. And in most cases I would say the ‘experts’ espousing them don’t really have the right to. Sure, they know the ‘concept’ and the ‘actions’ but do they really, really understand the cultural underpinnings that lead to them?

So along these lines, I’m going to start stealing from the Japanese. Here are my credentials…

• I did Karate for eight weeks
• My friends in highschool were ninjas
• I did Kendo for a year
• I have seen hundreds (and hundreds) of hours of Anime
• My son now takes Karate so I’m hanging out at a dojo again (if only in the lobby)

Infinitely qualified, see?

First, I think we should steal the Tea Ceremony for its ceremony and presentation. Specifically designed rooms, lots of steps in specific order, repeatability and lots of history make it a fantastic specimen.

Oh.

Wait.

The Factory School already got to it.

Next then let’s steal Anime, or more specifically its Tropes. But let’s steal in a sensible way. Rather than apply things directly, each of these should be considered heuristic. And some of these are almost designed to baffle management. Imagine seeing ‘Quivering Eyes’ or ‘Snot Bubble’ or ‘Idiot Crows’ in a status update on the techniques used on a project. Priceless!

When I first thought of this post, I’m sure I had something on Japanese Gardens but can’t remember what, so that is left to the reader.

The final thing to steal is the notion of Shinbutsu shugo which is the syncretism of Shinto and Buddhism. In a testing context, there is Agile, Context-Driven, ATDD/BDD and Artistic camps. Each has its own priests and followers which get involved in religious debates. But the reconciliation and idea sharing is important regardless of religious differences. And now we can give a Japanese name for it.

And if we’re really good, we’ll mispronounce it.

We recently had a case of debugging a testcase where Load Tester reported an error with a page that looked and appeared just as it should, except it was missing a link the user needed to move forward. Debugging this case was easy: examine the replay two pages prior to the error, and the server had added a red error message to the page indicating the selected criteria was not valid. So why didn’t Load Tester flag this error from the server? Beneath the error message, the page was exactly identical to the expect page: form fields were present and filled … Continue reading »

So previously I posted that factors outside our own control make accurately estimating the total test schedule impossible. In addition, I posted that even estimatingf simple, known, static things by breaking them into tasks, then adding up the tasks is much less accurate than you might think.

Example Three:

Imagine your typical workday that ends at 5:00 think about it. Sure, I'll likely be home from my drive at 5:30, but I might hit traffic, my boss might ask me a question at 4:55PM, someone in the parking lot might need a jump start. So if you want a commitment from me to predict when to invite the neighbors of when to pull the turkey out of th oven, I should likely say 6:00PM.

That's a 100% timing padding.

One hundred per cent.

On a task I've done hundreds, if not thousands of times with known physical 'resources' and very few dependencies.

Which reinforces the conclusion that test accurate estimation is impossible. At least in the sense of estimating a single number for total time without context.

Yet we are tasked with doing it anyway -- better yet, creating that context. So where do we start?



About a year ago my good friend Ben Simo pointed out to me that even if you know nothing at all about a project, you can always estimate it. Here's an example:

Q: "How long will testing take?"
A: "Two Weeks."

See that? I know absolutely nothing at all about the software. Nothing at all. But I made up a number.

For any given project, we likely know more than nothing. So we can certainly do a better estimation than "two weeks."

Next: I'll start talking about how I do that.

Readers of the uTest blog are no doubt aware of the ongoing (some would say never-ending) debate on tester certifications. We occasionally write about this topic ourselves and frequently ask our Testing the Limits guests for their thoughts on the matter as well. Some have opposed them unequivocally, arguing that a certificate can never validate the passion, curiosity and other intangibles so important to the craft. Others consider them useful given the right circumstances. uTest has taken a “neither promote nor prohibit” stance on the issue. How’s that for a political answer?

Well, no matter which camp you fall into, the debate over certifications in the testing space could get A LOT more interesting. But instead of testers being certified by an independent body (like ISTQB), we’re now talking about testing companies being certified by the federal government. But certified for what exactly?

We recently learned from DOTmed.com that the U.S. Department of Health and Human Services has named two private testing companies as the first “certified bodies” for testing the use of electronic health records (EHR). Writes Brendon Nafziger:

Doctors and hospitals angling for incentive payments, offered under the Health Information Technology for Economic and Clinical Health provisions of the American Recovery and Reinvestment Act, must show they’re “meaningful users” of EHR. And these EHRs must themselves be certified by an ONC-approved body that they can meet the criteria to support meaningful use…

As mentioned, only two entities meet the criteria of “ONC approved” so far. No word yet on whether there will be more. By the way, ONC stands for Office of the National Coordinator for Health Information Technology. I had to look it up, too.

Without getting into any more specifics, I wanted to briefly touch upon the theme of certifications in general, and what it could mean for the software testing industry.

Let’s start off with a hypothetical. Assume a law gets passed that requires all software testers to obtain a “testing license” before they could legally practice the craft. What would that do to the number of available testers in the industry? How would it impact the development cycle in small and medium sized firms? What about the cost of obtaining this license – both for the testers and for the employers? And most importantly, what effect would it have on the quality of testing in general?

Needless to say, the introduction of licensing into the software industry – especially that which is backed by law – would greatly increase the cost and complexity of testing.

Lucky for us, the greater testing community, such certifications are not (yet) required by law. Testers are free to choose which certifications, if any, they wish to obtain and companies are free to contract with whomever they want – certification or not.

Contrast this to EHR situation. Imagine how expensive and time-consuming this process is going to be with only two certified bodies! The transition to electronic health records is a massive undertaking and companies in this space are going to need all the help they can get. But if this certification trend gains traction, there will be a fewer and fewer available testers. There could even be a shortage of software testers in this particular instance.

Anyway, I’m interested to hear what you think about this development. Will certifications work for testing companies better or worse than they work for individuals? Is this a positive trend for the industry?

Psion Teklogix, the pioneer of rugged mobile computing, has utilised Original Software’s newly launched Application Quality Management (AQM) solution, Qualify, to enable significant time savings, process improvements and re-use of assets; accelerating innovation on ground-breaking new products and features. Shahe Altounian, Manager for Product Support and Services at Psion, leads a team of Test and [...]

- Is the size of a test important?
- How big and comprehensive should a test case be?
- Should tests be always small, or should you write comprehensive tests that cover end-to-end and complex scenarios in “one-go”?

All of these are important questions, and even if there is no text-book answer for them there are some guidelines you can consider when wondering about this subject.

My thoughts on test sizes were prompted by 2 separate events. This week we are starting a long-range project to re-factor PractiTest’s test automation framework (the one we use to test ourselves!). This time we are making a conscious effort to keep our tests as short as possible and to re-use them as part of more complex test-sets.

I am also guiding one of our PractiTest users on his custom deployment, and we are trying to define how deep and extensive should the tests be in order to answer his needs and constraints. With this customer we are reaching the conclusion that most of the tests should be kept small and high level; but still a limited number of regression scenarios will be written very detailed and covering the system from end to end.

When considering the right size and depth of your test the spectrum is extremely wide.  On the one hand you can write a word-by-word scripted test, allowing you to take ‘any person from the street’ and use him to test an application he hasn’t ever seen.  On the other hand you can give an application to a tester and ask for her “professional opinion” without giving her any indication to what the AUT should do or what you want to check, then come back 3 days later to see what she has to say.

I personally think that both of these extremes are wrong! But there is a large space in the middle to choose from, and many times the right choice will not be easy to find or trivially clear.

I personally prefer my tests to be small and as high-level as possible. I believe that if I take good testers and we give them an introduction on what the AUT should do and who it should serve, then I don’t need to provide them with step by steps cases in order for them to do the job right.

Instead of creating extensive test cases I prefer to user generic check-lists and heuristics to help me go over my application. These tools allow me and my testers to test thoroughly and to find the important stuff; but on the other hand it doesn’t drive us mad by writing kilometric tests with low-level steps that no one really needs and just frustrate testers simply by needing to read them.

There is also an added point to having high-level tests, they force us to think when we test!  I’ve seen that if you give even an experienced tester a very detailed test she will turn into a kind of live-test-automation-tool, performing each and every step but being so focused on it that she doesn’t pay attention to the “interesting scenarios” lying right next to her predefined test.

When your tests are high-level steps or even charters or threats (like a couple of interesting articles I read this week about Exploratory Thread-based testing by the Bach Brothers – here are both by James & Jon) you actually motivate your testers to THINK BIG and take more responsibility for the test execution, overall achieving better results and finding more interesting and severe bugs.

On the other hand, having no tests at all and trusting on the intuition of your testers is a sure recipe for failure and important escaping defects; specially when not all your testers are experts in all the system, and even worst if they need to work under pressure when needing to provide results quickly.

So I guess that at least for me size does matter in my tests! I try to keep them small and smart as possible.

What do you think?

• Subscribe to the comments for this post?
• Share this on del.icio.us
• Digg this!
• Post on Google Buzz
• Share this on Reddit
• Stumble upon something good? Share it on StumbleUpon
• Share this on Technorati
• Tweet This!
• Share this on Facebook

Related posts:

1. Walking a mile on a Developer’s Shoes – why we should not assume they can test like testers

FxObjects is JavaFX Application Framework that enables methodical, pattern based and test friendly way of developing JavaFX applications based on ideas and best practices distilled from real life usage

The last webinar in our series, Build Promotion with Nexus Professional, was a huge success. This week Sonatype’s Senior Systems Engineer Blaine Mincey is back to host another Nexus Professional webinar.  This webinar will be held this Thursday September 9 at 10:30am PDT, 1:30pm EST.

This webinar is an introduction to the Build Promotion capabilities of Nexus Professional, the industry leading repository manager to help companies acquire, manage, and report on open source software artifacts in their software development projects.

If you release software, you will often need to test a release before deploying it to a production system or an externally accessible repository. For example, if you are developing a large, enterprise web application you may want to stage a release candidate to a production system and perform a series of rigorous tests before a release manager makes a decision to either return a system to development or deploy a system to production.

The Build Promotion feature in Nexus Professional allows an organization to create a temporary staging repository and to manage the promotion of artifacts from a staging repository to a release repository. This ability to create an isolated, release candidate repository that can be discarded or promoted makes it possible to support the decisions that go into certifying a release.

To register for this webinar, please click here.

 

Unit Testing in C++ – it’s even harder than Inidian Pole Gymnastics

I mean seriously - CPPUnit is the devil. What kind of test framework do you use in C++?

Cloud washing is in full swing this year with many traditional application performance management (APM) vendors claiming to have a solution that is specifically designed to meet the challenges in the cloud but yet is largely based on the same old approach used in previous incarnations that targeted data centers running in particular high latency applications deployed to standard application server runtimes.

So what exactly constitutes a cloud monitoring solution? That’s a pretty hard question to answer. We still do not have a clear definition for the cloud and any attempt would be out-of-date very quickly as the cloud is likely to undergo significant and rapid change over the next few years as we move beyond provisioning of bare metal in (vm) units at coarse grain periods (hours) to an ecosystem (grid) that supports mobility of code and data that is context (service, location, capacity,…) and cost (quality, financial,…) aware and which gives greater prominence to service interactions and service supply chains.

It is much easier to say what a monitoring solution for the cloud is not. Here are some examples.

1. Off-site (central) storing of monitoring data for single-point of failure viewing in a browser

This type of cloud washed APM vendor confuses the pushing of traditional response time monitoring data on a per request basis to a remote web site with cloud computing. The site might not be using any cloud infrastructure itself. More importantly none of the data collected pertains in anyway to important execution aspects of the cloud itself such as cost, collocation, and capacity (reserved or variable). Incredibly whilst you might have managed to scale your own applications you cannot be certain that the same applies to the monitoring solution. When it goes down you are completely blind because that important data that is secretly pushed out from your application runtimes is lost in the cloud – another cloud (if at all).

Our Vision: Monitoring data must travel first class with its the request/response payload. It must be accessible to multiple interception (metering) points up and down the technology stack and across execution boundaries. It should not be pushed to (or consumed by) only one (owning) destination point leading to a complete disconnect back through its chain of events & activities. It should be accessible in real-time within the context of the execution and flow supporting immediate problem determination, cost based optimization in call routing and fine grain capacity leasing. It should not measure execution behavior in one dimension.

2. Gimmicky usage of monitoring data for “shoot yourself in the foot” elasticity

This type of cloud washed APM vendor offers the ability to blindly execute a script that fires up one or more machines and servers on a publicly accessible IaaS platform such as Amazon’s EC2 based on some performance threshold being exceeded. This fire-and-forget-and-hope-for-the-best-without-a-care-in-the-world-for-cost approach is based on naive view that “more is better” whether it has any influence on the particular performance issue detected, positive or negative, is irrelevant. Who cares this [elasticity] is cool. Ops can just sit back and look at those large RED circles on the dashboard, through the hole in the donut their eating, and watch them turn GREEN as the monitoring software now becomes the master controller. There is no time for thinking, reasoning or policing. Aim and fire – AMI and EC2!

Our Vision: A monitoring solution should not be the controller especially when such solutions are focused on a single concern – performance. Monitoring data must feed into a model that unifies multiple management concerns including quality, cost, availability and value (profit) providing insight to both operations and business personnel some of which will eventually be used to introduce very specific automation but only after knowledge acquisition has taken place and derived models validated and verified. A monitoring solution should focus on ensuring data captured is sufficient, accessible, immediate, accurate and cost effective.

3. Meaningless publication of monitoring data to fake value and context

This type of cloud washed vendor has been around for countless iterations of computing. They report news for the sake of reporting. They lure a customer to a site offering flashy monitoring dashboards that present arbitrary performance & availability metrics pulled from one or more cloud platform. Whether such metrics have any relevance whatsoever to the performance and availability of other applications in such clouds is never communicated or related. The reporting seems to imply that the majority of applications in the cloud are much better engineered than the cloud itself. That might indeed be the case if you are pulling such data from a hello world web application deployed to such clouds that does nothing of significance and is rarely changed. Sadly in the real world the availability of the typical business application lies squarely with its own engineering. Such applications are routinely (daily) restarted in private data centers today around the world. It be unfortunate for a cloud platform vendor to have an outage when one of these applications was actually running reliably. So what’s the point? Marketing!

Our Vision: Metric monitoring is not suitable in effectively managing applications in the cloud. Monitoring needs to focus on the cause of costs (performance, expenditure, loss, capacity) in the cloud – users, activities, and resources. This requires a monitoring solution to be integrated at some point in the service supply (and resource consumption) chain and to be able to collect sufficient measurements. Monitoring data must reflect real life events, actions and workloads – not artificial requests. Monitoring must be performed at the point of interaction (and consumption) and then to be directly attributed to its causes (cost objects) for the purpose of cost management, charge backs, and resource metering & control.

It’s Labour Day here, so this post was written to backdrop of Sponge Bob. In case you were wondering.

• selenium-java-evidence appears to be another logging framework, this time with an emphasis on auditability. Personally, the reports from a CI server should be all that are required, but to each their own…
• taza is an opinionated browser-based test framework
• webdry is a Python (jquery-inspired) wrapper for DRY:er access to the Selenium WebDriver
• Self-testing AppEngine apps with Saucelabs seems like a cool little (if very low level) trick to using Se inside AppEngine
• Sauce Labs released their Getting Started With Selenium guide and is looking for feedback on it
• Need to interact with a web service during a BrowserMob run? Hybrid BrowserMob Scripts explains how to do it.
• Self-Verifying Page Objects is Patrick Wilson-Welsh’s first post in the series patterns that made up his and Dawn‘s Agile 2010 session.
• Felipe is starting to explore Se and TestNG (which is timely for me) starting with Using TestNG to launch your tests (and the Selenium server)
• Hudson create Kohsuke Kawaguchi did a webinar for Sauce Labs on using Hudson with their OnDemand service that was recorded. I haven’t watched it yet, but it looks like it has general Hudson and Se information as well.
• Se’s CSS support got and overhaul this week as Sizzle replaces CSSQuery as the way of finding things via the CSS locator. Sizzle is the CSS engine that powers JQuery.
• JBehave 3.0 has been released – for those who use it
• Simon wrote another technical piece on some of the inner workings for Se2 in Going Atomic: How

In this session we review the top 10 ways you can cause unit testing to fail to work for your project. From readability problems and lack of trust to mock object abuse and semi-integration testing – these gotchas can save you lots of time, sweat, and tears on your current and next projects. Download video in [...]

In this interactive exercise, you will learn what makes a good acceptance test and how to solve the most common problems with such tests. The participants will work in groups on restructuring several tests and then discussing the results. Gojko Adzic will run around facilitating the exercise and helping you if you get stuck. http://skillsmatter.com/podcast/agile-testing/anatomy-of-a-good-acceptance-test

Goanna Studio 2.0 has been a great hit, we have gotten a lot of positive feedback and we have also acted on very nearly all suggestions and bug reports to produce a shiny new Goanna Studio update 2.1. This means that a great many false positives have been eliminated, greater accuracy has been achieved and we have also fixed some bugs and there is even a performance improvement or two.

The fixed false positives and changes to checks include:

• ATH-sizeof-by-sizeof - a false positive involving array sizeofs has been fixed.
• FPT-cmp-null - a false negative when warning about using a function pointer directly in a condition, not using a condition operator.
• RED-unused-var-all now considers sizeof(x) to be a use of x.
• MEM-stack-global, MEM-stack-param and MEM-stack-param-ref now take into account re-assignments for globals or parameters
• SPC-uninit-struct now considers a struct with a field which is an array to be accessible without warning.
• ATH-cmp-unsigned-pos and ATH-cmp-unsigned-neg take into account comparisons with (unsigned)-1
• MEM-free-some doesn’t warn when you check the result of a malloc and exit (or return) if it is invalid
• RED-dead no longer warns about goto’s and breaks
• PTR-null-const-pos doesn’t warn about string literals
• EXP-null-stmt no longer warns about a non-empty else block or when there is an assignment or function call in the condition
• SEM-nonconst-call has been renamed to SEM-const-call
• SEM-global-write has been renamed to SEM-pure-global
• SEM-impure-call has been renamed to SEM-pure-call
• ATH-shift-bounds now warns when you shift by 64
• COP-assign-op-ret now warns about assignment operators that return a non-const reference to this

The Goanna Studio for Visualstudio also has some improvements and bug fixes:

• Support for additional options and CL environment variable support.
• Fixed bug involving a non Win32 or Win64 build target and the verbose flag
• Support for environmnent variables with non-alphabetic characters
• ProjectDir and SolutionDir macros now expand with a trailing backslash

We’ve also worked hard to squeeze even more performance where we can, and overall the Goanna analysis engine now scales better with the number of checks that are being used, and Interprocedural analysis is now slightly faster.

Other minor improvements and bug fixes are:

• Better handling of types that have been typedef’ed
• Better handling of implicit this parameter in member functions
• Better handling of simple conditions containing only variables
• Better identification of compound declarations
• Better handling of throw and catch statements
• Support for -std=gnu++0x and -std=c++0x command line arguments
• Improved support for c++0x standard (still incomplete)
• With Goanna Central Linux - ability to use specific rc files
• Goannacc now correctly identifies versions of GCC that are built for different targets

As usual, all current users get the upgrade for free. If you were a trial user in the past and need a trial extension visit: http://redlizards.com/trial-extension

This is the second of my technical posts. Again, if you’re interested in the internal workings of Selenium 2, then please skip straight to something else. If you’re interested in how and why we made some of the technical decisions on the project, keep reading….

We left our intrepid heroes in a tight spot: they’d decided to write a shared library of code, to be used by the various webdriver implementations and selenium core, but the requirements for doing this seemed to be at odds with it actually happening.

Fortunately, at about the same time we started down this path, Google Open Sourced the Closure compiler. This is a Javascript compiler that takes as input a set of Javascript files, and which outputs Javascript. It can be configured to either pass the code through untouched into a single file, or it can compile a script aggressively, removing unused code-paths and minifying the output as much as possible. The Closure compiler is used on a lot of Google products, so we know that it’s reliable and consistent.

In order to get the best out of the Closure compiler, we’re writing the atoms using the Closure library. This isn’t as well known as some of the other JS libraries out there, but it’s solid, well tested and is being actively developed. It also features an easy-to-use extension of JsUnit, which makes writing tests a far simpler task than might otherwise be the case, and it has an easy to use mechanism for modularizing code.

So, given that we could compile a single Javascript function (and it’s dependencies) into a minified fragment of JS, we were all set, right? Not quite.

The problem is that the atoms are being extracted from two frameworks that have a different way of viewing the world. As an example, Selenium 1′s “getAttribute” method only returns the value of a particular attribute, whereas WebDriver’s “getAttribute” method will return the value of either a property or an attribute (because sometimes it’s hard to remember whether something is an attribute or a property of an element)

As with all problems in computer science, an extra level of indirection is used to solve this issue.

We’re busy implementing the expected behaviour of both WebDriver’s and Selenium’s API on top of the atoms.

There is, of course, the obvious question about how we get this carefully compressed JS into a driver. One option would be to include the raw Javascript as files in each language binding, and pull them in as required. That’s possible, but it would make each language binding bloated, and would introduce a lot of duplication. The alternative is to push the atoms as far into the driver as possible, and this is what we do. As part of the build process for webdriver, we take the compressed JS and convert it into a form that can be consumed by a particular driver. For example, for the IE driver, we convert them into constants in a C header file. These constants can then be referred to by the driver and converted back into a script than be executed via the same mechanism that is used by “executeScript”.

What do we gain from this seemingly baroque approach? Other than the ability to share the same code between drivers? Many things. The cost of maintenance drops dramatically as we can fix a bug in one place and have that fix be picked up by every driver. Because we’re working in pure JS and just querying the DOM, we can run the unit tests in a browser whilst we’re developing the code. This leads to a very tight feedback cycle. It also makes it easier for developers not familiar with the code to take a look at how we do things, and send us patches (always appreciated!) Finally, we can ensure a consistency of result.

Right, any questions?

Last time I talked about HudsonMobi 2.0 there were some hiccups with the QR code which made me pretty irritated. Now the leading mobile app for Hudson users has regained my trust with their lastest release for iOS 4, the latest incarnation of Apple's mobile operating systems for iPhones and iPod Touches.

Since I fall into the category of subhumans who for one reason or another choose not to own an iPhone, I can't verify the awesomeness of this new HudsonMobi release. If it is anything like the 2.0 release for Android you can expect enhancements such as:

• Tighter integration with iOS 4
• Embedded artifact viewer! For archived build artifacts that are text-based, you can view them directly on your handset.
• Access to the build history and changes for a job.
• Quick and easy access to a job's last build

If you own a device with iOS 3.0 or higher you should be able to download HudsonMobi 2.0 from the appstore, if you give it a whirl, please leave a comment to let me know how awesome it is :)

Just wanted to let everybody know that I've gone ahead and added a Calendar for all the upcoming Hudson-related events.

Hopefully we'll be able to add more and more events for the rest of the year including seminars, more meetups and potentially a few drink-ups!

Worth mentioning that I've not yet tested the iCal feed so if you have troubles with it, let me know (via the comments).

[Part of a series of posts on Java Selenium RC patterns I find useful.]

What we Want: Expressive, Succinct Tests

Let’s say I want to test that I can log into an app. I want my Selenium RC Java test code to look something like this, because I want it to read, aloud, the way I would describe actually logging in and checking that everything went mostly OK:

@Test
public void canLoginToFatFreeCRM() throws Exception {
  LoginPage loginPage = new LoginPage();
  DashBoard homePage = loginPage.login(BrowserDriver.DEFAULT_USERNAME,
                          BrowserDriver.DEFAULT_PASSWORD);
 
  assertTrue(homePage.isLoaded());
}

Several techniques make this possible, but the one we will focus on in this blog post is the notion of a self-verifying PageObject, which is essentially a “testable representation” of an actual web app page we wish to traverse, manipulate, and test. So my LoginPage class, above, allows me succinct manipulation of the corresponding login page in the system under test.

Note that this test knows nothing about Selenium. Indeed, neither does the LoginPage class (below). More on that later. Note that this test does not include any waitUntilPageIsFrickinLoaded() calls, nor (worse) Thread.sleep() code, and very little direct instantiation of PageObjects. The page flow mechanics is encapsulated elsewhere. Note also that as I go from page to page, I don’t explicitly assert that I got there. I shouldn’t have to. My test should be able to take that for granted. It turns out that assertions are being made under the covers about having arrived successfully on intermediate pages, but that code is abstracted away. That’s what this post covers.

The only useful assertion in the test proper is that, once I have logged in with proper credentials, I arrive safely at the homepage (and even this assertion in the test is redundant, and included here only to reveal intention; the DashBoard page is also self-verifying).

Glance-Readability

A test as short and expressive as the one above passes the “glance readability” test. Anyone familiar with automated testing semantics, and the biz language of this CRM domain, ought to be able to grasp what this test does at a single glance.

Suffice to say, we want our Selenium RC tests, as much as they can, to be this succinct. And, again, that requires that they stick to expressing page traversal and manipulation in the local biz-specific language of the web app’s business domain. So, finally, let’s see the under-the-covers page self-verification mechanics.

PageObjects as I Use Them

The selenium-rc-patterns sample Java codebase that illustrates all of the patterns in this series of posts talks to a localhost copy of a Rails project called Fat Free CRM. If you want to learn a bit about this app, you can play with a hosted sample version here (you have to sign up for an account first, which is easy).

This CRM system has discrete web pages like a Dashboard (home page, essentially), and pages for Tasks, Leads, Campaigns, Contacts, Accounts, and Opportunities. For each of these actual, production pages, my selenium-rc-patterns Eclipse project contains a matching PageObject with methods and fields that provide access to the services on that page.

Above, when you ask an instance of a LoginPage to login(username, password), then the TextFields and PageLink type know how to return us a PageObject that should then be cast to a DashboardPage and returned. (More on that in another post.)

Selenium RC developers have been using a PageObject pattern for awhile. The pattern dates back, at least, to the original HTMLUnit, whose API rather strongly encourages you to represent your pages under test as extensions of what it calls an HtmlPage. I don’t use Se 2 yet (might someday, might not), so I don’t use its PageFactory pattern.

Instead, I use my own PageObjects that extend a BasePage (or BasePane), partly so that I can control how and when PageObjects are instantiated, and verify automatically and ambiently on instantiation that Selenium is indeed on that page. Again, that’s much of what keeps the above test code so simple: I’m not explicitly waiting for a page to load, and I’m not asserting all over the place that I’ve successfully arrived on a given page.

So, my LoginPage class has a login() method that accepts a username and password (we’ll cover the return types of these fields and methods in another post that describes how page-flow works):

package bizdomain.pages;
 
import util.BasePane;
import util.elements.PageLink;
import util.elements.TextField;
 
public class LoginPage extends BasePane {
  public static final String PAGE_IS_LOADED_CSS
                                           = "input[id=authentication_username]";
  private TextField userNameField;
  private TextField passwordField;
  private PageLink loginButton;
 
  public LoginPage() {
    super();
    userNameField = new TextField("input[id=authentication_username]");
    passwordField = new TextField("input[id=authentication_password]");
    loginButton = new PageLink("input[id=authentication_submit]",
                                                                                DashBoard.class);
  }
 
  @Override
  public String getPageLoadedCssLocator() {
    return PAGE_IS_LOADED_CSS;
  }
 
  public DashBoard login(String userName, String password) {
    userNameField.enter(userName);
    passwordField.enter(password);
    return (DashBoard) loginButton.clickToNewPage();
 
  }
}

Verifying That the Production Page has Been Loaded

The real point of my flavor of PageObject pattern is the self-verifying bit. Let’s dive into that — not later, but now.

Note that each of these PageObjects extends BasePane, and has a PAGE_IS_LOADED_CSS constant:

package bizdomain.pages;
public class LoginPage extends BasePane {
  public static final String PAGE_IS_LOADED_CSS = "input[id=authentication_username]";

In the LoginPage constructor way above up there, you can see we first explicitly call super() on BasePane. Here is BasePane:

public abstract class BasePane {
  public BasePane() {
    waitUntilLoaded();
  }
 
  public final boolean isLoaded() {
    return BrowserDriver.isElementPresent(getPageLoadedCssLocator());
  }
 
  public final boolean isVisible() {
    return BrowserDriver.isElementVisible(getPageLoadedCssLocator());
  }
 
  public final void waitUntilLoaded()  {
    BrowserDriver.waitForElementVisible(getPageLoadedCssLocator());
  }
 
  public abstract String getPageLoadedCssLocator();
}

You can see that this constructor calls waitUntilLoaded(), which makes a static call to a method on our BrowserDriver (the Facade / Decorator that handles all of the actual Selenium calls) in order to loop until our LoginPage (in this case) actually is loaded. The argument supplied is the result of calling getPageLoadedCssLocator().

But wait! That’s an abstract method! So Yes, we have something very like a template method pattern here in waitUntilLoaded(): the concrete implementation of getPageLoadedCssLocator() on LoginPage returns that PAGE_IS_LOADED_CSS constant String.

Deep, deep under the covers, the BrowserDriver.waitForElementIsVisible() method looks like this:

public static void waitForElementVisible(String cssLocator)  {
  for (int second = 0;; second++) {
    if (second >=
 Integer.valueOf(BrowserDriver.STANDARD_DHTML_LOAD_WAIT_  TIME) / MS_PER_SECOND)
 
  fail("Timeout waiting for element " + cssLocator + " to become visible.");
    if (isElementVisible(cssLocator)) break;
    sleepForASecond();
  }
}
 
public static boolean isElementVisible(String cssLocator) {
  injectJqueryIfAbsent();
  return executeJavascript(JqueryCodeFactory.getVisibilityCode(cssLocator)).equals(TRUE);
}

We’ll discuss the BrowerDriver class at length elsewhere. And we’ll also discuss the injectJqueryIfAbsent() method, which we hope an upcoming Selenium RC release will obviate.

The upshot of the waitForElementVisible() method and the LoginPage and BasePane code above is that the LoginPage object will not successfully finish instantiating until Selenium can successfully verify that a unique element on that page has been loaded. In other words, once a LoginPage instance is loaded in memory, we actually are on the LoginPage, by definition, as long as that CSS element selector syntax is correct. Voila, automatic, ambient page flow assertion.

This is much of how we get our test methods so succinct. The PageObjects take care of verifying for us, at instantiation, that we have safely arrived on their corresponding production app web pages.

Caveat Lector: there is a flaw in my code I have yet to squeeze out:  duplication between a BasePane, used above, which I usually use for dynamic changes in the HTML, and a BasePage, which presumes that a real HTTP Request/Response cycle has occurred. I will collapse those two together shortly.

Next: we’ll talk about reusable HTML Element objects, and how the linkish ones know, in my code, how to return the PageObject we wish to traverse to next.

There’s a mildly rollicking little discussion going on the in the Software Testing Club at the moment, in which Rob Lambert observes, “I’ve seen a couple of conversations recently where people are talking about red, green and yellow box testing.” Rob then asks “There’s the obvious black and white. How many more are there?” (For [...]

Let’s imagine your team is making a chat application. Nothing too fancy, just something simple in which you need to make sure multiple users get together, talk, and read each other.

Doesn’t sound that bad, huh? But how about coordinating those browsers? You need to have different sessions running at the same time in coordination and interacting with each other through your web app.

Now, when we think about parallelization and multiple browsers, Sauce OnDemand is, of course, our first answer, but here, there can be a small limitation. Sauce OnDemand has a special timeout that waits for new commands for no more than 90 seconds*. If it doesn’t get any new command, it will kill the job and assume something went wrong.

This effects us, as we need to get multiple browsers running in parallel and in coordination. As we request them one after another, depending on the amount of browsers and the load our service is put under, the first ones might timeout while waiting for the others to come.

To avoid this, the best solution is to write a multi-threaded script, in which browsers will send heartbeat commands while they are waiting for the rest. Multi-threading can be done with every programming language, but I decided to write my example in Python because I think it’s understandable for almost anyone, even if you’ve never seen Python before. The complete example is in github as a gist, and here is a brief description of what each interesting part is doing:

def get_browser_and_wait(browser, browser_num):
    print "starting browser %s" % browser_num
    browser.start()
    browser.open("/") # if we get here, OnDemand already gave us the browser
    browsers_waiting.append(browser)
    print "browser %s ready" % browser_num
    while len(browsers_waiting) < len(browsers):
        print "browser %s sending heartbeat while waiting" % browser_num
        browser.open("/")
        time.sleep(3)

get_browser_and_wait is the function that will take care of the ‘start the browser’ request in OnDemand and wait till it comes back. Once the server is ready, it will keep sending open commands as heartbeats and waiting until the other requested browsers are ready and waiting too.

thread_list = []
for i, browser in enumerate(browsers):
    t = Thread(target=get_browser_and_wait, args=[browser, i + 1])
    thread_list.append(t)
    t.start()

for t in thread_list:
    t.join()

This is the magic multithreading part, in which we iterate over every Selenium instance and call get_browser_and_wait with it. Once we send each browser to a thread, the main thread will continue and get to the t.join() part. By this method, the main thread will wait for every browser thread to complete in order to proceed with the rest of the code.

Again, the full example is up in github: http://gist.github.com/511658. If you’re interested in learning what you can use this kind of test for, stay tuned!

* This helps avoid running (and charging) tests that get disconnected or crash.