The 2012 San Francisco Selenium Meetups kicked off last week with a great talk by Denali Lumma (@denalilumma), who is the QA Lead at Okta.

Okta, an on-demand identity & access management service for cloud/SaaS applications, is notable for that fact that they manage to keep 1,000 Selenium Tests 100% blue – and they don’t use any manual testers. To get an inside look at how they accomplish this impressive feat, take a look at the video below.

For more info about the San Francisco Selenium Meetups and to join the group, visit our meetup.com page. And for help with starting a Selenium Meetup in your own city, check out my blog post on the subject :-)

Over the last year or so, interest in and development for Node.js has accelerated dramatically. There is no shortage of excitement and promise in this burgeoning community and for many of the same reasons cited by others, about six months ago we decided it made sense to dive in ourselves.

Coming from a committed testing background on multiple web frameworks, including Rails, we are strong adherents to BDD. Naturally the first thing we did when spinning up on the Node stack was look around for good BDD acceptance testing tools. Not finding anything that fit for us, we went ahead and rolled our own. In this post, we’ll introduce GivenWhenThen, a DSL and runner that allows anyone to easily construct acceptance tests with BDD semantics in straightforward, sentence-like statements and then run them against Sauce Labs.

Writing a Story

Although you can also directly access the Selenium RemoteWebDriver that is under the covers, GivenWhenThen is primarily intended to provide a DSL for writing executable stories in the Dan North format. To write a story, begin with a BDD description in a xxx_test.coffee file:

 

story 'Executing a Google search',
  """
  As a human
  I want to perform a search
  So that I can access the world's information
  """, ->

 

And then add one or more scenarios:

scenario "Search for info about Node.js", (browser) ->
  browser
    .given "I am on the homepage", ->
      browser.step(steps.visitHomepage)
    .when "I enter search terms", ->
      browser.typeInElement('q', 'nodejs', using:'name')
    .and "submit the search", ->
      browser.clickElement('btnG', using:'name')
    .then "I see search results", ->
      browser.assertTextPresent('results')
    .and "the results contain information about nodejs", ->
      browser
        .assertTextPresent('node.js')
        .assertTextPresent('nodejs.org')

 

Each scenario has “given“, “when“, and “then” steps.

• Given: Set up the initial conditions for the scenario.
• When: Take the action the scenario is testing.
• Then: Assert the conditions expected after taking the tested action.

Each step contains one or more chained calls to WebDriver commands in the form of browser.someCommand.

Each step (given, when, then) can have an arbitrary number of and steps following it (see above example).

Steps Often there are steps that are repeated throughout many scenarios, for example “visit homepage” or “sign in”. This kind of functionality can be defined in steps and referred to in scenarios via browser.step():

 

scenario "Search for info about Node.js", (browser) ->
  browser
    .given "I am on the homepage", ->
      browser.step(steps.visitHomepage)

 

Steps are defined in *_steps.coffee files. Multiple steps per file can be defined as follows:

 

steps.visitHomepage = (browser) -> browser.get 'http://www.google.com'

 

Multiple steps files can be created to organize your steps in a manageable way.

Configuration Configuration is simple and contains:

• Overall story and Sauce Labs configuration.
• Browser / OS definitions. Stories will be run against each browser/os configuration defined.

For example:

config.credentials =
  'username':   'sauce_labs_username'
  'access-key': 'sauce_labs_access_key'

config.settings =
  'max-duration': '180'

config.browsers = [
  {
    'platform':     'VISTA'
    'browserName':  'firefox'
    'version':      '7'
  }
  {
    'platform':     'LINUX'
    'browserName':  'firefox'
    'version':      '7'
  }
]

 

Subtitles In Test Videos One of our favorite features of Sauce Labs is test videos. It is incredibly useful to be able to view a video of a failed test and quickly understand what went wrong. However, one issue that has been raised about the videos is that they could sometimes benefit from some context about what is happening on screen.Well, it just so happens that this context is conveniently codified in the steps of BDD scenarios! So, by flipping a switch in GivenWhenThen, you can add BDD subtitles to your Sauce test videos. When the subtitle flag is passed to the GivenWhenThen runner, subtitle divs will be inserted into the test browser, which will show up in the Sauce Labs test videos and illustrate which of the BDD steps is currently being executed:

 

 

Design Background

In addition to being built on top of Sauce Labs and Selenium 2, GivenWhenThen is heavily influenced by the BDD movement.

While it is also clearly influenced by Cucumber, aficionados will surely notice differences. Most obviously, in GivenWhenThen there is no prose-to-code translation step. Cucumber uses the Gherkin language, which parses the prose into features, scenarios, and steps.

In GivenWhenThen, the DSL is embedded directly in the host language. We do realize there is real benefit to the full Cucumber vision – including Gherkin – but we took a leaner approach to getting the BDD structure without completely porting Cucumber.  To us, the core power of the BDD approach comes from the conceptual framework Dan North developed. Just stating the story and scenario in that formal way and using the “given, when, then” flow gives powerful structure and coherence to your specifications.

Conclusion GivenWhenThen fills a need we have by providing a Selenium 2 driver and Sauce Labs integration for Node.js, and adding BDD structure to our tests, while staying light and keeping us close to the host language. We hope you find it useful as well!

Integration for Selenium testing running with your continuous integration builds with Sauce OnDemand and Sauce Connect is made easy through the plugins we’ve developed for Bamboo and Jenkins.

These plugins allow you to specify the browser to be used by your tests and launch Sauce Connect prior to the running of your tests.

They set several environment variables that include the settings for your build, and your tests will then need to reference these environment variables as part of the Selenium setup.

However, explicitly referencing these variables can make your test code cluttered and it can be cumbersome to get the tests structured so that they work against either a local browser or Sauce OnDemand.

Rather than updating your code to explicitly reference the environment variables the plugins set, you can instead use the Selenium Client Factory library.

This library will read the environment variables set by the CI plugin, and instantiate a Selenium/WebDriver instance that will run your tests against Sauce OnDemand.

Selenium Client Factory is a Java library, which can be incorporated into your tests by referencing the jar file or if you’re using Maven, by including the following dependency:

<dependency>
    <groupId>com.saucelabs.selenium</groupId>
    <artifactId>selenium-client-factory</artifactId>
    <version>2.1</version>
    <scope>test</scope>
</dependency>

While this is great for Java applications, Selenium and the CI tools themselves support a number of languages.

One of our users has developed a Python implementation of the Selenium Client Factory code.

The factory object reads environments variables setup by the Bamboo plugin and creates a remote Sauce OnDemand session accordingly. If the Sauce-specific environment variables aren’t set, the library will create a local Selenium configuration.

In addition, the factory will output the Sauce OnDemand Session Id to the build output, and the Sauce CI plugins will parse this id and associate the Sauce Job with the CI build.

- The Selenium Client Factory library makes integrating your tests with Sauce OnDemand and your continuous integration build tool easy – you can configure your local tests to run against a local browser, and use your CI environment to access Sauce OnDemand, all via properties and without having to change any code.

To use Selenium Client Factory under Python, just instantiate Selenium via:

from SeleniumFactory import *

#For selenium 2 webDriver:
webDriver = SeleniumFactory().createWebDriver()

#For selenium 1 RC:
browser = SeleniumFactory().create()

The code for the Java and Python implementations is open source, so if you’d like to implement the Selenium Client Factory in your language of choice, either take a look at the code and have a go, or let us know and we can take a look at it.

As you all know, the Selenium development team keeps moving at an incredible pace and we do our best to keep up.

This time, we have the pleasure to announce that OnDemand now supports Selenium 2.16.1! Selenium 2.16.1 is the release I personally have been waiting for as it considerably improves native interactions, and fixes some pretty gnarly issues with scrolling and clicks. Here’s the changelog for more information.

Along with Selenium 2.16.1, we included ChromeDriver 18.0.995.0 (changelog). We’re also releasing Firefox 8 and 9 support for both Selenium 1 and Selenium 2 users. To use these browsers, you need to make sure you specify 2.16.1 as the Selenium version to use, as our current default (2.6.0) won’t support them.

You can start using this new version right now by adding the following Desired Capabilities/JSON key-value:

"selenium-version": "2.16.1"

During the next few weeks we’ll be testing this version and depending on how it does, we’ll consider moving our default version to it. If you see any issues after moving your tests to this new release, we definitely want to hear about it.

For more information about the current Selenium version that is used by Sauce, how to use other Selenium versions, and the selenium-version flag, have a look at our docs in the Sauce OnDemand additional configuration section.

Abstract:

The key benefits of Behavior-Driven-Development (BDD) practices are communication enhancement and customer satisfaction. You can read more on that by Dan North and Gojko Adzic. Perhaps the biggest practical challenge that stands in the way of reaping those benefits is the burden of provisioning, installation and maintenance of requisite complex and fussy infrastructure. The recent availability of CI servers such as Jenkins & cloud-based testing services such as Sauce Labs  carries the potential to remove that barrier. This post discusses and shows how to integrate Behat, an emerging BDD framework for PHP, with Jenkins and Sauce Labs.

What is Behat?

Behat is a BDD framework for PHP. There are some tools available for BDD like Cucumber for Ruby, SpecFlow for .NET and Lettuce for Python. Behat is the first BDD tool for PHP applications. Developers can also use the  PHPSpec framework to implement classes within the Behat projects. Behat is written in PHP by Konstantin Kudryashov. With Behat, you can write human readable stories, which are used as tests to run against your application. Behat can be used for API testing, functional testing and data-driven testing. Developers will do API testing and we will carry on with functional testing (web acceptance testing) with Behat.

Functional Testing with Behat and Mink

Behat is used for acceptance testing (any tests) by executing a Gherkin scenario. Developers can implement integrated classes. Testers start thinking of more workflow level and technical level steps (actions), which turns scenarios to features. Once a tester starts to think of Web Acceptance Testing (functional testing) with browser interaction, another tool called “Mink” comes into the picture.

Mink is used for browser emulation (functional testing) where browser interactions takes place. As of now, these are the following Selenium drivers available for browser emulation.

• Selenium provides a bridge for Selenium RC  (Selenium 1).
• Webdriver provides a bridge for Selenium 2. (Facebook webdriver) for PHP.

Behat In Action:

You must first have pear installed in order to proceed with Behat installation. Now we will run these commands from your terminal window:

$ sudo pear channel-discover pear.behat.org
$ sudo pear channel-discover pear.symfony.com
$ sudo pear install behat/gherkin-beta
$ sudo pear install behat/behat-beta

 

Test your installation by running this command:

$ behat --version
Behat version 2.2.0

 

Now let’s install Mink and run the following commands from the terminal window:

$ pear channel-discover pear.symfony.com
$ pear channel-discover pear.behat.org
$ pear install behat/mink

 

Mink is ready to use. We have to include “mink/autoload.php” in your “bootstrap.php” file as shown below:

require_once 'mink/autoload.php';

Start Your Project

Navigate to the project root directory and initialize Behat by running these commands:

$ cd /path/to/my/project

$ ls
application

$ behat --init
+d features - place your *.feature files here
+d features/bootstrap - place bootstrap scripts and static files here
+f features/bootstrap/FeatureContext.php - place your feature related code here

$ ls
application	features	

$cd features

$ ls
bootstrap

$cd bootstrap

$ls
FeatureContext.php

 

This will create a “features” directory and “bootstrap/FeatureContext.php” for you. Now we will jump directly to the project created with a feature file.

You can use NetBeans with installed Cucumber plugin for Gherkin syntax highlighting. Project structure will look like this:

Directory Structure

Behat has already created a “features” directory and “features/bootstrap” directory with “FeatureContext.php” in it.

bootstrap.php

We can use this create file to define some constants and some third-party libraries that need to be included in class files. This file should look like this:

<?php
date_default_timezone_set('Europe/London');
 require_once 'mink/autoload.php';
 require_once 'PHPUnit/Autoload.php';
 require_once 'PHPUnit/Framework/Assert/Functions.php';
 require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
 require_once 'PHPUnit/Extensions/SeleniumTestCase/SauceOnDemandTestCase.php';

behat.yml

This file is a default config file that Behat uses to execute features. An example of behat.yml is shown below:

default:
    context:
        parameters:
            javascript_session: selenium
            base_url: http://en.wikipedia.org/wiki/Main_Page
            browser: firefox
            show_cmd: open %s

 

You can change the drivers by changing the “javascript_session” parameter. It can be “selenium” or “webdriver”.

sauce.yml

This file is used for running features on Sauce Labs. The code for this file is explained in the “Behat and Sauce Labs” section below.

build.xml

This file is used for running features with ANT. We can use this ANT file to plug into Jenkins. Simple ANT file should look like this:

<project name="behat" default= "behat" basedir="">
   <delete dir="${basedir}/report" />
   <mkdir dir="${basedir}/report"/>
   <target name="behat">
    <exec dir="${basedir}" executable="behat" failonerror="true">
     <arg line="-f junit --out ${basedir}/report"/>
    </exec>
   </target>
   <target name="create-test-report"  
          description="Generate reports for executed JUnit tests.">  
       <junitreport todir="./report">  
          <fileset dir="${basedir}/report">  
             <include name="TEST-*.xml"/>  
          </fileset>  
          <report format="frames" todir="./report/html"/>  
       </junitreport>  
   </target>  
</project>

report

This directory is used to store *.xml reports generated by Behat’s “junit” formatter. We can use this report to plug into Jenkins.

Start your Engine

Remember, you have to download the latest version of Selenium server. Now navigate to the directory where you saved Selenium server .jar file. You have to launch it using command shown below:

java -jar selenium-server-standalone-2.15.0.jar

 

Behat & Sauce Labs

Sauce Labs is a cloud testing service that allows Selenium tests to run in the cloud. Sauce Labs allocates machines and browsers for your tests, captures screenshots for every step and records video of all jobs (tests). You don’t need to set up separate machines to run tests. Sauce Labs helps us to write tests without complex infrastructure.

In order to integrate Behat with Sauce Labs, you need to have an account with Sauce Labs. You’ll need your “Username” and “API Key” to plug them into a config file.

Behat executes features with “behat.yml” file by default, but we can run features with any other configuration file. We can create another configuration file like “sauce.yml” to run features on Sauce Labs. Example “sauce.yml” should look like this:

default:
  context:
    parameters:
      default_session:    goutte
      javascript_session: selenium
      base_url:           http://en.wikipedia.org/wiki/Main_Page
      browser:            firefox
      selenium:
        host: ondemand.saucelabs.com
        port: 80
        browser: >
          {
            "username":         "your username",
            "access-key":       "your API key",
            "browser":          "firefox",
            "browser-version":  "7",
            "os":               "Windows 2003",
            "name":             "Testing Selenium with Behat"
          }

 

We will use “sauce.yml” as a config file to run features on Sauce Labs. If you wish to run all features from the “features” directory on Sauce labs, you can use this command:

behat -c sauce.yml

 

When to implement step definitions?

• If you can speak fluent Gherkin, then you don’t need to write code. Behat/Mink will understand Gherkin and run your features without suggesting step definitions.
• If the features are written by someone else, you can take full advantage of Mink’s APIs in order to implement step definitions suggested by Behat/Mink.
• It’s very important to write good Gherkin to write minimum code.
• It’s very easy to access Mink API’s by writing simple code.

Now we will see how you “click” particular elements on page. You need to use Xpath as a locator for that element. Mink will suggest some step definitions and you’ll need to complete it like this:

/**
* @Given /^I click Something$/
*/
public function iClickSomething()
{
$this->getMink()->getSession()->getDriver()->click("//Xpath");
}

 

Example  : Feature wikiSearch

We will write a simple feature to add a product into shopping cart. The feature will look like this:

 Feature: wikiSearch
  In order to search information on wiki
  As a Wiki user
  I want to get sensible results from site

 @javascript
  Scenario Outline: Search Keywords on Google
    Given I am on "/"
    And I fill in searchBox with "<input>"
    When I press search button
    Then I should see "<output>" 

    Examples:
      | input       | output         |                                   
      | London      | lʌndən/        |
      | NewYork     | nɪu ˈjɔək      |
      | Sydney      | sɪdni/         |
      | Mumbai      | मुंबई            |
      | Bejing      | 北京            |
      | Tokyo       | 東京            |
      | Lahore      | لاہور            |
      | Paris       | paʁi            |

 

Feature Explained

This feature file is written in “Gherkin” DSL (Domain Specific Language). The feature file mentioned above is a good example of data-driven testing. This feature will execute our scenario for 8 different data sets mentioned in the example section. This feature will have the following steps:

1. User enters “London” in search box.
2. User will check if that page has city name in their local language as described in output.
3. This test will run for 8 different cities as shown in “examples” section of feature.

This feature is also a good example of testing internationalization as it consists of test data (examples) in different languages.

Now we will run this feature using command:

behat --name wikiSearch

 

Remember, we are running it locally for now using default config file “behat.yml” with Selenium driver. After executing the above command, we will get some step definitions suggested by Behat/Mink

Feature: wikiSearch
  In order to search information on wiki
  As a Wiki user
  I want to get sensible results from site

  @javascript
  Scenario Outline: Search Keywords on Wiki # features/wikiSearch.feature:8
    Given I am on "/"                         # FeatureContext::visit()
    And I fill in searchBox with "<input>"
    When I press search button
    Then I should see "<output>"              # FeatureContext::assertPageContainsText()

    Examples:
      | input   | output    |
      | London  | lʌndən/   |
        Undefined step "I fill in searchBox with "London""
        Undefined step "I press search button"
      | NewYork | nɪu ˈjɔək |
        Undefined step "I fill in searchBox with "NewYork""
        Undefined step "I press search button"
      | Sydney  | sɪdni/    |
        Undefined step "I fill in searchBox with "Sydney""
        Undefined step "I press search button"
      | Mumbai  | मुंबई     |
        Undefined step "I fill in searchBox with "Mumbai""
        Undefined step "I press search button"
      | Bejing  | 北京        |
        Undefined step "I fill in searchBox with "Bejing""
        Undefined step "I press search button"
      | Tokyo   | 東京        |
        Undefined step "I fill in searchBox with "Tokyo""
        Undefined step "I press search button"
      | Lahore  | لاہور     |
        Undefined step "I fill in searchBox with "Lahore""
        Undefined step "I press search button"
      | Paris   | paʁi      |
        Undefined step "I fill in searchBox with "Paris""
        Undefined step "I press search button"

8 scenarios (8 undefined)
32 steps (8 passed, 8 skipped, 16 undefined)
0m15.771s

You can implement step definitions for undefined steps with these snippets:

    /**
     * @Given /^I fill in searchBox with "([^"]*)"$/
     */
    public function iFillInSearchboxWith($argument1)
    {
        throw new PendingException();
    }

    /**
     * @When /^I press search button$/
     */
    public function iPressSearchButton()
    {
        throw new PendingException();
    }

 

As you can see from above, Behat/Mink have suggested some step definitions for undefined steps. We can implement these step definitions using Mink in “bootstrap/FeatureContext.php” file. We can implement the first step definition “iFillInSearchboxWith($argument1)” like this:

/**
     * @Given /^I fill in searchBox with "([^"]*)"$/
     */
    /*
    public function iFillInSearchboxWith($input)
    {
        $this->fillField("searchInput",$input);
    }

 

Now, when we run “behat” command again:

behat --name wikiSearch

 

You can see now sixteen step passed. The terminal window output should be like this:

Feature: wikiSearch
  In order to search information on wiki
  As a Wiki user
  I want to get sensible results from site

  @javascript
  Scenario Outline: Search Keywords on wiki # features/wikiSearch.feature:7
    Given I am on "/"                       # FeatureContext::visit()
    And I fill in searchBox with "<input>"  # FeatureContext::iFillInSearchboxWith()
    When I press search button
    Then I should see "<output>"            # FeatureContext::assertPageContainsText()

    Examples:
      | input   | output    |
      | London  | lʌndən/   |
        Undefined step "I press search button"
      | NewYork | nɪu ˈjɔək |
        Undefined step "I press search button"
      | Sydney  | sɪdni/    |
        Undefined step "I press search button"
      | Mumbai  | मुंबई     |
        Undefined step "I press search button"
      | Bejing  | 北京        |
        Undefined step "I press search button"
      | Tokyo   | 東京        |
        Undefined step "I press search button"
      | Lahore  | لاہور     |
        Undefined step "I press search button"
      | Paris   | paʁi      |
        Undefined step "I press search button"

8 scenarios (8 undefined)
32 steps (16 passed, 8 skipped, 8 undefined)
0m25.568s

You can implement step definitions for undefined steps with these snippets:

    /**
     * @When /^I press search button$/
     */
    public function iPressSearchButton()
    {
        throw new PendingException();
    }

We have to repeat this process until all steps get “passed”. We can implement these steps by adding some code in “bootstrap/FeatureContext.php” file. The code will look like this:

<?php
use Behat\Behat\Context\ClosuredContextInterface,
    Behat\Behat\Context\TranslatedContextInterface,
    Behat\Behat\Context\BehatContext,
    Behat\Behat\Exception\PendingException;
use Behat\Gherkin\Node\PyStringNode,
    Behat\Gherkin\Node\TableNode;
use Behat\Mink\Behat\Context\MinkContext;
use Behat\Mink\Session;
use Behat\Mink\Driver\DriverInterface;
require_once 'bootstrap.php';

/**
 * Features context.
 */
class FeatureContext extends MinkContext
{
   /**
     * @Given /^I fill in searchBox with "([^"]*)"$/
     */
    public function iFillInSearchboxWith($input)
    {
        $this->fillField("searchInput",$input);
    }

    /**
     * @When /^I press search button$/
     */

    public function iPressSearchButton()
    {
        $this->getMink()->getSession()->getDriver()->click("//*[@id='searchButton']");
         $this->getMink()->getSession()->wait("3000");
    }

 

After implementation, we have to run “behat” command again. You will see the feature running in the browser for all the inputs mentioned in the feature file. You will have to wait until the browser finishes all the examples. You will see the terminal output look like this:

 Feature: wikiSearch
  In order to search information on wiki
  As a Wiki user
  I want to get sensible results from site

  @javascript
  Scenario Outline: Search Keywords on wiki # features/wikiSearch.feature:8
    Given I am on "/"                       # FeatureContext::visit()
    And I fill in searchBox with "<input>"  # FeatureContext::iFillInSearchboxWith()
    When I press search button              # FeatureContext::iPressSearchButton()
    Then I should see "<output>"            # FeatureContext::assertPageContainsText()

    Examples:
      | input   | output    |
      | London  | lʌndən/   |
      | NewYork | nɪu ˈjɔək |
      | Sydney  | sɪdni/    |
      | Mumbai  | मुंबई       |
      | Bejing  | 北京      |
      | Tokyo   | 東京       |
      | Lahore  | لاہور       |
      | Paris   | paʁi      |

8 scenarios (8 passed)
32 steps (32 passed)
0m42.794s

 

We managed to get all our scenario/steps “passed” so now it’s time to login into your Sauce Labs account to see this scenario running on Sauce Labs.

Now we have to use “sauce.yml” config file. We will run below mentioned command from terminal and we can see output on Sauce Labs as shown below:

 behat -c sauce.yml

 

Screenshots for the feature running on Sauce Labs:

                                                Watch video and Setps of this job on Sauce Labs

You can see detail steps, screenshots and video of this job on Sauce Labs
See terminal output as shown below:

Building Features with Jenkins

Continuous Integration

Continuous Integration (CI) is one of the best practices in agile projects to detect bugs early. Each integration is verified by an automated build to detect integration errors as quickly as possible. Now we will see how we can integrate Jenkins to our behat project.

Don’t forget to start your engine before building a project in Jenkins. You have to start Selenium/WebDriver before starting Jenkins.

 

There is “/report” directory to save JUnit reports in xml format generated by Behat. You can generate reports using this command:

 

behat -f junit --out ~/pathto/report

 

To continue, you need Jenkins installed on your machine. Now start Jenkins by executing *.war file from terminal on port 8080. You also need to visit “http://localhost:8080″ to see Jenkins GUI.

$  java -jar jenkins.war

• Visit http://localhost:8080/ and you should see Jenkins Dashboard.
• Create “New Job” for behat project.
• Specify your SCM to create Jenkin’s workspace on your system.
• You can specify build file as per your project structure. In here I have configured in “Acne” project by specifying build file path and test reports like this.

• Save your configuration. Go back to project and click”Build Now”, you will see:

• You can see features running in browser.
• Sit back and enjoy your features running on Jenkins till it finishes job.
• Few minutes later, you will see your build is “Green”:

• Watch your test results:

Behat, Sauce Labs and Jenkins

You can run all your features on Sauce Labs just by updating ANT file like:

<project name="behat" default= "behat" basedir="">
   <delete dir="${basedir}/report" />
   <mkdir dir="${basedir}/report"/>
   <target name="behat">
    <exec dir="${basedir}" executable="behat" failonerror="true">
     <arg line="-c sauce.yml -f junit --out ${basedir}/report"/>
    </exec>
   </target>
   <target name="create-test-report"  
          description="Generate reports for executed JUnit tests.">  
       <junitreport todir="./report">  
          <fileset dir="${basedir}/report">  
             <include name="TEST-*.xml"/>  
          </fileset>  
          <report format="frames" todir="./report/html"/>  
       </junitreport>  
   </target>  
</project>

 

Alternatively, you can configure Sauce On demand tunnel into your Jenkins.

Conclusion:

We can write web acceptance tests with Behat and Mink combination. We can plug them into Sauce Labs with config file (“sauce.yml”) and run them on a CI server. Now you can implement BDD practices for PHP applications by utilizing the benefits of Bahat and Sauce Labs.

Demo

To see a video demonstration of this blog post, visit on Vimeo and YouTube.  To read a little more about me, click here.

As we tie the bow on another year, on the behalf of everyone here at Sauce Labs, allow me to thank each of you for participating in the Sauce story this past year—and take a moment to share some of the highlights.

It has been a thrilling and humbling 12 months for all of us here at Sauce. Our user base grew dramatically, we expanded our product offering to include manual testing in the cloud, and continued to deliver on our vision for making software testing as fast and easy as possible.

On the humbled side, we learned we need to do better at staying ahead of fast-rising demand for our service. We know our users place great importance on and expect scalability and reliability from us. Being surprised by unexpectedly strong growth does not relieve our duty to deliver on that. We’re taking steps to ensure we meet that expectation going forward.

In 2011 we saw rapid change and broke many records. Just to share with you a few of our favorites, we:

Record usage levels:

• Passed the 10,000,000 completed tests mark on Sauce OnDemand
• Watched as over 2,000 different customers used our service every month
• Experienced a 21% month-on-month growth of cloud revenue since Jan 2011
• Counted our 300th paying customer

Product expansion

• Introduced Sauce Cloud, which enables us to deliver 500 virtual machines in under 50 seconds (currently limited to 150 per user)
• Added Selenium 2 support
• Added Sauce Scout to our product offering for visual / manual testing

Market and community context:

• Began hosting the integration tests for the Selenium 2 project as a service to the Selenium community
• Watched Selenium eclipse HP’s QTP on job posting aggregation site Indeed.com to become the number one requisite job experience / skill for on-line posted automated QA jobs (2700+ vs ~2500 as of this writing)
• Lead the community effort to hold the first-ever Selenium Conference
• Organized 16 Selenium meetups spanning CA, MA and NY, and inspired the creation of five new Selenium meetup groups world-wide.

The dramatic growth we experienced this year—and the strength of the user communication received when we creaked under load—have underscored for us the importance of the Sauce Labs mission. We love what we do and so appreciate knowing how much it matters to you, our customers. So again, we want to extend our heartfelt thanks for supporting us, for giving our mission great meaning.

From all of us here at Sauce, we wish you a healthy, happy and prosperous 2012.

Happy New Year!

- John Dunham
CEO and Co-Founder

We’ve been working to implement a new timeout that will prevent Selenium failures from consuming many of our customers’ minutes. This new timeout will detect issues in Selenium’s end (e.g. browser or Selenium crashing before answering back) and will stop and fail the job when that happens.

(Notice: this timeout is only triggered when Selenium doesn’t send a reply back to your end after a command has been requested for a certain amount of time. Without this feature, we’ll wait for 30 minutes, or until the full job exceeds the maximum duration, before stopping the whole job).

While this feature has not yet been made publicly available, you can start using it now by setting the command-timeout capability to any value you believe reasonable:
https://saucelabs.com/docs/ondemand/additional-config#command-timeout

Starting next week, and as our docs state, we’ll be setting this to a default value of 300 seconds for all jobs that don’t explicitly specify a timeout limit . Once this happens, you may find that some of your tests get caught and fail. We believe the default value – five minutes – is long enough so that in those special cases, you’ll be able to revisit the cause of such a long command and either fix the problem or extend the timeout.

You can read more about timeouts and changing its value in our documentation here:
https://saucelabs.com/docs/ondemand/additional-config#command-timeout

Please let us know if you have any questions or concerns about this.