Let’s imagine your team is making a chat application. Nothing too fancy, just something simple in which you need to make sure multiple users get together, talk, and read each other.

Doesn’t sound that bad, huh? But how about coordinating those browsers? You need to have different sessions running at the same time in coordination and interacting with each other through your web app.

Now, when we think about parallelization and multiple browsers, Sauce OnDemand is, of course, our first answer, but here, there can be a small limitation. Sauce OnDemand has a special timeout that waits for new commands for no more than 90 seconds*. If it doesn’t get any new command, it will kill the job and assume something went wrong.

This effects us, as we need to get multiple browsers running in parallel and in coordination. As we request them one after another, depending on the amount of browsers and the load our service is put under, the first ones might timeout while waiting for the others to come.

To avoid this, the best solution is to write a multi-threaded script, in which browsers will send heartbeat commands while they are waiting for the rest. Multi-threading can be done with every programming language, but I decided to write my example in Python because I think it’s understandable for almost anyone, even if you’ve never seen Python before. The complete example is in github as a gist, and here is a brief description of what each interesting part is doing:

def get_browser_and_wait(browser, browser_num):
    print "starting browser %s" % browser_num
    browser.start()
    browser.open("/") # if we get here, OnDemand already gave us the browser
    browsers_waiting.append(browser)
    print "browser %s ready" % browser_num
    while len(browsers_waiting) < len(browsers):
        print "browser %s sending heartbeat while waiting" % browser_num
        browser.open("/")
        time.sleep(3)

get_browser_and_wait is the function that will take care of the ‘start the browser’ request in OnDemand and wait till it comes back. Once the server is ready, it will keep sending open commands as heartbeats and waiting until the other requested browsers are ready and waiting too.

thread_list = []
for i, browser in enumerate(browsers):
    t = Thread(target=get_browser_and_wait, args=[browser, i + 1])
    thread_list.append(t)
    t.start()

for t in thread_list:
    t.join()

This is the magic multithreading part, in which we iterate over every Selenium instance and call get_browser_and_wait with it. Once we send each browser to a thread, the main thread will continue and get to the t.join() part. By this method, the main thread will wait for every browser thread to complete in order to proceed with the rest of the code.

Again, the full example is up in github: http://gist.github.com/511658. If you’re interested in learning what you can use this kind of test for, stay tuned!

* This helps avoid running (and charging) tests that get disconnected or crash.

Watch Rob Fan, founder and CTO of Sharethrough, talk about how instrumental Sauce OnDemand has been in solving many of their testing woes. Sharethrough, a popular video distribution network, has been a Sauce Labs customer since the beginning, and we’re just thrilled to be part of their testing process!

As part of our featured webinar series, Kohsuke Kawaguchi, creator of Hudson and founder of InfraDNA, recently led an interactive demonstration on using Hudson with Selenium and Sauce OnDemand. In case you missed the live presentation on September 1, check out the recorded video to learn more about:

1. Setting up the Hudson plugin for Sauce OnDemand
2. How using Hudson for CI will drastically speed up your testing process
3. Tips for integrating Hudson with Selenium Grid
4. Tips for integrating Hudson with Sauce OnDemand

Sauce Labs is based on open source and we use a lot of it. We also do our best to contribute back to the community by working on Selenium, organizing #SFSE meetups, answering questions on the messages boards, and more.

Today we are happy to announce selenium.saucelabs.com, a searchable archive for the Selenium IRC channel. selenium.saucelabs.com is powered by “selbot,” and it hangs out at #selenium on freenode.

“selbot” has two main functions:

• It keep logs of the channel and makes them searchable
• It announces the latest Selenium bamboo build results in the channel

“selbot” is made of open source components, mainly Supybot and xapian. An open source project itself, you can find the code on github.

“selbot” is a humble start, and we have great plans for it in the future. For now, log in to #selenium on freenode and have a look. And while you’re there, answer a few questions – it’s a great way to give back to the community.

Rafael Ruffolo reports Aug 13, 2010 – San Francisco-based Sauce Labs Inc. has updated its cloud-based, cross-browser testing platform to allow Web developers to watch live video of their programming tests.

The new Sauce TV feature means application developers using the company’s Selenium software testing framework will be able to watch their off-site tests in real-time as opposed to waiting for reports or recorded video. The functionality aims to shorten the debugging cycle and get instant feedback to developers as they work.

Prior to the Sauce TV addition, developers had to wait for the cloud-based service to send them back recording of these tests.

Read more at IT World Canada

Today we officially unveiled Sauce TV, a great new feature that allows you to watch live video of your tests running on Sauce Labs‘ cloud machines.

Sauce TV provides a secure portal to your cross-browsing testing cycles, and drastically shortens debugging time by letting you catch issues as soon as they arise. Additionally, Sauce TV’s remote desktop player works right in the web browser, so there’s no need to install extra software.

Check out the screencast below to see Sauce TV in action. If you’ve given it a try, let us know how it’s working. We’d love to hear if you’re as excited about this feature as we are!

Now available for current customers, new real-time video capability speeds development by giving developers secure visibility into all their tests on Sauce OnDemand

SAN FRANCISCO, CA—August 12, 2010—Sauce Labs, the Selenium company, today rolled out Sauce TV to customers of its Sauce OnDemand cross-browser testing service. Designed to eliminate a critical ‘blind spot’ when testing applications in the cloud, Sauce TV gives developers a live, secure view of their tests running on Sauce Labs’ cloud machines. Developers can shorten their test debugging cycles with the confidence of knowing what’s going on at all times – and share testing videos with their teams.

With almost two million tests already served in the cloud, Sauce OnDemand is the premier cloud-based Selenium service. Created to meet customer demand, Sauce TV delivers full visibility into cross-browser testing cycles in a secure environment. Users have access to only their own tests and do not have to wait until the end to discover issues, allowing for faster debugging cycles. Sauce TV’s remote desktop player also works directly in the web browser without any extra software needed. Read more here

A few weeks ago, we announced the Sauce Flash/Flex Testing System, a new and innovative way to automatically test your apps built with Flash and Flex.

The first functional testing product to support test automation for hybrid web apps, Sauce Flash/Flex Testing System bundles Flex-Pilot’s object explorer API into Sauce IDE, and uses Sauce OnDemand to run tests across multiple browsers in the cloud.

Flex Pilot co-creator Adam Christian, a developer here at Sauce Labs, recently led an informative webinar demonstrating how to setup and use this complete testing system. Watch the video below to learn how easy it is to start testing your Flash and Flex apps today!

Using Sauce OnDemand with Hudson to Accelerate Your Test Cycles
Wednesday, September 1, 2010 at 11 am PST
Register Now

Join Kohsuke Kawaguchi as he leads this special webinar on the benefits of the new integration between Sauce OnDemand and Hudson, the most widely used CI server. Kohsuke is the creator of Hudson and the founder of InfraDNA, the Hudson Company.

Hudson supports distributed builds and multiple extensions, including those for Selenium. Now with the Hudson-Sauce OnDemand plugin, it’s even easier to check the builds of your Selenium tests, thanks to the automatic setup and teardown of SSH tunnels to Sauce OnDemand. Tune in to to learn about:

* Accelerating development, testing and release speed with Sauce OnDemand
* How to setup and use the Hudson-Sauce OnDemand plugin
* Current Selenium integrations and upcoming Hudson features

Who should attend: Anyone interested in improving their continuous integration system and testing their apps automatically in the cloud.

Earlier this week, the UK government posted their reply to a petition, stating that they will not upgrade their systems away from Internet Explorer 6 (IE6) anytime soon. The online petition, created in February 2010, encouraged all government departments to upgrade their browsers. More than 6,000 UK citizens signed the petition.

The petition explains its reasoning: “Most creative and software development companies are forced by government department clients to build websites for IE6 when most of the industry has moved on.”

We at Sauce Labs feel for the citizens of the UK, and we’d like to help. Unfortunately, in its reply, the UK government cites the high cost and time involved in testing an upgrade as the reason it will not upgrade IE6. The official government response states: “Upgrading these systems to IE8 can be a very large operation, taking weeks to test and roll out to all users. To test all the web applications currently used by HMG departments can take months at significant potential cost to the taxpayer. It is therefore more cost effective in many cases to continue to use IE6″

Sauce Labs is kindly offering Her Majesty’s Government some help in upgrading, and a free evaluation of Sauce OnDemand, which will save development time and cost the citizens of the UK far less.

With the Selenium cross browser testing tool, any UK government department can test their site in IE6 and IE8 in one shot. Since Selenium is free and open source software, licensing fees won’t be an issue for UK taxpayers. And with cloud computing services like Sauce OnDemand, there’s no need to invest in expensive and labor intensive test lab infrastructure. Instead, Sauce Labs provides a low-cost cross-browser testing service in the cloud.

We challenge the UK government to create an account at saucelabs.com now and try Selenium today. To back this up, we’re offering any UK government department a free trial account to try out the service to test their department websites in IE6 and IE8.

All in favor, petition your parliamentarian directly and Demand Your Sauce!

Mitchell Pronschinske reports on 2010/07/28 – Greg and Steven Hazel from Sauce Labs have recently built what they call, “An async programming framework with a blocking look-alike syntax”. This framework, named Monocle, is focused on being portable between event-driven I/O frameworks. Currently, Monocle supports the Twisted and Tornado frameworks.

For those who haven’t heard of Sauce Labs, they were co-founded by John Huggins, the creator of Selenium. Sauce Labs’ free and commercial tools build on top of the core Selenium testing framework.

The emergence of Monocle could indicate that Sauce Labs is taking a more focused look at event-driven code and its role in concurrent web performance. Event-driven code is efficient and intuitive, but sometimes procedures are are split up and code is expanded in a not-so-good way.

Read more at Dzone

David Rubinstein reports on July 21, 2010 —  To ease automation of cross-browser Web application testing, startup company Sauce Labs today is introducing Sauce for Flex and Flash. Now, the company said, testing of Flash, HTML and JavaScript applications can be done in one browser page.

Built on top of the open-source Flex Pilot library for automating Adobe Flex and Flash testing, Sauce for Flex and Flash can natively fire events into those types of Web applications for analysis, according to Adam Christian, JavaScript architect at Sauce Labs.

Read more at SD Times

The San Francisco Selenium Meetup Group gathered last week for a meetup at Digg’s headquarters, and WOW! What a fantastic night it was.

In the hopes of learning how members were specifically using Selenium, and to foster informal conversations around this, we decided to have “Whiteboard Night,” where we invited #SFSE members to volunteer to share their Selenium testing structures, philosophies, best practices, etc. The setup was similar to a poster session, with different “whiteboarders” presenting 5-10 minute sessions simultaneously and then engaging in organic discussions with the “minglers” who circled the room.

With 70+ plus attendees and 10 great whiteboarders from companies such as Mozilla, Pivotal Labs, and PB Works (to name a few), the conversations that resulted were highly thought-provoking. Most whiteboarders shared their existing test structures, while others proposed new ideas to get feedback from the group. Even Sauce Labs made an impromptu whiteboard visit, with Developer Miki Tebeka sketching out our testing system.

The responses afterwards were widely enthusiastic, with one newcomer to the group writing on the meetup page: “Awesome event! Great to talk to folks working ‘in the trenches.’”

With the success of Whiteboard Night, we’d love to have more in the future. We just need a big enough space to make it happen. If you think your company would be interested in hosting Whiteboard Night, please let us know. (To note: Digg’s digs were perfect. Plenty of open room, multiple whiteboards, and a huge fridge to store our beer :-) Thanks, guys!).

If you haven’t joined the #SFSE meetup group, please do so, as it’s the best way to stay in the loop. Next meetup is August 24 at LinkedIn – details coming soon.

Below are some photos from Whiteboard Night. For more, check out our new Sauce Labs Flickr Page. We’ll hope to see YOU at the next one!

The war between Adobe’s Flash and Apple’s support for HTML5 continues to heat up. The first set of battles between Apple and Adobe have reached a standstill for the moment. Adobe is pushing Flash for Android (which is awesome, on my new Google Nexus One, by the way). Meanwhile, the Apple-backed SproutCore project makes HTML5 a viable platform for developers to adopt today.

In spite of the Apple marketing machine, we’re betting that Flex and Flash still have their parts to play in the future of the web. After all, how can we just forget about the massive installed base? Flash is now on 98 percent of computers. More importantly, Flex and Flash have comprehensive tool sets that developers still need. Flash is the single browser plug-in that provides consistency for a chaotic and scattered web world. There are things that HTML and Javascript can’t do well, like webcam capture, media-streaming, and full-screen video. And let’s not forget — a huge segment of the web is gaming, which wouldn’t be possible (so far) without Flash. As YouTube recently noted: “While HTML5’s video support enables us to bring most of the content and features of YouTube to computers and other devices that don’t support Flash Player, it does not yet meet all of our needs.” Should we just abandon everything that we’ve known for HTML5 because it is new and exciting?

I’ll admit that even I have drunk the Apple Kool-Aid and see HTML5 for its role in the future of the web. Sauce specializes in cross-browser web app testing and HTML5 is easier to integrate with Selenium. However, the web is bigger than HTML and JavaScript. HTML5 is the future, but we live in the now. Though growing, HTML5 does not yet have the adoption that Flex and Flash have.

It’s this installed base that Sauce Labs is committing to with our announcement of the Sauce Flash-Flex Testing System.

The newest Selenium-based solution, the Sauce Flash-Flex Testing System provides testing of Flex applications on the Flash platform. Other tools in the marketplace can only test Flex applications, but the Sauce Flash-Flex Testing System covers the entire Flash platform. While Flex is a great tool for many developers, the two biggest markets on the web — gaming and advertising — are based on Flash, not Flex.

So, while we’re all excited for HTML5, rumors of Flash’s death are greatly exaggerated. Demand for Flash wont die. To a lot of people, Adobe may seem like yesterday’s news. But since not everyone has an iPad or an iPhone, the answer is still Flash and Flex, even if it’s not the new hotness.

To this strong base of Flex and Flash users: try out our Sauce Flash-Flex Testing System and give us your feedback.

At our last #SFSE meetup at StumbleUpon, Kohsuke Kawaguchi, the creator of Hudson, gave a fantastic presentation on Hudson and the integrations with Selenium. He also unveiled a new Hudson plugin for Sauce OnDemand, documented here in an earlier blog post.

Hudson is an easy-to-use, open-source Continuous Integration (CI) server that supports distributed builds and multiple extensions, including those for Selenium. The first video below gives background on Hudson, the reasons for its development, and the features included. The second two videos walk you through using the Hudson Sauce OnDemand plugin, which makes the Hudson & Selenium integrations quicker and easier to utilize.

If you’d like to see this presentation in its entirety, check out our Sauce Labs YouTube channel. Thanks again to our friends at StumbleUpon for providing a great space for us to use for the event. To hear these talks in person, join our Selenium meetup group. We generally host meetups once a month around the Bay Area, featuring different Selenium topics and speakers, so join in the conversation!

Hudson Background:

Sauce OnDemand Plugin:

By Kohsuke Kawaguchi

As promised in my earlier post, today I’ll talk about the library that does the heavy-lifting for the Hudson Sauce OnDemand plugin. This library is called Sauce REST API, and defines a Java API to access Sauce OnDemand REST API. It’s built by Maven, and deployed to java.net Maven2 repository.

The library is capable of basic CRUD (create, read, update, and delete) operations over SSH tunnels. To use this library, you create a SauceTunnelFactory instance, then use the CRUD methods defined on it.

For example, to set up a tunnel and tear it down, do as follows:

SauceTunnelFactory tunnelFactory = new SauceTunnelFactory();

// create a new tunnel for the intranet host foo.corp.infradna.com
// wait until the tunnel is ready
SauceTunnel t = tunnelFactory.create("foo.corp.infradna.com");
t.waitUntilRunning(30000);
assertTrue(t.isRunning());

// reverse port forwarding so that foo.corp.infradna.com:80
// will come to localhost:80
t.connect(80,"localhost",80);

// do something with the tunnel
...

// destroy the tunnel (and all port forwardings that go with it)
t.destroy();

You can also list the current active tunnels, if you’d like to.
This lists all the tunnels that belong to your account, not just the ones
created by the current process:

for (SauceTunnel t : tunnelFactory.list()) {
    System.out.println(t.getCreationTime()+" "+t.isRunning());
}

In these examples, you see I didn’t pass in the user name and API key explicitly. This is because the library reads the credential from a property file at ~/.sauce-ondemand, which should look like this:

username=kohsuke
key=12345678-abcd-cdef-1234-1234567890ab

This encourages all the applications that use this library to share the same credential source, which is more convenient for users. Of course, if your application wants to control the credential in its own way, you can use a different overloaded version of the SauceTunnelFactory constructor to pass it in explicitly.

The Maven-generated documentation and javadoc are also available.

Kohsuke Kawaguchi, the creator of Hudson, wrote the majority of Hudson’s core single-handedly. He is a founder of InfraDNA, which provides products, services, and support for Hudson.

A new survey by ActiveState shows Selenium to be the most highly adopted testing framework among the automated testing community.

According to the author’s analysis, the “results were a mixed bag as the test automation community has a lot of frameworks and testing approaches from which to choose,” but we read things a bit differently.

With 40% of users preferring Selenium, and the next most popular approach being “other,” it’s clear who the winner is. But of course, there are “lies, damn lies, and statistics” to keep in mind, and at Sauce Labs, we admit to having a special bias towards Selenium :-).  It’s hard not to, though, when Jason Huggins, our co-founder, is the original creator, and was recently named one of the 25 “Best CTOs of 2010″ by InfoWorld.

With that being said, it’s great there are many choices for automated testing, and we’re glad that our favorite testing platform is at least winning “by a nose.”

If you’d like to see useful tips and tricks for using Selenium, check out some of our other blog posts under the category “Selenium Knowledge.” Also, make sure to keep your eyes peeled, we have some good stuff planned for next week.

The good people behind Bromine, an open-source QA tool that allows you to easily run and view your Selenium RC tests, recently created a Sauce Labs integration. This means you can now run your Bromine tests with Sauce Labs’ in-the-cloud service, Sauce OnDemand, with the added bonus of having a jobs queue AND video playback. Exciting news for all!

Check out the demo below, and download Bromine here to start using it with Sauce OnDemand today.

By Kohsuke Kawaguchi

There has always been some overlap between Hudson users and Selenium users. As a result, the support for Selenium in Hudson, such as the Selenium Grid plugin, has generally been good. But while this Selenium Grid support, for example, does make it easier for you to run cross-platform, cross-browser tests, setting up the right OS/browser combinations, especially in the environment that doesn’t use virtualization, can be tedious.

This is where Sauce OnDemand comes into play. Sauce OnDemand solves this problem by offering browsers running in the cloud. But these browsers need to be able to talk to the web server being tested, which is most likely inside a firewall. Although Sauce OnDemand offers a SSH port forwarding option to handle this, setting up and and then tearing them down (or keeping them going all the time) is time consuming, especially when you are running your tests on a CI server that spans across multiple computers.

I wrote the new “Sauce OnDemand” plugin to fix this problem. With this plugin, Hudson will set up and tear down tunnels automatically for you.

Once you install a plugin from Hudson’s update center, go to the system configuration and specify your Sauce OnDemand credential:

Then, for each job that uses Sauce OnDemand, tell Hudson to set up SSH tunnels before the build. Hudson will tear them down at the end of the build.

Your projects don’t need anything in their build scripts to do this, which saves a lot of time if you have multiple projects/modules that use Sauce OnDemand. This also avoids the complexity of maintaining a persistent connection, which has its own complications. SSH tunnels on Sauce OnDemand starts very quickly, normally about 10 seconds, so the impact of this to the total build time is minimal.

Once you start using Sauce OnDemand extensively in Hudson, you’ll notice one problem. Normally you hard-code the host name that browsers connect to (which points to the cloud side of the tunnel and then forwards it back into your intranet to your webapp), but if you do that, you won’t be able to run such tests concurrently on Hudson, since the tunnel setting is global to your account.

In other words, imagine hard-coding your test to send HTTP requests to “example.org”, and you have two active branches of that project in development now. If the builds of those branches happen to run at the same time, their tunnel setup collides!

In the Hudson plugin, I added a mechanism to let Hudson auto-generate a unique host name when it sets up a tunnel. To do this, specify “AUTO” where you normally specify the domain name (“example.org” in the above example). Hudson will generate a random host name and make that host name available as the environment variable named SAUCE_ONDEMAND_HOST.

Your test should then look up this value and pass it to Selenium when you connect to it.

The implementation of this is done in two parts. First I wrote a library that performs SSH tunnel setup and tear down, and then I wrote a Hudson plug-in around it. The former is reusable on its own, and would be useful if you want to take the same idea and apply it elsewhere other than Hudson. My next post will talk more about this library, so stay tuned!

Kohsuke Kawaguchi, the creator of Hudson, wrote the majority of Hudson’s core single-handedly. He is a founder of InfraDNA, which provides products, services, and support for Hudson.

While FIFA officials agonize whether to offer instant replay as part of their soccer match officiating regimen, Sauce Labs users experience no such stress. Sauce Labs users enjoy as a default, instant replay on all their tests, making judgment calls about what when wrong both easy and repeatable.

So for all those England fans, cheer on the mantra with us “Replay or Regret” and Sauce will keep the noise going on our end (we promise not to use vuvuzela’s).