Skip to content

Feed aggregator

Very Short Blog Posts (16): Usability Problems Are Probably Testability Problems Too

DevelopSense Blog - Wed, 04/16/2014 - 21:04
Want to add ooomph to your reports of usability problems in your product? Consider that usability problems also tend to be testability problems. The design of the product may make it frustrating, inconsistent, slow, or difficult to learn. Poor affordances may conceal useful features and shortcuts. Missing help files could fail to address confusion; self-contradictory […]
Categories: Blogs

Saving you from Heartbleed

Kloctalk - Klocwork - Wed, 04/16/2014 - 18:39

Last week we talked about the flaw in OpenSSL known as “Heartbleed” and it’s massive impact on websites and users around the world. We also mentioned how open-source scanning and support tools, such as OpenLogic, report this flaw. Today, we look at how Klocwork handles the issue.

Out of the box

The root cause of the Heartbleed issue is that a request to retrieve server memory for OpenSSL’s heartbeat function isn’t validated, resulting in an array access through memcpy() potentially accessing data beyond the length of the array (and into memory that could contain sensitive information). Due to the use of a macro in the OpenSSL code that performs this function, this “tainted data” breach wouldn’t be discovered by Klocwork’s analysis out-of-the-box. The good news is, unlike other static code analysis tools out there, a few simple overrides performed by any user would discover the flaw.

Here is the relevant OpenSSL code with some interesting lines highlighted (click to enlarge):

Heartbleed code

The main culprit is the memcpy() on line 1487 where payload bytes is copied from pl to bp. Since payload isn’t validated anywhere, it’s possible that more bytes than pl contains are copied into bp, resulting in unknown memory copied into bp. This memory is eventually transmitted to the outside world (see lines 1492 and 1495).

Looking at line 1464, you’ll see why Klocwork doesn’t find this problem out-of-the-box. The difficulty here is the n2s() macro used to extract payload from p, where p is defined on line 1457 as &s->s3->[0]. This macro effectively “hides” the propagation of data through the function and, of course, Klocwork can’t report what it can’t see.

Start tuning

Using an override file and a custom knowledge base record, it’s pretty simple to tune Klocwork’s analysis to find this flaw. You don’t need any special version or upgrade to do this and, in fact, this is fairly common practice for ensuring the analysis understands your project’s unique code. First, create an override file that contains this macro override (the file must have a .h extension, so you can call it n2s.h):

#kw_override n2s(p, num) n2s_func((p), &(num))

This tells the Klocwork compiler to expand all instances of the n2s() function in the source code to the analysis-specific definition specified here, n2s_func(). Then, create your own knowledge base record to help Klocwork understand this particular macro (you can name the file n2s.kb):

n2s_func - TaintedIntData *$2

This record says that the function here returns potentially tainted integer data if the second argument points to a buffer that may contain tainted data (you can learn more about this syntax here). Once this tuning is done and an analysis performed, Klocwork reports the flaw as SV.TAINTED.CALL.INDEX_ACCESS or, an unvalidated integer is being used to access an array (click to enlarge):

This tuning of Klocwork’s analysis isn’t specific to the Heartbleed problem but it does provide a real (and unfortunately, popular) example of how static code analysis can prevent some fairly serious issues.

And, in case you’re wondering, here’s how a comprehensive open source scanning and support tool like OpenLogic flags the issue (click to enlarge):

Categories: Companies

Improving Software Quality with SonarQube

Testing TV - Wed, 04/16/2014 - 18:22
Software quality is about a lot more than slinging good code. As a developer you use numerous tools, techniques, processes and frameworks as you write, organize, build, test, refactor, and continuously improve your applications. SonarQube (formerly Sonar) is an open source quality management platform, dedicated to continuously analyze and measure source code quality. It makes […]
Categories: Blogs

Ask a Selenium Expert: Selenium Test Report Example

Sauce Labs - Wed, 04/16/2014 - 18:00

selenium testing & sauce

This is part 2 of 8 in a mini series of follow-up Q&A’s from Selenium expert Dave Haeffner. Read the first Q&A here.

Dave joined us and led our recent webinar, “Selenium Bootcamp“, wherein he discussed  how to build out a well factored, maintainable, resilient, and parallelized suite of tests that will run locally, on a Continuous Integration system, and in the cloud.

He’s also agreed to respond to 8 of the many follow-up questions we received post-webinar. Below you’ll find the second Q&A. Stay tuned next Wednesday for the next question.

2. ­Can you please show some examples of a Selenium test report?

Here are two examples:

-Dave Haeffner, April 9, 2014

Can’t wait to see the rest of the Q&A? Read the whole post here.  Get more info on Selenium with Dave’s book, The Selenium Guidebook, or follow him on Twitter orGithub.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!)  here.

Categories: Companies

Active Directory plugin improvements

One of the few plugins that I still personally maintain is Active Directory plugin. In the past few months, I've been making steady improvements in this plugin, thanks to various inputs and bug reports given to me from the ClodBees customers.

One of the recent fixes was to get the "remember me" feature finally working for Active Directory. This requires a relatively new Jenkins 1.556, but it eliminates the need to having to constantly type the password in.

Then I've rebumped the version of COM4J, which was causing a thread leak when Jenkins runs on Windows. If you are running a Windows deployment with lots of active users, this probably would have contributed to the instability of Jenkins.

And then lastly, a small but crucial improvement was made to the way we search group membership, so that we can avoid recursively searching AD. This should result in a significant speed improvement when you are logging into Jenkins through AD.

The latest version of the plugin as of writing is 1.37. I hope you'll have a chance to update the plugin soon.

Categories: Open Source

Collective Noun for Testers?

The Social Tester - Wed, 04/16/2014 - 17:57
The other day I tweeted a picture of our test team testing together as a group (for the first time) in our new test lab. I asked “What do you call a collection of...
Categories: Blogs

Health care devices introduce cybersecurity complications

Kloctalk - Klocwork - Wed, 04/16/2014 - 14:30

The world is becoming more digital, and the health care landscape is no exception. While the proliferation of mobile devices in hospitals and other caregiving facilities can improve the productivity of workers and make it easier for physicians to access critical information on patients, the unfiltered presence of those platforms and the applications they utilize can introduce substantial cybersecurity concerns.

InformationWeek recently highlighted a new SANS Institute survey that looked into millions of endpoints throughout the health care environment, revealing numerous patching problems and fundamental vulnerabilities in the mobile device realm that could compromise personal and financial data integrity, as well as the reputations of the organizations in question. Specifically, experts told InformationWeek that approximately 375 health care networks were compromised by attackers during the 13-month research period.

"We were shocked at [the number of] devices that were wide open to the Internet that would provide adversaries with considerable power and access not only for a breach, but – for those who are skilled – even to conduct malicious acts," cybersecurity expert Sam Glines told InformationWeek.

While there seem to be a number of factors contributing to the growing risk facing the health care IT security environment, poorly configured and generally unsafe devices are largely to blame.

The problem with careless devices
While the term "device" is widely associated with smartphones and tablets, the concept also includes routers and other technologies used to connect to the Internet. Routers in particular are a common link in the poor cybersecurity chain. Citing experts from security firm Tripwire, InformationWeek noted that many platforms are unsafe straight out of the box, which suggests that if organizations do not take any proactive measures to mitigate risk, they may encounter severe vulnerabilities down the line.

"All of these compromised devices, not only are they available to be used for a breach of data, but they're also used as attack points against other adversaries," Glines told InformationWeek.

In other words, unsafe devices provide cybercriminals with more launch​ pads for attacks on confidential networks and IT environments that house highly confidential information. One of the best ways to combat these challenges and other vulnerabilities being introduced to the health care industry is to ensure the embedded software within those devices is secure and robust enough to mitigate risk on multiple levels.

Developers who are charged with the responsibility of creating and launching applications in the health care realm must be sure the end products they create are not vulnerable in any way. This means utilizing code review platforms that provide developers, quality assurance and other professionals insight from multiple perspectives throughout the development process. When these technologies are used correctly, applications are generally produced with fewer weaknesses than if developers streamlined the process without any input from others.

As the Internet of Things grows more pervasive, health care executives must be prepared to combat an increasingly sophisticated digital threat landscape. Taking a robust and comprehensive approach to cybersecurity will be among the only ways to mitigate risk.

Categories: Companies

Top 3 PHP Performance Tips for Continuous Delivery

Are you developing or hosting PHP applications? Are you doing performance sanity checks along your delivery pipeline? No? Not Yet? Then start with a quick check. It only takes 15 minutes and it really pays off. As developer you can improve your code, and as somebody responsible for your build pipeline you can automate these […]
Categories: Companies

Conducting a Root Cause Analysis in TestTrack

The Seapine View - Wed, 04/16/2014 - 12:30

Is your team currently conducting root cause analysis (RCA) investigations? If not, they’re a great way to identify deep-rooted issues that have the potential to create multiple and/or recurring defects in a product line. Of course, you can do RCA sessions with documents or spreadsheets. But using TestTrack makes the analysis process less time-consuming and, therefore, more likely to actually be done.

This short video provides a high-level overview of how to conduct an RCA investigation with TestTrack. 

Share on Technorati . . Digg . Reddit . Slashdot . Facebook . StumbleUpon

Categories: Companies

As Retailers Boost IT Spending, Be Mindful of the Minefields

By George Wilson Research out this week has earmarked the retail sector for soaring IT investment in 2014, with websites, mobile and IT system replacement being top of their wish lists. Law firm TLT has found that two thirds of the UK’s top 60 retailers expect their firms to grow this year and 80 per […]
Categories: Companies

2,600 Take Our “Achy Breaky Heartbleed” Survey

Sonatype Blog - Tue, 04/15/2014 - 23:42

It turns out, our timing could not have been better…

On April 1st, we launched our annual open source development survey. The five minute survey asks participants several questions about their open source security policies, practices, and experiences. Questions include:

  • Have you ever banned an open source component?
  • Do you have an open source policy and does it address security vulnerabilities?
  • Do you track changes in vulnerabilities in production apps?
  • Are applications developed with open source just as secure as COTS applications?

As bad fortune would have it, or perhaps it was luck, the Heartbleed bug was announced on April 7th and the notice went viral almost immediately. During that first week of April (pre-Heartbleed notice), we had over 1,500 participants in the survey. Post-heartbleed, we have had another 1,100+ participate.

This means we have perhaps the best and broadest stats on the state of open source application security at the apex of Heartbleed awareness. While it started as the “open source development survey”, it has quickly turned into the industry’s largest, most current “achy breaky Heartbleed survey”.

The survey runs through the end of April, so there is still time to take it here.

What have we learned thus far? Here are some of the preliminary results:

  • 64% of organizations don’t actively monitor open source components for changes in vulnerability data (That’s right, these companies would not be watching for the next Heartbleed, Struts, Bouncy Castle vulnerability…perhaps waiting for word-of-mouth to reach them)
  • 52% of organizations do not keep a record of all open source components used in production applications (For the next Heartbleed-like vulnerability, how would these companies be able to ascertain if they had ever used the component somewhere?)
  • The top 3 challenges of open source policies named were: (1) No enforcement / workarounds common, (2) Does not address security vulnerabilities, (3) Not clear what’s expected of us (Does your organization face similar challenges with open source policies?)

You might be asking yourself if there were any notable differences in answers following the April 7th announcement? To our surprise, a few responses have seen a post-Heartbleed bump while responses to other security-related questions barely moved the needle.

If you would like to assess how your organization is faring compared to 2,600+ peers in the midst of the heartbleed announcement, please take our survey — and invite others to participate as well. All participants will receive a copy of the survey results shortly after the survey closes.

THE MOST IMPORTANT POINT : Take the survey, get the results, and then…start a conversation, spark a debate, or ask if your organization should be considering new actions. As with all surveys, it is not the stats that matter…it’s what you do with them.

Categories: Companies

Do Testers Still Own Landline Phones?

uTest - Tue, 04/15/2014 - 22:22

Testers, especially those within the uTest Community, are at the forefront of mobile technology. From iPhones, to Android tablets, to even the latest smartwatches and fitness devices, uTesters often are armed with 5, 10, even 20 devices at a time for various testing projects.

So one would think that if anyone on the planet was going to own a ghastly piece of 1990s technology like a landline phone, it wouldn’t be testers.

But you’d be wrong.

According to a recent poll kicked off in the uTest Community, in fact, 64% of uTesters have landline phones still in their homes, and it’s not just for nostalgia.


According to a couple of testers, there are still practical uses of the landline phone:

I like having an alternate number to give to folks who don’t really need to have my cell phone number, to interrupt every minute of my day just to find out if I was “very” satisfied with their customer service… and enjoy screening those calls appropriately! – Brian S.

I do, but I find myself more and more looking at getting a VoIP solution, as my landline keeps getting more expensive for basic service. I have a cell and do most calls that way, but for the wife and kids, they use the landline. – Tim F.

Given the diverse tester base making up the Community, some of it just boils down to the popularity of landlines in certain countries:

 I have one. They still are pretty common in my country, but I mostly use it for Internet, since the only broadband in my area is a DSL service. – Caio B.

It’s still pretty much common in Argentina to have a landline phone. Mostly all families and companies have a landline phone. – Anahi G.

So based on all of this, is the landline phone (dare I say, rotary dial?) making a big comeback like vinyl music sales? Not quite. It’s fair to point out that some of the testers with landline phones begrudgingly own them (including this guy right here) because they’re part of a cable package that includes phone, Internet and TV at reduced rates. Additionally, amongst those in the uTest Community that cut the landline cord, many of them did it upwards of 10 years ago.

Nonetheless, the 64% of testers surveyed that still own landline phones was a surprising number highlighting a bygone area, one that is increasingly becoming irrelevant in a mobile world. I wouldn’t suspect the one-trick-pony in landline phones to be quite as popular in the 2015 or 2016 version of this poll.

Tester Community outside of uTest: Do you still own a landline phone? Do you actively use it? We’d love to hear from you in the comments below.    

Categories: Companies

Upcoming Jenkins Office Hours: Acceptance Test Harness

One of the new efforts in Jenkins this year is the acceptance test harness for Jenkins.

We will be doing the Jenkins office hours next week to go over this and sync up and coordinate between people in the community that are trying to work on this.

It'll be April 23rd 11am PT (see what this time is in your time zone) on Google Hangout at If you are intereste in hacking Jenkins or if you are a large user of Jenkins who have acceptance tests, we are looking forward to seeing you there.

For those of you who haven't looked, this test harness allows you to write blackbox tests of Jenkins and its plugins. It was originally used to test LTS releases, but over the time, it acquired a number of features, such as ...:

  • Docker support for launching complex fixtures to test Jenkins with.
  • Pluggability to launch Jenkins under test (JUT) in many different environments
  • Pluggability to provision Jenkins and slaves from EC2 to test large deployments
  • Choice of cucumber or JUnit to write test scripts

We are working on porting over existing test cases, but we'd like to work with users to move their acceptance tests on top of this same harness. The idea is to pool those test cases in the community so that we can test Jenkins and its plugins as we develop them. For this to work, we want tests to have lots of metadata (such as what plugins it touches), and for the harness to have sufficient modularity that different people can run the same scenario against different deployments, including existing instance.

Categories: Open Source

LoadRunner 12.00’s new network profiles make it easier to connect to your LGs, MI Listeners and MoFW

HP LoadRunner and Performance Center Blog - Tue, 04/15/2014 - 19:00

HP LoadRunner 12.00 adds a new feature called Network Profile, which makes it easier for customers to configure their network environment for load testing. The new feature enables the Controller to communicate with other components such as Load Generators, MI Listener and Monitor Over Firewall via a proxy.


LoadRunner 12.00 allows you to export and import these configurations, making it even easier to configure the LoadRunner network environment.


Continue reading to learn about Network Profiles and how you can use them.


(This post was written by Lin Chen (Charlie), from the LoadRunner R&D Team)

Categories: Companies

Meet the Bees: Félix Belzunce

Félix enjoying the sun in Namur.At CloudBees, we have a lot of seriously talented developers. They work hard behind the scenes to keep the CloudBees Platform as a Service (PaaS) and our on-premise Jenkins solutions up-to-date with all the latest and greatest technologies, gizmos and overall stuff that makes it easy for you to develop amazing software.

Félix’s main role at CloudBees is to improve the user experience on the CloudBees platform. He is native of Cartagena, Spain, where he attended the Polytechnic University of Cartagena. While at University, he researched the synchronization problem for Wireless Sensor Network (WSN), which helped him to develop a WSN localization application. After graduating, he worked as a Java developer coding some modules for a healthcare management software. He also taught technology in Andalucia for three years helping students to improve both their knowledge and their work methodologies. He educated his students about the benefits of living in a multicultural society, in which everybody can make positive contributions.

You can follow Félix on Twitter.

Who are you? What is your role at CloudBees?
Félix likes cinema. Here, he gets to watch original
version English films with French and Dutch subtitles
at the same time. I work for CloudBees as a developer advocate, helping users to be successful in leveraging the services we provide. If we help them, they should be happy and then we will see an increase in the number of active users on the platform. In my opinion, one of the most interesting things about this position is that you really need to be able to understand how all the services work, which means I get to multi-task and not just do routine work. For that reason, the developer advocate team is usually in touch with most of the development teams on the platform, so we have an understanding about what is happening throughout the company. I also like that most of us do not have a really fixed role at CloudBees, so we can spend some time in other roles we like. In my case, I use this time to take a look at some support tickets, design some funnel processes or to improve some of the services we provide. Specifically, I have been contributing to the runtime environment by updating some ClickStacks and creating new ClickStarts. Next step will be to develop a simple Jenkins plugin. In my spare time, usually on the weekend, I am currently developing a small cloud Java application, which is helping me to improve my skills on CCD (Continuous Cloud Delivery).
What does a typical day look like for you? What are CloudBees customers like? I get up really early in the morning to avoid the disgusting traffic jams in Brussels, so I usually arrive at the office when the building is closed and the cleaning lady is finishing her job at our office. Sometimes I take advantage of having a short conversation with her to improve my French language skills. Being serious now, the first thing I usually do is take a look at StackOverflow and at the support platform to see the new customer questions, so they can receive an answer as soon as possible. Also, I usually get some emails from customers who need help in achieving their goals, and this takes me some time, as most of the time I need to reproduce a specific Jenkins configuration or need to recreate a similar environment. Sometimes, it is just a frequently asked question which doesn’t take me a lot of time to answer.

As a member of the pre-sales team, I also help sales out with the technical questions they have or that they receive from their prospects. This is also a really good way to learn, since all the pre-sales engineers participate in the discussions, so at the end we create a forum where we discuss what is the best approach for user issues and questions.  What suggestions do you have for someone who is just starting to use CloudBees? On runtime, one of the most common mistakes is to think that the file system is persistent on a multi-tenant environment, so users expect to be able to store files there.  Instead, if you really want to have your application ready for the cloud, you should use a persistent file-system like Amazon S3 in the same datacenter where you plan to deploy your web app. Enjoying a Belgian brew. "Drinking a Leffe with summer clothes and
eating mussels in a terrace is priceless in this country."

I also recommend to use some add-ons which will help you to know how much heap memory your application needs, or to alert you when your app is down for any reason. These kind of automations are really easy to set-up on a PaaS and will help you to get information about how everything is going. For the development environment, the answer is pretty easy: just use Jenkins. Jenkins will allow you to get feedback every time your team pushes a change into the repository, so you can quickly fix the issues and deliver your software faster. But Jenkins is not only about delivering faster. It is also about improving some other aspects such as the quality of the code. With the great quantity of Jenkins plugins, you will for sure get an idea of the grade of customization you can get. Do you have any advice for someone starting a career in the cloud space? If you plan to work as an engineer for a cloud company, the best idea is to start contributing to open source projects related to the cloud: Jenkins plugins, cloud projects like jclouds, or even coding your own stack for the CloudBees Platform. It is also a really good idea to have your own personal blog in which you can talk about topics related to the cloud.  Nowadays, on interviews, managers may start asking you for your GitHub/StackOverflow profile and also for any link where they can read something you wrote where they can have you answer questions about it. What is your favorite form of social media and why? I like Facebook as it is a simple way to be in touch with your friends, sharing what you are doing and what are your thoughts about a topic.  Obviously, I am talking about “real” Facebook friends and not the ones you have, but have never met in your life ;-) It is true I use a fake name, with a fake location and a fake email address to guarantee my personal private information on Facebook. Also, I could in this way add the people I want without receiving so many friend requests. I was working as a teacher in the past, so maybe it explains why I didn't want to receive 300 student requests every time I was moved to a new school. Vanilla or chocolate or some other flavor, what’s your favorite ice cream flavor and brand? What is better than eating a waffle with chocolate
in Brussels after a great bike ride?Definitely it is chocolate. I eat chocolate almost every day. Belgian chocolate is especially good, so it seems I am in the right place. I also like drinking hot chocolate and I have discovered that hot chocolate in Spain is much different than the one is served in the rest of the world. If you like chocolate and you come to Spain, you should taste it.

If you could eat only one meal for the rest of your life, what would it be? Difficult question. I don’t have a favorite food as it depends on the day, if it is summer, winter or if I ate too much the day before. What I can say is that it would probably be a Mediterranean meal.

Categories: Companies

You’re Invited: Appium 1.0 Launch Party! RSVP Today

Sauce Labs - Tue, 04/15/2014 - 18:00

Sauce Labs is thrilled to announce we’re sponsoring and hosting the Appium 1.0 launch party at our new San Francisco office on Friday, May 2, from 7-10 PM!

Sauce Labs’ own Jonathan Lipps, chief architect of Appium, will be releasing Appium 1.0 at the party. We’ll celebrate with food, refreshments and merriment – and if you’re an Appium or mobile enthusiast, we want you there, too! Appium 1.0 could not have happened without massive support and work from the Appium community, and we want to toast the launch with all of you.

Mark your calendars for May 2. Space is limited, so make sure to RSVP here for a ticket. See you there!


Categories: Companies

Make your Ranorex Report fully comprehensible for Jenkins by using xUnit

Ranorex - Tue, 04/15/2014 - 16:53

A Ranorex test automation project creates an executable file which returns a two-state test result indicating a success or a failure. Continuous Integration (CI) tools like Jenkins, Hudson, TeamCity, CruiseControl, Bamboo etc. are able to call automated tests and do understand these two-state test results. This is similar with ALM tools like Polarion or Jira. In short, in the world of CI and ALM, test automation results are reflected by a single flag, success or failure.

This blog describes how a Ranorex Report can be automatically transformed into xUnit format by using an XSL Transformation (XSLT). This transformation allows the CI tool to fully understand the result of each test case, not only whether the entire test run passed or failed.

The following example shows how UI test results can be interpreted following the same standards as xUnit tests do. Furthermore an example implementation of an XSL transformation file can be downloaded at the end of this blog.


The blog post “Integrating Ranorex Automation in Jenkins CI Process” is introducing the entire CI topic and gives a step by step introduction on how to integrate Ranorex into Jenkins CI. On the top of that, optimizations can be done supporting the machined understanding of the Ranorex Report.

Nowadays most of the modern unit testing frameworks such as JUnit, NUnit etc. do implement the xUnit notation (list of implementations). This comes with the advantage that tests and their results are defined once following the xUnit standard, and can then be interpreted and understood independent from the respective target language (such as Java, C#, etc.). In addition, continuous integration servers like Jenkins do use xUnit to interpret the results of a test in a unified way. Although Ranorex is a UI testing and not a unit testing framework, there still is a way to also interpret the Ranorex test results (Ranorex Reports) in the same way as you might be used to from your (automated) unit tests.

Install xUnit Plugin for Jenkins

First, the xUnit-plugin needs to be installed on Jenkins. Therefore please navigate to “Manage Jenkins” in the main menu and click the “Manage Plugins” menu item.

The tab sheet “Available” now provides the possibility to install the “xUnit plugin”.

Please proceed by installing the plugin.

Activate Custom Transformation (XSLT)

Next, the transformation of the Ranorex Report to a xUnit-based report format has to be activated. Therefore navigate to your Ranorex project and bring up the project configuration.

Start by adding a “post build action” called “Publish xUnit test result report”.

Two new panels appear allowing to define the thresholds for failed and skipped tests (both, total and new). More importantly click the “Custom Tool” menu item in the drop down:

The file name of the Ranorex Report needs to be defined in the setting “Custom Tool Pattern”:

Please make sure that path and file name of your report file pattern match the definition of your build step that calls the Ranorex executable. Here is the one used by this example:

TestCIProject.exe /rf:Report-Build-%BUILD_NUMBER%

Note: The Ranorex command line argument “rf” (reportfile) allows defining the report file name including the extension. If no file-extension is provided, the default extension “rxlog“ will be used for the report frame file. Additionally, the XML file containing the relevant data will be created with a file-extension “” then. For details on all command line arguments please refer to the user guide, chapter Running Tests via Command Line.

Note also: The above used placeholders %BUILD_NUMBER% and ${BUILD_NUMBER} are provided by Jenkins. At the beginning of a test run they will be replaced automatically by the current build number. Please take care to use these notations correctly.

Another relevant setting is the path to the transformation file (XSLT). A sample file is shared on GitHub and can be downloaded here:

Given these settings, right after every Ranorex test execution triggered by Jenkins the Ranorex Report is automatically transformed into a xUnit-based XML file. This allows a deeper integration of test results into CI which will be described in the next section.

Exploring Test Results

Having applied the mentioned settings to your Ranorex project on Jenkins, a section “Test Result” will appear for every new test run.

Following the “Test Result” link, an overview site is presented providing useful information like “All Failed Tests” including their duration and age (since how many builds does test fail). Additionally, a short summary of all passed, skipped and failed tests is presented.

By clicking the plus icon of a failed test Jenkins presents error details and the stack trace additionally:

Also, Jenkins can now distinguish on the test case level whether a test case passed or failed. Before the conversion to xUnit format, Jenkins was only able to interpret the entire test result.


Calling your Ranorex test automation project from a Continuous Integration (CI) environment provides a lot of benefits and allows you to represent the results of the entire build process over time. However, even more information of every single test run can be gathered out when the Ranorex Report is fully understood by the CI tool. This understanding can be achieved by a transforming the test report into xUnit format. As visible above, there are only a few steps needed to see the results of every single test case instead of only one result of the entire test run. Furthermore these test case detail results can be used easier for further processing.

Although the sample above is using Jenkins as a CI tool this approach is the same for all CI tools understanding xUnit.

Further Reading


Categories: Companies

StrongLoop Releases StrongOps 2.0 DevOps Service for Node.js

SQA Zone - Tue, 04/15/2014 - 16:10
StrongLoop has announced the general availability of StrongOps 2.0, the first Node.js specific application lifecycle management (ALM) solution to include both monitoring and DevOps capabilities for Linux, Windows and Mac. Node.js continues to gai ...
Categories: Communities

Knowledge Sharing

Telerik Test Studio is all-in-one testing solution that makes software testing easy. SpiraTest is the most powerful and affordable test management solution on the market today