The difference turns out to be a lot! To start with load and performance testing goals are very different.

 

Load testing goals usually seek to answer questions regarding system capacity, throughput, and client response times for multiple concurrent users. For example, how many concurrent users can the system successfully accommodate? Or how many transactions per second can the system successfully process? Or can the system successfully scale to 1,000 users? Whereas performance testing goals seek to answer questions regarding server processing, and client response times for a single user. The primary goal of performance testing is to tune the system to achieve maximum performance, [e.g. execute a given amount of work (transactions) in the least amount of time (transactions per second (TPS)].

 

Load and performance testing are both challenging; however, performance testing is the easier of the two because only one user is tested. Whereas in load testing there are hundreds or thousands of concurrent users tested.

 

Additionally the implementation of performance testing is much less complicated because, again of the fact that only one user is being tested versus hundreds or thousands for load testing. Also, each user type in load testing must have a scenario designed, implemented, and modified, to execute and simulate live users as accurately as possible.

 

When modifying and enhancing test code for load testing the test engineer must consider concurrent users and adjust the test code accordingly. This will mean modification of session IDs, user form data, and input data, whereas in performance testing there is only one user and typically no session IDs, etc., need to be handled.

 

Because the performance test only involves one user, it is typically a much shorter test than a load test. Load testing involves hundreds or thousands of concurrent users and is usually a much longer test and thus much more data to be collected and saved.

 

When it comes to results analysis and correlation of test data, the performance test data volume is much less then the volume of load test data. Again, because performance testing executes only one user, thus there is much less data to be analyzed for performance vs. load testing, even though the same or similar metrics are collected.

 

During the remediation phase of testing, performance testing is a much smaller problem space than load testing. With remediation of performance tests, typically metric ‘outliers’ are identified, then the engineer drills down to determine root cause. Load testing remediation is not that straight forward or easy. Load testing results may show errors. However, there may be bottlenecks which are more difficult to identify and more difficult yet to remediate.  So, it turns out there is a great deal of difference between load and performance testing.

 

Test Studio, from Telerik, has long been capable of performance testing. Further, any functional test can also be used as a performance test. To execute a performance test in Test Studio, there are just three steps: 1) Create or select an existing web functional test that satisfies the test scenario criteria, (e.g. user actions, and or transactions to be executed), and open that test, 2) configure performance test properties such as location to store results then select the computer to collect performance metrics from, 3) execute (run) the performance test.

 

In the Test Studio 2012 R1 release just out, load testing has been added to the capabilities of Test Studio!

 

Through load and/or performance vulnerabilities of software comes financial vulnerabilities of the company, which both directly affects company financials and customer satisfaction. Test Studio can help mediate these risks and reduce vulnerabilities.

 

Last week I wrote a quick summary of the Telerik Testing Summit.

I’m happy to announce we’ve got a short video posted up showing some of the highlights of the discussions during the first day of the conference.

If you’re not already, why not follow our Test Studio YouTube channel?

Are you completely new to functional web testing? Don’t know where to get started? If so, please mark off your calendar for a three day series of webinars June 26th, 27th, and 28th. I’ll be hosting these webinars from 11AM – 12PM Eastern US.

This series is targeted to individuals or teams just getting started with web automation—although more experienced folks may learn a few things too!

I’m still fleshing out the agenda and content, but here’s a few of the things you can look forward to learning about:

• Automation’s purpose/fit in your overall quality and testing strategy
• What a web page looks like under the covers, and how Test Studio interacts with it
• What a page’s lifecycle is, and why you need to know about it
• Understanding element locators, and why it’s so important to understand them well
• Dealing with dynamic content like AJAX
• Designing and implementing great test cases
• Why you’ll need close collaboration with your entire team
• Learning to build a backing framework
• Keeping your tests reliable and lean
• Learning why it’s important to not test everything—and learning the sorts of things to not test!
• Data driving tests to flesh out coverage
• Working with baseline data sets
• Understanding teardown and setup actions
• Working with source control
• Surviving and keeping your sanity as your test suite grows in size

These webinars are intended to be “Lunch and Learn” sessions: informal but with a lot of information flowing. I’ll be leaving plenty of time for questions, too!

More information, and registration links, will follow soon. For now, just block off those dates if you’re interested!

Telerik’s TeamPulse is a terrific project management tool for boosting communication in and between projects. We’re constantly looking for ways to better integrate Test Studio with TeamPulse to clearly communicate quality metrics around your automated tests back to the projects.

We’ve had integration tying tests to acceptance criteria in TeamPulse for some time, but with Test Studio’s 2012 R1 release we’ve kicked up the integration another notch: You can now push bug reports from Test Studio directly to your TeamPulse projects from anywhere you can examine a failing test: the test explorer, test results calendar, or test list details.

(BY THE WAY: We’ve also got the same bug reporting functionality for Team Foundation Server. I’ll post on that one later!)

The user interface is the same at any of these places in Test Studio:

From the failure details screen you click the Submit Bug icon at the bottom, select where you want to submit the bug report to, edit the title and description as desired. Checking the “Attach failure details” checkbox will create a zip file with the exception details, screenshot of the UI at the point of failure, and enclose the page DOM at the moment of failure. That zip gets attached to the bug report in the target system—folks working triage and bugfixes will have everything right at their fingertips!

You can also use Test Studio Explore to file bugs during your exploratory testing sessions! (Read more about the Explore tool in a previous post.)

I’ve just posted up a short video walking through all of this. Please have a look at it. I think you’ll find it extremely useful!

Join us Wednesday, 16 May, at 11am US Eastern time, to learn how Test Studio’s performance testing and just-released load testing features can help you detect bottlenecks in your system’s performance and answer business-critical questions about your site’s ability to handle large amounts of traffic.

This webinar will focus solely on Test Studio’s Performance and Load Testing features. We’ll be drilling down in to some great specifics of how these features can help you respond to questions from management like “Will our site be able to support traffic after this new product’s launch?” and “Are our customers happy when they’re using our site during normal and high traffic times?”

You’ll learn how you can use one functional test and run it as a performance test to give you great breakdowns for information about specific use cases – and tie in detailed metrics from Windows Performance Monitor counters.

You’ll also discover how easy it is to construct complex, realistic load test scenarios. We’ll also walk through analyzing results from those tests using Test Studio’s clear, understandable reports!

Interested? Why not head over and register for the webinar right now?

Are you working with iOS applications? Do you need to get some automated tests around your native (UIKit), web, or hybrid applications?

Have a look at Test Studio for iOS! I’ve written about it beforehand, and tomorrow I’ll be hosting a webinar giving an introduction to our great testing tool for apps on iPhones, iPads, and iPod Touches.

I’ll show you how to set up your Xcode projects, and run through a few highlights of working with Test Studio for iOS. The webinar is at 11AM Eastern US.

Interested? Why not go register for the webinar?

Telerik Test Studio has been nominated in the Debugging & QA category in this year’s Code Project Readers’ Choice Awards. If you enjoy using Test Studio; appreciate your experience with our customer & technical support, please, do cast your vote for us!

Telerik has been nominated in 14 categories. If some of your favorite products are in that list you can vote for them until Friday, May 11.

Thank you for your support!

A good test case is neither too small nor too big. A good test case stands alone, sets up pre-test conditions, exercises the application under test (AUT), verifies one thing, reports the results of the verification, then if applicable, cleans up (usually data) and reports the overall test case result; pass or fail. 

If you were following Twitter last this last weekend (4 and 5 May) you may have seen a number of Tweets tagged with #TelSum. That was the Telerik Testing Summit, a peer conference held in Austin, Texas. Telerik welcomed a number of thought leaders in the testing and software industry to Austin, Texas, for two days of discussions on the state of software development. More importantly, we as a group were focused on what we could do to try and improve things.

Telerik was represented by myself and Phil Sams, plus Holly Bertoncini, our event organizer. Joining us were Charley Baker, Jeff Morgan, Lanette Creamer, Lisa Crispin, Marlena Compton, Dawn Test Code (yes, that’s her real legal name!), Matt Barcomb, Adam Goucher, and Chris McMahon. Austin locals Brett Pettichord and @rubytester joined us for various parts of the conference as well.

The summit was completely free-form in nature. We built the agenda Friday morning and adjusted it on the fly as we progressed. We had several hours of focused discussion Friday morning and afternoon, a site visit to Telerik’s Test Studio office in Austin, open space discussions Saturday morning, lightning talks, two hours working on the Weekend Testing project for Wikipedia, and a few hours closing out to summarize and create action items. We also had some practical sessions on things like using GitHub (fork, clone, fix, commit, push, pull), and a number of folks paired up to solve practical problems for their “day” jobs.

I mention action items, because one thing we felt critical was to ensure we left the summit with some concrete things we could do to improve, change, or elevate testing and software development.

I’ll be posting up more details on various things over the next few days, but here’s the rough agenda as it shook out: Brainstorming topics Discussions

• Whole Teams (Benefits of, how to get teams working better, etc.)
• UI Automation 101 (why so many don’t understand, what to do)
• Big Frameworks (Why so many, how to understand differences)

Open Spaces

• Data for automation
• Visualizing QA activities
• Page Objects
• Spurring ideas for testing

Weekend Testing #27

• We took two hours to test and debrief as part of the Weekend Testing Americas project.

Cheesy’s GitHub walkthrough Lightning Talks

• Lanette on Agile/cultural antipatterns. Great humor!
• Me on It’s Not About You
• Adam on Selenium v1 vs v2
• Chris (and his ukulele) on getting folks moving as part of an icebreaker
• Cheesy on requests for his Relish tool he’s evolving to handle reporting for Cucumber suites

As I mentioned, I’ll be posting up more details on all this over the next week or so. In the meantime, I thought it would be interesting to post up images of the sheets we used for brainstorming discussion topics. We took 20 minutes or so where each person wrote down items they were interested in discussing. Everyone had a few moments to describe their topics to the group, then we did a bit of consolidation. The group voted on topics to discuss, and off we went!

Without further ado, here’s the pictures of our lists.

Page 1

Page 2

Pages 3 and 4

Pages 5, 6, and 7

Page 8

And finally, page 9, the last one.

 

Chris McMahon was kind enough to post up his initial thoughts on the event here on his blog. He hits one thing that’s central to why I came to work at Telerik: The company’s culture of understanding we gain so much by focusing on industry-wide issues instead of just trying to push our products.

The conference/summit turned out extraordinarily well. We had some amazing discussions, everyone learned something, and we came out if it with many concrete things to work on over the next year. It’s going to be very exciting to see how everyone approaches their various action items!

UPDATED: I goofed and listed Quality Center as an option for pushing bug reports to—but we only integrate bugs with TeamPulse and Team Foundation Server.

As much as I love them, automated tests (be they functional, integration, or unit) aren’t the end-all-be-all for your testing approach. Exploratory testing, particularly for any system of any consequence, is a critical part of your overall “Deliver Great Stuff!” strategy.

Getting your team freed up from repetitive regression testing empowers them to put their brains to use as they poke around the corners of your system. Exploratory testing gives you tremendous value as your team uses their knowledge of the overall problem domain and the system.

We’ve shipped a small, lightweight feedback tool with Test Studio 2012 R1 in order to help you out with quickly filing bugs during your exploratory sessions! Installing Test Studio adds the Test Studio Explore launcher to each of your installed browsers' toolbars. Here’s what it looks like in Firefox:

Click that button and you’ll find the Explore app appears – and it’s nicely done to stay out of your way while you’re doing your testing! Drag it around your screen via the grab handle. (That’s the part with all the dots on the left of the tool.)

With Test Studio Explore you can capture and annotate screenshots. Give it a good title and great description, too!

Once that’s done, push it to your bug tracking system. We now link directly to TeamPulse, and Team Foundation Server!

Test Studio Explore is a great lightweight tool to help you quickly document issues as you’re doing your exploratory. I’ve always been adamant that automation’s only part of your overall testing strategy. We’re here to help with your exploratory testing, too!

What does one second of performance degradation in web and e-commerce applications cost?

It depends, but according to the Aberdeen Group (June, 2008 Benchmark report), customers are won or lost in just one second, based solely on the performance of the web or e-commerce application. Just this single second of delay for an online retail site equates to a “seven percent reduction in customer conversion” and thus lost revenue, seven percent! 

Telerik Test Studio Performance test can help customers achieve a higher level of web or e-commerce performance, and mediate this risk. 

I and four other Telerikers (Holly, May, Dan, and Jasper) just returned back to our respective homes after a few days at StarEast. Our booth at StarEast was busy the entire time!

We all had an extremely productive trip in a great many ways. Holly’s our organizer extraordinaire, May (our UX dev) got a lot of insight to how our users see the product, Jasper had many conversations with users about our feature set and potential new features, and Dan spent a lot of time talking with folks to introduce them to the awesomeness that is Test Studio.

We spent a lot of time covering our brand new feature releases: iOS testing, load, exploratory testing feedback tool. Folks seemed very excited, and we got some great feedback on things, too.

I had two different talks to give: Making Distributed Test Teams Work, a regular breakout session I had selected for StarEast, and Real World Problems, Real World Load Testing, an Industry Technical Presentation (read: “vendor session”).

We also raffled off an iPad 3 at our booth. It drew a bit of a crowd…

Here’s our winner: Sathya Betha!

Thanks to the many attendees who stopped by and spoke with us. We really enjoyed chatting with you!

It’s that time of the year: the latest, greatest release of Telerik’s Test Studio is due out Wednesday, 11 April!

We’ve got a number of great features lined up for this release: integrated bug tracking, exploratory testing tooling, better support for HTML controls, and of course, the big feature: load testing!

Grab the release bits tomorrow, but be sure to stop by the Release webinar on Wednesday, 25 April at 11AM Eastern US. You’ll need to register first, but there are plenty of spaces!

Load testing can be a complex, intimidating subject. If you’d like some more information, I encourage you to check out the following resources:

• My series on Performance/Load Testing
  • An Introduction to Performance Testing
  • Performance Testing: Laying the Groundwork
  • Wrapping Performance Testing Introduction: What to Monitor
• Steve Smith’s blog (lots of good development stuff, some great performance content)
• Scott Barber’s blog. Prolific Tweeter & blogger.
• Scott’s book Web Load Testing for Dummies.
• Microsoft’s Performance Testing Guide for Web Applications

I look forward to seeing you at the release webinar!

 

PS: Working with developing apps on the iOS platform? Have you signed up for our Test Studio for iOS beta program yet?

Interested in learning more about how Test Studio for iOS can help you create great automated tests for your iPhone, iPad, or iPod Touch applications?

Have a look at the just-uploaded webinar video!

I’ll walk you through getting Test Studio for iOS installed and configured in your Xcode workspace projects, then I’ll show you how easy it is to get started recording and playing back tests. You’ll also learn about error reporting and troubleshooting failed tests, plus you’ll find out how we work hard to make your tests extremely flexible with element location.

You’ll also discover how to deal with handling asynchronous delays in the UI (a big part of creating robust tests on any platform!), and you’ll see how you can adjust target elements right from within a step.

Additionally, you’ll see how Test Studio for iOS lets you test completely disconnected without any tethering or wifi link required. (No testing on unicycles while riding on a rooftop, please. [Watch the video to learn the context of that line…])

There’s a passle of other great content in the video. I hope you find it useful!

If you haven’t already, why not sign up for the Beta release and give it a try?

This morning I gave a “Speed Geeking Breakfast Talk” at the Software Testing Professionals Conference in New Orleans.

Speed Geeking talks at STPCon are eight minute lightning format talks. There’s not really any time for Q&A during the talks, but they’re great for laying out important points and starting a conversation after the talk.

My talk was whimsically titled “Brittle Automation or a Sharp Stick in the Eye? Wait, Let Me Think.” My point with the title is that badly done test automation can become such a time-consuming, frustrating task that the pain of it is metaphorically worse than  a sharp stick in the eye.

I thought I’d post up photos of my flip charts along with some quick explanation of the points!

What’s brittle automation?

Brittle automation are tests which intermittently fail for unclear reasons—the epic “Works on my machine” kind of test! Brittle automation tests are those which are high-maintenance and require a lot of time to update when the system under test changes. Note that’s when the system changes, not if! I also lump slow test suites in with brittle tests because it’s an indicator of root cause issues with your testing approach.

What are some of the impacts of brittle automation?

Brittle tests impact your velocity by soaking up time and effort as you run around fixing failing tests. Brittle tests erode your team’s trust in the value of automation, and can even leak over in to eroded confidence in your testers/QA. Worst of all, teams take a huge morale hit playing “Whack a Mole” with the automation suite.

Why do we have brittle automation?

Brittle comes about because we often try to test too much, or make our tests too complex. Duplication is perhaps one of the largest causes of brittle tests, or at least the pain associated with fixing broken tests – you have to touch tens or hundreds of tests when one locator on the UI changes, or one piece of a workflow changes. Manual pauses in functional automation are a huge source of brittleness. Timing delays you craft for your environment will inevitably fail in other environments. Finally, especially in UI-based automation, coupling yourself to hardwired, brittle locators for UI elements (Xpath, for example) is a recipe for pain.

Fixing brittle automation

Save yourself the pain and frustration around brittle automation.The reality of automated testing is that it’s a software development task – you need to use many of the same approaches in your automation work as in the systems we’re building.

Work closely with your developers to learn what parts of the system under test can be leveraged for speeding up tasks like creating baseline or setup data. Collaborate with them on creating great test cases – they’ll be able to help you understand good software design concepts like the “DRY” (Don’t Repeat Yourself) principle.

Above all stay lean and focused on writing only high-value automation. Not everything can or should be automated. Think about what’s the most important piece to automate. Do that, then move on to something else that’s critical.

Wrapping up

Automation can be incredibly frustrating and time-consuming if you don’t step back and carefully consider how you’ll accomplish the work. Avoid brittle automation and instead focus on writing high-value, low-maintenance test suites.

That way you’ll never have to pause when considering a choice between automation and a sharp stick in the eye…