Skip to content

Feed aggregator

Authors in Testing Q&A With Penetration Testing Expert Peter Kim

uTest - 8 hours 42 min ago

Peter Kim has been in the information security industry for the last 10 years and has been a penetration tester for the last seven. He is the author of the becoverst-selling computer hacking book, ‘The Hacker Playbook: Practical Guide to Penetration Testing.’ He was the lead penetration tester for the U.S. Treasury and Financial Management Systems.

In addition, he was a penetration tester for multiple utility companies, Fortune 1000 entertainment companies, government agencies, and the Federal Reserve. He also gives back to the security community by teaching penetration testing courses at a community college, and creating and maintaining one of the largest security communities in the Santa Monica, CA area. He has also spoken at multiple security conferences. You can find him at his blog Secure Planet

In this Q&A, uTest spoke with Peter about some of the more memorable vulnerabilities he has come across while hacking web apps, what he thinks of Apple Pay, and why his book is used in college coursework. Stay tuned at the end of the interview for a chapter excerpt from ‘The Hacker Playbook,’ currently the number one-selling software testing book on Amazon.

uTest: You’ve been in security and pen testing for a while now. Without giving out too many specifics, what was one of the more surprising or memorable lapses in judgment you have come across while ethically hacking web applications?

Peter Kim: I could write a book just on this question. I mean, I’ve seen it all, from a single company having 20+ different SQLi vulnerable public web applications, default credentials into their whole camera system, PII data leaks from major e-commerce sites, all the way to having access into equipment that controlled certain types of SCADA utility networks.

The funniest one I came across was about five years ago. A major AV vendor had all their clients talking back to their central web application over HTTP APIs. Sniffing the traffic, I was able to gain the administrative credentials in clear text from a client. Once I logged into the web application, I was able to modify the update agents within the web interface to force the end user to download a malicious file and execute them on the host systems.

We all had a good laugh, because what was meant to protect the network allowed us to compromise the network, and, ironically, the companies that advocated security had one of the worst IT security practices.

uTest: With all of the data breaches in the news, are organizations not investing enough money in their security strategies, or are they just not investing enough in the right security strategies/programs such as extensive penetration testing?

PK: This is a tough question to answer. I think everyone is looking for the golden egg answer, but it’s much more complex than that.

What I’ve been seeing as the problem is that corporations are becoming tool-dependent. We have host/network-based monitoring, antivirus, malware detection, vulnerability scanners, managed services, application filters, email proxies, and web proxies. Yet, our users are still getting infected with malware, clicking on spear phishing emails, and aren’t able to detect and stop C2 traffic properly.

People focus too much on words like APT, zero-day, PCI, and checkboxes. I’ve worked with security teams where the analysts spent most of their type fighting adware and junk. This isn’t where we should be today, and we should have our analysts focused on identifying anomalies and locking down networks.

With the recent large breaches, like on those Point of Sales (PoS) devices, those networks and systems were only designed for a single purpose. Time should have really been spent detecting any anomalies and alerting on any changes on those systems. If systems are specifically made to do XYZ, it should be very easy to identify and alert when a system decides to anything suspicious.

I also believe we are still failing at user education. This isn’t just the responsibility of the security department, but it should be everyone’s job to be part of the solution. Users need to be able to identify malicious attacks, know how to report these incidents easily, and to stop clicking on malicious email links.

uTest: Do you think programs like Apple Pay are going to be a savior for a retail industry that has been so hard hit with breaches at Home Depot, Kmart and Target, amongst others?

PK: The great thing about hacking is that it’s always about doing what they say is impossible. With that said, what Apple is doing with things like Apple Pay, is in the right direction. By removing the need for third-party credit card number storage, requiring multiple factors of authentication, and not having to hand your credit card to a random stranger for purchases (like at restaurants, grocery stores, and gas stations), it provides many different additional layers of security for the end user.

Just remember that the bad guys adapt just as quickly, if not quicker, than the good guys. So if credit card cloning becomes hard, what about spoofing NFC, what about attacking jailbroken devices with financial-purposed malware, or attacking iTunes accounts associated with your credit cards?

It also really comes down to adoption. With Google going in one direction with payments and Apple going in another, without mass adoption, we might not see the full potential benefits of these systems.

uTest: You’ve mentioned that your book ‘The Hacker Playbook’ has been used as core university materials in some colleges. Could you tell us a bit about which programs it is used in, and where it fits in with the curriculum as an educational resource?

PK: Although the book wasn’t originally developed to be used as college resource, it seems to have ended up aligning with many different undergrad and graduate programs.

Graduate courses like “Advanced Topics – Penetration testing forensics” at George Mason University have incorporated it as the core book for their course. In addition to being added to multiple U.S. universities, it has also been incorporated in multiple universities in other countries (Sheffield Hallam University, Asian Institute of Technology, and Algonquin College). The great part about security is that it isn’t language/culture-bound. Attacks in one country are just as prevalent in another country.

I see this book as a good fit in the advanced network security courses. Whether it is forensics, incident response, or penetration testing, this book gives students a real-world view in what both professionals and unethical hackers are doing. Being able to understand and replicate these attacks allows students to prepare for the types of attacks they’ll encounter in their professional career.

uTest: The book doesn’t read like an encyclopedia – it’s a story walking a tester through the entire penetration testing process from network layer to Web application layer. Could you describe why you laid the book out the way you did, and whether it’s designed for the security rookie or a seasoned veteran?

PK: I’ve read a ton of different security books and they were always laid out by tool or by protocol. I never really came across a book that walked me through an actual penetration test. The other thing I didn’t see too often was a book breaking out of the norm by trying to incorporate and push creative attacks that might not have been fully polished. This allows the reader to continue his/her research and progress their own skills.

The layout was also developed based on needs I’ve had when performing my own penetration tests. Many times, I’ve gotten stuck at a particular point during a test. For example, I might have compromised a host as a regular domain user, but wasn’t able to get that domain admin account. I just pop open ‘The Hacker Playbook’ and skip to “The Lateral Pass” section, and review all of the different options I have. Other times, I get caught up by a certain AV vendor and turn to the “Quarterback Sneak” section and bypass AV.

As the book was originally written as a collection of my lifetime of notes and tips, it’s not targeted for those without any experience. Those that benefit the most are the ones that have played around with Metasploit and Meterpreter. The most surprising part was that a lot of senior penetration testers have come back to me and said that they were really surprised to have learned a bunch of new things from my book. That alone makes it all worth it.

Excerpt from The Hacker Playbook: Practical Guide to Penetration Testing:

Hunched over your keyboard in your dimly lit room, frustrated, possibly on one too many energy drinks, you check your phone. As you squint from the glare of the bright LCD screen, you barely make out the time to be 3:00 a.m. “Great”, you think to yourself. You have 5 more hours before your test is over and you haven’t found a single exploit or critical vulnerability. Your scans were not fruitful and no one’s going to accept a report with a bunch of Secure Flag cookie issues.

You need that Hail Mary pass, so you pick up The Hacker Playbook and open to the section called “The Throw – Manual Web Application Findings.” Scanning through, you see that you’ve missed testing the cookies for SQL injection attacks. You think, “This is something that a simple web scanner would miss.” You kick off SQLMap using the cookie switch and run it. A couple of minutes later, your screen starts to violently scroll and stops at:

Web server operating system: Windows 2008
web application technology:, Microsoft IIS 7.5
back and DBMS: Microsoft SQL Server 2008

Perfect. You use SQLMap to drop into a command shell, but sadly realize that you do not have administrative privileges. “What would be the next logical step…? I wish I had some post-exploitation tricks up my sleeve”, you think to yourself. Then you remember that this book could help with that. You open to the section “The Lateral Pass – Moving through the Network” and read up and down. There are so many different options here, but let’s see if this host is connected to the domain and if they used Group Policy Preferences to set Local Administrators.

Taking advantage of the IEX Power Shell command, you force the server to download Power Sploit’s GPP script, execute it, and store the results to a file. Looks like it worked without triggering Anti-Virus! You read the contents of the file that the script exported and lo and behold, the local administrative password.

The rest is history… you spawn a Meterpreter shell with the admin privileges, pivot through that host, and use SMBexec to pull all the user hashes from the Domain Controller.

Categories: Companies

HP Announces HP Mobile Testing Center

Software Testing Magazine - 9 hours 58 min ago
HP has announced HP Mobile Center, a new on-premise software solution for testing mobile application functionality and performance across real-world network conditions on real-devices. The solution is designed to allow customers to deliver high-quality mobile applications and ensure a consistent, positive user experience under any circumstance. HP Mobile Center is designed precisely with these new realities in mind, giving developers and test teams the ability to launch amazing apps with confidence and speed. The solution provides instant feedback to a DevOps lifecycle team, helping them understand how an app will perform ...
Categories: Communities

SmartBear Launches LoadComplete 3.0

Software Testing Magazine - 10 hours 30 min ago
SmartBear Software has announced a new and rebranded version of LoadUIWeb called LoadComplete 3.0. This new version should help organizations drastically reduce the time required to test and optimize application performance. Customers deploying the new version accelerate load testing cycles and significantly decrease performance testing time. LoadComplete 3.0 helps test business-critical rich Internet and mobile applications in shortened performance testing cycles, providing advanced analysis and reporting, which allows testers to compare results of different tests side by side. Analyzing server and browser side metrics across different load tests becomes convenient as ...
Categories: Communities

EuroSTAR Conference, Dublin, Ireland, November 24-27 2014

Software Testing Magazine - 11 hours 57 min ago
The EuroSTAR Conference is a four-day conference focused on software testing. Global and European software testing experts propose a program full with tutorials and presentations. In the agenda of EuroSTAR Conference you can find topics like “Testing in the World of Startups”, “Testing the New Web – Tackling HTML5 with Selenium”, “Adapting Automation to the Available Workforce”, “Beacons of the Test Organisation”, “How We Transformed the Traditional Software QA by Getting Rid of the Central QA Group”, “Leading the Transition to Effective Testing in Your Agile Team”, “Testing Off-Shore Equipment – ...
Categories: Communities

Proper configuration for running PHP on Nginx

When I started looking into Nginx, I was very impressed by the high performance of this lightweight HTTP server. But more and more I’ve become keen on the ease of its configuration. I have successfully used Nginx for serving PHP applications for quite a while, this article is about the lessons I have learned. I’m […]

The post Proper configuration for running PHP on Nginx appeared first on Dynatrace APM Blog.

Categories: Companies

How to help your recruiters create a great first impression

The Social Tester - 16 hours 5 min ago

At some point during your recruitment drive you’ll likely use recruiters. The problem is that some recruiters are creating a bad first impression of your company. Your recruiters are often the first point of contact a potential hire has with your company. Here are some ideas on how to help your recruiters create a great … Read More →

The post How to help your recruiters create a great first impression appeared first on The Social Tester.

Categories: Blogs

Candy or Death: The Automatic Halloween Candy Dispenser

Radyology - Ben Rady - Thu, 10/30/2014 - 05:59
Let's start with a word problem. Assume you live in a busy trick-or-treating neighborhood and that, on average, a group of four rings your doorbell every minute and takes 1/2 oz of candy per person. If you leave a bowl... Ben Rady
Categories: Blogs

On-Demand Webinar: Managing Automotive ISO 26262 Compliance with Seapine TestTrack

The Seapine View - Thu, 10/30/2014 - 00:13

Thanks to everyone who participated in the “Managing Automotive ISO 26262 Compliance with Seapine TestTrack” webinar. The webinar recording is now available if you weren’t able to attend or if you would like to watch it again.

Share on Technorati . . Digg . Reddit . Slashdot . Facebook . StumbleUpon

Categories: Companies

Testing Examples

Thoughts from The Test Eye - Wed, 10/29/2014 - 22:36

I believe we need a lot more examples of software testing. They will be key in transferring tacit knowledge (they will not be all that is required, but an important part.) They work best when done live, so you can discuss, but that doesn’t scale very well.

So I have created a few examples in video or text format:

Exploratory Testing Session – Spotify Offline

Bug Finding – Spotify Space, Image Galumphing

Scenario Testing – LibreOffice Compatibility

Test Analysis – Screen Pluck

I am very interested in knowing if they are useful to you, and how.

Which other public testing examples are your favorites?

Categories: Blogs

How Sesame Street Can Help You Become a Better Software Tester

uTest - Wed, 10/29/2014 - 22:09

indexAll I really need to know, I learned in Kindergarten.

STARWEST presenter Robert Sabourin – a 30+ year veteran and well-respected member of the software development community – took that nugget of conventional wisdom and put his own unique tech spin on it in his course on Testing Lessons Learned from Sesame Street.

While the topic was fun and lighthearted, Rob took his subject matter seriously and impressed on attendees just how important it is to learn and master the basics. But what are “the basics”?

Let’s take a closer look at what you really need to know to build a solid software testing foundation.

Rob’s presentation focused on two main areas of professional – and personal! – development: cognitive skills and social skills. Developing your cognitive skills allows you to think more analytically, to develop efficient models and lay out precise explanations for your processes and reasoning. Strong social skills elevate your ability to collaborate to a whole new level of effectiveness and can help grow your reputation as a thought-leader.

Think of your team as a “neighborhood.” To be successful, people take on many diverse roles which require them to focus on various short-term goals. You may all be working toward the same end goal, but to get there, you’ll have to be sensitive to diversity and understand how successfully navigating it can enhance the overall quality of the end product.

Much as John F. Kennedy asked in his 1961 inaugural presidential address, “ask not what your country can do for you; ask what you can do for your country,” Rob Sabourin encourages testers to ask themselves and their teams:

Ask not what the system can do for the user; ask what your user does with the system.

Testers often find themselves in the position of defending their work, proving the whats and whys and hows. We can learn a lot from Big Bird and his struggle to help justify the existence of his imaginary friend Mr. Snuffleupagus. How did Big Bird prove he was real? Garnering and presenting proof of his friend’s existence parallels a tester’s role in reporting an issue found while testing a product. It requires persistence, evidence, and continuing advocacy for the bug.

Oscar the Grouch – while seen negatively by many – can actually be a great role model for software testers! Oscar thrives in a messy environment, where there’s plenty of “trash” (bugs!) and he excels at breaking things (knowledge of inducing failure modes). Oscar’s goal is to travel the unhappy path since that’s where he can best employ his talents in disrupting the flow, similar to how a great tester will learn to step outside the comfort of the happy path, dig deep and think creatively to uncover valuable issues.

And, of course, there’s Kermit the Frog. We can’t forget him! Like a good bug-hunter, Kermit is the ultimate reporter; he knows how to observe, blend in, gather facts, and report them in a factual (non-emotional) way. One of the biggest lessons we can learn from Kermit is that “it’s not easy being green,” where “green” is synonymous with the progressive, persistent investigation that makes excellent testers so valuable.

Kudos to Rob Sabourin on creating and delivering an excellent – and enjoyably creative – STARWEST 2014 session presentation!

Want to see the full slide deck from Rob’s presentation? Check it out on the uTest forums here, then answer our poll asking which Sesame Street character you best identify with and chat with other testers about it!

Categories: Companies

Azure Mobile Services with AutoMapper

Jimmy Bogard - Wed, 10/29/2014 - 15:23

At the recent Xamarin Evolve conference, Paul Batum gave a great talk on Azure Mobile Service in cross-platform business apps, part of which included a piece on how AutoMapper fits in with their overall system:

There were sooooo many of those C# shirts at the conference, I felt a little left out without one.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

uTest Announces New Software Testing Career Mentoring Program

uTest - Wed, 10/29/2014 - 15:00

mentoringACEing your work as a software tester just got a little easier.

uTest is proud to introduce the beta version of A.C.E. (Assisted Continuing Education), a new software testing career mentoring initiative beginning November 1. The program will be available to all members of the uTest Community.

The mentoring program is designed to help software testers build a solid foundation of testing education. By honing these essential skills, participants will be well-equipped to grow their testing careers and strive for professional success on many levels. This will be achieved through participation in various course modules, each geared to the software testing professional at various stages of his or her career.

At the November 1 beta launch of the program, A.C.E. will offer the first two modules of the program, How to find valuable bugs and How to write great bug reports. Testers will have the option of signing up for one (or both) of the course modules.

Both courses will consist of a brief independent study, along with a graded homework assignment. The core concepts gained through the two modules will then be tied together in a live webinar with a uTest Community testing expert and Test Team Lead, and a hands-on, live exploratory testing session where participants can practice their new skills.

Sound like something you want to be a part of? A.C.E. is free for members of the uTest Community. Check out the full announcement now (requires uTest login) for further details and to sign up today. Space is limited.

If you’re not a uTest member, you can sign up free of charge to attend the mentoring program, and get access to free training, discussions with peers, the latest news, and opportunities to make money by working on testing projects with some of the top brands in the world.

Categories: Companies

What’s your testing approach?

PractiTest - Wed, 10/29/2014 - 12:04

testing approachToo often we complain about our daily routine. We wake up most everyday of the week and go through the same motions and home and at work. Our mood might change, so we approach our daily tasks differently perhaps, but the routine remains the same. However, this is not a bad thing!

You don’t realize how great routine can be until it is gone or taken away, and I don’t mean because you’ve gone on vacation. We are creatures of habit, and rightly so. Routine gives us structure and makes us feel safe, confident and is comforting. The same applies to our work practices and the routine approaches we use on the job.

This got me thinking about our different testing approaches as testers, and how they dictate the routine of our work. While we probably vary in our approaches, we all have the same professional goal – to do the best job possible, “leave no bug unturned”. So what is your testing approach?

For instance, when you get a new feature you :
1. Go over the documentation for it
2. Run a short exploratory testing session to get to know it first hand
3. Create some high level or low level testing scenarios for it
4. Run your scenarios while taking notes on how to improve them
5. Run a short session with your team to give your feedback on the feature, its stability and some functional improvement ideas
Then, move on to the next feature…

Here is another related example:
Test Plan Recipe for a Mixed Formal & Informal Testing Approach

What are your successfully proven routines?

Categories: Companies

Integrate Ranorex Tests in andagon’s ALM tool aqua

Ranorex - Wed, 10/29/2014 - 10:40

Guest blog post by Stefan Büssemaker, Innovation Manager at andagon GmbH

This blog post will illustrate the interaction between Ranorex and the application lifecycle management solution aqua.

In a few simple steps you can expand your Ranorex test cases with the powerful ALM tool aqua.

What is aqua? What is ALM?

aqua is an ALM (=Application Lifecycle Management System), which is a software product to control and manage IT projects in one solution. aqua is based on five pillars:

  • project management for initialise, planning, controlling and finalising projects
  • requirement management for document requirements, analyse risks and control changes
  • test management for manual and automated software tests
  • defect management for preventing and minimize the impact of bugs
  • reporting to summarise the state of developing process


Requirements, tests and defects are clearly structured by visually hierarchic structure for a comprehensible traceability. aqua is intuitive and easy to use, because it is based on the Microsoft look and feel environment, which everybody knows of products like Microsoft Word.

All components can be specified by attributes for uniform documentation. Test cases can be parameterized with test data for manual and automated tests. Automated test scripts expand test cases with tools like

  • Ranorex: functional record and replay test automation tool
  • JMeter: non-functional load and performance tool
  • SoapUI: functional web service testing tool
  • Databases: for checking database queries
  • Shell: for executing Unix commands


Moreover an agile board is available for working in an agile scrum developing process.

The integrated report module allows the management to create customized reports from the current state of developing process within a few seconds.

The Integration of Ranorex in aqua? And what is an Agent?

Ranorex is completely integrated in aqua, which gives the testers more opportunities to work with Ranorex.

A Ranorex automated test script is stored behind a documented test case in aqua, which represents the abstract functionality.The test case is expandable with other automated test scripts or manual test steps.

This automated test case can be executed by aqua on a distributed pool of agents. An agent is a simple program which bidirectional connects aqua and Ranorex Runtime together on a distributed operating systems.

This results in several advantages.

  • the tester’s desktop is not blocked while test case execution
  • the duration of execution can be divided by the number of agents
  • scripts can be automatic connected with a documentation
  • test cases can be specified and bundled for test management
  • adjustment of test scripts can be distributed to different testers
  • best workload of agents is ensured of aqua


With this functionality of aqua it is possible to bundle selected test cases dynamically into test scenarios for Regression Tests of Unit Tests, Feature Tests, Smoke Tests or Acceptance Tests.

This allows to faster receive the actual quality condition of the system and find bugs early after implementation. With the traceability of test cases, requirements and test cases are easier to review and further to find duplicated or obsolete bugs. The results of executed tests are stored historically in aqua.

Reporting with aqua

In aqua you have two possibilities to report the actual state of the developing and test process.

On the one hand you have a real time dashboards like pie charts, burn down charts or column charts, which can be customized for your needs.

On the other hand you have the possibility to create a customized report with even more details.

You can, for example, summarize your status of all test cases which are connected to all requirements in one sprint or release. Summarized you have the possibility to track your development project in real time and can compile key figures with one click.


Aqua and Ranorex are a great combination between automated functional testing and test management.

It is very easy to expand your test framework with requirement management and defect management.

The test framework scales with the number of test cases and number of testers, which helps to control the testing process quick and easy. The time of monitoring and reporting will be reduced as well as the duration of test cases.

For more information please have a look at the following video:


Categories: Companies

Is DevOps Killing Some Types of Jobs?

IBM UrbanCode - Release And Deploy - Wed, 10/29/2014 - 00:14

At the Enterprise DevOps Summit last week, there was a great deal of conversation around finding the right kind of skills, and taking different approaches to getting work done. Forrester VP and Director serving Infrastructure and Ops professionals had a slide that was rather blunt and received a great deal of attention once tweeted out.


“Roughly speaking if your job title includes ‘administrator’ you’re in trouble.” @glennodonnell #DOES14

— Eric Minick (@EricMinick) October 22, 2014

Clearly a nerve was struck. Listening to the crew at Hangops discuss this, it was clear that the tweet lacked some of the subtlety that Glenn discussed. So let me first try to describe what I heard*. Glenn addressed the fear than many have that they’ll automate themselves out of a job. While some jobs may diminish with DevOps, others will emerge. If you automate yourself out of a job, you will also likely position yourself very well for the next job, often with the same company. Companies that routinely lay off people who deliver huge efficiency gains tend not to prosper. At the same time, those job titles on the left need to prepare for significant change.

My understanding is that while knowing how to tune a thing is still going to be a key skill, it’s going to be equally important to know how to express that tuning in a way that is versionable, and can be applied to dozens, hundreds or thousands of similar systems.

“Dying” is too strong

I meet people all the time who do jobs that a slide like this would have flagged as “dying” ten or twenty years ago. As the Hangops guys pointed out, mainframe development was supposedly dying long ago, but you probably get better paid to be a Cobol developer than a .Net or Java developer. At the same time, there is probably more new code being cranked out in either of those languages than Cobol.

I don’t expect mass layoffs

I’m sure there will be shops that have hired a Sysadmin for every handful of servers who put in place something more scalable and lay off a bunch of admins. Some executive will get a bonus. What I hear though is usually a different story. A team will start to have way more servers than they could manage the old way, probably “because cloud” and learn to automate. Or modern development methods like Agile or Kanban will cause a rate of change that’s intolerable with existing methods. Either way, the volume of stuff we are asking our admins to manage is often growing extremely quickly, and they are forced to look to DevOps techniques just to keep up. The biggest risk is that the admins fight these changes as unreasonable, a rogue group starts taking over bits of their work within the subset of applications that are undergoing the cloud or agile transformations and eventually takes over.

Neighboring Examples

DevOps has been driven partly by Agile speeding up development enough to break operations. Looking at what happened to jobs closer to developers a few years ago should shed some light on what lies in store.

The SCM Team

Ten years ago when I talked to someone who worked on the “Software Configuration Management Team” they usually focused on the care and feeding of the source control system. They helped devs with big merges, crafted config specs, and helped with the nasty build processes. As Agile took over, the trend was towards simpler and lower maintaince source control tools, frequent small merges, and continuous integration tools that ran the builds. Most SCM teams eventually took over the build systems driving rapid feedback to developers, and making sure that a builds were done in a traceable way. Some continued towards continuous delivery and are probably rebranding their team the DevOps Team now.

Project Management

Lots of Gantt loving project managers got lost in the swing to Agile, especially in those shops that shifted away from projects and towards a product view of their applications. At the same time, organized, responsible people who understand the business reasonably well are generally in demand. So we saw a range of conversions. Some retrained and became Scrum masters. Others took roles as business advocates within IT. Others shifted towards release management as that discipline emerged.


See a lot of people following a written test script by hand and recording the results? That kind of tester probably represents the worst case scenario for the “administrators” that Glenn identified as at risk. The low value that many shops put on quality is partly to blame. Those with engineering skill have often found a home writing a lot of test automation, or others are focused more creative exploratory testing.

General rule: work gets more interesting

Jobs that are in danger of being automated are usually the ones that require the least humanity. Machines can follow a script well. People problem solve and use creative judgement. To the degree that you are problem solving something other than remediating errors in a manual process your job is probably secure. Otherwise, it is probably time to begin automating yourself out of your job. Testers provide the map of the road forward. Those who are just skilled at diligently following a script are at the greatest risk. If you don’t automate yourself out of a job, someone else will. The people who build the automation of the future are probably going to stick around expanding that automation, optimizing it, and caring for it. That’s more fun than following someone else’s poorly written instructions anyway.

Ironically, automating yourself out of a job is probably the best thing you can do for your job security.


* Correction: I’m pretty sure Glenn said “watch out” not “you’re in trouble”.

Categories: Companies

Lean, metric-driven product development is the next frontier for all SaaS

Assembla - Tue, 10/28/2014 - 22:56


Many SaaS development teams have enough data about usage to see that "a very large percentage of our ideas are bad."  The challenge is actually using the data to drive a move away from vague product strategies, to directly seeing what customers care about.  They have "Lean Startup" on their minds.  They hunger to deliver work that has significance and meaning.  I had at least a dozen conversations about this topic in the last month.  It's the next frontier for all product development.

Here is the guide that I wrote for my team.  Please feel free to use it, modify it, and improve it.  Our internal version listed pecific examples of time that we wasted, which  have been removed to protect the guilty, which was often me.

....We do a lot of work, and we do a lot of good work. However, that is not enough. We want to do work that is meaningful for our users. We can do less SaaS development work, and deliver more things that people care about. We can use two principles – lean management, and measurement.

Lean management means that we get product improvements finished, released, used, improved, and sold, as fast as possible. This reduces the amount of waste because it reduces the amount of work that we did, that is not being used. Lean management increases focus. We can reduce the number of things that we are working on (WIP), so that we can finish tasks faster.

Metric-driven means that we look at numbers to see if our users care about what we are doing. Are they increasing their usage? Are sales going up? Has speed and reliability improved? It’s not enough to improve the product. We want to improve the product in ways that people care about. If we don’t get increased usage of our beautiful new feature, people don’t care.

It’s easy to ignore this type of data. So, we need to decide to use measurements (metrics) from the very beginning. When we start working on something new, we should have a hypothesis about how our work is going to change our metrics. A lot of the time, even an optimistic assumption about measurable results is not very impressive, and we can decide to skip the task. That’s a good result that gives us more time for important work, or a night out with friends. The remaining work is worth doing. The hypothesis about its result becomes very valuable later on, when it forces us to go back and check the measurements. This will lead us to abandon the initiative (if there are no results), or intensify our work to maximize a good result. Both results will increase the time we spend on meaningful work.


We can achieve our goals if we use these steps in our development process:

1) Find a story owner. The story owner does not need to be a product manager or a designer. The story owner can be a programmer. The story owner is anyone who will take the time to understand our goals, and measure the results. The first step for the story owner is to understand the use case, and create a measurable goal. Then, the story owner will lead development with the skills he/she has (design, or programming and prototyping), and get help from the other team members to finish a complete release. The last step is to unveil, and look at the results. Then, we make a decision to stop working (because people do not care), or to continue making improvements (because people do care). This is how we will get releases and results that are important.

2) Make a roadmap in tickets. We do not need a lot of tickets. However, we need enough tickets to explain the work to new people who work on the story. If we have some tickets, we can work with our distributed team, and we can bring in new team members. We should also make drawings, take pictures of the drawings with a phone, and attach the drawings to the tickets.

3) Do continuous integration.  This is built in to our core SaaS development, but we should be making sure to get it on everything else - mobile and integrations.

4) Do frequent releases. We can use feature flags to hide changes, but we should be doing releases to some group of people.  Please follow our continuous delivery policy and release every change.

5) Unveil and look at the reaction from users, and the measurable results

6) Decide to stop, or to accelerate work on this improvement.


We need story owners. Anyone can be a story owner. Programmers, UI designers, marketing, and QA people can all be good story owners. They only need to agree to do some things:

  • Understand the use case and the measurable goals
  • Work on the story in an appropriate role, like design, or programming and prototyping. Organize development AND release.
  • Measure the response in some way


If we find that a project is taking longer than we expect, we can often analyze it with these two cases:

1) We are distracted by other tasks, and we are not committed to completing this task. In this case, we can stop working on some other tasks, and finish the first things that we started.

2) We don’t know how to do it. In this case, we should stop working on it, and wait until we have a clearer path to success. There are lots of wonderful things that I would like to do, such as cure cancer, that I don’t know how to do. I’m not going to waste time on them right now. The same strategy applies to things that seem much more achievable, like “Improve the user experience”. Of course we want an awesome user experience, and we have a lot of ways to achieve it. However, it is a truly hard problem. I’m not going to ask people to work on it until we can find a plan or vision that is likely to succeed.

Sometimes, we work hard on an important improvement for a month before we realize that we aren’t getting results, and we aren’t sure how to finish, and we are just arguing or asking questions. In this case, EVEN IF THE TASK IS IMPORTANT, we should put the work aside and focus on something that we can finish. Enlightenment will come to us in time.


Categories: Companies

STARWEST 2014 Interview: Qualities Test Managers Seek in Software Testers

uTest - Tue, 10/28/2014 - 20:00

Recently at STARWEST, I caught up with professional software testing team lead Richard DeBarba.

Rich gave some great insight into what qualities he looks for in a team member and how attending STARWEST helps him focus on the most optimal training materials and direction for his team. By attending conferences like STARWEST, Rich is able to keep up with recent trends in software testing and learn about progressive new tools or practices.

So what types of testers do team managers like Rich look for? Check out the video below to see what he had to say!

Categories: Companies

BugBuster Version 3 Released

Software Testing Magazine - Tue, 10/28/2014 - 19:31
BugBuster has announced the general availability of BugBuster v3. This version addresses several shortcomings of BugBuster v2 and includes new productivity tools such as the Scenario Recorder. Using this recorder you don’t need to be an engineer to create in minutes functional test scenarios for your web app. And without writing a single line of code. BugBuster v3 is also packed with some unique features such as content language and spell checking, an application environment manager, the amazing timing insensitive API, form-to-email end-to-end testing and more. The BugBuster Web Testing platform allows ...
Categories: Communities

Ranorex 5.2 Released

Software Testing Magazine - Tue, 10/28/2014 - 19:20
Ranorex is pleased to announce that Ranorex 5.2 is now available for download! This latest release considerably extends the object recognition capabilities of Ranorex for desktop, web and mobile application testing: out-of-the-box test automation for Qt applications on the desktop and in web environments as well as a lot of newly supported web development frameworks… and of course full support for iOS 8. In addition to the newly supported technologies, the out-of-the-box robustness of the Ranorex Recorder has been dramatically improved. Finally, the release Ranorex 5.2 introduces more smart and flexible ...
Categories: Communities

Comparing processing times of NServiceBus saga patterns

Jimmy Bogard - Tue, 10/28/2014 - 19:16

A few weeks ago I gave a talk at NSBCon NYC on scaling NServiceBus, and one of the pieces I highlighted were various saga/processing patterns and how they can affect performance. It was difficult to give real numbers as part of the discussion, mostly because how long it takes to do something is highly variable in the work being done and environmental constraints.

I compared three styles of performing a set of distributed work:

And highlighted the performance differences between them all. Andreas from the Particular team mentioned that some work had been done to improve saga performance, so I wanted to revisit my assumptions to see if the performance numbers still hold.

I wanted to look at a lot of messages – say, 10K, and measure two things:

  • How long it took for an individual item to complete
  • How long it took for the entire set of work to complete

Based on this, I built a prototype that consisted of a process of 4 distinct steps, and each variation of process control to track/control/observe progress. You can find the entire set of code on my GitHub.

Here’s what I found:

Process Total Time Average Median Observer 6:28 0.1 sec <0.1 sec Controller 6:25 3:25 3:37 Routing Slip 2:57 2.6 sec <0.1 sec

Both the observer and controller styles took roughly the same total amount of time. This is mostly because they have to process the same total amount of messages. The observer took slightly longer in my tests, because the observer is more likely to get exceptions for trying to start the same saga twice. But once an item began in the observer, it finished very quickly.

On the controller side, because all messages get funneled to the same queue, adding more messages meant that each individual item of work would have to wait for all previous items to complete.

Finally, the routing slip took less than half the time, with higher total average but comparable median to the observer. On the routing slip side, what I found was that the process sped up over time as the individual steps “caught up” with the rate of incoming messages to start the process.

This was all on a single laptop, so no network hops needed to be made. In practice, we found that each additional network hop from a new message or a DB call for the saga entity added latency to the overall process. By eliminating network hops and optimizing the total flow, we’ve seen in production total processing times decrease by an order of magnitude based on the deployment topology.

This may not matter for small numbers of messages, but for many of my systems, we’ll have 100s of thousands to millions of messages dropped on our lap, all at once, every day. When you have this situation, more efficient processing patterns can alleviate pressure in completing the work to be processed.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

Knowledge Sharing

SpiraTest is the most powerful and affordable test management solution on the market today