In Part I of this month’s Testing the Limits series, Jonathan Kohl talked the life of the Agile methodology, his relationship with and advice about mobile app testing, and how new and season testers alike can advance their skills and feed their passion.

Today, he’ll talk about the biggest hang-ups teams have when it comes to testing, how to overcome those hurdles, how he became a fan of “gamification” in testing and a few other words of wisdom.

********

uTest: When speaking or consulting on mobile application testing, what’s the most common question you encounter and what’s your answer?

JK: This is the question that comes up the most after people have heard me talk about testing on mobile projects:

“Is it true that we need to test on real devices, incorporate movement, and test out in the real world, outside of the office?”

My answer: “Yes.”

Mobile technology has been developed to support movement, and it has a lot of dependencies. For example: networking (while moving you change networks, technology, and encounter lots of errors or weak signal conditions), location services (all about movement and spending time in different locations), weather and environmental conditions (temperature and light have an enormous effect on how some apps work), and movement (the devices have movement sensors, and are interacted with using touch screens and voice control. Combining inputs and sensors getting triggered from movement can be tough for apps to handle.). I know of no tool that helps us sort that out without getting physical with the devices. Hopefully we get better ones soon that focus on mobile technology, instead of treating it like a small PC/web browser. 

uTest: You work a lot with startups, when there’s a shoe string budget how do you communicate the importance of testing and QA?

JK: A successful business requires execution on several dimensions, for example, understanding your customers and their needs, having a great product or service offering that solves real problems for people, having enough investment to enable you to deliver, and a strong grasp of your financials. These require different skills and focus, but there is an over-arching need to create value for your customers, and part of that is great customer service. Technology fits directly into that, especially as we view organizations through the window (screen) of a smartphone or tablet. A poor technology experience is another type of poor customer service, so there is a direct line there. If you let down your customer or provide them with a poor first experience (which we are finding is increasingly on mobile devices), you can lose them forever. So not only do we need to have great people skills and a good strategy for satisfying and impressing our customers in our human interactions, but in our technology interactions as well.

A lot of people don’t realize that we are evaluated on our ability to deliver on the technology front, and that technology is a major part of customer service. It fits into the customer experience, but also in our offering of a product or service. Does the technology help, or hinder? Do we have an awkward experience, or is it seamless. It also has an effect on investment and financials, even if it is indirect. A quality product and technology stack is vital to the success of a business over the long term, so we need to invest in ways to make sure that we meet our customer’s needs there as well. Good quality processes, development and testing are part of that, and we ignore them at our peril.

uTest: Part of your consulting work is to help teams adjust to methodology changes. What’s the biggest hurdle these teams need to overcome and how do you help them do that?

JK: Dealing with people issues – different individual people with different agendas, motivations and fears. On one hand, you have people who have bought the marketing of a methodology and think that it will cure all ills. On the other, you have people who are deeply threatened by the change the methodology represents. It is a difficult balance to reign in the exuberance and unrealistic expectations of one group, while encouraging the laggards to get on board and actually try it out. I spend a lot of time myth busting on both groups. The methodology will not make everything perfect (and it won’t work forever), but it will also not cause you to lose your job, or cause bad things to happen to you. I work with teams by doing – getting in there and demonstrating, and helping address their fears. If I can do it, anyone can do it.

It’s also important to help them be able to measure the success of their methodology adoption in real terms: better product, better customer service, saved costs, etc. Just measuring success on adopting a new process misses the point, and will lead to some difficult situations down the road, particularly if that new process isn’t helping you create value for you, for your team, for the company, and for your customers.

People issues are hard to cope with. They aren’t fun, and it means we have to confront people on difficult topics. However, healthy disagreement and healthy confrontation are good, healthy things. If we suppress conflict, it just goes underground. We reach peaceful solutions through negotiation and co-operation, not by suppressing them in a need to “be positive.” There is also nothing quite like trying something out and seeing whether it works or not. Evidence, generated by a team that is committed to getting results, not just in implementing a methodology change, is powerful, and helps build teamwork and consensus towards reaching goals. Talking is one thing, doing and getting feedback on how we are doing helps enormously.

uTest: You’re a fan of “gamification.” What drew you to this concept and why do you think it works?

JK: My colleague David McFadzean can take the blame for that. David and I were looking at various topics: using uncertainty factors and monte carlo simulations when estimating software projects, game theory ideas for helping teams organize their mix of processes, tools, technology and internal policies, and engagement, particularly with regards to mobile apps. I do work in mobile strategy, mobile app design and product management, and I was working on formulating my thoughts on mobile design and dependencies. David kept pointing out how I had used game theory ideas, or gamification ideas in my work without realizing it, so eventually I gave in and started looking at it more seriously. At first, I thought gamification was childish, but he urged me to look into it more deeply, and I was surprised at what I learned.

David is a video game enthusiast, so we started looking at video game design ideas and how they could be applied to a project we were considering working on. I then discovered Jane McGonigal’s book Reality is Broken, and I loved her game creation, the research she had done regarding video games, and her positive urging to help solve real world problems with games. I’m not sure she would agree with how I implement game concepts on software projects, and how I have worked to make software testing activities more like a game (she’d likely prefer we made a real game out of it, rather than gamify something else), but I owe a lot of my energy and inspiration in this space to her. I am drawn to her positivity and her challenge for us to do better things in our lives and to help harness collective productivity to solve problems.

It works because it is a powerful engagement model. Just look at how much time people play games on mobile devices. The game designers are tapping into something powerful, and they know how to develop great apps people want to engage with. Gamification when done well helps motivate people to return to use our apps over and over, and they feel drawn to the activities within the app. Games help provide intrinsic rewards – Jane McGonigal calls games “hard work that we like to do.” Many times, especially in software testing, we avoid the hard work because there is no inner motivation. We just have to do it because it is our job, or it is someone else’s job. I have had success in the past by making testing interesting and even fun on software development teams. It is not uncommon to work on teams I lead where everybody tests, and sometimes it’s hard to get people to do their other jobs like programming, analysis or project management because they are having fun testing.

Gamification isn’t the only engagement model, and because it became a bit of a buzzword, there is an enormous amount of bad gamification out there. But, if you take the time to analyze it systematically, determine that it will be a good thing to try in your software or system, and put the time in to create it from the ground up, it can be a useful tool.

uTest: You’re quite the prolific writer. What’s your favorite topic to write about – the one you never get tired of?

JK: Anything that helps people look beyond their current problems, to look at the big picture and inspire them to create great software. It’s easy for us to get caught up in good practices and tools, and forget about the people we are developing software for. While tools, practices, processes and things like that are very important, they should all serve our efforts to create great software. We often forget that and put other things ahead of that effort.

One thing I never get tired of hearing from people who read my work, is when it helps them. Especially if it inspired them to change their world for the better, even in some small way. That’s always my goal – to inspire people to do something better, and hopefully my experience resonates with them and they can enrich their own experiences by building on those ideas.

uTest: What new thing/trend/innovation are you keeping your eye on at the moment?

JK: I’m watching mobile or pervasive technology that utilizes movement (gestures, eye movement, etc.) rather than depending on keyboards and touch screens. This is a fascinating space, as devices become more embedded in our lives, and (hopefully) easier to interact with. It is also interesting to imagine the unintended or unwanted consequences. In many ways, we are just scratching the surface of what mobile technology can do for us. Mobile technology has taken a big step forward with smartphones and tablets, and network speeds and reliability are getting there, but have a long way to go yet. It reminds me a lot of the rise of the world wide web in the 90s. There is so much potential for fascinating tools, usage and effects both positive and negative on human behavior and interactions.

uTest:   Any brand new advice or words of wisdom that we can be the first to share?

JK: This is a tough question to answer. I think I will leave you with a thought about the lack of permanency in our industry, particularly with regards to software development processes. As a consultant I see a lot of people working very hard to implement a tool, practice or process change, expecting it to solve a lot of their problems. People will push to get something through, hoping for great success, sometimes sacrificing their health, their emotional well-being, and sometimes even political capital or their reputations to try to get adoption. While that particular solution may help, it won’t last forever. We need to adapt and change, and sometimes it really isn’t worth that personal sacrifice. If the team doesn’t want to adopt a tool or practice or development philosophy, there are most likely deeper issues that need to be addressed first.

The hardest problem to point out to people who are desperate for change is that no one solution will last forever. Everything changes, so this new state is not static and is not permanent. Once you have overcome this hurdle, there will be new ones. Many lottery winners have to go back to work, rock stars have to release an album and go on tour when their kids need an expensive university education, etc. There isn’t a lot of permanence in life, so we need to be open to adapting and changing. Getting to X, whatever X might be, is not a permanent solution. It’s just the best next step of a journey. This also applies to our software development processes. What worked last time may not work next time, so we need to carefully evaluate what we do on the basis of how it helps, and to be able to know when it isn’t helping us anymore. Implementing Scrum, buying a new productivity tool, or creating a fantastic new mobile app isn’t the end of the road for us. We haven’t arrived, it’s just (hopefully) the right thing to do at the time.

In this journey of life, we don’t really arrive until our journey ends at the end. Similarly, on software projects, if we get arrogant or complacent and think we have arrived and don’t need to adapt, we are probably already dead. We have to constantly change and adapt to new technology, to market and economic forces, to evaluate new tools, practices and processes and adapt our mix of those elements frequently, or else we will fall behind.

Our Testing the Limits guest this month is Jonathan Kohl, a consultant and technical leader who writes and speaks on a wide range of software development and testing topics.

In this interview we talk to Jonathan about his passion for the field, what’s changed over time, his take on mobile app testing and his advice for new and seasoned testers. To learn more about Jonathan visit his website or follow him on Twitter @jonathan_kohl.

********

uTest: I think it’s safe to say that you’ve jumped into the world of software development and testing whole-heartedly. What drew you to this field and how do you stay passionate about it?

JK: The bottom line: I love to build things, and creating great software with a talented team is incredibly rewarding. Knowing that we have created something that helps make people’s lives easier is gratifying. I have found an ideal mix of problem solving, technology, creativity and satisfying and impressing customers in the software development industry.

I have great friends in the industry, and we keep each other sharp, and work on side projects together. We can talk about the latest technology, tease each other about programming tools we like and dislike, or branch off into just about any topic when we’re together. We also serve as a great support network for each other when someone is struggling, either technically, or personally. When we are working together to solve a difficult problem, nothing describes that energy of collaborating and working past your weaknesses, and that triumph of shipping working software at the end.

I ended up in this field by accident – I got lost on my way to law school. In the mid 90’s, I had three university professors who were incredibly influential: Dr. Michael Kubara (Philosophy), Dr. John Rutland (Management) and Dr. David Cowan (Mathematics). I was taking logic and philosophy classes from Dr. Kubara, and he told me to talk to a Math professor because I was delving into territory that he didn’t have expertise in. Dr. Cowan was teaching me Linear Algebra at the time, and jokingly said: “What are you doing with this ancient logic when the real research is in computer science?” Dr. Rutland had us study Data General for his class, and I was reading Tracy Kidder’s book The Soul of a New Machine.  The quirky nature of the technical people in the book appealed to me (I was playing in a band at the time, and read legal journals and philosophy papers in my spare time, yes, I was pretty quirky myself) and I loved the elegance of logic, and the questioning nature of philosophy. I agreed to try out a computer science class, and ended up getting sucked into an internship at a software company not too long after learning to code.

The software company reminded me of what I had read in Kidder’s book Soul of a New Machine, particularly the interesting, intelligent technical people. I enjoyed hanging out with them, and working on really difficult problems. I loved the fast paced, work-hard, play-hard culture of a software startup. I’ve stayed here ever since, with a brief trip back to university to finish off my undergrad degree. Once in a while I am tempted to pursue the law degree again, but I am invariably sucked back into a great project or opportunity.

uTest: You were an early adopter of Agile and later started talking about “post-Agilism.” What are your thoughts on the Agile movement over the years? (Since you could probably write an entire book on this topic give us the dust jacket version.)

JK: I started looking into Agile methods when I was working on a team in 1999 that was trying a combination of Rapid Application Development (RAD), Open Source, and iterative/incremental models like Spiral. The lead developer sent us this article called “Chrysler Goes to Extremes” which described the very basics of Extreme Programming. We liked that it was open and free (we didn’t have to purchase some tool chain and coaching we couldn’t afford) and we started trying some of the ideas out. We then bought the book “Extreme Programming Explained” by Kent Beck, and started implementing XP as best we could. A year later I was introduced to Scrum, and we had good success with it. After a while, I started noticing failures on Agile projects, and people moving on and doing other things. That was fascinating to me. Sadly, instead of learning why there were failures, there was an overwhelming urge to suppress them, or worse, dehumanize people by blaming them for doing it wrong.

As for Agile methods in general, I am ambivalent.  I am glad that lightweight methodologies are much more common place than they were in the ‘90s, and we have benefited from creating a common language around practices, and our tools are so much better now than they were ten years ago. I enjoy working on many Agile projects, since they fit my process and personality, and how I like to work.

However, there is a lot of snake oil out there, with proponents claiming that merely adopting Agile methods will lead to a successful business (What about having a great product, great customer service, skilled people and strong financials? Don’t those sort of matter too?) That really turns me off.  Furthermore, for years, any Agile failure would inevitably involve blaming the victim: you did it wrong, you don’t get Agile, if you were really Agile it would have been a success, which often sounds like another variation on the “no true Scotsman” argument fallacy. Sure, sometimes people fail because they made some mistakes, or didn’t commit, but every Agile project that fails can’t be blamed on the people.

We should elevate people above processes, practices and tools, not make them subservient to them. The people come first, not the process. Also some of the Agile gatherings feel very religious to me, with conversion stories, wild eyed claims that aren’t questioned and lots of zealotry. Those sorts of meetings make me feel very uncomfortable.

There is also a lot of rigid thinking about implementing Agile methods themselves – do it by the book, or the way the coach describes, and once you have reached a level of mastery, only then may you adapt. That smacks of a priesthood, and that bothers me. I believe teams know what is best for themselves, and should be self-determining in a mix of tools and processes that helps them create valuable software for their customers, value for the shareholders and owners of the company, and value for their teammates and themselves. No one should have the right to dictate how you do that. It’s a personal decision, and in an industry with so many organizations, people, fields and different contexts, there is no such thing as a “one size fits all” solution. By all means gather expertise and learn from people with more experience and different ideas, and really try out concepts instead of just prefixing “Agile” in front of whatever you are doing, but don’t stop there. The world is not a static place. What worked last release may not work the next one, and you have to adapt in this business or fall behind.

Post-Agilism was something I observed that reminded me of post-modernism in architecture and art. Modernism is more rigid in rules of how things should form, while post-modernism rebels against that restriction, and people use combinations or mashups of styles to solve a problem. The problem is solved in the end, just by different means. We see this in music, where much of popular music is a fusion of what has come before, and in much of the food we eat, which is a fusion of many cultures and tastes. I saw this group of people who were grasping their own freedom to create value, not just follow a process. They were not being dictated to by consultants, coaches and books, and were taking information and ideas from many sources creating mashups of processes, tools and technology to create great software. They often had high morale and a good deal of creativity and retained staff, so they were also creating value for themselves, something that gets overlooked if you just think of value as “business value.”

That was interesting to observe, especially at a time when “being Agile” was held up to be the most important thing, and people were losing sight of why Agile was supposed to help us. I felt so strongly about this, in 2009 I presented a keynote at the Better Software conference: “What’s More Important: Being Agile or Creating Value?” I’m happy to see “value” as an overarching concept now on Agile projects, but we still have a ways to go. (There is more to value than just “business value” for example.)

Post-Agilism, as I observe it, is about freedom, about giving you and your team permission to experiment and figure out your current mix of processes, practices and tools to help create value. There is nothing wrong with Agile methods out of the box if they work for you. But if they don’t, or they stop working for you, don’t be afraid to experiment and find your own way, no matter what the Agile experts say. If what you do works, but your Agile coach doesn’t like it, seize your freedom anyway. Post-Agilism is about using what works for that release, and jettisoning what doesn’t.

That way of thinking is heresy to some of my Agile friends. They are great people, but they prefer if you do things the Agile way. I like to see a mix of old practices, new practices, and things in the middle on a team that is doing well. If whatever weird and wild combination of what they are doing works, that’s great. If it works and I haven’t seen it before, that’s even more fascinating. I don’t need to call it Agile or something else. It’s your process, name it what you want. Many of my Agile colleagues are more purist, like the modernist architects. It seems like they are saying: “We want you to create valuable software, but we think it is better if you do it according to the commonly accepted practices of this community, using these approved tools, techniques and practices.” While they might be right in many cases, I want to let people know they have the freedom to choose what they want as well, even if it isn’t something I or my colleagues personally like.

One of my colleagues told me about a team that had implemented Scrum, but were having some quality issues, so they implemented software inspections as well, a very old school tool. They had great success with this mashup or fusion of old and new. My Agile friends are rarely very supportive of this sort of thing, (it isn’t Agile) but I think it’s great. I love to see creative solutions that teams use to create great software, and value for themselves as well as the business. There is so much we can learn from others who do things differently. Furthermore, new technology changes the dynamics, and Agile methods were created in the 1990s, so some new technology projects find they need to adapt processes as well because they may not have the easy fit with Agile process or tool support they had when developing with older technology.

Another area of contention in the Agile movement for me is testing. Many Agilists think manual testing is a bad thing, that all tests should be automated. I prefer a blend, especially when humans are the customers of the software we develop, we need some human involvement, since they use the software. Especially with pervasive computing, like mobile technology, movement and human reactions to the technology are very important to capture, along with tech solutions like test automation. The technology augments, it can’t replace. I have also worked on systems that bridged different computing systems, and most of the testing was automated – our customers for our software were other computers. We still had some humans doing work though.

If labels matter to you, Post-Agilism has a home for you if the Agilists don’t like your weird mashup. If the Agile world works for you, then more power to you and your team. If you are doing something that defies categorization, then that is even better! Take pride in your individuality and uniqueness, don’t feel like you have to follow the crowd. The software we create should be valuable for all the stakeholders – that’s the most important thing, and all processes should help serve that, not the other way around. Be different. Be you. Do some amazing things, and then tell the rest of us about it.

uTest:  Your eBook Tap Into Mobile Application Testing is extremely thorough and easy to consume. What motivated you to create this ebook?

JK: Thank you for the kind words. I couldn’t find a book that dealt with testing modern touchscreen smartphones and tablets. I had experience, and a lot of people asking for help, so I agreed to write a book to help people who are either transitioning to testing mobile from traditional projects, or people familiar with mobile technology, but new to testing. I also created a 2 day training course for the same reason, to help spread knowledge and experience for those looking for a head start.

My wife Elizabeth was the primary motivation behind the book. She really encouraged me to just do it. I had started a book on Agile testing back before I started to lose my faith in Agile methods. I found that if I deleted the word “Agile” out of my manuscript, it was really about testing in different environments.  The key with being a great extreme ironing practitioner isn’t about where you do the ironing, someone has to teach you how to iron clothes first. I wanted to talk about the nuts and bolts, not the environment. That wasn’t my passion – that was someone else’s book. Next, I tried to write a book on exploratory testing, but that’s really a topic I thought James Bach or Cem Kaner would be better suited to tackle. I lost motivation on it and it sat there. Finally, I had a co-authoring project fall through after a lot of effort. That was quite painful, so I wasn’t sure I had a book in me or not. Elizabeth really encouraged me to just write, so I did, and soon I had 400 pages of material. I hope people find it useful.

uTest:  If someone is looking to become a mobile app tester where do you suggest they start?

JK: I would look at resources like my book, look at online forums, discussion groups and articles on blogs you can find, and try a Weekend Testers virtual meetup when they have a mobile theme. Weekend testers is a fantastic place to start and learn. Also, consider contributing to open source projects, they always need help, and it’s a great way to build your experience.

On your own, watch your own mobile usage, and for anything that you find confusing, or note specific examples of when mobile technology lets you down. The systems and software should work for us, not make us feel confused, stupid or frustrated. Combine testing (careful observing and evaluating) with what you know already, and learn about the technology so you understand why or how it can fail, and note anything that bothers you and let the team know about it. After all, James Bach has said, a bug is anything that bugs you. And if it bugs you, it will bug lots of other people.

uTest:  On the flip side, if someone has been a software tester for decades, what’s the best way for them to keep their skills sharp?

JK: It’s important to develop an awareness for new technology information, and just try things out. Don’t be afraid to say you don’t know something, and to get in over your head and ask for help. Testers often take a wait and see approach, because we get brought in to projects once technology questions have been settled, but that means we often get left behind, or aren’t consulted early on in projects. I do the following:

• Read continuously about new technology and announcements.
• Encourage my techie friends to share interesting tech stories with me, and I share with them.
• Try something out that interests me – download a tool or try something out in a store.
• If I get stuck, I ask for help. I have a great network of friends with diverse skills, so I can usually find someone to help bail me out, and then we learn from it.
• Strategically position myself to work on new technology projects that I find interesting.

There’s more! Read Part II of the interview.

I would imagine that completing a software project ahead of schedule is like being told by your doctor that you’re too healthy, or having your accountant tell you that you have too much money, but apparently it does happen. The forums on Dice.com prove it.

So what should one do if they find themselves in this situation? Play Spider Solitaire? Chit chat with co-workers? Name all the South American countries on Sporcle.com? While those ideas are well and good, there are slightly more productive ways of spending your free time. Here are a few suggestions:

Make sure you’re really done! Of course, in the broader sense, there is no such thing as “done” when it comes to software development projects. Certain tasks might be done – iterations even – but there is always work to be completed. This especially true if you’re working in an Agile environment or adhere to continuous integration and testing. Here’s an excerpt from a recent Gartner report on agile development on the importance of properly defining “done”:

It is important that the definition of done be comprehensive. Because of agile’s focus on finding the simplest solution to the problem, there is a tendency to try to have no design, end user or operation documentation. This should be addressed by including the required documents in the definition of done. Done can include a review by stakeholders such as architects, database administrators and security/compliance officers.

Consider the corner cases. This one is specifically for testers. It can be tempting to complete a set of test cases and consider it a job well done. But as others have pointed out, many serious issues won’t be discovered this way. As Michael Bolton once wrote on our blog:

“Real testing, to me, should be based on investigating how the software allows people to deal with what we call ‘exceptions’ or ‘corner cases.’ That’s what we call them, but if we bothered to look, we’d find out that they were a lot more common than we realize; routine, even.”

Figure out what’s next. This is for the independent contractors out there. This very honest answer comes from the Dice.com thread mentioned earlier. Take a look (emphasis added):

All of my projects in the last two years have been completed on time, ahead of schedule or way ahead of schedule. As a consultant working for a client finishing a project early can end my gig early. Try looking within the company for the next project to manage or at least get involved in some existing project as a facilitator. The bottom line is you want to do the best possible job for your client, even if that means you’re looking for another engagement sooner than you had planned. If you just keep busy you hurt the client and yourself. Keep your mind fully engaged and active. Even if the client doesn’t mind paying to keep you on hold until another project comes along, unless you’re in your cubicle studying for a certification or earning PDUs, you’re going to disadvantage yourself in the end.

What do you do when a project is completed ahead of schedule? Let us know in the comments section below.

Earlier this year the clever minds at uTest Labs launched Applause – a mobile app analytics product that crawls reviews in the Apple App Store, Google Play, and now Windows Phone Store to provided developers, marketers and brand owners with insightful, actionable, explicit data about app user sentiment.

Since then, Applause has received quite a bit of attention. It is currently a finalist for: the 2013 CTIA E-Tech Awards in the “Enterprise Solution – General Business” category; the 2013 MITX awards in the “Most Insightful: Big Data and Analytics Innovations” category; and the 2013 Stevie Awards in the category of “Tech Innovation of the Year (up to 1,000 employees).”

Well we’re proud to add to that with some great news: Applause has been selected as a “Cool Vendor” by Gartner in the *”Cool Vendors in Mobile Marketing, 2013” report. The recognition comes less than two months after the launch of the Applause Index, the first-ever stock market-like index for the mobile apps economy, and just a few weeks after Applause expanded to begin crawling apps in the Windows Phone Store.

Gartner’s “Cool Vendors in Mobile Marketing, 2013″ “reflects a focus on the areas  Gartner believes form the foundation for any good mobile marketing strategy: platforms, analytics and the ability of a brand or service provider to tie online and offline worlds via mobile devices. Each of the vendors profiled in this year’s research enables or provides one or more of these areas.”

As we know here at Blog Central, mobile apps play an increasingly vital role in how users consume content, conduct commerce and interact with companies.

”Mobile marketers, mobile app developers and digital marketing leaders all have a substantial stake in understanding user engagement with their mobile apps. Flying blind or looking at download metrics alone isn’t sufficient for managing growth and monetization of mobile apps,” stated Gartner in the report.

 “In today’s apps economy, users expect apps to perform flawlessly and intuitively – the first time and every time,” added Doron Reuveni, CEO of Applause’s creator, uTest. “Applause fills a massive blind spot for brands enabling them to go beyond simple download counts and star ratings in order to make informed decisions about how to improve app quality and user satisfaction.”

On behalf of everyone at uTest and Applause, we’re grateful to the folks at Gartner for this esteemed recognition. As always, keep your internet tuned here for upcoming Applause news!

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose

*”Cool Vendors in Mobile Marketing, 2013” – Mike McGuire and Jake Sorofman, April 26, 2013

You’ve talked; we listened. Over the past few months our engineers have been hard at work tinkering on the latest features for the uTest platform. Based on your feedback we’re excited to announce that we’re now rolling these out. Starting earlier this month, we began the first of a few big rollouts to bring you a more streamlined uTest than ever before.

If you haven’t noticed already, be on the lookout for:

• New Streamlined Organization and Navigation – We’re retiring projects as an organization method and now enable you to assign your internal team members at a product level.
• A New Start Screen – When logging into uTest you’ll now be greeted by the new Products page. Be on the lookout for additional changes here soon.
• Improved Search – You can now search for Product and Test Cycles from anywhere within the uTest platform.
• New Team Management Controls – Teams can now be created and assigned at the company level.

Coming soon:

• A brand new testing scorecard view
• New filters for sorting issues
• One-page test cycle description reviews
• New ways to manage and select your favorite testers
• The ability to customize your notification preferences

Learn more and stay up to date about the latest features >>>

And of course, if you’re new to uTest and don’t know what this all means, be sure to check out our product tour and intro video.

Are you a Linux system administrator? Are you running a kernel that’s newer than at least 2.6.37 (or 2.6.32 on CentOS)? Then you might want to pay attention. A newly discovered kernel vulnerability allows users to escalate their privileges to root level. That means that anyone who has access to the command line can gain root access on just about any recent Linux system.

Ars Technica outlines some of the major issues, while Reddit has a deeper and more technical explanation. Here’s a quick summary:

Deep inside the kernel, in the performance counters subsystem, is a rather innocuous looking signed integer variable. However, in other portions of the code, the same variable is treated as an unsigned integer. The problem is that it’s possible for a user to provide a very large unsigned integer to the process, and when that unsigned integer encounters the signed integer, it gets transformed into a negative number. That means that while I might input BIG_NUMBER, once it finally percolates through the code it becomes -DIFFERENT_BIG_NUMBER.

If you’re a former C developer, you’re probably starting to see how this goes wrong. This particular integer just so happens to be used as an array reference, and C is perfectly happy to reference anywhere in memory an array reference says to look. A clever attacker can use the incorrect negative index to write data into invalid portions of memory which are eventually executed by other processes. Both the bug and the exploit are pretty classic and standard stuff.

The fix is remarkably simple: change the signed integer to an unsigned integer. Most people will need to contact their Linux vendors to get an updated version of the kernel, while those who are true diehards can of course compile a kernel themselves. Either way, if you manage a Linux machine of any kind, you should definitely upgrade as soon as possible. An exploit is already floating around in the wild.

More details are also available from Red Hat. This is CVE-2013-2094, for those of  you keeping track of such things.

Over the years, we’ve thrown a considerable amount of nouns, verbs and adjectives at the topic of development methodologies. We’ve focused mainly on that of agile and waterfall, as they are not only the most popular, but they are (in my opinion) the two most polarizing approaches.

We’ve examined and listened to arguments in favor of one versus the another – agile is more fluid and responsive, waterfall is more practical, agile people are flakes, waterfall people are dinosaurs, etc. etc. etc. We tried to be neutral, giving equal weight to the arguments of both sides. Our aim was to educate you – dear reader – in the hopes that we could help you make the right decision when it came to choosing a methodology.

What a waste of time that was! Well, only if you believe that the war of methodologies are over - an idea that was raised in a recent article on PCadvisor.com, and one that I felt was worthy of further consideration. Here’s the gist:

The wars over development methodologies — agile, XP (extreme programming), waterfall, and so on — are  fast giving way to a more fluid and flexible approach to producing and refining  a product. Telerik’s Semeniuk is one of many in the modern development world who  sees development methodologies not as dogmas to be followed to the letter, but  toolkits to be raided for what’s useful. Confining a development team to one methodology is becoming a thing of the past.

In other words, if you are a methodology fundamentalist – that is to say you never waver from the playbook – then you are likely missing out on a number of tactics that can improve the development (and testing) process. Case in point:

Some, however, caution that agile can’t simply be sprayed onto an existing  development process. A former program manager who has declined to be named but has five years of experience as a Scrum master has time and again seen agile used in development, but with no corresponding changes in other facets of  bringing software to market.

“There’s no intermittent QA; instead, there’s old-school ‘toss it over the  wall to QA’-style QA,” he says. “Instead of regular releases, they’re using agile to get a release out, then having the schedule disrupted by support.” In  his purview, there has been a battle between traditional software releases and agile, with a lot of people simply using agile merely to drive old-school models.

So what now? On the one hand, you can continue to stick to the rigid rules of the methodology of your choice. Or you can accept the notion that every methodology has something of value, and to borrow those elements as they are needed.

What are your thoughts? Do you believe the methodology wars are coming to an end?

Please sound off in the comments section below.

How do you make a ‘killer’ application? This was the question uTest’s CMO, Matt Johnston, addressed yesterday on NBC’s Press: Here in the San Francisco Bay Area.

For those unaware, Press: Here is a Sunday morning roundtable discussion show featuring the top names in Silicon Valley’s tech industry. Yesterday, Press: Here host, Scott McGrew, along with Fortune writer, Michal Lev-Ram, and eWEEK editor, Chris Preimesberger, asked Matt about uTest’s expert-sourcing model, the apps economy and the factors that make an app stand out and succeed. Check out the video here:

How many of you software testers secretly wish these “toys for testers” really existed? (Now how many of you are scheduling time this weekend to paint your Newton’s Cradle like bugs?)

From Cartoon Tester:

If anyone creates Software Bug Twister be sure to stop by the uTest offices so we can play!

Since the age of the smartphone, our society has replaced innumerable activities in favor of their digital counterparts. It’s difficult to think back to the way things were before these mini-super computers were on hand at all times. Before the modern cell phone or smartphone, cellular devices were used to make calls and that was about it. Eventually we began seeing ringtones, games, cameras and text messaging being added to the earlier devices. These building blocks allowed for more sophisticated mobile operating systems to reach the point they have today. As a result, our smartphones have all but done away with traditional versions of:

1. Alarm Clocks – The days of slamming your fist into the audible beacon of the waking world have been dwindling into obscurity for years now. Your smartphone can be programmed to go off to any song you desire at multiple times to allow for maximum snoozing. Alarms can even be set to go off based on the user’s location, so if you’re getting some shut eye on the train, you can set that Prince ringtone to go off when you reach your stop. It’s a pretty amazing world we live in when you can reign supreme over the infamous sandman.
2. Newspapers – Almost every aspect of printed media has made its way to our tapping fingertips, which means the newspaper has migrated in a big way. Developers now create apps, e-readers, RSS feeds and social media tools to push information to the masses. Nearly every major newspaper or magazine in the country has made their content available on mobile devices at this point. At long last, you can read the latest issue of ‘Grapes Monthly’ when you’re not within throwing distance of the nearest magazine stand.
3. Asking “What band is this?” – Let’s be honest, smartphones replaced the Walkman a long time ago, but what improvements has it made to music on-the-go? No longer simply playing content loaded on the device itself, we’re able to stream almost any form of aural media, from music or podcasts to audio books or even police scanner feeds. The most mind-blowing aspect (in my opinion) is the app ‘Shazam’ which prompts you to hold your phone to the sky so it can analyze the waveform and spit back the song artist and title. You can then choose to purchase the newly identified song and stream it immediately, how convenient.
4. Postal Service – Gone are the days of paying bills by mail and writing checks now that everything can be done through your smartphone. When someone might have sent a postcard in the past, now they can Instagram a picturesque landscape and tag the other person in the photo as a “Wish you were here” gesture of the utmost impersonal degree.
5. Losing Weight – Smartphones have a bevy of weight loss apps that help you track your run times, distances and calories burnt. Logging the food you eat and your daily workout regimen is a huge win for those who want to get in shape and stay fit.

Staying on top of almost every aspect of your daily life has been made easier by orders of magnitude with the help of smartphones. This can sometimes be a vast improvement or a burden, it all depends on your perspective. I think it’s fair to say we’ve welcomed these streamlined apps and updates for the betterment of all humanity. The future is looking so bright, you may need to bring your shades, or maybe there will be an app for that?

What if most of the negative stereotypes associated with software testing were actually those of software checking? In other words, what if most people’s view of testing was completely wrong? Before one could answer those questions, they would first have to know the difference is between testing and checking.

Michael Bolton and James Bach have an idea or two on the difference between testing and checking. In fact, the two co-authored a post on that very subject for the Satisfice blog. I highly encourage you to read their post in its entirety, but I did want to briefly highlight their formal explanation, as I think it will be a subject we come back to in the near future. Here is how they define it:

Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference.

Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

And then, to give a clearer picture, they offer some key distinctions/assumptions:

• Testing encompasses checking, whereas checking cannot encompass testing.
• Checking is a process that can, in principle be performed by a tool instead of a human, whereas testing can only be supported by tools. Nevertheless, tools can be used for much more than checking.
• Testing is an open-ended investigation– think “Sherlock Holmes”– whereas checking is short for “fact checking” and focuses on specific facts and rules related to those facts.
• Checking is not the same as confirming. Checks are often used in a confirmatory way (most typically during regression testing), but we can also imagine them used for disconfirmation or for speculative exploration (i.e. a set of automatically generated checks that randomly stomp through a vast space, looking for anything different).
• One common problem in our industry is that checking is confused with testing. Our purpose here is to reduce that confusion.
• A check is describable; a test might not be (that’s because, unlike a check, a test involves tacit knowledge).
• An assertion, in the Computer Science sense, is a kind of check. But not all checks are assertions, and even in the case of assertions, there may be code before the assertion which is part of the check, but not part of the assertion.
• These definitions are not moral judgments. We’re not saying that checking is an inherently bad thing to do. On the contrary, checking may be very important to do. We are asserting that for checking to be considered good, it must happen in the context of a competent testing process. Checking is a tactic of testing.

As I alluded to in the beginning, I find it interesting to note that many of the negative stereotypes of testing actually apply to what Bolton and Bach define as checking, for instance:

• Testing is a not a boring task. Checking, on the other hand, very well could be.
• Testing is not easy. Checking, on the other hand, is sometimes quite easy.
• Testing cannot really be done by computers alone. Checking can (and probably should) be automated.
• Human testers will never be obsolete, but human checkers (for lack of a better word) might be.

Do you have your own definitions of testing and checking? If so, please share them in the comment section below.

Until next time, happy testing (happy checking, too).

Fringe use cases and obscure bugs are always going to pop up – even the most intense testing won’t necessarily turn up every little issue. Sometimes the tiniest mistakes made during development can turn into embarrassing, public bugs. To catch these bugs before launch testers would need to be very, very meticulous about testing every combination of actions (something that’s just not practical) or pour over every line of code with a sharp, critical eye. Here’s the most recent real-world example:

Sophos’ Naked Security blog dissected a bug in Apple iMessages that has conspiracy theorists going crazy. If you send a message reading “I could be the next Obama” with an extra space at the end of the sentence the message will be delivered with the word “Obama” missing. As it turns out, the bug is likely the result of a very tiny mistake in the code.

The most credible explanation I’ve seen is that the code that presents the message reckons that it will just fit on one line, and prepares a one-line bubble for the purpose.

But the code that actually formats the message reckons that it won’t quite fit on one line, and thus renders it with the last word on a second line.

In short, the word Obama is there; you just can’t see it.

You can imagine how this might happen: a bug that’s a relative of what’s called an off-by-one or fencepost error, because a fence that is X sections long actually needs X+1 fenceposts to finish it off.

Here’s a visual example of this sort of programming mistake:

1 2 3 4 5 6 7 8 9 /* Is this a one-line case? */ if (pixelsize(msg) <= pixelsononeline) {    specialonelinemessage(msg); } . . . /* Later, processing the message */ if (pixelsize(msg) >= pixelsononeline) {    multilinerender(msg); }

The two comparisons have a nasty discrepancy.

The first considers it a single-line message if it’s no longer than number of pixels available on one line.

The second test looks very similar, but expressed the other way around: it’s checking that the message won’t fit on one line instead of that it does.

But the opposite of “less than or equal to” is “greater than,” not “greater than or equal to.”

In our synthetic example, only a message that is exactly the same pixel length as a line will be treated differently by the two code fragments and trigger the bug; all other messages will be handled correctly.

By the way, that’s one reason why software testing is hard.

In this case, for example, it isn’t enough just to test lots of different messages of randomly-varying length; you also need a structured test where you generate and test messages at all possible pixel lengths.

Read the full article at Sophos >>>

It’s a little mistake that testers were not likely to find. We’re not talking a specific number of characters that’s triggering the bug, it’s pixel length so finding the bug would have been the result of incredibly meticulous testing or a totally random coincidence.

Don’t start hyperventilating and panicking that your testing isn’t good enough. I highlighted this story to remind everyone that QA doesn’t mean software is perfect and bug free – that’s not the point of quality assurance. While QA serves a slew of different purposes, when it comes to testing and bugs, make sure you have people and practices in place that will help you find as many bugs as possible before release. Just as importantly, fix as many of these bugs as possible before launch. More bugs will always appear once your software gets into the hands of users – if you are aware of and/or have fixed as many bugs as you can before launch you’re in a much better position to address these new comers.

Imagine that you stumbled upon a casino slot machine containing a software bug like no other. A glitch that could make you rich in minutes. Would you pull the lever and take the money? If you did, is that a crime?

This is what federal courts are debating after a Las Vegas local named John Kane uncovered a firmware bug present in 10 casino machines. The glitch had been hidden for seven years.

The casino, known as the Silverton Casino Lodge, housed these faulty machines and had been suspicious of Kane for quite some time. On his last day reaping the benefits of the faulty software, Wired’s Kevin Poulsen, says Kane scored some big wins. In about one hour he scored five jackpots, and after Kane didn’t collect his last win of $8,200, the Gaming Control Board removed the machine and took it to the lab for testing. According to Poulsen:

“Now Kane and the bug he exploited are at the center of a high-stakes legal battle before a federal judge in Las Vegas. The question: was it a criminal violation of federal anti-hacking law for Kane and a friend to knowingly take advantage of the glitch to the tune of at least half-a-million dollars? Prosecutors say it was. But in a win for the defense, a federal magistrate found last fall that the Computer Fraud and Abuse Act doesn’t apply, and recommended the hacking charge be dismissed. The issue is now being argued in front of U.S. District Court Judge Miranda Du, who’s likely to rule this month.”

This incident has to be having a major effect on slot machine-makers worldwide. How many more of these real world bugs exist? Can the Gaming Control Board get to them before users do?

With any software comes the usual bugs, and as developers increasingly use higher technology for slot machines, the bugs themselves are becoming more advanced. Cases like this remind us of the immense value that real world testing provides. It’s near impossible to replicate a real world software bug inside the lab, and these are bugs that users can and will stumble upon if developers don’t catch them first.

Slot machine makers need to reconsider their current QA processes; Who is testing their machines?  If developers are testing their own work – or worse, if they aren’t doing any QA – they need to make changes fast.

What do you think of John Kane’s case? Is it a crime to discover a slot machine bug and make money off of it? Share your opinion in the comments section.

What a week it’s been for us down here in air-conditioned Orlando, Florida. Once again, uTest was a proud sponsor of STAREast – one of the premier software testing conferences in the known universe. There’s a lot to love about the event, but if we were forced to narrow it down to 5 things, they would include:

Tutorials: Often times, conferences feature nothing but quick sessions that give you a view from 40,000 feet. Not so much at STAREast. I was lucky enough to attend some of the full-day and half-day tutorials earlier in the week. First up was the Rapid Introduction to Rapid Software Testing by our good friend Michael Bolton. He addressed some the remedies for test teams that are under tight deadlines, but who might also have limited information with which to test. Informative and entertaining, as always. Other notable presenters, who also happen to be former Testing the Limits guests, included James Bach, Scott Barber, Jon Bach, Karen N. Johnson, Dan Bartow and Cem Kaner.

Sessions: Offering the best of both worlds, STAREast 2013 had a terrific line-up of one-hour presentations, covering just about every major trend in the space today. Highlights for me included Reports of the Death of Testing Have Been Greatly Exaggerated, Presenting Test Results with Clarity and Confidence, Building Successful Test Teams and New Testing Standards Are on the Horizon: What Will Be Their Impact?. If I were a betting man, I’d say several of these presenters will be making appearances on our blog in very near future (maybe even our own Fumi Matsumoto).

Innovation: Remember the software testing space from six months ago? Yeah, it’s completely different now. Few industries evolve at such a rapid pace as that of software development and testing – something that was in full display inside the exhibitor’s hall. Without naming any names – we are a commercial enterprise after all, and we don’t give free advertising – I will say that there are a lot of companies doing some really amazing things. It was great to walk around and talk about how others are disrupting the space.

Networking: STAREast 2013 gave us a great opportunity to meet and network with testing professionals from all over the globe. We spoke with testers in the insurance industry, biogenetics, defense, travel and many others. We even met some current uTesters at the uMeetUp, and our own Jessica Fleet persuaded a few newbies to sign up and give it a try.

Prizes, Games and Other Fun Stuff: We were able to give away a metric ton of free uTest SWAG – t-shirts, mouse pads, pens and other items. Thanks to everyone that stopped by the uTest booth. We also awarded two lucky attendees a Kindle Fire and an iPad. And finally, kudos to the SQE team for keeping things fun with their Shake-Rattle-And-Roll event, along with a few other casual gatherings.

I could go on, but needless to say, it was a wonderful week. Hope to make it to the next.

Until next time, happy testing!

If you’re involved with development or testing for all things internet or mobile, you’re sitting in a sweet spot in the economy, according to IBIS World. While the economy as a whole is starting to do better then it was five years ago, the industry research company says that many web-based industries are thriving and growing at an impressive rate. Not surprisingly, mobile will be an increasingly important driver for these industries.

IBISWorld has compiled a list of standout industries based on their annualized revenue growth over the past 10 years, as well as performance expected through 2018. One trend stood out above all: The internet is playing an integral and deciding role among these industries. For example, IBISWorld expects the number of mobile internet connections in the United States to rise at an annualized rate of 7.9% through 2018. Thus, a rise in the number of mobile internet connections, higher internet traffic and the proliferation of smartphones, tablets and other mobile devices across the US will help these 10 industries continue a meteoric rise that will outpace the rest of the economy.

Here’s a look at the fastest growing industries, according to IBIS World:

Social Networking Sites
Over the past decade, SNS [social networking sites] has exploded, recording 74.4% annualized growth since 2003, which included the beginnings of early SNS like MySpace and Friendster. … Over the next five years, SNS will further develop new ways to monetize, tracking user data and leveraging more lucrative advertising contracts. As a result, the Social Networking Sites industry is expected to generate growth of roughly 25.0% per year on average over the five years to 2018.

Social Gaming
Companies, such as Zynga and EA (Electronic Arts), have transformed well-known, classic games like Scrabble and Battleship into social attractions where consumers can play head-to-head over the internet. This trend has been enhanced by a surge in social game applications, which are being developed at a rapid rate to cater to those consumers looking to play games on their mobile device. The industry hardly existed in 2003, but has since posted annualized revenue growth of 134.4% over the past 10 years. Moreover, the industry is expected to grow an additional 16.9% per year on average over the five years to 2018.

E-Commerce
Online sales of fashion samples, greeting cards, photo printing, shoes and household furniture have exploded in recent years and look to maintain their strong growth in the long run. … Aside from well established firms, however, many specialized websites are gaining greater traction and maintaining a robust, loyal customer base. … Moreover, these sites have been enhancing their mobile presence, as many consumers continually access mobile e-commerce content from their smartphones or tablets. Each of these sites has shown impressive, swift growth over the past decade in revenue, establishments and in their overall contribution to the economy.

Online Payment Processing
Online Payment Processing Software Developers are strongly benefiting from the ever-increasing e-commerce activity. As more brick-and-mortar retail establishments created online retail websites over the past decade, the need for online payment processing software has increased dramatically. As a result, the Online Payment Processing Software Developers industry has experienced average annual revenue growth of 43.2% over the 10 years to 2013. … Additionally, as more niche, specialized firms continue to enter the e-commerce realm, online payment processors will continue reaping the benefits, with expected growth of 17.2% per year on average over the five years to 2018.

E-Publishing
With increasing smartphone and tablet penetration in the United States, e-book publishers have flourished. … these factors have lead to strong growth for the industry over the past decade, reaching annualized growth of 88.3% since 2003. Over the next five years, e-book publishers will continue to benefit from the improvement in the prevailing economy and its effect on consumer disposable income, as well as the gradual shift toward more books formatted only for e-readers. IBISWorld expects E-Book Publishing to grow 7.5% per year on average over the five years to 2018.

Fantasy Sport Services
Fantasy Sports Services have been gaining popularity as more consumers go to the internet to compete in these social sports games. The industry develops software, and markets an online platform for multiplayer fantasy sports. … These games have been around for more than 25 years and played in tight knit social circles, but the internet really helped push the envelope for transforming Fantasy Sports Services into a full-blown industry. Over the past 10 years, fantasy sports services have experienced explosive absolute growth of 241.0%. Fantasy Sports Service firms will continue cashing in on the general move toward more mobile content, which will help bring revenue up at an annualized rate of 7.6% over the five years to 2018.

Online Recruiting
Online recruitment sites have been taking to the internet and revolutionizing the Employment Recruiting industry. … Online recruitment sites have grown an annualized 11.9% since 2003. Over the next five years, these sites will grow another 3.0% per year on average by penetrating specialized, niche job markets and catering to specific markets of job seekers.

Read the full IBIS World report >>>

If you’re involved with any of those industries, hold on for the ride – things are only going to continue to grow! Also let this be a pre-warning: As the industries continue to grow, more competition will enter the ring. You need to stay vigilant and continue to put out the best product you can, don’t rest on simply being one of the first. Listen to users and make sure you’re continuously giving them what they want. If you don’t, someone one will certainly swoop in to fill the void – and take your customers/users while they’re at it.

Also keep an eye on what’s coming. Mobile has a long way to go and it will certainly go through many changes during that process. Take leaps and adapt to new technology as it comes out, you don’t want to be left behind if some wild new gadget like Google Glass makes it big.

In the software development world – the pressure is on.

Management teams want software developed yesterday, and developers are struggling to keep their heads above water. The push towards agile methodologies and faster development has falsely suggested that quality assurance would soon become the developer’s responsibility, thus reducing downtime and speeding up the dev process. The assumption being that development teams would be the ‘keepers’ of quality, testing their own work. After all, this could save organizations a lot of time and money… right?

Wrong. The cost of testing is minor compared to the cost you’ll pay fixing critical bugs after release, handling customer complaints, pushing a patch and loosing revenue. It won’t save time either. All the bugs and glitches that are overlooked by developers will need to be handled, patched, re-released – and who knows if any of your software’s users will have stuck around when your done cleaning up the mess. Overlooking quality can double, even triple, your development time.

So will QA ever become the developer’s job?  As the SD Times Editorial Board says, “Sorry, that’s not going to happen.” According to the SD Times, quality software will always require quality testing:

“Sure, dev teams can and should focus on building quality into the product. Of course, code repositories can enforce standards for unit testing. Certainly, business stakeholders should regularly review the software to ensure that it meets their business requirements, just as architects should review against technical requirements.

All of those trends will improve software quality by identifying defects sooner so they can be fixed less expensively. All of those trends will shorten software release cycles and lower software costs.

However, no matter how you slice it, sophisticated software will require sophisticated testing that goes beyond what developers can do.

…someone, somewhere, needs to perform QA prior to release. All the automation in the world can’t eliminate the need for QA. Thank heavens.”

The SD Times Editorial Board is spot on  – nothing can overshadow the importance of in-the-wild testing. As a recent article on The Codist Blog says, “No matter how careful or artful a programmer might be about small testing the need for big testing is not diminished. Quality applications are built by a combination of skilled people and one of those needed skills is testing.”

In short, real world QA isn’t going anywhere. Software is so entrenched in consumers’ lives that they have little to no tolerance for buggy software and won’t hesitate to abandon your application if it hasn’t been properly tested. That’s a risk organizations can not afford to take.

In the following post, guest blogger Lucas Dargis outlines several reasons why he believes testers should use requirement documents for checking purposes, but should not rely heavily on them for testing. Lucas Dargis is a software testing consultant. He has led the testing efforts of mission critical and flagship projects for several global companies. He specializes in the development and implementation of testing strategies.

***

The prevalent idea that testers are dependent on a requirements document to do their job is a dangerous one. Requirements are not always needed to test. In fact, in many situations, they may actually reduce a tester’s effectiveness.

The process of deriving tests directly from the requirements has several names. The ISTQB uses the term “specification-based testing”, sometimes it’s referred to as “Happy Path” testing, but I think the most appropriate name is “checking”. Michael Bolton wrote a well-known post about this topic. Checking is confirming that what we believe is actually true. Products are built in accordance with the requirements, so the requirements are what we believe to be true. When we verify that our product meets the requirements, we are “checking” the product. When a tester relies on a requirement document to test, he isn’t testing, he’s checking.

When we test, we are exploring, investigating and learning. Our actions are influenced by new questions and ideas that haven’t yet been explored. The use of requirement documents while testing can cause problems because it can give a false sense of test completeness, it can steer testers in the wrong direction, and it can reduce the independent thinking of the tester.

Gives a False Sense of Test Completeness

If we have verified that our product meets all of the requirements, does that mean that the product has been well tested? True, you have verified that the product behaves the way we expect it to (in specific situations) but you still don’t know how the product will behave in situations not specified in the requirements.

“That wasn’t in the requirements” I’ve heard testers use this excuse many times and it drives me crazy. As a tester it is your job to investigate the product to learn about areas and behaviors outside of the requirements. A product is much more than its conformance to the requirements. It’s up to you to cover that gap between what is expected of a product and what actually is.

Checking that a product meets the requirements is necessary of course, but checking alone does not indicate test completeness.

Steers Testers in the Wrong Direction

When you first start testing a new product or feature, what should be the first thing you test? Some might say you should verify the requirements. I challenge that view. I think the requirements should be one of the last things you test. In my opinion, a developer who writes code that didn’t meet the requirements failed to do his job. Before the test team sees the product, the developer has already spent hours working and verifying that his work is correct. Although possible, a strong developer rarely produces work that doesn’t meet the requirements. With that in mind, there is little value in retesting his work, especially when you consider the other aspects of the product that haven’t been tested yet.

Requirement documents can stifle the creativity of exploratory testing. When testers have a requirements document in front of them, they may be more likely to verify the requirements first and focus on areas where the likelihood of learning new information is the lowest. Instead, they should focus their efforts on new tests and unexplored areas where the opportunity for learning is the highest. When testing without requirements, you eliminate its influence on your testing decisions; you have to rely on your own abilities, your knowledge, and your curiosity. You test.

All testers have heard the old saying “Trust but verify”. I’m not saying that we shouldn’t verify developer’s work, only that there’s more value in performing a test for the first time, then performing the same test multiple times. Focus on the activities that produce the most value first.

Reduces Independence

The idea of “independence” (See ISTQB Foundation) refers to how close a person is to a product. A developer who wrote the code would have the least amount of independence, while a person in a different company who has never seen the product would have the most independence. Independence can often be a great quality for testers. Testers with no preconceived notions of how the product is supposed to work are able to view the product with more objectivity.

Consider the common situation where a product was built according to incorrect requirements. The tester was able to verify that the product met the requirements, so was there an issue? In this case the requirements served as a false crutch to the tester. Just because the product met the requirements doesn’t mean it was working correctly. A tester with no knowledge of the requirements would have been better positioned to identify any errors because he wouldn’t have been comparing it to an incorrect “truth”.

We have now seen some reasons why testers should avoid relying on requirement documents. While they may be a necessity for different “checking” activities, testers who wish to provide value thorough “testing” must understand that their value is best realized when they test without requirements.

uTest is setting the pace in Massachusetts…and we’re not talking about uTest CEO Doron Reuveni’s ability to run the entire length of the state without breaking a sweat.

Yesterday morning at the Boston Business Journal’s Pacesetters breakfast event, the BBJ named uTest the #1 pacesetting company in the region. In case you were wondering, the Pacesetters list is comprised of seventy Massachusetts companies that recorded the highest three-year growth rate in terms of revenue. Not too shabby.

“We’re thrilled to be named as the #1 Pacesetter in the region by the Boston Business Journal,” said Doron. “It further validates our vision and execution, and is a distinct honor to be recognized alongside so many great companies. Congratulations to all of the Pacesetters.”

It is truly exciting to be a part of the top companies driving the economy in Massachusetts, and as uTest CMO Matt Johnston said earlier this month, “uTest is proud to call the Boston area our home base.”

What makes us even more thrilled is that yesterday’s Pacesetters Breakfast Event was able to raise $76,000 for the One Fund Boston. As BBJ Publisher Chris McIntosh said, “these companies displayed the kind of commitment that symbolizes Boston Strong.”

Congratulations to all the Pacesetters, and thank you to the folks at the Boston Business Journal for putting together such a great award and event!

There’s a certain idea making the rounds in the world of software development. Some people have implicitly accepted it, others have eagerly embraced it and some people hate it with the passion. So what is the big idea? It’s that software testing is essentially dead.

This argument has been made on numerous blogs, articles and presentations – each with its own variations – and so I won’t list or reference any one in particular. Instead, I’ll use this post to explain how I view the idea and why I consider it to have one glaring flaw.

So what does it mean to say that software testing is dead? Here is how I understand it:

Today, software applications have an enormous base of users, with an equal number of unique use cases. Rather than hire a traditional QA team to cover this increasingly complex matrix, it’s more effective/practical to find bugs through big data (i.e. real time metrics), real users (i.e. customers or beta users) and through the programmers themselves.

My thoughts:

The premise is spot on. The testing matrix is indeed more complicated than it has ever been, with an endless combination of hardware, browsers, carriers, languages, APIs and other criteria. And it’s also true that a traditional test teams can’t possibly account for all of these scenarios – and that a whole class of bugs are likely to go undiscovered until the app is released into the real world. For this reason, measuring performance in real-time (often referred to as Testing in Production or “TiP”) and beta programs are incredibly valuable, but they should not, in my opinion, be considered as a replacement for professional testing. Therefore…

The conclusion is wrong. The challenges of the new testing matrix are well documented and indeed quite daunting, but if one concludes that the solution is to abandon formal testing (and testers) altogether, then they can expect an entirely new set of problems.

Mainly, that if everybody is considered a tester, then nobody is really a tester. Rest assured, users WILL find the bugs, but from there, it’s anyone’s guess as to what happens. uTest CTO Fumi Matsumoto covered the many weaknesses of beta testing in a recent blog post, they included:

• Beta testing often generates too much noise (that is, feedback) that is not accurate and not actionable.
• Inconsistent participation — too much or too little — often administered with poor processes for collecting and analyzing feedback. Not all use cases get covered, so bugs slip through.
• Good catches but insufficient information: Even when bugs are identified, the reports are often not useful because they lack sufficient information to reproduce the defect.
• Delay: Beta testing slows the release cycle by having a dedicated phase before the production release.

Like any consumer of apps, I’ve encountered hundreds of bugs in my day. But how many of those have I reported to the developer or company? I could probably count them on one hand. Apart from the frustration of not being able to complete a task, I really had no reason to.

That’s obviously not the case for testers; they approach an application with a much different mindset.

So how can a company “keep testing alive” but still get feedback from a diverse, dispersed set of users? Through crowdsourcing, or as we we like to call it, in-the-wild testing. Fumi continued in his post:

With the advent of crowdsourced testing, or what is often referred to as “expert-sourcing” because it often utilizes vetted and trained QA professionals, development organizations can now get the benefit of in-the-wild testing without the downside of beta testing’s high noise level. This option offers companies the ability to test pre-deployment under real-world conditions and,  in particular, address the difficult problem of mobile device fragmentation: OS versions, mobile carriers, memory and other mobile device configurations, or location diversity. Typically, the vendors will hire a test company’s members in specific locales to beta test the software and report defects via agreed upon forms and channels.

So is traditional testing dead? For some that might be true. Is software testing altogether dead? Not by a long shot.

What do you think? Is there a case to be made that testing is dead, or perhaps a dying profession? Please share your thoughts in the comment section below.

Hardly a day goes by without a major software security breach – and the truth is, we are all at risk. With so much at stake it’s hard to see security testing take the back seat to other forms of QA. All the effort poured into developing an application can be taken away with one single hack.

However, companies don’t necessarily choose to let security take the back seat. Often they don’t have enough access to security professionals to properly test their apps.  According to Jaikumar Vijayan, in CSO, there aren’t enough skilled security testers and professionals available to fill the current employer demand:

“In 2012, there were more than 67,400 separate postings for cybersecurity-related jobs in a range of industries, including defense, financial services, retail, healthcare and professional services. The 2012 total is 73% higher than the number of security jobs posted in 2007, Burning Glass said.

By comparison, the number of job postings for all computer jobs grew by about 20% between 2007 and 2012. Posting for all jobs grew by only 6% during the period.

The two most sought-after jobs by employers were information security engineers and security analysts. Close to one in three of all computer security jobs advertised last year were for information security engineers. Nearly 25% of the job postings were for security analysts.“

We know that the growing demand for these experts comes from a constantly increasing number of cyber criminals, hacks and threats. Companies have to protect themselves, as consumers store more and more private information within all sorts of mobile or web applications. But why aren’t there enough security professionals to meet the demand?

Joan Goodchild, in CSO, believes it is a disconnect between the demand for security jobs and certifications:

“So, in one set of research, we have numbers that claim there aren’t enough skilled security professionals out there to fill the vast number of positions that demand their expertise. On the other hand, there is (at this point mostly anecdotal) concern that some certifications are out of date and no longer necessary.  Is one factor influencing the other? Are employers misguided to seek employees who only hold these industry-specific certifications?”

Nevertheless, security testers are hard to come by. This is partly because being an expert in security requires a vastly different skill set than that of a functional or usability tester. Knowing and being able to find/probe for the most common security attacks is very hard to do, and if companies can’t find an expert to do the job well, it’s no wonder their apps are suffering.

However, through utilizing crowdsourced security testing services – that have access to the ‘best of breed’ security experts all over the world – companies can get the testing coverage they need to protect their software, customers and their brand.

For more information on security testing, download this free security testing whitepaper.