The last few weeks I get a lot of requests to tell something about testing and cloud, like this interview for the Australian newspaper Sydney Morning Herald. And one of the most asked questions of course is what’s different in testing on the cloud and where do you need to pay extra attention at.

Cloud applications are still few compared to traditional applications, but they are the future. But what are cloud applications? When the question is asked “can you name a few cloud applications?” most people answer Salesforce.com, Facebook, Google Apps and even Microsoft Azure. Four hits (where one isn’t actually a cloud application), as the best known examples. Are there more? Yes and they are growing in numbers!

Note: Microsoft Azure is not a cloud application, but an infrastructure and platform.

But how do we test these cloud applications? What’s so special about them that they need a different type of testing than traditional applications? Cloud applications are applications that are created to leverage the opportunities the cloud gives them, but they also work with the disadvantages the cloud offers, like, for example, standardization.

The cloud is defined by its service model, deployment model and usage

Mostly cloud applications are based in the third cloud layer: SaaS. They are Software as a Service solutions that run completely on cloud infrastructure and platforms. And that is exactly the reason the testing of SaaS applications is different from traditional applications. When they are integrated in the current architecture they need to be tested on three levels: namely the infrastructure, the platform and the application itself, see figure. The usage of standard services of applications also means a change for system testing. Functional testing will be executed at a minimum, as the standard applications are already tested and approved by the supplier. But that doesn’t say anything about how it integrates into the client’s cloud.

But that’s my idea. How do you see this?

Do you know what Insanity means? As Einstein once said it’s “Doing the same thing over and over again and expecting different results”.  I already said this in an earlier post, but it also fits here perfectly. And we’re being insane in IT quality today. Every year, money is thrown away by implementing bad software. It’s either lacks quality or the quality use is not practiced to get the highest quality against the least costs and still be flexible enough.

Sounds familiar isn’t it? Are you also struggling with the right balance in cost and quality?

Quality IT supports the Business in achieving business goals. Aspects in realizing shorter lead times, more flexibility, higher quality and lower cost are crucial. Quality Assurance and structured testing have been contributing to determining quality for years. Now we can increase this value and move to ‘Business Quality’.

With a new process around Business Quality to see three major movements in quality. By using technological opportunities, like TMap and TPI and Lean thinking we can industrialize quality activities.

When trying to start the quality process before the project even has started the business case is the driver of the IT project. All activities are evaluated against the business case, because by paying attention to quality as early as possible the greatest savings are achieved.

Integrating (future) SMART innovations into the whole process will help being quality driven from the very beginning of the project; once the business case is made, quality is part of the solution!

This all helps the Business to determine to what extent the described process is complete and accurate and effective solution to an identified business need; Business Quality is created, with PointZERO!

With this we van stop the insanity and proving Murphy wrong!

If you have any other ideas or the answer just let me know…

I’m writing this again as a response to Andréas’ post (who wrote this as a response to mine).

As I’ve been looking at the ROI of testing the last few weeks I found out that the most used numbers are still based on the initial study of Boehm from 1979. He calculated the cost of change of a waterfall method and found out that ‘the earlier you fix the problem, the cheaper it is’. That idea is still the most relevant of all. If you look at the implementation of LEAN you get the same. Fix the issues when it occurs, instead of waiting till the end. It improves the lead time and reduces the costs.

But the numbers from Boehm are based on the waterfall method, as said. Now what is the cost of change for Agile? Of course you cannot really compare them both directly, but what I found out surprised me a little bit.

Both of them are based on studies, waterfall by Barry Boehm and STBC, and the Agile by STBC again. Why am I looking at these studies? Well people in QA always use Boehm to elaborate why they want to start testing as soon as possible and both of them are used in the IV&V method for the Public Sector in the US.

When you look at both you see a rising amount of cost after the requirements phase of the Waterfall method and after the test phase these costs just go sky high. When you look at agile there is a slight rise in costs after the coding phase in the Agile cycle, but when going into Production the costs will go sky high.

Why are the costs of Agile lower after the business case? Well, one reason is that with Agile documents are created as needed for the development process. So during the full Agile cycle there is not so much need in changing documentation, only the code when an error is found. But when the product is shipped for production, that documentation still has tob e delivered for the Management & Control team of the organization.

Note: This something that is often forgotten in Agile development teams not doing a good usage of the method. Speed is generated by producing as less as possible documentation, but it still has to be delivered at the end of the project. And it’s only human to ‘forget’ this!

But statements from Agile devotees like ‘collaboration between the various partners in the development process is what makes Agile better compared with Waterfall’ are bullshit. In Waterfall collaboration is also possible! That doesn’t have to be a difference. It’s only true that quite a lot of times this is not done in Waterfall. But maybe you have some ideas around this

But one thing is the same in Agile and Waterfall: issues found in Production are always very expensive to fix!

In his last post Ewald wrote about evaluations, about starting late in the project and wasting your time and money as a test expert. He also asked of you are able to make the right business case and explain why starting early with testing activities will be beneficial. This post will give and answer at his final question or statement in the post: maybe Agile can help?

There are for sure very good principles that make Agile as it is and will help you in lowering the cost and raise the quality as a team of the product. Let’s discuss 5 reasons for Agile Testing.

#1. In control by delivering the software sprint by sprint
Are you familiar with the feeling that you have to jump on the train that is already going? That you are involved to late? That the amount of work of a project is too big for you, not knowing where to start? Since the software will be created sprint by sprint the amount of work is small, you can oversee the amount of work. More important you can address the right testing techniques for each of the features realized in a sprint. And yes there are the aspects of (automated) regression testing with the tool and coverage related issues, there are issues of working with stubs because interfaces are not finished. But each sprint is small project at it own so it is possible to come in control.

#2. In control by lowering the risk profile sprint by sprint
The image below is showing how the risk of failure will be lowered each sprint. Where in traditional waterfall the software will be deployed in production once per year or two times per year in Agile in can be deployed each 4 weeks when it is shippable. Not every organization will put it in production each 4 weeks. Since Agile is focussed on creating high risk user stories at the beginning the risk profile will lower during the release with several sprints. For sure interface testing, integration testing are mitigations in waterfall projects to reduce the risk. They still exist as long as the software is not in production. By moving to production sprint by sprint the risk profile will be lower.

#3. Regular feedback from the business at the product delivered
Late involvement from you as a test expert is not useful but late feedback from the business will cost a lot more. Where other businesses very often work with prototyping, models and end user involvement is the involvement of the business in waterfall projects not a common approach (I know it is possible, addressing waterfall approach in the right way will involve the business, but we all know in large organizations that is not the case).

For each definition of done, and definition of shippable the business is involved, that will give us feedback. If needed these things are processed in the Product Backlog.

Besides the feedback after delivering a piece of software is the business also involved at the beginning. They define via the Product Owner all the different features and user stories at the Product Backlog and in more detail at the Sprint Backlog. Because of the short cycles (average 3-4 weeks) the feedback can be given very often to the team.

#4. Regular feedback from the team and improvements each 4 weeks
If you really want to improve testing, that is what Ewald states in his blog, you should have a mechanism of continues learning. Agile is offering an excellent instrument for this. This is the retrospective at the end of each sprint. Working in a team and working as a close team creates the environment to give each other feedback.

A retrospective has two different goals: See what you did well and see what you didn’t really do well. The first group of lessons you should continue, the second group are the real improvements, the points that need attention from you as a team. It is always a team effort!

#5. Grip at testing by a staged risk analysis
The four different reasons for doing Agile as described before are related to the quality of the software not directly related to real test execution. But for the right test execution you should have the right test strategy. How do you determine what tests you should execute? How to find out what the high risk objects are in the application. In waterfall project the Product Risk Analysis is and instrument used by test managers. In my experience this is an instrument that can work but because we do not really know how the product will look like we are not able identify all different risks at the beginning. That is where Agile can help with is sprint by sprint approach. The risk analysis should consist of at least two different stages.

The first one is at the features that are described at the Product Backlog, these are often on a high level but can be classified on a high medium or low risk profile. High risk features will have more impact and from a testing perspective probably have more story points during the sprint. This can be part of the Definition of Ready. Until the items at the Product Backlog do not have a risk classification we are not ready to start a sprint.

The second one is at the user stories that are described at the Sprint Backlog. These have detail enough for building and testing the software. They have enough detail to derive test case and also to determine the test effort and approach that is needed. In and before the Poker Session together with the team all different risks can be identified. Great discussion will happen in the team by determining the story points. Because a user story can cost just a couple of story points from a developer. But if it is related to interfacing it will cost a lot more effort to test this.

However, there is much more to say about risk analysis and classification in Agile projects. This is a topic for the beginning of next year.

Conclusions
There are at least five different reasons to start Agile project from a testing perspective. If addressed in the right way it will lower the cost for testing and higher the added value of software testing.

Finally, last but not least: we wish you all the best for 2012.
Happy reading,
Andreas Prins.

I’m still getting amazed on the misuse of evaluation opportunities within the ‘quality’ world. In 2009 I wrote several posts around the usage of evaluations in IT. And still I don’t see enough of it around me. There even is a clear business case behind it; by finding defects in an early stage of the project the solution of them cost less, then when you find them in a later stage in the project.

There clearly is an easy business case, but people are not using them at all. Or at least enough and that’s just plain dumb! And where is the fault to this? Who can I blame for these mistakes that cost time, money and resources. Sure it’s easy to say that the project manager is to blame. He or she is only focused on progress and doing evaluations is not beneficial to progress of delivering a project. But that’s how their managers manage them; their managers want them to stay focused on in time and within budget. As a result focus goes on time.

But hey, it also stated within budget. Why are they not focusing on this equally? And it this time of crisis, at least again in Europe, costs should be the main focus on management. Don’t go beyond the budget! Plan for the budget as you plan on time. Cutting costs hasn’t always only have to be done by doing less projects or cutting on features, it can also be done by focusing more on cost reducing activities for the whole project life cycle.

It looks like a clear case; in this hour of cost savings we save as much as possible. Wrong! As said we save on the projects that we do, we save on the features within those projects. All good options to save money, don’t get me wrong. But we can same more and even save on time. According to the STBC (The economics of testing) the ratio is 1:10:100 in costs (Requirements vs. Testing vs. Production).

So it’s up to us, people in QA, Testing, Users and other people that are working in some kind of quality service to deliver this message and deliver it in a project. If you cannot do this you’re making yourself irrelevant. Most project managers know they have to test. They need to comply with regulations and rules to show that a certain amount of quality is in the delivered system. An expensive option to accessing quality and prove its compliance.

There are less expensive options that even let you provide a higher degree of expected quality; using evaluations. And the ROI needs to be clearly shown, by people in QA. And if you don’t do this you’re making yourself irrelevant. Because some people in development, like this blog from Clemens Reijnen, are opening their eyes and are making a difference. So don’t be stupid, be relevant!

Maybe Agile can help????

I’ve been working on some abstracts for conferences I would like to tell something about my new assignment for 2012; PointZERO®. For people who don’t know anything about this I would refer them to a blog I wrote about “Cutting on quality? Or implementing quality?” One of the main features of PointZERO is the removal of what I would call ‘Insane Testing’, based on Einstein: “Insanity: Doing the same thing over and over again and expecting different results.” Why do testers (or other people in IT) have that insanity? Why don’t they cure themselves? Are they dumb?

Maybe they are, but at least they can act real dumb. I think testers are the only kind of people that do expect a different result when doing the same exercise. Partly because we think this could happen, and partly because that’s how we test. But let’s be honest, it happens a lot of times that when doing the same exercise something else happens! But we are still doing the same exercise, the same! By hand! Why are we doing that by hand, again and again and again?

These repeating tasks should be automated! Because when they are automated you are sure they are executed exactly the same and you’re not doing an insane task. This is where industrialized, or automated, regression testing comes to help out. Insane Testing done by tools!

These tools help us to industrialize testing. That sounds negative, but we need tools to do more, better and easier testing. Because Industrialization is the process of social and economic change that transforms a human group from an agrarian society into an industrial one. And we should make it a testing change!

Testers should have more knowledge of the tools available around them. Tools that let them executed tests faster. That let them create test cases automatically. And maybe even create them and execute them. Model-Based Testing can be the answer to that. But either way there options available to help testers with all in testing. Let’s stop the insanity and get other results…