Last week they published a new issue of the Security Acts magazine. You can download and read this magazine for free via this link. A couple of months ago I wrote a blog about 20 ways to test the login function. A lot of people have read this post so the idea comes up to write an article about this topic.

The result is the article in the 4th issue of the Security Acts magazine. You can find this article at page 34. If you want to read only my article, click at the image below.

Enjoy reading, I can tell you that there are quit some good articles in this magazine, so it’s worth to register and download the complete magazine.

The fun is, after writing the article I found out that there are only 19 ways described in the article and the blog. I had 2 choices, or writing an extra test case, or change the title. I choose the simple one.

Worth reading in this context:

• Hidden treasures for everyone, About vulnerabilities in applications
• A Letter to a friend, About common mistakes by building applications
• Threats are caused by a combination of defects!, about how threats occur by different defects together in one application.

Cloud computing is an emerging phenomenon that offers enormous advantages, such as shorter time to market, flexible computing capabilities and limitless power, but the cloud market, still in a very early stage, continues to grow and evolve.

But cloud computing is much more than technology, cutting costs or getting more agile. The cloud model is a new business model, a new way of thinking and doing IT as a business instead of a technology! The cloud business model. More modular, incremental, selective, collaborative and with a value proposition based on financial liquidity and operational flexibility [Forrester, 2010]. This has a lot of (side) effects, also on software testing…

The way we do business around testing will change also. Not the way we test, but how we deliver it. The market will be more focused on value driven, agile, modular and flexible solutions from test service providers. These providers need to create more ‘building blocks’ around how they deliver services: testing. Testing will also become a service, Software Testing as a Service (has a nice buzz ring to it).

Software Testing as a Service (STaaS) was coined by my colleague Leo van der Aalst in 2008, but his ideas didn’t go as far as I would think of today. In Leo’s proposal all testing was done by a service provider on demand at the scale as needed by the client. My idea goes further.

Software testing needs to be even more flexible, from the test manager to the test engineer, from the system test environment to the performance test environment, from the security scan to the full usability test and from audits to end-2-end tests. And everything can be done separate and fully integrated with one other. This creates a full cloud model for testing. With the possibility to be fully standardized, agile and elastic to help the client when needed; a full service consisting of several building blocks.

Let me share you my lesson learned of the last few days. How it happened, I don’t know, but I had some trouble with the communication. Afterwards is was not a big problem, but during the week it was hard enough. I’ll share with you the solution and some lessons I learned.

The project and the team
I’m in a small test team, with just a couple of testers, testing a smart card management system. The goal of this system is to manage all the smart cards with the certificates. These cards are used by almost 60.000 users with different roles in the Netherlands.

Our input documentation consists of a small functional design, a technical design (written after they build the software) and quite a few use cases. These are detailed enough for executing a kind of functional or acceptance testing. But we use these process oriented documentation for our system test (sound a little bit strange but it works).

To have, insight in the progress of the test team, they give me an update each day about the percentage of completion of the test based on the use cases they executed that specific day.

So far nothing strange; however I realized that the percentage of tests executed based on the uses cases were not the best way to measure the progress of the testing. (James Christie wrote a good post about why this type of measuring doesn’t work well)

The same message but different meaning
I requested my team to send me a daily update with an enumeration of the use cased followed by a percentage of completion. It looks like this:

Use case 010 request card: 100%
Use case 020 approve request: 100%
Use case 030 print smart card: 20%

The numbers where growing so we made some progress. The second part of the daily update are the defects found during the day.

This week a blocking defect occurred in the first use case, number 010. The result is that the last 70% of the use case can’t be tested because that part of the application is blocked.
The strange thing was that, the team reported a completion of 100% the day before, they found an blocking defect, still with a 100% completion. And use case 020 was also completed while this case get’s his input from 010.

As you may understand there was confusion between the test team and me. Because how can it happen that we have a blocking issue at 30%, we complete it 100% and also the next use case?

A little drawing gave us the solution
Clarity was needed in this discussion to be sure we’re heading in the same direction. I made a little drawing, based on what I expect what went wrong.

The thing was, when the team communicated to me that a certain use case was 100% completed, they mean they did all the things they could do in that specific situation. But what I understand was they are at 100% so they are finished.

A blocking issue at 30% of the test execution means from their point of view, we executed all test cases we can execute at this moment so we’re 100% complete.
I expected a completion of 30% in this case because the biggest part, 70%, isn’t covered.

Lessons learned from this incident
However this was not a big problem with effect on the project I had some general lessons learned These are my lessons learned on the way to successful communication:

• Be sure you have the same expectations
• Make drawings to clarify what you mean
• If you explained something, ask the other  what you mean
• Ask some questions about the progress the first times they delivered you the daily status update to be sure you understand the message
• Relate different aspects to each other, for example blocking issues in a certain part, and the completion of that part.
• Always keep in mind, in case of communication, nobody is right or wrong just be clear to each other.

Problem:
In an earlier post we’ve seen some security issues with the Electronic Health Record. To avoid this type of information leakage we need to improve the awareness among the people using these cards.

To try and solve this problem I want to introduce a little mix of existing components to improve the security level and security awareness for the people that are using these (smart) cards for authentication. For example in hospitals and pharmacies they are using cards to have access to patient data (see this blog). But people treat it unsafe. Maybe don’t see the value of this authentication card.

This post describes a solution, not in the field of software testing but more in the field of communication, a change of mindset to create awareness among the people. The final result is to improve health care. Because if people use their personal card in a safe way, information security will be on a higher level. This is also part of the health care, because if you’re a famous star, you don’t like it if your personal health record is known in the media. First of all some simple ingredients that are part of the combination to improve the world.

Let me give you an overview of the different ingredients.

Ingredient 1: A printed hard plastic credit card
I think for 2,5 year now, (why? I don’t know) I carry with me, in my bunch of bank cards, two plastic cards printed with a linguistic and mathematical mnemonic. This printed hard plastic card can be printed with your own design. We need this technique later on if we combine this with the other two ingredients. Know that it’s possible to print your own design.

Design your own business card

Ingredient 2: The existing authentication cards
The cards we issue to our medical staff for authentication of the medical systems, like the electronic health record. There are several types of cards used all over the world. Most of them have the size of a credit card. These cards are used multiple times per day by all these people.

The Estonia Health Card

The German gesundheidskarte

The Dutch UZI-pas

Ingredient 3: Five Golden rules
5 Golden rules, 5 ‘keep in mind’ statements to create security awareness about the use of the authentication card as mentioned in ingredient 2. Statements like:

• This card is like a key, keep it private
• This card is like underwear, don’t share it with others.
• This card is your personal secret, don’t tell it to others
• This card is like a glass, handle with care.
• This card is like a sport car, be aware of his power

The recipe
If we combine these three ingredients together the result is a card with the same functions as the authentication card already in use. But from now on printed cards with 5 golden rules to create awareness for the secure use of this card. Because if people will see these 5 statements every time they use the card, they will be triggered about the security. They become aware of the importance of the data and the possibilities of an incorrect use of the card.

The costs
Only a new design of the front or backside of the card.

A better world
If this results is a safer use of the card it will also result in better quality of health for the patients. The staff in for example hospitals will use the card in a safe way. Leaking information of Hollywood stars to the media will happen less. The privacy shall improve with this very simple mix of ingredients.

If this argument is too simple, please let me know why.
If you need more information please let me know and let’s make a new design for the backside for example.

Have you ever had the problem to setup a test environment? Or, even worse, to change the configuration of that environment? At a client I wanted to upgrade the test environment to match the storage capacity with the production environment. We were testing output documentation for customers and the client had a penalty by law of €120.000 for every not send or wrong output document. So we needed all the possible different options in the test database. Even though I had the budget to upgrade the environment, it still took me 3 months to persuade all stakeholders and to get it done! How can this be done better or easier or faster or…? A cloud?

Last April and May there was a perfect example of when a cloud came in useful. There was an ‘ash cloud’ hovering over Northwest Europe. Airlines couldn’t fly from and to any destination that was under the ash. As a result a lot of passengers were stuck on airports or couldn’t go on vacation. And what do most people do in such a case? They call the airline for information. Luckily the airlines were smart, they had a recording on the phone line to check their website. As a result thousands of people were checking their airline websites, which resulted in an overload of these websites. And… they went down.

If these website were cloud-enabled they could have added extra capacity to the site servers to help the overloaded servers and people could still access the sites!

In the beginning of this month IBM Rational had its innovation conference in Orlando, Florida. I had the opportunity to be there. One of the things I had to do was to take a seat in the Cloud Analyst Panel. In this panel I had to give a short presentation for an amount of analysts from the worldwide research firms. In this post I’ll tell you what I had to say.

This is an availability option, but what has this to do with testing? Well, the website testers could have tested for the load and stress on the site. But clouds can help testing also in other ways. They can provide the needed test infrastructure and test tooling. If you look at typical programs, they have a demand on the website certain parts of the year, when the tests are executed. Like for instance Amazon around Christmas or the Tax system (in the Netherlands) in April/May.  This is not only for one program, but also for more programs. But still there is no 100% demand whole year round, although it can happen there is a 120% demand.

Clouds can help! They give you a sheer number of environments for testing, when there is a demand for them. Standardization, virtualization and automation enable the use of clouds to create a multitude of (test) environments, when needed.

By the way: this is not only true for infrastructure, but also for the use of tooling in a SaaS option. This use of ‘software on demand’ creates a greater flexibility for using test tools.

The use of SaaS applications and environments in the cloud have a potential for testing. But we needed to put this in practice, instead of all this theory. We started a pilot project to use IBM services in the cloud, the Development & Test Cloud and their SaaS test tools, to see how it could support testing.

We created the test environment in the cloud and also used the tools from the cloud. Now we could execute the test scripts in the environments and connected them with the test execution and test management tools. Just like a normal project!

Well, what happened? It worked! Of course there are some strengths and some weaknesses.

• The set-up of the environment architecture is very fast
• The performance of the current cloud environment is sufficient for a normal- sized project. Performance is scalable – using more or less cloud resources (no figures available yet).
• Using integrated IBM Rational tooling, it was possible to do central management over all the cloud environments.
• The use of standardized test tools (SaaS) helps to cut time on set-up.
• There are system resources available in the cloud for applications and storage, freeing up system resources.
• Data location for storage, servers or applications is, in terms of compliance, traceable in the control panel of the cloud. It’s transparent where services are running and data is stored. Later this summer there will also be a possibility to use separate data (using VPN), for example on your own network.
• Instances can be saved and recovered.

• A test environment coordinator needs to have technical skills, because they are needed to install and manage some of the cloud services (on Linux).
• There is always a confidentiality risk when important business information is set in a hybrid or public cloud. As it should be, most of the security is left to the user, but tools are limited.
• Back-up is not ‘out-of-the-box’
• Networking is limited; no VPN, firewall, subnets are available. These are needed to control access to the cloud servers.

When we overcame these challenges we can use the cloud for our test environments. With cloud computing it’s possible to create multiple (testing) environments in and flexibly use tools from the cloud. These environments can therefore be used as testing and acceptance environments, so reducing the need for other expensive environments that have to be set up internally and are only used when tests are executed.

It’s not obvious when you’re a good tester. It’s even more difficult to point out who are excellent testers. It’s hard to measure the quality of a tester. Can you express this  in a certain amount of bugs, The amount of articles, reviews, visits of conferences or the power to convince people? It’s not easy to say when you really a master in the test profession.

This post is inspired by a lot of other blogs as you can see below the image. Reading all these blogs points out that masters the test profession exist in multiple parts.

However it’s hard to determine what makes you a good tester there are some characteristics. The figure below shows, in my opinion, the most important aspects of being a good tester. You can have a couple of them, but the more you have, the better a tester you are. All these aspects are related to each other. Indeed you can be a very good tester with a lot of method and tools and domain knowledge, but if there is a lack of communication skills it’ll be hard to make the results clear for, for example, the business.

Let me give you some interesting links, food for thought and food for reading!

Soft skill
A blog I will mention in this case is a post of Pradeep Soundarajan, in the post “Coaching testers on Bugs reports, Advocay & Credibility” he mentions a couple of things that are directly related to the soft skills you as a tester must have. Review, interview and communication skills, but don’t forget the ability to make estimations about the message you send out to the organization and the way you describe these things.

Open for feedback
A good example of helping each other with feedback is the post of James Bach, “How challenging each other helps the craft”. It not only helps the (testing) craft, but it also helps you to grow in skills and profession.

Domain knowledge
The better you understand a certain domain, for example payments, the better you can determine the impact of a certain bug. If you have to test, for example the applications at your EMV chip (the new chip of your banking card) of your payment card you need to know a lot. Think about the payment schemes,transactions in the background, how an ATM works, what the different applications in the chip are, how you can read that kind of data and what it means.

If it’s not clear what I mean go on reading the next paragraph, maybe this is applicable for you. Do you know some interesting posts about having domain knowledge? Please let me know!

Continuous learning
If we speak about continuous learning everything can be an object for study. The test object, as well as our own skills and knowledge. Challenge yourself, best practices, other testers and feel free to write about it. Rob Lambert points it out in short post “Don’t be a follower, be a tester” This post holds a couple of interesting links to other stuff and to the video of the buccaneer tester by James Bach.

“The Tombstone Puzzle” by Selena Delesie and the post “a testing challenge” by Matthew Heusser are good example of how we can learn continuously.

Another good movement is weekend testing. A lot of people are writing about this. Jeroen Rosink write about this topic in the post “Test fishing for bugs and mismanagement” and “Thinking about testing and learning”

Knowledge of methods and tools
The discussion about the value of certification and the different methods is a discussion I like. People have their pro’s and con’s. Check the blogs I mentioned above and find these discussion.

The knowledge of tools is a very important one. Not only IE or Firefox, but also spiders, analyzers and proxy tools for example can be useful. The more knowledge you have about this topic the better you can manage the test profession. Read my blog “Hidden treasures for everyone” to see how a free tool can help you finding security bugs.

To master the test profession you have to improve yourself for all these and more of these aspects! More about the characteristics of a tester can be found via the following links:

#1: Attitude or methods
#2: Testers are priests
#3: Focus on result starts with the business
#4: Master the test profession

As a tester or developer you start very often on new projects. Here are 5 easy steps to create an overview in the chaos and overload of information.

The 5 steps

1. Gather information;
2. Make an overview of the information;
3. Read important information;
4. Make drawings, and flows;
5. Summarize.

Some interesting links
Most of the time I directly start a mindmap via Mindmister.com, it’s free and you can access it from everywhere 24/7. It even has a collaboration options and the great benefit of this is, that you can work together with several people to get an overview.

The second link is a link I use for new projects where a lot of suppliers are involved. At this portal you can find the common criteria for certain projects.

For a lot of background information you can use, for example Wikipedia, but also communities like the ICMCC for the health market, these are all very useful (thanks to @ICMCC for the great job he’s doing all day).

Don’t forget thing like the patent database; via this link you can find the international database. It’s a very useful source about products and services.

And don’t forget the Twitter search function, for useful for the first steps.

What is your approach? What steps do you follow to get the right information? Please feel free to share your tips!

A performance test is usually done at the end of the test phase. This because of the need of a well enough performing environment. The environment that is most production-like is the acceptance environment. This creates a risk in applications that are highly dependent on a high performance, because the defects are found late in the process and are therefore expensive to fix. But how can we help this?

With a test cloud this can be done in every environment. Whenever the infrastructure needs to be upgraded to a production-like infrastructure this can be achieved in the cloud. After the needed performance tests the environment can again be decreased.

With the use of the needed infrastructure from a cloud a more ‘real’ load can be generated than the virtual load from arduous tools. A cloud enabled performance test tool (like the one from SOASTA, see image) works with the cloud to generate the needed load and stress to test an application (which is or isn’t in a cloud environment).

Certification of software can be useful
If you have a large network of applications connected to each other, exchanging critical information to each other and the software is of different vendors, certification of the software can be very useful. Take for example the electronic health record (EHR), all kinds of vendors are sending and receiving information to and from each other. This data is very critical in the meaning of life and dead of people.

For this type of software certification is useful. As a part of the assessment is the execution of a test set to validate of the software works according certain standards. To be sure that each tag in for example an SOAP message has the same meaning. But there is a little problem…

The pesticide paradox
If you’re using always the same test set to verify the software and you don’t change the test set you will not find any new bugs except for regression. This is according to the ISTQB source the pesticide paradox.

But what if you’re executing with for example the EHR an end-to-end test after you’ve certified the software? What if you find new bugs while you’re exchanging the information between systems of different vendors? What if you find issues when you’re live? Why haven’t you found them during the certification process?

If the test set of the certification process doesn’t change, you’re not able to find more issues in the software. Even if the software works according to certain standards this doesn’t mean the software works correct in each situation, it only works good enough according to the certification standard.

Bad certification if the test set doesn’t change
Does that mean that if you do not change the test set of the certification procedure of the software that the certification is bad? I don’t think so; it’s good enough according to the standards.
But failures in the software can lead to failures related to the dead of people why not improving the certification procedure?

If your software is certified with a certain version of the certification you know it works according to that certification. But this doesn’t mean it works fine in every situation.
If we give our software certificates must we change continuously our test set to improve the quality?

Continuous improvement of the set of qualifications
That’s why it’s important to change continuously the test set, make it bigger or better to cover more situations. If there occur bugs during the end-to-end test among different vendors, if there are new issues from production in for example the hospital, why not adding them to the test set?

If you improve the test set, change the certification, bugs from the past will not occur anymore in that situation because they are covered. And yes, existing software must be recertified maybe every half year.

Adding new tests to the test set cost time and money, but if you automate the test execution it’s not that much work. It shall give you more profit because the certification takes place before the end-to-end test with multi vendors and more important it takes place before you’re in production.

What I want to say is, learn from these important lessons learned. An extra test can save a life or a least a failure and money!

Visiting a conference about the Electronic Patient Dossier (EPD) or EHR (Electronic Health Record) gives a good overview of what’s going on in this field where hardware, software, people and processes come together. A lot of vendors are present and a lot of good discussions take place at this conference. It was also quit a modern conference with a kind of a twitter stream via #L2D.

However a lot of security aspects are covered I get a strange feeling during the conference. The question that came up in my mind is: how is the weakest link, the people covered in the security approach we follow at this moment. In this post I’ll explain some weaknesses of the EHR. What can we do as testers to make this weakness visible?

Security aspects of the EPD
The Dutch Electronic Patient Dossier (Electronic Health Record) is intended as a national infrastructure for exchanging medical patient records among authorized parties. The EHR is partially centralized but almost all data is stored locally is the xIS (x Information System) of hospitals and pharmacies. In some countries around us this takes already place.

As you can imagine are there a lot of parties involved in this type of projects. Not only the government but also a lot of software vendors, consultants, and hardware suppliers. The EHR has to address a number of requirements, ranging from scalability and performance to security and privacy etc. With so many parties it’s not easy to have all the responsibilities in place.

During the conference I get a feeling that a lot of security aspects are addressed to the different parties. Using them in practice is the next step, but we’ll see this in the next coming months to years.

The government for example makes the laws and legislations for this, a ministry is involved. One of the  qualifications is the GBZ (Goed Beheerd Zorgsysteem) translated from Dutch stands for well-managed care system. Also are some ISO certificates involved for the hospitals. Information security is addressed to the government.

The infrastructure security is mostly addressed to the parties that maintain the networks, they must deliver the right exchange, manage the connections and deliver a high uptime.

Building save applications is a task of the software vendors. The Secure Development Lifecycle, assessments and penetration tests must be in place there.

People weaknesses above these aspects
What I missed at the conference is the people aspect in this situation. Because if the information security, infrastructure security and the application security are in place but the people are not aware of security of misuse the information, they are in that case the weakest link. People are standing above these three pillars and have influence on all of them.

Here are some possible threats:

1. People use the UZI pass of each other (UZI pass is personal authentication card to get access to the systems). If people use the cards of each other together with the secret password of somebody else, all information is available for the person that isn’t authorized to see the data. (Remark: this is only the data the owner has access to)
2. 2. People don’t lock the PC. If people are working behind their desk with patient dossiers, they walk away without locking their PC everybody will have access to the open dossiers at the PC.
3. 3. Username and password under the keyboard. How often does this happen? Post-its under the keyboard with the username and password written on it? Keyboards, agenda’s and sometimes the monitor are the places to search for usernames and passwords.
4. 4. Use an MP3 CD to avoid the Windows lock. A friend of mine works in a hospital in the Netherlands. To avoid the Windows lock of the PC they put a MP3 CD, in the drive and play the whole day music with the sound of. The effect of this is that the PC doesn’t lock. PC’s unattended of medical personnel can be used by everyone. The local dossiers are all available. What if you just saw a celebrity walking in the corridor? Just open her dossiers to see what’s wrong?
5. 5. Printed dossiers at the desk of the employees, or what my neighbor told me a bunch of dossiers on the lap of patient in wheelchairs that is moved from A to B. Open and visible for everyone, We can do our best but they pass all security, firewalls and an authentication cards.
6. 6. Duplicate authentication passes, according to the formal process is this not possible, but my sources told me there are a lot of them in hospitals for example.

The role of testers in these weaknesses
As you can imagine it’s hard to test these vulnerabilities. Information security can be assessed with for example a security audit at the processes. The infrastructure can be tested with a penetration test and the application with an application security assessment. But the people aspect, the weakest link with those vulnerabilities cannot be tested in one of these types of tests.

Secondly all these issues occur after go live, how often are you still involved at that moment? Most of the testers act in earlier phases.

However, with the awareness in mind, with knowledge from earlier projects you can address some of these things during reviews of designs, processes or during the test phase.

The role of organizations and the government
Organizations, like hospitals and pharmacies, who use this type of software, have in my opinion the responsibility to make people aware of these vulnerabilities. Once I saw a campaign to make people aware of the value of passwords. A big picture of a piece of underwear with the text: “A password is like underwear, you don’t share them”.

The safety of the systems can be pushed from this side. Logging all the request and taking some samples of this data base to check of only people that are involved in a certain process can check this. This way can make it a little bit “Celebrity checking proof”.

The government has in my humble opinion besides a monitor function also the responsibility to make all parties that are involved aware of this aspect. Via standards like ISO and HL7, IHE can they motivate companies.

Because as long as human beings are involved in the process and they use of software they will always be the weakest link of the security spectrum. You can test what you want, 100% secure doesn’t exist.

Please let me know, what is your opinion about this topic? How can we make big projects like the electronic health record secure, if we talk about the people aspect?

This blog will show that the only way to get correct test data is to generate the data by ourselves.With the side effect that we save a lot of money, while gaining an incredibly better result.

Real data is too complex to handle
We all know the problem of data retrieved from the live system. It is hard to get, it is complex, it all looks identical (although it is not), it is a lot, it does not have the needed content and it is almost impossible to manipulate.
However, to be able to perform our tests, we need specific data, or files containing specific values and we need it in huge volumes. Additional, files used might need unique identifiers with the consequence that once a file is used, it has become useless for further testing or retesting. And due to the amount of data and the complexity of the data, it is a hell of a job to manipulate the files without making mistakes.

Create template files
Because we cannot retrieve the needed files with test data from the live system, we will generate ‘synthetic’ files.

A big risk with synthetic data is that it does not represent actual data and therefore should not be used for testing. The good news is that this can easily be coped with by using actual data files to create templates. This can be done due to the fact that all files used use a limited set of syntax definitions, otherwise it is too complex for the system to process the files.

Knowing this, we retrieve sample files from the live system for every type of data we need for the test.
Then we define what values in the data files are relevant for the needed variances for testing. This can be specific data values, but also uniqueness of identifiers and/or valid date-time stamps.
In the template file, we replace these values with variables.

Generate your own test files using the templates
Then we make a list of all combinations of data needed for these variables.
Finally we write a simple program that generates a new data file from the template for every combination, replacing the variables with the defined values.
This results in a bulk of files with the specified content. This method might also be used to create anonymous data by replacing sensitive information with dummy values.

Feed the test data to the system
Now we have our synthetic files these need to be fed to the system. This is a generic problem for using test data and has nothing to do with the way the test data is gathered. This might be as easy as saving the file in a specific directory, maybe using FTP, or POSTing it using a HTTP request, feeding it to a command line, inserting it into the database using SQL or even mailing it to a designated email address. As said, this is not specific to this way of creating test files, it is part of the testing procedure.

And happily use it
This way of creating test data makes it easy to test with thousands of variances in your test set. We only need to bother about the complexity of the files when creating the template files. After that is done, we only make a list of all combinations we want to test. And run the test.
It does hardly take time to define an additional test set. We have a simple overview available of the test set used for every test. And we generate every test set on the fly. If needed, thousands of files in seconds.

Simple tooling is needed to do the trick
As we assume that all data fed to the system is in, or can be represented by, plain text files (XML, SQL, CSV, Logs), we need a text editor to create the template files. Do not use MS-NotePad, but free editor like TextPad, MultiEdit or Notepad++ will do.
For listing all variances in the data, MS-Excel is to be advised. Excel has the additional advantage of Visual Basic for Applications (VBA) in the background. With VBA you can automate all tasks needed to create the files from the templates, replacing all variables.
And we can use VBA to automate most of the ‘feeding’ to the system. In some specific cases, additional tooling is needed to, for example, feed the data to a database.

This has been used successfully on several projects
At Vodafone this method has been used to simulate alarms from the mobile telephony network. This enabled us to test very specific outages without affecting the live network. In this case Network Events, Error Logs, Network Triggers and SQL files were created.

For several KPN projects we did the same for alarms from the fixed line network.
At UWV we tested several web services by generating and POSTing specific XML files.
At Intergraph, Klic (GIS) information requests were created, FTPed and mailed to the test system.
And at KLM this method was used to POST luggage information to the new luggage handling system.

This article has been contributed by Bert Veldman who realized tooling used at most of the mentioned projects. The KLM case is based on a separate development by Peter Wanders, using the same approach as described in this article.

Cloud computing brings a lot of opportunities for service providers that do software testing. There are many possible options to create work for the service providers. The cloud market is still in a very early stage and will continue to grow and evolve.

New possibilities infrastructure
As cloud computing evolves, it creates the global infrastructure for a new possibilities to be used in software quality assurance and testing. It gives our clients an option to use us as a specialized provider in the cloud, instead of trying to do everything themselves. It even generates a whole new greenfields market. Clients share public or hybrid clouds with each other or create private clouds to be shared within the whole company, instead of separate options for all different departments. Service providers can help these clients with the best advice on how they can integrate the cloud computing possibilities with testing.

According to Gartner 20% of the businesses in 2012 will have no IT assets. These businesses will use the cloud as an IT asset. These clouds need to be secure and readily available; it needs to be a commodity. Fellow IT researcher IDC expects the server revenue from the public cloud category will grow from €473 million in 2009 to €583 million in 2014. And server revenues generated as a result of the private cloud market is expected to grow from €5.9 billion to €9.6 billion within the same period. A growth of about 62 percent and significantly more than the 29 percent growth of the public cloud. In 2010 a growth of services and the Cloud subscription model, where you only pay for what you (re)use, are to be the key change factors.

Opportunities service providers
The many possible opportunities for software testing generated by cloud computing are as follows:

• Test environments in a private Cloud for test outsourcing projects;
• Test environments in a private or hybrid cloud at the client;
• Test environments in a hybrid Cloud, serviced by the service provider;
• Test environments in a public Cloud, serviced by service provider;
• Varieties on test levels (like performance testing from/to the cloud), and
• SaaS offers with tooling.

Virtualisation
All opportunities have a need for virtualisation (and standardization). When test environments are created in the cloud they need to be virtualised parts of that environment. This not only makes the implementation easier, but also the execution of the different instances in the cloud. The virtualised environments can be added or removed from the cloud when needed. Creating a flexible option to create test environments (including the needed configuration) whenever a test project starts.

Service providers needs to have an approach on how it can take a bite out of this possible revenue. It’s advised not to blindly jump on the cloud bandwagon, but to have a good approach. This can be different from client to client, and therefore we need to determine which types we are going to support that can help serve a wide range of clients.

Please help me with the next topic: result driven testers! Do they exist? When you are positive about that, what defines a result driven tester? This is the question I’m struggling with! Sometimes you’re listening to a presentation or reading a book about testing and they talk about result driven testing: Focus on the result.

I believe focus on result is important for a tester. This must be part of the testers attitude, the way to do the job. Always the focus on the result, especially the end result for the business. Having said that; this doesn’t mean that this is very clear. Let me give you an example with the following figure.

If you have to test the system under test you can have several test cases. The result can be (from a functional perspective) that everything works fine. But from a users point of view, the user friendliness is very bad. So is this application good for the end user, it does what it has to do, but it costs you double the time than the application you had earlier. What about the maintenance, the application is easy to maintain, but has a very bad security. What is your focus on result in this case? You don’t know the business goal in this situation.

The second issue I’m struggling is, what documents, which people and what systems or oracles offer you the fundament for the business goals? Where can I find the descriptions of these goals.

Thinking and reading about this topic gives me the figure below (this is not a formal scheme or something but just my thoughts). As a tester I have some influence at the left blue column. If I’m only the test engineer in a large organization, my scope is the Detail Test Plan. Depending on your role or the place in the organizations hierarchy you can affect the MTP, GTS or policy.

That means that if you create these documents they must be derived from other documents that tell the result, from a business perspective.

What about a Project Plan? Looks like a good one; normally there is a ‘business case’ inside. But is this business case well-balanced? Does it come up with the end user goals, the manager goals, or with the goals of the people that do the configuration management? Having said that, the question comes up in my mind, is this really the job of the tester? Or is this the auditors work?

Keep in mind that we’re still looking for the answer at the question: What is focus on result? Are In this perspective Project Plans, IT Strategies or the annual plans useful to get the information about the business goals? The more people that work on these documents for example, the bigger the chance that it’s well balanced.  But this is still no guarantee. More important is the thing that the relevant people must be involved, without for example important stakeholders you can’t get the right results.

To have the left blue column in place, you need a mature test organization. And to get insight as a tester into the information that is in the orange part you have to arrange a lot. This isn’t an easy job, and this figure doesn’t give a complete answer. If you think these types of documents can give you an answer to be a result driven testers try to get them. But focus on result. I you don’t expect important answer don’t invest too much time.

Having said that gives me the following question, hopefully you can help me. How can I as a tester in the first place find the business goals, in the second place translate them to testing, and third well-balance them if the organization gives me several answers?

As described above, focus on result is needed. This attitude is sometimes hard to practice because results are covered. Let me give you some tips I use to focus on the result. The things I do to learn about the business and try to get a clear understanding of the business goal are the following:

• Read domain related blogs, magazine and sites with the news about the domain you’re working in.
• Open via the customer site the annual plans, and scroll through the site of the customer you’re working for. You’ll learn a lot about his business.
• Try to find out something about the history of your company, is it a old organization, a new and dynamic one, …?
• Talk with the real end user, last Saturday I had a nice talk with my neighbor. He’s working in a hospital as an IT manager implementing health related systems. The information he gave to me was very useful since I try to get an understanding of the health market for my new job.
• Find other projects that are done before! Read the evaluation reports, lessons learned and other stuff.
• Try to search for laws and legislations of your domain, almost every domain will have some. Companies must be compliant to these, also their software.

With these things you’ll get a better understanding of the market and the customer you’re working for. Maybe you can come closer to the result you have to focus on. With this you can deliver added value to have a shorter time to market or better software, things that will sound familiar to the business.

Please let me know what you think about:

• Is focus on result an attitude for testers?
• How to get in touch with the result? The business goal? Do you have concrete tips

Earlier posts about the testers attitude:

#1: Attitude or methods
#2: Testers are priests
#3: Focus on result, start with the business

Software development should become cheaper! And with this, testing should become cheaper! That’s a line I’ve been hearing a lot the last few years, but even more now. But How can this be achieved?

I’ve been blogging quite a few posts (like here, here, here and here) about finding defects in an earlier stage in the development process. One of the most important activities that can be done to find these defects, is to integrate evaluations (like testing) in the quality strategy of software development projects. But as all systems are becoming more and more complex, this emphasis on finding defects early is also growing. And as Boehm told us: the earlier errors are found, the cheaper it is to fix them. But are there other options than evaluations to do this?

Yes, there are. Advancing technologies make this possible, even now. How? By using models! Models can be used to generate test cases automatically (MBT), evaluate the software and can help with understanding the functionality of the software. But it’s also possible to use it for other practices. In embedded software it’s a common use to use models for testing the software in an early stage of the project. This way errors are solved early before the software is implemented in the appliance, like a VCR, refrigerator, car or even an airplane. But how can we do this with non-embedded software?

With Model-Based Design (MBD)! The next figure is an example of an embedded system. But it’s also to create a figure like this for a non-embedded system. With using Object Oriented (OO) languages a model can be create, which can be executed.

Within the embedded world, various hardware components are processed in the design process. OO can also do this, by using different functions as the components. These (standard) functions can be incorporated into a model and connected with each other. As a result a model of a component arises which then can be simulated. And the functional correctness of the component can be determined.

Using models gives us (testers and developers) insight into the dynamics of the development process and algorithmic aspects of a system by simulation them. Therefore models can be used:

• as executable and testable specifications;
• to communicate (system) requirements and interface definitions;
• as a model of the entire system when provided with virtual prototypes;
• automatic code generation for software development (algorithm or logic), and
• automatic test case generation for testing the software.

Execute the functional design

As a tester I find the first option particularly interesting; ‘as executable and testable specifications’. With embedded software this is maybe easier to understand. There models are used to simulate the hardware or real world, in order to test the software of the embedded system. But with OO ‘normal’ software applications, such as administrative systems, can also be executed as a testable specification!

When using models it is now possible to simulate the (functional) design and visualise and debug the specification. In this simulation it is possible to integrate both the initial input of the test as well as the expected result, thus creating a functional test. Within the embedded world this is called model-in-the-loop testing (MIL). A term we can use also in non-embedded testing.

Development testing
A full exploration of a (functional) design based on testing scenarios by using simulations is a way to test requirements. Within these testing scenarios diverse subjects can be included, such as input parameters and environmental factors. And therefore covering all the possible input and parameters of a component or unit and this , subsequently, leads to an increased coverage of the development tests.

Best results can be achieved when the testing of the models (this is called simulation testing or virtual testing) is done parallel with the design process. When the models are created, the tests are focused on the design aspects and are simulated in the model. Since the design is still in a draft phase, you are not only able to adjust the (functional) design, but also the tests. And, as a result, a design model (at unit level) is created that enjoys the confidence of analysts, developers and testers.

Integration of the components

With MBD and simulation testing it is possible to continually test and validate the design. The next step is to integrate the validated models with other validated models and check the integration of the different components. The previously developed testing scenarios at an unit level can also be integrated with each other to use as a comprehensive integration test.

This can be done the same way as an end-2-end test of a built system. But when integrating various subsystems (models), it is important to have a good overview of the various requirements of each individual component. Or else the complexity will only get harder.

Reuse test cases

In the end a fully tested system can be built with good confidence. All system components and subsystems can be built, (re)tested and integrated in larger systems.

And with tooling it’s possible that the tests of the simulation can be reused during the  testing of the actual code and software. The use of identical test cases allows a comparison between the model and reality simpler (eg. Test Driven Development).

Simulation Testing

By using Simulation Testing (modelling and executing these models), it is possible to visualise and debug designs without any (newly) written code. Design errors can be removed quickly from the documentation. Moreover, a large part of the (development) testing can be performed based on simulations. By using simulations for checking the development process, models offer a different technique to find defects in the design at the earliest possible moment.

Last week I spoke a test manager who told me with his shoulders hanging low, that his test department had grown substantially, but hardly had developed. He realized that he was confronted with his own limits on his knowledge, experience and to a certain extent on his own personality. The organization had expressed her satisfaction with the test department on more than one occasion, but this was not the feedback he had hoped for. He struggled for a time with the question “what is the next step?”, not knowing the answer.

It troubled him that his own organization didn’t have an answer to that question. “Am I ahead of the rest of the organization?”

Talking the matter over we realized that the thing which makes you ‘big’ also keeps you ‘small’. To break through this dilemma you have to break through your own believes, pick up unconventional approaches and start experimenting with new methods. In that respect a test manager should not only be concerned with the test process, but also with his own personal development.

Where to start? How do you address today’s problems and at the same time develop towards future needs….as an individual, department and as an organization? Or is this question too far beyond the realm of testing?

We took a big sip of our coffee, noticed it had become undrinkable…. and started to ravage our brains and established some shared guidelines which might be helpful:

• Accept some ambiguity in the test profession. For instance be independent in your professional attitude, but accept that sometimes the organization only hears you instead of listens to you.
• Testing will never be ahead of organization’s IT processes in practice but only in preaching. Keep preaching until it becomes (best) practice. Be focused on adding value instead of entering a believer versus non believer discussion about testing.
• Test management (of just ‘process’) improvement might help you to formulate a roadmap for ‘the next step’, but only if it strengthens the whole software development lifecycle (SDLC). You need to focus on other best practices as well. Involve the stakeholders of the SDLC. Suppliers of software for instance can be supported with generic test agreements (see TMap NEXT) or with witness tests.
• There is an interesting similarity between auditing and testing: independent. An auditor needs to be independent and adheres to professional ethics. An auditor might lose his official status when he crosses the line of independency too often (on the penalty of being no longer registered). Testers claim to be independent but lack such a register and professional ethics. Still it might be beneficial to incorporate (into a test policy) for instance the ten test principles from TestGoal. These principles support the test manager in coaching his testers but also sets a certain level of service the organization can and in the end will expect from testing.
• Invest in personal development and not only in developing your test management skills. The effectiveness of the test process often depends rather on the form (how) than on the content (what). Release advice and finding defects are only effective if content and form are well balanced. A lot of books on personal development are available, but these lack interaction. Some people argue that effectiveness of a message does not depend on how, but rather on who. There is a case: Your reputation precedes you. It might be wise to invest some time in for instance a 360 degree feedback from direct colleagues, or a formal assessment to test your own skills….or consider joining the club and start playing golf.

We could have come up with many more or different `guidelines`…and probably will. We started our contemplative moment with “What is the next step?”….We realized that in this case the question itself is more valuable than the answer will ever be. It helps you to stay ahead, personally and professionally.

The bottom line is that you contemplate about your profession and personal development!

Do you have any personal guidelines you would like to share? What do you do to break through organizational limits or personal boundaries when it comes to the test profession?

Many thanks to Otto Vos, he has written this nice post! Follow him at twitter.com/ottovos, motivate him to be more active at twitter.

One of the most costly and difficult tasks my test teams need to perform are performance tests. Most of the time performance testing requires specific tools that are expensive, to say the least. Special engineers are needed to deploy these tests. And for online applications that have a great deal of visitors performance testing becomes more  important these days.. Unfortunately performance tests are not only expensive, but also difficult to execute. Because performance needs a well normal performing environment, most performance tests can only be done on the acceptance environment (where applicable). But this is mostly too late. But how can this be improved?

Performance test tools

The performance of an application is becoming more and more important. So to know what the performance is and test against it is a task worth doing. Performance, load and stress test tools like HP LoadRunner or Borland SilkPerformer® are pretty costly. Most of the time they are worth the expense, but in these economic days they’re still a large investment.

Cloud computing can offer a solution for this. A few tools are available these days that can use the cloud to generate the load needed for a sufficient performance test. And these tools are not the expensive tools I talked about earlier. Most of these tools are much cheaper because the technology isn’t as complex as in ‘normal’ performance testing tools. They use the availability and cheap pricing of the cloud to execute the tests at a fraction of the cost. In this post from Dion Hincliffe you can see what cloud computing actually costs and compare this to the licencing of HP LoadRunner and Borland SilkPerformer®.

Performance of the environment

But the availability of a cheap and good performance test tool doesn’t solve the issue of the test environment. As said is the environment best suited to execute performance tests the acceptance environment. This one should be ‘production-like’ in capability, suitability and performance. But most of the project I did either the acceptance environment wasn’t production-like or didn’t exist. When it was available it’s always too late to execute the performance tests. To solve an issue you have to dig deep in the code or database to upgrade the performance. Which results in a lot of retesting for all the areas of the application that have been adjusted.

For all this cloud computing can also offer a solution. When you have a testing environment in the cloud you can upgrade your environment to act performance-like at will. And decrease it when wanted. Just add or remove extra servers, nodes, disk capacity, memory, or… when you want this done. Thus creating a production-like environment when needed. Then you can execute your performance tests as early as you want and, subsequently, adjust it! This can be repeated when wanted.

Cloud computing can help the testing of the performance of your application. By giving the option of a less costly type of test execution tool. And giving a production-like environment when needed.

You are a kind of a priest, was the commentary of a new colleague. At first a strange remark, but I think it is partially correct. Last week I started at my new job and changed from Sogeti to Collis, a move because I’m curious how it is to work for a smaller company <<end advertisement>>. Everything these first weeks is new for me and I have had a lot of meetings to get to know each other. At the end of one of these meetings we exchanged these remarkable sentences:

Colleague: “Welcome at Collis, if there is anything wrong, and you’re not smiling anymore if you’re talking about your job promise to call me.”
Me: “That’s fine (with a smile on my face because until now I like my new job)”
Colleague: “What type of person are you? What drives you to do this job?”
Me: “Learning all day in this new business, motivating other to do so, working this kind of people that are here (I met a lot of highly motivated people)”|
Colleague: “You’re a kind of a priest?”
Me: “Huh?”
Colleague: “Yes a priest, how you act and what you’re doing.”
Me: “What exactly do you mean? Why a priest?”
Colleague: “You have a smile on your face like a priest when you talking about your job, your firm in what you do. It’s a kind of a faith I think for you. You like it to motivate others.”
Me: “Is that positive or negative?”
Colleague: “I think positive…”
Me: “Hmm, let me think about that, maybe it’s a good topic for a blog post.”

Et voilà, I think he is right, at least we can make a nice comparison to this kind of people. And learn from them on how they act and what they are doing for others. In this second post about attitudes of testers, let’s see what we can learn from them priests or preachers. I’ll try to point out some characteristics and translate them to testers but there are maybe more things we can learn from them. Let me know.

Faith
First of all, the most striking characteristic is faith. They are strongly related to a certain faith, and know it from inside out. They know often all the details of their faith, the pro’s and con’s. They trust on their faith and the things they do are based on the principles of their faith.

As testers we can believe in a certain approach (or methodology). Some of you have their own and are working with the best of several best practices (that’s why a best practice doesn’t exist ). You firmly believe in the things you’re doing. And if possible you try to convince others of your faith. I think it’s good to choose for yourself a certain approach. That doesn’t mean, just one simple approach but it can be a combination of a few. In different situations, you’ll need a different solution.

Belief the message
Can’t we say that most of the priests belief in the message they send out to other people? Based on years of devotion, a lifestyle and experience they form their opinion and send this message out to other people. That doesn’t mean this message can’t change over the years. But they believe in the message they send at a certain moment.

Believe in what you say testers! Especially during meetings. For example the project meeting with developers and the project manager. If you make a choice based on what you know, and your experience, dare to make those choices.  You’re often the test expert, so you can tell what to do in the situation. How often does it happen, if the developer says, it’s already tested that the project manager says testing that part isn’t needed anymore? So when are you finished?
Believe your message and opinion and do what you think that is needed related to test activities. Let the developer prove it’s tested properly. In this case act like a priest and practice what you preach.

Listen carefully
One of the quality that priests have is that they can carefully listen to others. They are able to listen, ask questions and search for the deepest motivation and thoughts people have (often in situations with a lot of problems).

We testers also have to listen carefully to a lot of different people. I think we have to learn to do a better job in learning to others. At the first place, listen to the end user. What do they want? When is a project ready? What is fit for use from their perspective?

You have to listen to project managers if they explain their choices. Listen to developers and architects when they explain the solution. Ask questions, listen to what they say, ask more questions go deeper and listen again. You cannot judge, even make choices, if you’re not sure what the drivers of others are.

Tip: The better you listen to your oracles the better choices you can make, the better end result you will have.

Ability to deliver the right help
Priests deliver after listening carefully. And hopefully give the right help at the right place and time. Based on their experiences, what they learned from other troubles, they can decide what to offer. Based on their faith with the principles the priests make choices.

As Michael Bolton says in his response at the earlier Attitude of testers post “do we as testers need the skill that gives us the ability to choose and apply techniques appropriately?”. I think that’s true. Why? Because you know things of different methodologies, experience is needed, and motivation but more important you’ll need to choose between these things.

What solution fits in which situation? Experience can help in combination with your ability to listen to what others have to say.
Some final remarks:

• Do the test rites that are needed to accomplish the assignment
• Know what’s going on in your surrounding
• Help others with for example improving their skills
• Last but not least testing is not just  a simple job, it’s a mission.

Maybe choosing this job (priest) is not the right example at this moment, with all the negative publicity around the Catholic Church. Sorry for that. But I had this conversation, and therefore I had these thoughts.

Earlier posts about the testers attitude:
#1: Attitude or methods
#2: Testers are priests

Are there must haves for testers? Things testers should know of methods they have to master or work with? Attitudes, habits or other things they should have in their set of (soft) skills?

Until now I haven’t joint weekend testing somewhere on the world. Maybe it’s time to organize this in the Netherlands. What I see in all these blogs around software testing is that those people are very motivated, to learn and improve their testing skills! In most of the post about WT you read a lot of lessons learned.

Jeroen Rosink shares with us a couple of these lessons learned in his post. In none of these lessons learned I see direct method related lessons, a lesson like, “As ISTQB Chapter X page xxx we must do this or that” . Also in other posts around weekend testing I can’t find these things. After careful consideration you can translate with some fantasy these things to a method like ISTQB, but this is not direct translation. That meant that the next question comes up in my mind today: Are methods enough? Is a tester that knows different methods, holds a couple of certifications in his portfolio able to the right tests?

Most of the people that join WT or other types of boot camps are often motivated. They want to share and collaborate. You can see that the attitude of these testers is a proactive one. There are motivated until the end. I know quite some people that are doing their job also on Saturday. The reason they are doing it is to earn money. But the driver of these testers isn’t the money.

They want to share their passion about the things they learned. But more important they use it for their customers. The attitude of these persons is: communicate about the lessons learned so everybody else can learn from them.

Sometimes I’m wondering why such a small part of the certain traditional test courses are focussed at the attitude of the testers. In my opinion are the testers of your organization the ambassadors by your clients.

Not the suit and the tie you wear but the attitude will determine how the client will judge you.

A focus at methods only isn’t enough, but is a focus at attitudes only enough? If you have the right attitude are you able to learn all testing skills that are needed to be a successful tester?

This post is the first one in a set of post about the testers attitude.

Please let me know what the most important testers attitudes are for you!

Today (May 4, 2010) I gave a presentation at the IDC Cloud Conference – Denmark in Copenhagen, Denmark.

I allready posted about my story but here you can find the slides on slideshare.

Next week I’ll be presenting on the use of clouds and testing. I’m giving a presentation at the IDC conference in Copenhagen, Denmark. Why am I presenting here? I already know something about the potential use of the cloud and at my company we are working on a pilot of proof of concept (PoC) about the possibilities of the IBM Development & Test Cloud. We all have heard a lot about the potential of the cloud and how it can be used, but we are trying to really use it!

What is the Development & Test Cloud from IBM? You can find more about it here, but I’ll try to explain it in my words. A number of technical tools from IBM are needed for software testing and these include test and acceptance environments and test tooling. However both test and acceptance environments and test tools are expensive to buy and, more importantly, to maintain. In addition, test and acceptance environments impose a large demand on the administration of the network infrastructure. Can the use of clouds help you in reducing the costs or complexity of testing or speed up your testing?

To be sure what I mean with cloud computing I first state this: “using computer resources on the Internet when needed.” Or according to Wikipedia: “Cloud computing is Internet-based computing, whereby shared resources, software and information are provided to computers and other devices on-demand, like a public utility.”

Test infrastructure on demand
Maintaining testing environments is expensive to, because you need to have environments that can facilitate all you testing needs the whole year round. But you don’t need that facilitation the whole year. So you pay for maximum need all year round, but you don’t have the need for it!

But that environments are quite expensive to buy and maintain is not the only problem that exists around the test infrastructure. The availability of (chains of) test environments is a regular source of problems for test teams. As a test manager I usually needed to spend a lot of time (and money) on preparing the test environment and even more effort in keeping the environment ‘up’, managing it. And even if it was up a lot of defects that were found, were a result of the environment!

As a result, automation projects must make choices. Take the system into production without adequate testing and therefore not knowing what the quality of the product is, with a lot of errors and what are the risks? Or delaying the go live date and spend more hours and money on levelling the quality? Even if the environment is up and stable to perform tests, this is done with the goodwill of people. The test team is actively managing the environment because tasks and mandates are not performed by the suppliers of the environments, but they are broken up and shared with the test team. At one client I had a very good testing coordinator who was an expert in managing the availability of the infrastructure.

And what happens when an error is found in the environment? You would think it would be resolved, but unfortunately organizations often assign low priority to solving a non-functioning test environment. Which results in a even larger delay of the testing activities.

Can clouds help with these problems? Well it can give you a large number of environments to perform all of your testing. With cloud computing and virtualization it’s possible to create multiple (testing) environments in the cloud. Clouds can provide an infrastructure on which tests can be executed. All needed environments can be created; development, testing, acceptance and production (DTAP).

With test environments all quality attributes like availability, reliability, security, performance, scalability and elasticity thrive around the infrastructure. When you look at the availability and elasticity a cloud can give you unlimited web services and applications when they are needed to test.

SaaS for test tooling in the cloud
Can clouds also help with test tooling and not only for the testing environments. If you combine the use of SaaS or Software as a Service with cloud services it’s possible to use the test tooling whenever it is needed. And thus creating a greater flexibility for using these tools. With this cloud services can help IT departments with two of their problems. Problem 1: the business demands more value at a lower costs. And problem 2: the (end) users of the tools (in this case testers) who demand a greater choice and flexibility. By making applications or tools available in the cloud you can satisfy both the business (tools are only costing money when they are needed) and the users (they can use the tools they need from the cloud).

Tip: This can also be done for mobile environments and the execution of performance and stress testing.

And with this Gartner also states that cloud computing is a catalyst and a perfect excuse for IT modernization and improving internal IT services maturity. Clouds have an indirect impact on other infrastructure activities. Like application consolidation and portfolio rationalisation. It helps you figure out the cost of providing services internally and improving the efficiency and transparency of IT operations.

Starting a pilot
So, the cloud has potential for testing. But this is all theory and that’s why we at Sogeti wanted to try this in practice. So we started a Proof of Concept together with the IBM Development & Test Cloud to try out test tooling and infrastructure in the cloud.

We are using a real testing project for the pilot. And for this project we created a test environment in the cloud and set up our testing tools within the cloud. This was easily done, because we could copy the existing tools (including data) into the cloud.

The test environment we created was made up of test management tools (collaboration tools a testing team uses to manage the testing) and test execution tools. These tools are emulated from a virtual image to be executed on your desktop. As a last step we  are finishing the setup of the test environment in the cloud to test the actual application. With this we can now execute our test scripts from our virtual environment within the test cloud and run a normal testing project.

Is there a light in a cloudy sky?
We haven’t finished the project yet. But what are our findings so far? Our experiences are based on the project as-is, being half way. It’s fairly easy to set up the environment and this can be done very fast. First time you need to adjust the whole image, but once that’s done it’s easy going from there. The performance of the current cloud environment is sufficient for a normal sized project. The performance is scalable when using more or less resources of the cloud, but we don’t have figures about that yet. It’s possible to do central management over all the cloud environments. There are already a reasonable number of default settings set in the cloud.

And the weaknesses? A test environment coordinator needs to have technical skills, because they are needed to install and manage the cloud. At this moment there’s no Windows environment available, only Linux. This is planned for the near future. When there are more servers available this would add more to the potential of the use. And there is a confidentiality risk when important information is set in a hybrid cloud (we are not using only a private cloud).

Cloud computing delivers us the opportunity to have all the testing and acceptance environments we want. At a fraction of the costs when we should want these configurations for real. It reduces the complexity and even can speed up the testing process because a test team can rely on the environment instead of lying awake about it. When you have the technical knowledge it’s only one mouse click away and you can set up multiple environments in the cloud when you need it!