Despite decades of investment in test automation tools, techniques and time, the majority of enterprise application testing is still performed manually. I have always believed it is because the test tools were too technical and time-consuming to use, and if we could only make them easy and accessible then things would change. And while there may be some truth to this, it is not the whole story.

To understand the whole story, you have to step back and look at why testing happens, who does it and what this means to automation.

Why Test?

Testing happens because something is changing. In most enterprises, there are two streams of changes. One is the stream of projects, usually planned and executed within functional silos to advance specific goals. The other is maintenance or sustaining, which responds to production issues or sudden business demands. In this latter stream, changes may happen daily or even more often. These typically unplanned, out of window changes are a subject unto themselves for the next blog installment. For now let’s focus on planned projects.

When a project is planned, an allocation of time and resources is made to testing based on scope and risk. Of course even planned projects can result in both expected and unexpected outcomes, but the scope constraint imposed by the project necessarily focuses attention on verifying the expected results. Potential unexpected results are so undefined and broad in scope that they are either ignored or given cursory attention in the form of minimal regression testing. This means that most project tests are tightly focused around the specific changes being introduced.

This focus on what is changing makes sense from a resource and time allocation point of view, but for automation it’s bad news. The reason is that highly focused tests have limited reusability as a practical matter.

Here’s an example: Let’s say the enterprise has established a new distribution center that services a particular line of products and geographic area, and a project is launched to implement it and all its attendant business rules within the enterprise systems. As you would expect, the project calls for tests that verify that indeed the new center appears in the system and that its products can be shipped to the areas it services.

And while this all makes sense within the context of the project, it makes less sense to automate these tests for the simple reason that once the distribution center is operational, future projects will likely focus on something else and there will be no reason – and no resources – to keep running the same tests. Even if the tests are automated, why would they be re-executed and who would do it? Why would that particular distribution center be any more important than any other?

In this case, it’s not the ability or technology to automate that is missing – it’s the lack of interest in reusing the tests. And that lack of interest is only worsened by the way projects are organized.

Who Tests?

Projects are, by their nature, transient; they have a limited scope and duration. They are formed, executed, then concluded. The test resources assigned to enterprise application projects are typically business process experts borrowed for the duration from the operational area and functional silo that initiated the project. Thus, project test team members are also transients and scatter to other projects or back to their day jobs when it is concluded.

The upshot is that new project members may be unfamiliar with the artifacts and tools left behind by any previous project, or uninterested in them because of their irrelevance to the planned changes.

Unfortunately, this discontinuous approach to changes results in a low level of reuse, even when it might be justified. For example, I am familiar with a project that flatly refused to even explore, let alone use, an extensive library of automated tests created by a previous project team on the grounds that the previous project was concerned with a completely different set of changes and therefore the tests were irrelevant to them and not worth the effort of engaging the extra skill sets needed to take advantage of them.

Even documented manual tests may fall into disuse. Ironically, the more tests that are accumulated the less likely they are to be reused. New project members may not have the time or motivation to review hundreds or even thousands of previous tests to look for relevance; it is often easier and faster to just create something new than to locate and understand something old.

Who Cares?

In the end, manual testing predominates in this environment because there is little to no incentive – or budget or schedule – within most projects to invest in reusability. The project team is constrained by its own expertise and logistics to focus on the functionality at hand, with limited if any understanding of the potential impact on other areas or future projects.

Clearly there are some projects that are broad enough in scope to warrant automation, such as an initial roll-out or an upgrade that has wide-ranging potential impact. And just as clearly there are some aspects of almost every project that would justify reuse on the grounds that they are adding to the overall functional base and therefore represent a new component of risk. But the problem is that the transient nature and structure of the project itself does not lend itself to investing in reusability.

What is especially ironic about this conclusion is that in most enterprises, the only formal testing that ever happens is inside the project stream, yet the majority of changes actually come from the sustaining stream where testing is minimal and informal at best. But that’s a whole other subject for the next installment.

Enterprises of any significant size depend on extensive software portfolios to operate. I know of several who manage more than two hundred applications and at least one whose inventory exceeds 500. Granted, some of these are small departmental systems, but many support mission critical operations. And, of course, most of these are undergoing continuous changes in the form of corrections, enhancements and updates. This results in a constantly shifting production landscape.

Yet software testing disciplines are almost exclusively focused on individual applications within the context of a given project. Traditional V-model approaches track the software development process for a particular system, for example, and while they may touch upon interfaces it is only for those systems directly connected. Even user acceptance testing is usually organized and performed within silos, with each participant focused on his or her area of expertise.

But the reality is that many of these systems are interconnected. Even if there is no direct interface, they may share data that originates elsewhere and terminates somewhere else, perhaps several applications removed. Thus, a change to one area often has unforeseen effects in another, and generally accepted test practices are not designed to uncover them.

The result is that even a well-tested application can wreak unexpected havoc once in production, and this is what keeps management up at night.

The solution, obviously, is testing end-to-end. That is, following information all the way from its origination to its termination, as in Order to Cash, Procure to Pay, and Hire to Retire. And even among these mega processes there are interrelationships: raw materials are procured that are processed by plant employees into finished goods for sale. Extensive integration is both the benefit and the curse of enterprise applications.

So why isn’t end-to-end testing widely practiced?

First, because it’s complicated. The fact is that enterprises are organized around functional silos and few, if any, truly understand all of the interconnections and relationships among them. Not everyone may even be aware of the existence of some systems, let alone how they affect, or are affected by, others. It takes a lot of diligence to identify the right end-to-end scenarios, then ferret out the right resources for each silo, then coordinate them all into a meaningful workflow.

Second, because it’s hard. Effective end-to-end testing requires a robust and controlled test environment that can reproduce – or at least simulate – production. This means installing the vast majority of the software portfolio and providing for both internal and external interfaces, either live or simulated. It also requires provisioning test data that is both comprehensive and coherent so that the flow of data from one end to the other is consistent. None of this is easy.

And, finally, it takes a lot of time. Just the logistics of coordinating each functional resource to perform their respective tasks in the right order is time-consuming. In one case it took five calendar days to execute one cycle of Order to Cash, most of it lost to the hand-off between people from role to role. This is often the showstopper, because once an application has completed its project and related testing, it is usually on a fast track to production. Or, even worse, the change is actually a fix to a problem that has created an emergency. But whatever the reason, changes may be made daily and typical end-to-end testing takes far longer. The result is that most end-to-end issues aren’t discovered until production.

But it doesn’t have to be that way.

End-to-end testing can be achieved and performed regularly, even daily. The key is to adopt the right strategy, implement the right tools, and enlist the right team.

Adopting the right strategy means including end-to-end testing as part of an overarching approach for every application. This means doing the analysis to understand the sources and uses of data for any application to reveal both immediate and far-flung interfaces. It also means applying risk analysis to understand which systems have the potential for critical operational impact, and making sure the test repository is kept current and updated as changes are made.

Implementing the right tools means deploying an automation solution that enables extensive end-to-end testing to run lights-out every night if necessary. In the above example of Order to Cash taking five days to execute manually, the same process was automated using Worksoft Certify…and it executes in less than a minute. How? No wait time, think time or hand-off time. Just rapid, error-free execution that is fully documented. This type of speed means hundreds of extensive end-to-end tests can be executed for each and every change before it reaches production.

And automation isn’t just for execution. Just knowing what changes have been made and what impact they may have on critical business processes is itself a challenging task. Wading through arcane release notes or change notifications or IT tickets is tedious enough, but even assuming you can understand what they say it doesn’t mean you can tell what the impact might be. Solutions like the Worksoft Transport Analyzer can simplify and accelerate the entire change impact analysis process so you know for sure what to test.

Enlisting the right team means not just finding and recruiting the right experts for each functional silo, but also assembling a centralized test team that can coordinate the human, hardware and software resources necessary to make end-to-end testing a reality.

Granted, even with these factors in place it won’t happen overnight. The key is to remember that even a little end-to-end testing is better than none, and all the time you will save from avoiding even one production crisis can be reinvested to continually expand your scope. With patience, effort and commitment you can eventually arrive at the point where your end-to-end execution validates hundreds of your most critical business processes...while you sleep, soundly.