We wanted to reach the moon,
so why did we build a train?

By Colin Armitage

On this 21st anniversary of its inception it would have been nice to wish ‘Happy Birthday’ to the software test automation industry. But why should we offer felicitations to this misguided runaway monster that should have derailed a long time go.

No-one can take issue with the laudable objective of improving software quality and it is the difficulties in achieving it that created the opportunity for technology based solutions. However, if we remind ourselves of what we were trying to achieve the catastrophic flaws in the solutions on offer become all too clear.

Many different approaches were taken to testing in 1985 but in general it had been accepted that it was a good idea to let someone other than the developers try a new application before it went live. Typically this was achieved by taking knowledgeable users, what are now described as Subject Matter Experts (SMEs), out of the line of business while they performed an acceptance test.

Let us remember that this was before Y2K, an event that fundamentally altered users expectations of the quality of application that should be initially delivered by the development teams. Prior to Y2K the proposition to the users was that if you give us your time to undertake a User Acceptance Test (UAT) then you will receive a new, or much enhanced application. Y2K changed all this as typically the new version offered not one iota of new functionality, just a promise that if would continue to work once everyone had recovered from the Millennium celebrations. The reaction of many user teams was to insist that the IS function took much greater, or even total responsibility as the business functionality remained the same and the testing did not require their expertise.

This enhanced level of expectation for the quality of application entering UAT was accompanied by a Y2K driven realization in development teams that effective testing was tough to achieve, repetitive and if undertaken manually, extremely laborious. And it is the issue of repetition that is at the heart of the drive for test automation.

It is the interests of everyone involved in the application lifecycle to see the number of test repetitions kept to an effective minimum as excess repetitions lead to a loss of motivation, a loss of confidence in the new application, missed implementation timescales and most importantly, a danger that application quality will actually reduce!

A loss of motivation and confidence are easily understood if the application is constantly bouncing between the UAT group and development because too many bugs exist which should have been identified and corrected during unit and system testing. Test repetitions, whether manual or automated, take time so the effect on timescales is inevitable. It is also a fact that UAT happens at the end of the development lifecycle when previous phases will have probably overrun and an implementation date been set. However my personal favorite is the way in which excess repetitions can lead to a reduction in application quality. It is all down to boredom.

Consider the keen sense of anticipation, even excitement in SMEs that accompanies the chance to see and test a new application. This application could potentially solve a number of issues in the line of business, perhaps increasing the company’s competitiveness and financial success. This initial enthusiasm takes a bit of a knock as the tester experiences a crash on the Log-in dialog and it is distinctly on the wane as each test cycle follows another.

Testers are only human and as the implementation date approaches the excess repetitions have taken a potentially fatal toll. Driven by time pressures and an everincreasing level of disenchantment they inevitably start to test only the fixes development have provided. So the depth and effectiveness of testing actually deteriorates in this final stage and that is when the dreaded collateral damage occurs. This need for testing to be equally effective and complete on every repetition is another key driver for test automation.

So the justifications for test automation are eminently sensible, so why have so many companies given up, not bothered or are incurring such significant costs in its support? This is a riddle that is easily solved simply by examining what the users actually wanted in the key areas of usability, capability and survivability – if you want to go to the moon it’s a good idea to build a rocket!

The multi-stage Titan rockets that supported the lunar missions seem the most appropriate analogy so let’s start with the 1st stage, the big one at the bottom. For us the first challenge is usability as unless the solution can be utilized by its intended SME audience then it will never leave the launch pad. Even at this most basic level the flaws in the available technology become apparent. Instead of empowering the SMEs, the programming constructs underpinning automated solutions has taken automation away from its target audience and instead created a new industry and a lucrative career path for people with the appropriate technical skills.

You can throw money at the 1st stage problem, you can accept the limitations of your 2nd stage testing but it is the 3rd stage that has proved decisive and has consigned so much software to the shelf. Change is a fact of business and IS life and unless a test automation solution can adapt to new versions of software the benefits disappear. Creating automated tests will always take more time than a single manual test cycle - the benefits of automation are only realized through re-use. Take this away and the corporate asset you were hoping to create becomes an expensive liability.

So go on, take a look out of the window. If you are aiming for the moon, what are you driving?

Original Software has set out to prove its point and is hosting live webinar demonstrations. To book a place on one of these webinars call (630) 321-0092 in the US. or +44 (0)1256 338666 in the UK. You can also email requests to solutions@origsoft.com