Computers, productivity, and the benefits of
proper software QA

When thorough software testing is performed, the benefits can be in the millions

A National Institute of Standards and Technology (NIST) sponsored study in 2002 claimed that nearly $60 billion in productivity is lost each year because of software bugs. It said end users shouldered 64% of the cost, and the balance was the burden of developers and vendors. That's a whole lot of money.

As we all know, new software programs rarely work as well as they should. At some point in computing's awkward adolescence, we all rationalized away its potential for infallibility. In fact, some experts say perfect software can't exist, yet engineers who build things like high-rise buildings, bridges, and jetliners say they would never be given the latitude that software engineers enjoy.

Then there's the problem of legacy applications. Many of these were originally developed decades ago and have now become complex, bloated algorithms with modifications made by the hands of several generations of programmers who have all done things a different way. You would think that as a package progressed past release 1.0, it would gain stability, but unfortunately the converse is actually true. That's because it's risky to make modifications to a program when it has matured to a double-digit release level; one small tweak could cause a nasty, unintended ripple.

The Reality

Programming is a profoundly manual process, and programmers do make mistakes. While programmatic blunders can stem from flawed logic, unrealistic timetables, lean budgets, and shallow skill sets, the mighty forces of sales and marketing sometimes push developers beyond rational limits to finish projects in order to be the first to market with a new bell or whistle and capture greater market share. On top of this, the risk management people also need to shoulder part of the blame since they say it's irrational to spend $100,000 on QA if, in the worst case, only $90,000 is at risk if a new system fails. In the end, the blame for marginal quality ultimately goes to the programming managers because they have complete control over the end product. Indeed, marketeers and risk managers usually can't write even the first line of a simple query program, let alone code for a complex application.

When Help Desk Drives Development

Typically, unsuspecting users find things that don't work and report them to system administrators, who key incident reports into a help desk management system. Then development managers take programmers off of other important projects and feed them black coffee until the patch is done and the program works as it should.

If you analyze the initial event that triggers this cascade of activity many questions arise: Why are internal and external customers treated like a commodity in the first place? What happened to the Church of the Customer? Why is it OK to pull software out of production and disrupt developers while they make repairs? What sense does it make to fly a technician to a remote manufacturing or distribution outpost 1,800 miles away and pay hotel, car rental, and meal expenses for a week to fix something in an application that wasn't ready to be loaded onto a production box in the first place?

We all know that it's cheaper to prevent problems than repair them. New software shouldn't have any defects upon its release. The best IT departments in the world test their software extensively, and they regularly release bug-free software. A rational business view of software testing should regard it as an investment, just like every other phase of the development cycle. And as business works to minimize the costs associated with raw materials, manufacturing, logistics, and labor, it should work toward minimizing the cost of software quality. The NIST study says improvements in testing could reduce the $60 billion tab to the economy to about $22.5 billion. How much of that savings could be realized at your company?

Good Enough?

A reformation of the perception of acceptable quality criteria is essential. There is no technical barrier on the road that leads to better-quality software. All of the practices and tools necessary to produce zero-defect code are readily available. It is only a question of willingness. And for those who are on the fence, federal and other mandates may eventually force them into better development, QA, and documentation practices.

Stuart Bishop, Director of Product Development at iSeries test automation specialist Original Software, is an expert in software QA. According to Bishop, quality control unfortunately starts too far into the development cycle. "Development managers and programmers don't begin testing until applications are almost finished. Unit testing should begin at the beginning so problems can be found more easily as the program builds. Early testing keeps programmers from having to rework more code than they need to." Bishop adds that IT shops lack specific quality metrics and do not review new code in an efficient methodical way. "Thorough planning, better communication, and automated tools are characteristic of the best development shops."

Testing Options

The choices are to avoid testing, to test manually, or to use automation tools to test.

If you don't test, you have no investment in labor, additional infrastructure, or tools, but it is likely that your help desk will be very busy fixing problems and calming the frazzled nerves of users. In the process, many dollars will be burned.

If you choose to test software manually, you'll incur expenses related to labor and infrastructure but not tools. With manual testing, many bugs will be discovered, but others will be missed because manual testing is not as thorough as automated testing because testers cannot see the underlying programmatic relationships between functions. Your code will certainly be cleaner than it will be if you don't test, but the help desk will still be dealing with the problems that testers missed.

With automated testing tools, you'll incur expenses related to labor, infrastructure, and automation tools; therefore, your up-front investment will be the highest of the three options, but bug-related incident reports made to your help desk will drop to nil. And, since test scripts can be reused in perpetuity, many can be used throughout the product lifecycle. Automated testing solutions can help plan and organize testing and then streamline the processes of unit, system, regression, and load testing among others. Automated testing tools also produce extensive documentation to satisfy internal and external audit requirements.

With test automation, an investment in education is typically necessary, but the more that technicians know about the testing tools they use, the more they will be able to do with them. David Burgher, Group IS Manager at a large steel manufacturer in Michigan, wants his developers to receive advanced training in the automated testing tools they use so they can link several unit tests together and use the new test array for complete system tests. "We'll be able to test the entire ERP system in an hour," he says, adding, "The return on investment is definitely tangible."

Raising Standards

By appropriately assessing quality risks and deploying the right QA strategy to fit your organization's capacity to test, you'll noticeably raise the reliability standard of your computing environment in the eyes of your users. Today, users are not just employees of the company; they are anyone who accesses or interfaces with your systems, including suppliers, distributors, and even customers.

Perhaps your ultimate objective should not be to make your production code perfect. Making it better than it is now is quantifiable progress. If at least you analyze your risks and consider your options, you'll be on your way to having a much better understanding of what software defects truly cost your company.

Original Software has set out to prove its point and is hosting live webinar demonstrations. To book a place on one of these webinars call (630) 321-0092 in the US. or +44 (0)1256 338666 in the UK. You can also email requests to solutions@origsoft.com