Test/QA Report
August 16, 2005

Making Products Testable

That testability is a fundamental factor in producing quality products is a given. Untestable products play Russian roulette with quality. That's why, for example, it's common practice to add test points to a printed circuit assembly to improve its testability. Similarly, complex systems of all kinds have built-in provisions to make them more testable. But are software systems made equally testable? Some people think not.

"Why is it that a car that costs tens of thousands of dollars has testability built into it, yet software applications costing hundreds of thousands or even millions of dollars do not? asks Linda Hayes, the CTO of Dallas-based Worksoft.

Hayes sees automation as the key to software testability. She notes with dismay that each generation of software development technology delivers applications more rapidly, and even though these applications have more functionality than products a generation older, they are imbued with less automated testability, which is likely to result in a sacrifice of quality. "The annual cost to the U.S. economy of software failures is estimated to exceed $100 billion," she says, "and the internal cost to fix these errors is $59.5 billion."

Even though quality has to be designed in for huge systems under maintenance, there is no substitute for comprehensive regression testing to discover unintended impact, she says. And that's because you cannot test years of accumulated functionality by hand within a reasonable budget and on schedule.

While test automation is the solution to testing complex software products, is it meeting the challenge? In order for an automated test to execute against an application, it must be able to perform the test in a predictable and repeatable way, say Hayes. "This means it must be capable of recognizing the application's components and interacting with them." She gives as an example, the testing of an online trading application. "The test must be able to log into an account, navigate to the page for executing a trade, enter the data for the stock symbol, number of shares, and type of trade, press the button to execute the trade, and verify that it is confirmed," she says.

"Modern dynamically generated applications defy test automation," says Hayes, "because you can't read your tests, maintain or re-execute them without constant, massive effort." She questions what good it does to develop applications faster, if by doing so, we slow testing by forcing it to be done manually, and consequently degrade quality because we can't get it all done by hand in a reasonable period of time.

Her solution is for the industry to require and enforce the discipline to produce software that can be tested efficiently and comprehensively through automation. If the industry were to impose testability on software products, she says, test automation would become an accepted, expected discipline as an integral part of assuring the timeliness and quality of the final product. The billions of cost savings could then be funneled into new products and functionality.

The alternative, she says, "is to sacrifice quality for development expediency until the lawyers step in to punish us for costly catastrophes."

Hayes is not optimistic. "Unfortunately," she says, "If the past twenty years is any indication, we are firmly committed to the latter path. In this case, it is only a matter of time until the government steps in to regulate liability in the software industry or we are trampled by foreign competition."

--Lindsey Vereen

Test & QA Report is published weekly by BZ Media LLC.

Advertisers in Software Test & Performance Magazine: Hot Test & Performance Links!

Automated QA
Checkpoint Technologies
Empirix
HP
IBM
iTKO
Parasoft
Pragmatic Software
Seapine
Tomos