Bugs are small. Usually. Unless, of course, they are of the software kind, in which case they can grow quite large and become hugely expensive to fix. The Sustainable Computing Consortium, a collaboration of major corporate IT users, university researchers and government agencies, estimates that buggy or flawed software cost businesses $US175 billion worldwide in 2001. In the US, software bugs cost companies nearly $US60 billion per year, according to the Commerce Department's National Institute of Standards and Technology (NIST). More important is that one-third of these costs could be eliminated with improved testing that catches errors earlier in the software development process, NIST says.
Smart CIOs are creating comprehensive strategies to test for and fix bugs in both off-the-shelf software and applications created in-house. They know that bugs, like infections, fester the longer they hang around and, as a result, cost more to deal with when left unchecked.
Gartner analyst Theresa Lanowitz says a software defect left unfixed until late in the development cycle costs 80 to 1,000 times more to fix than it would if it was dealt with earlier. No company is immune to the potential costs of software bugs, which is why a comprehensive plan for dealing with them is critical.
"Bugs are a way of life. You have to accept that," says Vivek Wadhwa, CEO of Relativity Technologies, a software vendor in Cary, North Carolina.
"Every piece of software will have bugs in it," agrees Lanowitz.
Some are more easily recognised than others. For example, CIO Jose Marrero and his team at Agco, a $US2.5 billion manufacturer and distributor of agricultural equipment in Duluth, Georgia., recently thought they were updating a particular record in a database when, in fact, because of a bug, the software was updating a different field. The cost to analyse and fix the problem: $US30,000 to $US40,000.
Human Bug Busters
Experts say software quality assurance is as much a people issue as it is a technical one. "If you don't change the way people work, it won't help. To solve quality problems, you have to change what software people do," says Watts S Humphrey, a fellow at Carnegie Mellon University's Software Engineering Institute.
The institute trains programmers and engineers to work in self-directed teams and to manage their work. They become owners of their plans and processes and therefore take more responsibility for the quality of their products upfront. As a result, says Humphrey, engineers, who might spend upward of 50 per cent of their time testing software after it's developed, now devote as little as 10 per cent of their time to postdevelopment testing.
Regular communication among software developers and all stakeholders, including users, is also essential. Lanowitz recommends that software development project teams, which should include the development manager, test manager and business champion, meet weekly, with the same participants attending each meeting to ensure that everyone involved knows what progress has been made and what problems have arisen. If bugs are holding back a project, everyone needs to know as early as possible.
This strategy is typically used by commercial software development companies, but not by user companies, says Lanowitz.
Foster Wheeler, a design, engineering and construction services company in Clinton, New Jersey, has created Web-based software for internal use called Bugtracks. It tracks all information about bugs in real time, including who is working on any given bug, who is ultimately responsible and who the project managers are on both the business and technical sides. Bugtracks, says Foster's corporate IT director, Victor Slaicetti, "allows business users to see what the status of bugs is. It ties the actual developer to the actual business user. The benefit is, you put everybody on the same sheet of music."
But harmony isn't always the result. One reason is that some developers don't like to admit that they are the source of bugs, which is understandable. "Developers like to hide their bugs. They regard them as a weakness or inferiority," says Wadhwa. Observers say this is especially true once the software has gone into production.
To counter this problem, Jonathan Clark, chief technology officer at software tester VeriTest North America in Morrisville, NC, offers financial incentives to developers who find bugs in other developers' software during the development process — say, $US50 to $US100 per hit, depending on the severity of the bug. Wadhwa gives similar incentives, plus a party to the team that finds the most bugs by the end of the project. Such incentives motivate developers to find bugs before the software is deployed.
But first, bugs must be defined "Sometimes the [internal] customer thinks it's a system issue, but about 90 per cent of the time we have bugs reported by customers, it's more a process or data issue," says Art Data, vice president of IT at International Truck and Engine in Warrenville, Illinois.
Defining Your Terms
When someone on the business side can't do what he expected to because of software limitations, he calls it a bug. But the developer usually views this as a change request, because the functionality wasn't initially specified.
For example, IDC analyst Christine Tenneson says she has seen cases in which the software for a database covering multiple regions was designed with selections for only general geographies, such as the Americas, but users wanted to select by country. Users call this a bug; developers call it a change request.
For this reason, Andrei Chivvis, CIO at Converium Reinsurance in Stamford, Connecticut, does what might seem obvious: He requires users to sign off when they define their requirements and after they test in terms of performance. This forces users to think upfront about what they want, Chivvis says.
Before he implemented this formal sign-off process, users were involved with software development but frequently didn't think things through to the finished product. It was believed that if changes needed to be made later, that was fine. This tended to shorten the development time but caused more headaches after the software was created. Chivvis is trying to change this problem with his new strategy, which he acknowledges is a difficult culture change. Users need to understand that once the testing begins, specifications don't change.
A debugging strategy must also include the programmers, notes David J Agans, author of Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems (Amacom Books, 2002). "The assumption is that because they have a degree in engineering, they know how to debug." Not true, Agans says. Developers often approach debugging as puzzle-solving, but it's actually more investigative, a la Sherlock Holmes, he says. So train your developers in debugging.
To accommodate the growing number of external users, Bob Grawien, vice president of application development at trucking firm Schneider National in Green Bay, Wisconsin, implemented a comprehensive testing procedure to be used during the in-house development process. It begins with testing specific features in an isolated environment on a developer's workstation, followed by testing in an isolated system-level environment. Next comes application load testing, scalability testing and, finally, a user acceptance test. All of this is aimed at preventing bugs from making it into production.
When bugs do creep through Schneider's process and are found in production, there's an IT help desk for users to call. If that group can't determine the cause of the problem, it moves up to the next group, whose job is to keep production running. If the bug is still running around unchecked, the development team is brought in. This escalation process lets the development team focus primarily on development, while others deal with bugs first.
Schneider's bug management program has also implemented consistent ways to notify users of problems. For example, if a logistics application has a bug, IT has a list of users it calls to alert them to a problem in production, let them know what the ramifications might be and tell them when to expect the software to be up and running again.
"We tend to view the world as an iron triangle: people, process and technology," notes Grawien. Project managers, for instance, are given well-defined deliverables and responsibilities.
The company recently built isolated systems for a wide range of tests, with the goal of providing consistent quality improvement. And, on the technology side, it has invested in hardware and software, including a Segue Software tool that simulates multiple users, and a Sitraka software tool that analyses software performance. Schneider also provides training on the tools for its development teams.
Menasha, a manufacturer of packaging materials in Neenah, Wisconsin, goes a step further.
"It's not enough [to test] that the software ran to completion. We put a business model against it to validate it will respond in production," says CIO Ed Wojciechowski.
Testers consolidate a number of scripts for business processes, such as processing a sales order to ship a product. They then compare the software against historical trends, which, among other things, indicates how well current software works for the same business processes from a profit-and-loss perspective. It's not enough that the new software works technically; it must perform better than what it's replacing in the real-world production setting, he says.
It's estimated that less than 3 per cent of insects are actually pests in the natural world. VeriTest's Clark says something similar happens with software: "One per cent of software bugs tend to cause 90 per cent of the problems. [Fixing] your top 10 bugs would solve so much of your problems."
Software bugs can't be eliminated any more than all the insecticides in the world can eliminate creepy bugs. But experts conclude that if you focus on the peskiest ones, you'll go a long way toward eliminating most of your bug-related headaches.
Horowitz is a freelance business and technology writer in Salt Lake City, Utah.
Dealing With Packaged Bugs
Developers of third-party software are human too, which means that their products will inevitably be buggy. Consistent communication is key to getting fixes, IT managers say.
— "Report the problem, provide information as requested, and follow up, follow up, follow up." That's what Dan Bent, CIO at Benefit Systems in Indianapolis, says he does when he encounters third-party bugs.
— It helps to have one person in IT who works with each major third-party vendor and who knows the people there, advises Bob Grawien, vice president of application development at Schneider National.
— Agco CIO Jose Marrero participates in vendor advisory panels and gets involved with other users of the same software. When you get enough customers together, vendors will do just about anything for you, he says.