The widespread disruptions caused by this week's SQL Slammer worm demonstrated yet again the importance of proactive vulnerability patch management, users and analysts said.
Slammer, a self-propagating worm also dubbed Sapphire and SQL Hell, surfaced Jan. 25. The worm infected computers by means of a known flaw in Microsoft's frequently patched SQL Server database software. Slammer works by copying itself onto vulnerable computers and then using those systems to scan for and infect other machines running SQL Server.
As was the case with predecessor worms like Nimda and Code Red, Slammer could have been thwarted if users had applied a patch that Microsoft issued more than six months ago.
The administrators of affected servers "most certainly hold some responsibility for their negligence," said Mike Tindor, vice president of network operations at First USA Inc., an Internet service provider in St. Clairsville, Ohio.
Several of the network performance problems First USA suffered because of Slammer resulted from three unpatched systems that it was co-locating for customers. "Obviously, [the problems] could have been avoided if our customers had performed the proper security updates," Tindor said.
Yet despite the need, few companies have the resources it takes to keep current on all the recommended patches and security advisories that inundate them almost daily, users and analysts said.
"Systems administrators spend a lot of their time addressing day-to-day problems, so routine things such as updates get pushed into the background," said Jesse Fussell, president of Information Security Systems, an Edgewater, Maryland-based consultancy.
Software patches themselves are often unwieldy and difficult to apply and sometimes can break the systems they are intended to fix.
For instance, the patch that Microsoft had made available for the hole Slammer exploited involved in some cases a "brutally slow and manual process," said Chip Andrews, owner of SQLSecurity.com, a site dedicated to securing SQL servers.
As a result, at least "some administrators put off the patch because of the sheer time it would take to patch a production machine," said Ben Koshy, technical manager at W3 International Media, a hosting company in Vancouver, British Columbia.
Pat Hymes, vice president of corporate information security at Wachovia Corporation in Charlotte, North Carolina, said maintaining patches can be a challenge for any organisation.
"It can take a great deal of time and energy to download, test and implement service packs and hot fixes, especially in large organisations where they can impact hundreds of applications and thousands of servers," Hymes said. "The total cost of ownership for servers running some of these distributed [operating systems], databases and Web software is\ going through the roof due to the manpower being expended to maintain patches and respond to events like the SQL Slammer worm."
Claude Bailey, an IT security analyst at one of the nation's largest financial management firms, said the problem lies not in detecting the vulnerability but in deploying the patches and fixes across an organisation of 50,000 employees and guaranteeing that the patch won't cause more problems.
"We tested the original patch [for the SQL vulnerability], and it had problems," said Bailey. And now, in the middle of tax season, there's too much to lose in deploying patches that break other parts of the network, he said. As a result, the financial firm has placed a freeze on any such maintenance until tax season is over.
The patching issue becomes even harder when dealing with patches that touch core systems like a database server, said Eric Block, information security officer at Dallas-based Mary Kay.
"Database administrators can get very nervous when you tell them that a security patch could break their server," said Brock. As a result, decisions about patches sometimes can become a "risk-rewards judgement call," he said.
Even Microsoft itself wasn't above such oversight last week, with several unpatched systems becoming infected by Slammer.
"We struggle with the same issues as the rest of the industry," said Rick Miller, a Microsoft spokesman. "Some don't patch for time management reasons, some out of oversight. At the end of the day, it should have been patched."
Vendors have contributed to the problem by failing to provide enterprise-class patching and updating processes, said Paul Schmehl, adjunct information security officer at the University of Texas in Dallas. The university lost Internet connectivity for about 13 hours because of Slammer, according to Schmehl.
"Most vendors are still writing software for individual boxes instead of thinking about scaling processes to make them usable," he said.
It is in response to such concerns that Microsoft is revamping its processes for developing and distributing patches, Miller said.
For instance, the company has begun to make available easy-to-use installers for automating much of the patching process, Miller said. Microsoft is also working on tools that help companies scan their networks and identify vulnerable systems more efficiently, he added.
"We recognise that we need to do a much better job developing and delivering patches," Miller said. "We are working on it."