New approaches to malware detection coming into view

New startups play the malware defence game differently to the established security players

The traditional signature-based method to detect viruses and other malware is increasingly seen as an insufficient defense given the rapid pace at which attackers are churning out virus and spyware variants. All of which raises the question: What's next?

The three security vendors that dominate the antivirus market today, McAfee, Symantec and Trend Micro, say they have no intention of abandoning signature-based defense, which calls for identifying a specific malware sample to create a matching signature in order to detect and eradicate it. However, the big three vendors acknowledge there's a need to augment this decades-old methodology, and some of the new techniques they're devising will be unveiled as products this year.

"Everyone agrees signature-based defense is not enough," says Brian Foster, Symantec's senior director for product management, who notes the security firm receives 200,000 submissions of potential malware each month. "The number of variants is increasing."

To augment signature-based detection in its next enterprise antivirus release planned for this summer, Symantec will include whitelisting technology for policy-based control of applications down to a software-component level, says Foster. This future-looking malware protection from Symantec will also make use of behavior blocking that promises to be able to stop at least some malware from executing, holding it "in a frozen state on that machine," says Foster. "The core of our strategy is, we will change the game."

Security startups to watch

In the meantime, some brash start-ups say they realized years back the malware-defense game had changed -- and they're now elbowing their way in by playing it differently.

One is SignaCert, launched earlier this year to market enterprise desktop and server software that can be used to create a white list that only allows specified applications and files to work.

"We've definitely reached a point of diminishing returns with traditional signatures," says SignaCert chair and CEO Wyatt Starnes.

With its Enterprise Trust Server product, SignaCert has created encryption-based signatures of binary-software releases obtained directly from vendors, including Sun, Microsoft, IBM and Intel. "Because you know what the good code is, you don't let the bad code run," Starnes says.

Page Break

Another newcomer eager to shake up the old order is Robot Genius, which is making its formal debut on April 30.

"The problem we're trying to solve is malware," says Stephen Hsu, co-founder of the company, which has 10 employees and US$2 million in venture-capital funding. "Traditional approaches rely on reactive protection methods, which protect users only after a new threat has been discovered."

By contrast, says Hsu, "We're coming up with a new kind of security client that has behavior capabilities to identify malware. And we have a Web crawler that looks at Web pages to find executables that are malware, and we will warn you, or block, when you're about to do something deemed to be unsafe."

The Robot Genius client software, called Spyberus, uses a driver-based filtering technique to monitor and track like an audit trail all installed files on a system.

Spyberus detects malware and reverse malware infections using a Take Control feature to stop malware-hijacked processes.

Robot Genius plans to make Spyberus available as a free client for Windows XP and 32-bit Vista in early May. "Spyberus can't run on 64-bit Vista because of what Microsoft did with kernel protection," says Hsu.

Robot Genius plans to release a free browser plug-in later in May. Hsu adds he's working with a "major search engine," which he declined to name, that is sharing a subset of Web crawl data it collects so Robot Genius can identify harmful executables through what is says is a largely automated method.

Hsu says the McAfee, Symantec and Trend Micro, as well as other security vendors, are working on similar methods of malware detection, and he intends to show Robot Genius will do it better.

"We have their scanning engines, and the best antivirus products only catch 60% of the malware we detect," boasts Hsu.

Robot Genius's strategy calls for licensing its technology to antivirus companies, network firewall vendors and search engines that want to block Web-based malware which is often found in online games, screen savers, toolbars and small applications dispensed over the Web.

Some search engines have already started getting aggressive about keeping malware from hitting their users. Google, for example, is a supporter of the coalition, and last year started warning users about malware when search queries turned up links to sites that the coalition cites sources of "badware" deemed harmful to users.

Hsu -- a professor in theoretical physics at the University of Oregon who found earlier success selling a company he founded, SafeWeb, to Symantec -- may not be easy for the larger security vendors to dismiss.

"They've come up with some interesting technology," says Gartner Research Director Peter Firstbrook. "We see this as being in the area of the 'secure Web gateway.'"

Firstbrook says about two dozen vendors, including Websense, SurfControl and Secure Computing's SmartFilter, have all devised some method for detecting or blocking malware downloads through URLs. "The antivirus vendors also are doing things here, such as Trend Micro with its reputation services," he says.

Robot Genius has some "realistic market opportunities" but as a small start-up could be "spreading themselves pretty thin," Firstbrook says.

Page Break

Trend Micro, McAfee make plans

Trend Micro's director of Internet content security, Paul Moriarity, says the firm is looking beyond signature-based defense, which he says "has utility but some limitation."

He says Trend Micro is investing in technologies to determine malware based on patterns of traffic to desktops or servers. In addition, Trend Micro's researchers are increasingly convinced that Internet users can be saved from stumbling into Web-based malware by just keeping them away from Web sites whose domain names have existed less than five days.

"You should generally be skeptical about domains less than five days old," says Moriarity. Web sites containing malware are often established using what he called "domain kiting," registering a domain for free for five days and then cancelling, and then re-registering as another entity.

Moriarity says he applauds Robot Genius for "taking a different tack to solve the problem," noting that "scanning the Internet for malware is a very good approach." But he voices doubt that the Spyberus client's behavior-based detection would prove viable.

In the industry, "there's a lot of talk about looking at the behavior of malware, but I'd say that's false hubris," Moriarity says.

At McAfee, the focus remains on signature-based detection, augmented by host-based intrusion prevention, which was added to the eighth version of McAfee's antivirus products.

"I understand why some would think signatures are dying," says David Marcus, McAfee's security research manager, adding, "but it goes back to someone not really understanding what a signature is. Some cleaning and repair can't be done without them."

McAfee identifies 125 to 130 unique new forms of malware each day and turns around virus signatures in as little as two to four hours. "It's definitely manageable," says Marcus.