Malware Defensive Techniques Will Evolve as Security Arms Race Continues

When it comes to fighting malware, researchers have to both keep their eyes on the present and foresee what the future may hold before the next threat is on their doorstep.

While the majority of malware attacks stick to tried-and-true methods, malware authors are getting better at being stealthy and finding ways to fight back against the security pros trying to thwart them. A well-known example of this is the Conficker worm disabling antivirus updates by blocking infected computers from accessing the Websites of security companies like Symantec.

Looking ahead, several security pros said they expect some of the defensive mechanisms of today’s malware to be refined and enhanced as cyber-criminals try to sneak past researchers and network defenses alike. Right now, for example, there are malicious programs capable of recognizing when they are being run in a virtual environment. By exploiting bugs in commonly used virtualization software, malware could potentially infect researchers as they begin their poking and prodding, noted Danny Quist, CEO of Offensive Computing.

“There have been some talks at Black Hat that result in exploiting some of the VMware drivers used on a system,” he said. “By doing this, the malware is theoretically able to infect the root machine that's being used for malware analysis.”

“Others have exploited buffer overflows in reversing tools [like OllyDbg] to gain control of a system,” Quist continued. “One new technique uses thread local storage [TLS] system inside of Windows to execute code before the expected breakpoint. TLS is being used in the unpacking process, which is typically outside the regular analysis domain. It's important to realize that reverse engineering is a cat-and-mouse game. As one side gets better at something, the opposing side improves too.”

The standard right now is for malware that detects that it is being run in a virtual environment to cease/exit, explained Patrick Martin, senior manager at Symantec Security Response. Attackers are also using more advanced cryptography (Conficker used MD6) as well as so-called "spaghetti code," which involves a code path that jumps all over the place.

“Trojan.Linkoptimizer has a rather unique anti-antivirus defense,” he said. “It involves creating an admin privileged user on the infected system and then leveraging that to encrypt itself and run at boot time.”

The best defense for malware is of course to go undetected.

“In the past, malware would talk as fast and loud as it could over the network to get its traffic connection,” Quist said. “Lately, many samples have been using trickier techniques to hide themselves. The most devious is to not talk unless necessary. I was dealing with one sample recently that did exactly this. You would see no communication except for when the user was actively using the system. At that point, it would call home and do its communication. Any sort of network forensic analysis was made extremely difficult until the root cause of the traffic was analyzed.”

The one threat multiple experts agreed was the stealthiest is Mebroot, also known as Torpig. Mebroot has been linked to the theft of mountains of data, including some 10,000 bank accounts and credit card numbers, during a 10-day period.

"The MBR [Mebroot] rootkit is the most advanced malware in terms of stealth techniques,” said Patrik Runald, formerly of F-Secure but now senior manager of security research at Websense. “It’s basically a platform to hide anything from the operating system. It doesn’t store its 'files' as files but as data in sectors. It loads before Windows does, and it took a pretty long time before AV vendors were able to detect it while active.”

For security vendors, this means it’s time to focus on more than signature-based detection, argued Gartner analyst John Pescatore. That process has already begun via a combination of in-house development and a spate of security-related acquisitions over the past few years.

“The combination of the uber-whitelists, like the Signacerts and the Bit9s out there, and some of the more advanced real-time malware detection techniques that look for these advanced capabilities does a pretty good job of limiting the success of this stuff—but the standard outbound URL blocking, signature AV at the desktop has no chance,” Pescatore said.