Mac OS X 'Mountain Lion's' Gatekeeper Not Enough to Fight Malware

Apple's latest attempt to combat Mac malware with Gatekeeper in the upcoming Mac OS X 10.8, or "Mountain Lion," doesn't go far enough to protect users, security experts said.

Apple and Microsoft have added security features to the next versions of their respective operating systems in an effort to combat new, more complicated security threats. However, security experts remain skeptical the steps taken would be enough against these malware threats.

On Feb. 16, Apple previewed the new Gatekeeper security feature for its new operating system, Mac OS X 10.8, or €œMountain Lion,€ which is due this summer. The new security setting lets users define what sources would be able to install software on the system. Gatekeeper would prevent users from downloading and installing malicious software from uncertified and pirated sources.

By default, Mountain Lion would allow users to install only applications found on the Mac App Store, the application store Apple launched a little over a year ago for desktop and laptop software. While the Mac App Store offers "maximum security," users can download software from sites that have a signed Developer ID certificate, or from any source.

Gatekeeper is "designed to drive up costs and effort" for developing malware on OS X, said Roel Schouwenberg, a senior researcher with Kaspersky Lab. However, he didn't think Gatekeeper would "bring a stop to OS X malware."

With Gatekeeper, Apple is tacitly admitting that Mac malware does exist, and that it's increasing. Apple is trying to counter the threat by making it more expensive and difficult for cyber-criminals to develop malicious applications.

Developers can either go through Apple's vetting process to get listed in the Mac App Store or sign up for a developer account and receive a valid digital certificate to sign the software. If Apple finds out a developer is releasing malicious programs, it can revoke the certificate, forcing the developer to try to obtain a new certificate.

However, cyber-criminals have in the past successfully posed as legitimate companies and tricked certificate authorities into issuing digital certificates, Schouwenberg noted. There's no reason they wouldn't be able to pay, or use a stolen credit card to pay, the $99-a-year fee to join the Mac Developer Program and get a valid digital signature. The criminals can also steal someone else's certificate and use it to sign their malicious software if they can't create an account.

It's not that far-fetched to consider criminals would be able to take someone else's certificate. In the case of Stuxnet, its creators signed with a stolen digital certificate, said Schouwenberg.

Another problem with Gatekeeper is that Apple is making software development more expensive for legitimate developers as well, said Schouwenberg. It was possible that the developers who don't want to deal with paying for the certificate, or figure out how to use it, would ask users to temporarily change Gatekeeper settings "for compatibility reasons" in order to download software, he said.

Gatekeeper is a "pretty good idea," but the implementation is "flawed," Chester Wisniewski, a senior security advisor at Sophos Canada, wrote on the Naked Security blog. Gatekeeper is based on the LSQuarantine technology that powers XProtect, a rudimentary scanner integrated into Mac OS X to check whether a file being downloaded is a known piece of malware. Gatekeeper would help reduce user exposure to known Trojans by reducing where they can download from, said Wisniewski.

"It's what Gatekeeper doesn't catch that might inspire budding criminal authors to take the next step in creating more advanced malware for OS X," Wisniewski wrote.

At the moment, if the source of an infected file is a USB drive or networks share, and not the Internet, Gatekeeper won't be able to detect the malware, Wisniewski said. Digital signatures apply to only executable files, which means users remain vulnerable to malicious PDFs, Flash, shell scripts and Java. There are plenty of ways for malicious developers to keep creating new ways to attack Mac OS X.

Apple is clearly "betting on reputation" to fight malware, said Schouwenberg. While reputation plays a significant role in anti-malware efforts, it is not enough, but it just encourages criminals to adopt more "anti-reputation" techniques, said Schouwenberg.

There may be an uptick in the number of "Trojanized applications," where a perfectly legitimate download has been modified to include malware, said Schouwenberg. There have already been a few such cases, although they remain rare.

"It makes sense for the malware evolution to go this way," said Schouwenberg.

Apple is not the only one trying to beef up the operating system's security capabilities.

Microsoft is integrating antivirus software into its Windows 8 operating system, which the company plans to release later this year. The existing Windows Defender program, which Microsoft first began shipping with Windows Vista, will be expanded to incorporate the existing Microsoft Security Essentials malware scanner. With Windows 8, users will be getting out-of-the-box protection against malware and a desktop firewall.

The problem is that many users will think that since they have built-in security software, they don't need to get a comprehensive security application. Considering that many malware developers test their latest creations to make sure they can't be detected by popular antivirus software, criminals would start targeting the broad segment of users who will just have the built-in option, Schouwenberg predicted.

Considering that Windows Defender hasn't "done too much" to impact the security landscape in regard to reducing threats since its inclusion in Vista, it's not likely the new features in Windows 8 will make that "much of a difference," said Schouwenberg.