Apple's new OS X tightens screws on some malware

Mountain Lion's default 'Gatekeeper' setting allows only approved Mac software to be installed

Apple will introduce a new Mac security model with OS X Mountain Lion this summer that by default lets users install only programs downloaded from the Mac App Store or those digitally signed by a registered developer.

Some experts called Gatekeeper -- Apple's name for the model and technology -- a game-changer while others criticized it as less than watertight.

Gatekeeper will block the installation of the most common kind of Mac malware yet: Trojan horses unwittingly executed by users who have been duped into downloading and installing fake software.

Last year, several campaigns of "scareware," programs that posed as antivirus software but actually infected systems with attack code, made headlines. Apple responded to the scareware threat by repeatedly updating a rudimentary blocking list that debuted two years earlier.

Apple even took the trouble during the skirmishing to issue a tool that scrubbed infected machines of the "Mac Defender" malware.

Mountain Lion, which Apple said Thursday will ship late this summer, uses a new mechanism to bar malicious applications from most Macs.

By default, only software downloaded from the Mac App Store -- the Apple curated market that debuted in January 2011 -- or signed with certificates Apple provides free-of-charge to registered developers can be installed on Mountain Lion.

Because each digital certificate is linked to an individual developer or company, Apple will know who was responsible for, say, sneaking a malicious app by users, and be able to revoke the certificate and ban the developer from its program.

Apple will not review these digitally-signed third-party programs, but Gatekeeper lets the company retaliate against malicious application makers, and by revoking certificates, gives it a way to block new installs and stifle a malware campaign in its early stages.

Mountain Lion's Security & Privacy preferences screen also has options for tightening or loosening Gatekeeper's vigilance. If "Mac App Store" is selected, only software downloaded from Apple's mart can be installed; choosing "Anywhere" lets users install programs obtained from, well, anywhere. The latter is the wide-open model that Macs -- and Windows PCs -- have used since personal computing's infancy.

At its default setting, Gatekeeper, which has roots in moves Apple has been making with OS X for several years, is a set-and-forget "whitelist," or list of approved programs. "It's like a giant whitelist button," said Andrew Storms, director of security operations at nCircle Security, of Gatekeeper.

Some security experts were enthusiastic about Gatekeeper.

Rich Mogull, a security consultant and former Gartner analyst, called it a game-changer in a post he wrote for the TidBits blog Thursday. And in a more technical description of Gatekeeper on his firm's blog, he argued it would attack hackers where it hurts.

"Gatekeeper attacks the economics of widespread malware," Mogull said. "If most users use it, and as the default, that's extremely likely, it will hammer on the profitability of phishing-based Trojans."

Steven Frank, a co-founder of Portland, Ore.-based Mac application developer Panic, judged Gatekeeper as "quite a nice compromise" between the locked-down model Apple uses in iOS -- which some Mac developers feared would migrate to OS X, requiring all software to come through the Mac App Store -- and the free-for-all of installing anything found anywhere.

"We think Gatekeeper is a bold new feature that should do wonders for the security of your Mac for years to come," Frank wrote Thursday on Panic's blog.

Others were skeptical of Gatekeeper's ability to stymie determined attackers.

"Checking signatures [the] first time a downloaded program runs is not the same as mandatory code-signing like iOS has," said Charlie Miller, a principal research consultant for Denver-based security consultant Accuvant and noted Mac and iOS vulnerability researcher, in a tweet yesterday.

Miller was referring to Gatekeeper's practice of examining the certificate of a Mac app only once, as it's downloaded and before it's installed: If the certificate is invalid, or has been revoked, the user cannot install it. But already-installed applications remain on a Mac and can continue to be run even after Apple revokes the certificate.

That's different from the code signing required of iOS apps, which prevents them from using unauthorized commands once they're downloaded and installed. (Miller knows code-signing: Last November, he demonstrated a bug in iOS that let him circumvent Apple's iOS App Store code signing.)

And security researcher Chet Wisniewski, who works for U.K.-based antivirus company Sophos, pointed out that Gatekeeper won't stymie several malware attack tactics.

"[Because] Gatekeeper code signing only applies to executable files ... anything that is not itself a Trojan, like malicious PDFs, Flash, shell scripts and Java will still be able to be exploited without triggering a prompt," Wisniewski said on the Sophos blog late Thursday.

Gregg Keizer covers Microsoft, security issues, Apple, Web browsers and general technology breaking news for Computerworld. Follow Gregg on Twitter at @gkeizer, send e-mail to gkeizer@ix.netcom.com or subscribe to Gregg's RSS feed .

From CIO: 8 Free Online Courses to Grow Your Tech Skills
Join the discussion
Be the first to comment on this article. Our Commenting Policies