Security software firm Malwarebytes revamped its process last week for testing its malware-definitions file after an error allowed an over-broad check for malicious software to quarantine system files, disrupting thousands of computers.
On April 15, the company posted a regularly scheduled update for its definition database, the file its product uses to identify potentially bad software. The update caused Malwarebytes’ Anti-Malware (MBAM) to quarantine valid system files, disabling computers protected by the software.
The company pulled the update from its servers within eight minutes, but thousands of computers had downloaded the new definitions and subsequently crashed, the company said in a blog post.
“It’s been a rough week here at Malwarebytes, and I’m sure for many of you as well,” the company’s founder and CEO Marcin Kleczynski stated in the blog post. “We’ve spent the entire week focused on supporting the users affected by Monday’s false positive, as well as implementing systems to prevent this type of problem from ever happening again.”
On April 18, the company posted more details about the incident and its response. The update that caused MBAM to identify system files as malicious code—a “false positive” in industry parlance—was caused by a corrupted file, not a developer mistake, according to the post. Yet, other mistakes compounded the issue, Kleczynski told eWEEK in an email.
“We were in a rush to update a zero-hour exploit that was not detected by any other virus engines on Virus Total, and in our rush, we made several critical mistakes,” he said. “Again, nothing is more important to us than the trust and safety of our customers, and we are putting the necessary processes and systems in place to stop anything like this from happening again.”
The company provided a tool to help users fix the issue but many customers had trouble recovering their systems, according to forum posts. Affected systems could be booted into safe mode or, in the worst case, the system only displayed a black screen and a mouse pointer after booting, according to one post.
In the future, the company will test its updates against a collection of virtual machines designed to replicate the most common configurations used by its customers. In addition, the company plans to increase its support staff and have a plan in place for phone support.
Anti-malware software has to walk a tight line. Software providers must blacklist malicious programs and be aggressive enough to catch as much malware as possible, but without designating good programs or services as malicious.
In March, the infection of an ad network’s home page led Google’s automated Safe Browsing system to identify the entire ad network as malicious, a designation that rippled out to every client and customer, leaving major Websites—such as ZDNet and The Guardian UK—with a malware alert when viewed by Google’s Chrome browser.
Whitelisting technology provider Bit9—whose software allows companies to just allow known, legitimate programs—suffered the opposite situation, when attackers stole a digital certificate and signed their own malicious programs to pass them off as legitimate.
In the latest incident, Malwarebytes promised to work to improve the quality of its products and its process.
“We are building more redundancy to check our researchers’ work and improving our peer review,” the company’s CEO said.