Behavior-Based Anti-Malware Approaches

By Somesh Jha  |  Posted 2009-02-25 Print this article Print

Behavior-based anti-malware approaches

Behavior-based approaches to malware detection monitor behaviors of a program to determine whether it is malicious or not. The behavior of a program that is typically monitored is the stream of system calls that the program issues to the operating system. Since behavior-based techniques monitor what a program does, they are not susceptible to the shortcomings of signature-based detection discussed earlier. Simply put, a behavior-based detector determines whether a program is malicious by inspecting what it does rather than what it says.

It is clear that the industry needs to move beyond signature-based detection. But how that will happen is still very much a debate. Several types of behavior-based detections exist.

Anomaly detection

One major approach to behavior-based detection is anomaly detection. In this approach to malware detection, a profile of normal program behavior is constructed. Any deviations from that profile are flagged as anomalous and thus suspicious. Anomaly detection is analogous to credit card fraud detection. Credit card companies maintain "spending profiles" for their customers. Any significant deviation from these profiles is flagged as suspicious.

For example, if a credit card company notices a large expense in a shop in Europe, and the customer has not shopped in Europe in the last few years, they will flag that transaction as anomalous. Similarly, let's say a program, during its normal execution, never writes to a certain sensitive directory. If the monitoring system notices writes to that sensitive directory from the program, the detection system will flag that behavior as anomalous. Anomaly detection has the following two shortcomings:

Shortcoming No. 1: It is susceptible to false positives

Normal behavior for complex programs is very complicated. For example, the set of behaviors of Internet Explorer are very complex. Therefore, it is very hard to construct a model of normal behavior of a complex program. An inadequate model of normal behavior can lead to false positives.

Shortcoming No. 2: It is susceptible to mimicry attacks

It has been demonstrated that anomaly detection-based techniques are susceptible to mimicry attacks. In a mimicry attack, an attacker transforms his attack into another equally-malicious attack, but the transformed attack is allowed by the model of normal execution of the program.

Dr. Somesh Jha is co-Founder and Chief Scientist of NovaShield. He has more than 19 years of research and development experience (both academic and industrial) in security and IT. He is currently a member of the faculty at the Department of Computer Science at the University of Wisconsin-Madison. He focuses primarily on computer system security and is a frequent speaker at security-related conferences and events across the United States. Dr. Jha has been recognized through numerous awards including the NSF CAREER award, ACM SIGSOFT distinguished paper award and best paper award at ACSAC. He also conducted four years of advanced research during a postdoctoral fellowship at Carnegie Mellon University's Computer Emergency Response Team (CERT). He also serves on the editorial board of the Journal of Computer Security, and on the program committees for WORM05, RAID05, USNIX Security Symposium and WWW (Security and Privacy Track). Dr. Jha completed his PhD in Computer Science at Carnegie Mellon University and B.Tech in Electrical Engineering from IIT-Delhi, India. He can be reached

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel