Behavior-Based Anti-Malware Approaches
Behavior-based anti-malware approaches Behavior-based approaches to malware detection monitor behaviors of a program to determine whether it is malicious or not. The behavior of a program that is typically monitored is the stream of system calls that the program issues to the operating system. Since behavior-based techniques monitor what a program does, they are not susceptible to the shortcomings of signature-based detection discussed earlier. Simply put, a behavior-based detector determines whether a program is malicious by inspecting what it does rather than what it says.Anomaly detection One major approach to behavior-based detection is anomaly detection. In this approach to malware detection, a profile of normal program behavior is constructed. Any deviations from that profile are flagged as anomalous and thus suspicious. Anomaly detection is analogous to credit card fraud detection. Credit card companies maintain "spending profiles" for their customers. Any significant deviation from these profiles is flagged as suspicious. For example, if a credit card company notices a large expense in a shop in Europe, and the customer has not shopped in Europe in the last few years, they will flag that transaction as anomalous. Similarly, let's say a program, during its normal execution, never writes to a certain sensitive directory. If the monitoring system notices writes to that sensitive directory from the program, the detection system will flag that behavior as anomalous. Anomaly detection has the following two shortcomings: Shortcoming No. 1: It is susceptible to false positives Normal behavior for complex programs is very complicated. For example, the set of behaviors of Internet Explorer are very complex. Therefore, it is very hard to construct a model of normal behavior of a complex program. An inadequate model of normal behavior can lead to false positives. Shortcoming No. 2: It is susceptible to mimicry attacks It has been demonstrated that anomaly detection-based techniques are susceptible to mimicry attacks. In a mimicry attack, an attacker transforms his attack into another equally-malicious attack, but the transformed attack is allowed by the model of normal execution of the program.
It is clear that the industry needs to move beyond signature-based detection. But how that will happen is still very much a debate. Several types of behavior-based detections exist.