Detection of Heap-Based Overflow in binary Codes

Abstract

According to SANS Common weakness Enumeration Heap Overflow vulnerability is among twenty five most dangerous software errors which if exploited in an organized manner aids the attacker to gain privilege escalation. Detection of malwares that can exploit this vulnerability requires the combination of datamining and machine learning techniques. Our work presents a hybrid malware detection technique that is the combination of both data mining and machine learning approach. For overcoming the absence of typical anti-virus software we have used static analysis technique to extract features of malwares. We extracted features from malware binaries then calling frequencies of the raw features are collected to select valuable features. Feature engineering technique is used for the reduction of the selected features. The created feature set is used to train three classifiers J48, K-Star and Simple logistic for the detection of malwares that exploit heap based overflow vulnerability. By embracing the notion of machine learning and datamining a static malware detection technique is proposed. The proposed technique is easy to implement in operations of cyber security to comprehend the behavior of malwares targeting their organizations.