RESEARCH DESIGN AND IMPLEMENTATION OF ZERO-DAY MALWARE DETECTION SYSTEM USING WINDOWS API CALLS

Abstract

The traditional anti-virus software using signature based approach cannot cope up with the unprecedented threats like zero day malware. The malware growth phenomena these days necessitate behavior based anti-virus systems. We provide a novel solution that monitors process functionality at thread level and classifies them as benign or malware at runtime. Runtime monitoring of program execution behavior is widely used to discriminate between benign and malicious processes running on an end-host. Towards this end, most of the existing run-time intrusion or malware detection techniques utilize information available in Windows Application Programming Interface (API) call arguments or sequences. In comparison, the key novelty of our proposed tool is the use of statistical features which are extracted from both spatial (arguments) and temporal (sequences) information available in Windows API calls. We provide this composite feature set as an input to standard machine learning algorithms to raise the final alarm. The results of our experiments show that the concurrent analysis of spatio-temporal features improves the detection accuracy of all classifiers. We also perform the scalability analysis to identify a minimal subset of API categories to be monitored whilst maintaining high detection accuracy. Our system logs and analyzes the Windows Application Programming Interface (API) calls being made by running processes categorized according to their threads. Newly spawned processes are also immediately passed through the test. The API hooking is achieved using DLL injection and IAT patching technique. The sequences of API calls as well as their arguments are used in classification. This provides for a more robust detection system, relying on both statistical features and n-grams of relevant API calls