Detection of rootkits using memory analysis

Abstract

Rootkits are malicious software designed to hide their presence and activities on compromised systems. Traditional detection methods often struggle to identify rootkits due to their ability to mimic normal files and evade detection. However, volatile memory analysis has emerged as a powerful technique for monitoring system activities. In this thesis, we propose the use of memory analysis combined with machine learning and deep learning to develop an effective rootkit detection model. By analyzing the contents of the system’s volatile memory, our model aims to identify suspicious patterns and behaviors that indicate the presence of rootkits. We employ machine learning and deep learning approach to train the model on a comprehensive dataset of known rootkit samples, enabling it to learn and recognize the distinct characteristics associated with these stealthy malwares. Through extensive experiments and evaluations, we assess the performance and accuracy of our model in detecting various types of rootkits. The results demonstrate the effectiveness of memory analysis combined with machine learning/deep learning in rootkit detection, offering a promising solution to combat the ever-evolving threat of these elusive malware. This research contributes to the development of advanced defense mechanisms and enhances the security posture of systems against rootkit attacks.