Investigating Adversarial Attacks in Hypervisor-based Intrusion Detection Systems using an Intrusion Detection Bayesian Game

Abstract

In recent times, cloud computing has become significantly popular as it offers on-demand access to computing resources via internet. Despite its widespread use, security remains a significant concern due to cyber-attacks. Intrusion Detection Systems (IDSs) are vital for identifying and mitigating these attacks in the cloud. Despite their effectiveness, they lack the perspective of adaptive and smart adversaries, leading to a gradual performance deterioration. Game theory is a valuable tool to model the strategic interaction between cloud-based defenders and attackers. However, existing Game Theoretic IDSs employ non-comprehensive utility functions with limited parameters, failing to capture the com plexities of the real-world cloud environment. We aim to enhance the security of the virtualization layer in cloud through Game Theoretic Hypervisor-based IDS (GHyIDS). Our objective is to enhance the efficiency and effectiveness of GHyIDS by formulating comprehensive utility functions for both hypervisor and adversary. The proposed utility functions incorporate crucial parameters such as VM trust, attack risks, vulnerability, damage severity, adversary means, opportunities, and access level, VM worth, attack penalties, and success rates. We formulate a Resource-Aware Static Intrusion Detection Bayesian Game (S-IDBG) between the hypervisor and VMs, extending it into a Dynamic Multi-Stage IDBG (D-IDBG) to adapt GHyIDS to the evolving attack environment of cloud computing. Additionally, a belief update model enhances the adaptability of the proposed IDBG model, enabling the hypervisor to refine its understanding of VM types based on observed behaviors, thereby improving detection accuracy. The effectiveness of the proposed model was evaluated across 50 VMs over 100 game stages. The IDBG model demonstrated promising outcomes even as the attack threshold increased pro gressively. Experimental results showcased the enhanced detection rate, reduced false positive and false negative rates, and efficient performance when compared to state of-the-art cloud-based intrusion detection models, namely, Trust-based Maxmin Game (TMMG) and Repeated Bayesian Stackelberg Game (RBSG).