AI-Driven Malware Detection and Uncovering Anomalous Traffic Patterns with SIEM Integration

Abstract

The amount of network traffic continuously rises, and network services are becoming increasingly more complex and vulnerable. Intrusion detection systems are employed to safeguard these networks. Signature-based intrusion detection cannot detect new attacks, so anomaly detection is necessary. Also, most of the traditional Security Information and Event Management (SIEM) doesn’t have any live dataset creation and anomalous data logs detection. In this solution, we propose a framework for detecting anomalies in Hypertext Transfer Protocol (HTTP) logs which would enable fast detection of anomalies, visualization of network traffic, and integration with SIEM solutions. This solution gets HTTP logs from the network and preprocesses them. Then, the solution checks the traffic for anomaly detection by applying the Isolation Forest algorithm using unsupervised learning. Later, we will then explore those anomalies with clustering using the K-means method. The proposed the solution uses no predefined dataset file. But, it trains on the live data logs, where the user checks for any anomalous data logs within a given timestamp. The logs from this specified timestamp serves as the dataset for the model’s training. The solutions are on live network HTTP logs, but in this thesis, we are using this solution on a test network where we have 153 logs, of which 32 are declared anomalous by our proposed solution and after that, the clustering on those anomalous data logs were applied. Finally, the results are sent to the SIEM solution, which visualizes the network and the anomalous traffic. Besides this, the solution is quick to detect intrusion attempts. This method is significant in the sense that it contributes to proactive cybersecurity strategies specifically designed to meet the requirements of Security Operations Center (SOC) analysts and threat-hunting teams.