Abstract:
The amount of network traffic continuously rises, and network services are
becoming increasingly more complex and vulnerable. Intrusion detection
systems are employed to safeguard these networks. Signature-based
intrusion detection cannot detect new attacks, so anomaly detection is
necessary. Also, most of the traditional Security Information and Event
Management (SIEM) doesn’t have any live dataset creation and anomalous
data logs detection. In this solution, we propose a framework for detecting
anomalies in Hypertext Transfer Protocol (HTTP) logs which would enable
fast detection of anomalies, visualization of network traffic, and integration
with SIEM solutions. This solution gets HTTP logs from the network
and preprocesses them. Then, the solution checks the traffic for anomaly
detection by applying the Isolation Forest algorithm using unsupervised
learning. Later, we will then explore those anomalies with clustering using
the K-means method. The proposed the solution uses no predefined dataset
file. But, it trains on the live data logs, where the user checks for any
anomalous data logs within a given timestamp. The logs from this specified
timestamp serves as the dataset for the model’s training. The solutions are
on live network HTTP logs, but in this thesis, we are using this solution on
a test network where we have 153 logs, of which 32 are declared anomalous
by our proposed solution and after that, the clustering on those anomalous
data logs were applied. Finally, the results are sent to the SIEM solution,
which visualizes the network and the anomalous traffic. Besides this, the
solution is quick to detect intrusion attempts. This method is significant in
the sense that it contributes to proactive cybersecurity strategies specifically
designed to meet the requirements of Security Operations Center (SOC)
analysts and threat-hunting teams.
