Enhancing Multiclass Anomaly-Based Network Intrusion Detection Systems with State-of-the-Art Deep Learning Techniques on Contemporary Large Imbalanced Datasets

Abstract

The rapid expansion of digital communication and the concurrent sophistication of cyberattacks have underscored the critical need for robust, adaptive network intrusion detection systems (NIDS). This research presents the development of an advanced NIDS framework that leverages state-of-the-art Transformer-based deep learning techniques to address the shortcomings of traditional methods. The study is built upon a comprehensive approach encompassing detailed data preprocessing, the use of various input encoders, effective classification head strategies, and techniques such as SMOTE to address common challenges like class imbalance and feature overlap in benchmark datasets. Two widely recognized datasets, UNSW-NB15 and CIC-IDS2018, serve as the empirical foundation for this work. These datasets, characterized by diverse and large volumes of network traffic data, facilitate a rigorous examination of the model’s capabilities in both binary and multi-class classification scenarios. The research begins with an in-depth exploration of existing detection techniques ranging from signature-based to anomaly-based methods—and moves towards the implementation of a Transformer architecture designed to capture intricate long-range dependencies through self-attention mechanisms. Critical enhancements to the baseline model include the integration of multi-head self-attention, advanced input tokenization, and a multi-class classification head supported by robust loss functions. The proposed system not only achieves high detection accuracy and significantly reduces false alarm rates but also demonstrates scalability and the efficiency required for real-time deployment. Detailed experimental evaluations, which include quantitative performance metrics such as precision, recall, and F1-scores as well as qualitative analyses through confusion matrices and ROC curve analyses, validate the effectiveness of the enhanced model. The study also addresses inherent dataset limitations by implementing advanced synthetic data generation and hybrid sampling techniques, which help to mitigate issues arising from class imbalance and overlapping feature distributions. Despite these challenges, the results clearly indicate that the proposed method outperforms several state-of-the-art systems on both the UNSW-NB15 and CIC-IDS2018 datasets. In summary, this research not only contributes a robust Transformer-based model for network intrusion detection but also provides a detailed methodology for data preprocessing, model training, and evaluation. Despite limitations in the data sets, we improved detection rates up to 6% compared to the baseline study and maintained multiclass detection accuracy above 85%.