Performance Evaluation and Accuracy Improvement for Anomaly and Bot Detectors

Abstract

Due to the ever-rising network security threats, the domain of intrusion detection is progressing at a fast pace, and new detection techniques are being proposed quite frequently. However, little e ort is being expended into carrying out comparative evaluations of the techniques and summarizing the body of knowledge that exists. Consequently, there is a lack of guidelines based on the state of art, which can help shape the design of future detectors. In this thesis, we make an e ort to address this gap in the research literature in two sub domains of intrusion detection: (i) entropy based anomaly detection, and (ii) botnet detection. First part of the thesis focuses on entropy based anomaly detection systems (ADSes). Although entropy based measures have been widely used in ADSes to quantify behavioral patterns, and these measures have shown promise in detecting diverse set of anomalies present in networks and end-hosts, it is unclear if full potential of the entropy tool is being exploited. We survey and investigate the usage of entropy for anomaly detection and show that the full potential of entropy-based anomaly detection is currently not being exploited because of its ine cient use. We highlight three important shortcomings of existing entropybased ADSes and propose e cient entropy usage to mitigate these shortcomings. Second part of the thesis focuses on botnets, which are regarded as the most signi cant security threat facing the Internet today because of their massive computing power and bandwidth. iii iv Botnets are used today to launch large scale distributed denial of service attacks, harvest loads of personal information and generate vast amounts of spam. Security response to the botnet threat is however in its infancy; over the past years, few host and network-based bot detection techniques have been proposed in research literature. However, a comprehensive and judicious performance comparison of these techniques has not been performed, mainly because of the unavailability of open-source bot detector implementations and labeled bot datasets. We perform the rst comparative evaluation of prominent bot detection techniques on a dataset containing tra c patterns of a diverse set of IRC bot malware, and release our dataset and open source detector implementations publicly for future performance evaluations by the research community. Based on the evaluation results, we highlight strengths and weaknesses of di erent techniques and outline the state-of-the-art in bot detection research. We also propose promising guidelines that can be used to improve the performance of bot detectors.