Encrypted Traffic Analysis for Detecting Malicious Insider

Abstract

Recently the use of secure protocols on web such as HTTPS (Hyper Text Transfer Protocol Secure), instead of HTTP (Hyper Text Transfer Protocol), has increased widely. HTTPS provides confidentiality of information between the two parties. This increase in encrypted traffic has forced organizations to use network firewalls along with Intrusion Detection and Prevention Systems (IDPS) to analyze the network traffic for detecting attacks and vulnerabilities inside the network. Generally to inspect or govern HTTPS or encrypted traffic inside the network, the organization relies on the unencrypted traffic to be inspected by firewalls and intrusion detection system (IDS). A Virtual Private Network (VPN) is a service which hides even the unencrypted traffic of the user by creating a secure tunnel, generally protected by HTTPS, between the service provider and customer. This allows any VPN service to bypass the filters or signatures applied on any network security appliances. In addition to this, these services may be used to leak any sensitive information or an entry point for any new threat for the network. In this study we have proposed a novel approach to safeguard the network from such VPN activity. The communication between the client and the server is analyzed and multiple features are extracted from network (IP), transport (TCP, UDP) and application layer (HTTPS, DNS). These extracted features are not encrypted and helps the system in classifying the network traffic. By analyzing DNS (Domain Name System) packets and HTTPS (Hyper Text Transfer Protocol Secure) based traffic the traffic is classified. Once the traffic is classified, server’s IP, TCP port connected, domain name of each connection is analyzed. Based on the analysis the system is able to differentiate between legitimate and VPN-based connection. Our proposed system has no added overhead in terms of network traffic and is light weight due to the analysis on plain traffic only