URL-Analyzer Intelligent Detection and Classi fication of Malicious URLs using Natural Language Processing

Abstract

Uniform Resource Locators (URLs) have been a basis of the Web since its origin. They are the reference point to any resource in the cyber space. Ac- cording to Verizon DBIR 2016 report, attacks on web applications are the single-biggest source of data loss and they account for over 40% of incidents resulting in data breaches. The main challenges are that: Firstly, URLs are sometimes hidden, shortened or encoded which humans cannot readily identify as legitimate. While this is typical for URLs, the attackers utilize it to their advantage. Secondly, the automation of attacks using domain generation algorithms (DGA) and exploit toolkits have led to the need for the automated and proactive protection system for malicious URL detec- tion. Thirdly, the attackers can manipulate the users to redirect them to the intended URL without the need to click by a variety of attacks such as Phishing attack and drive-by-download attacks. Existing malicious web site detection techniques have limitations in terms of accuracy and time that have inverse relation and are di cult to achieve at a good rate. Also, few research works are focused on speci c attack types such as domain typosquatting. The aim of this work is the development of a system, URL-Analyzer, for malicious URL detection with good accuracy and time trade-o . This novel contribution is based on Natural Language Processing(NLP) technique and a breadth of features: (i) URL lexical, (ii) host based, (iii) social reputation and (iv) time-based features for the detection of phishing, malware, drive by download and typosquatting. The input to the URL-Analyzer comprises of known benign and malicious URL datasets. To detect signs of malicious- ness in the URL, static analysis technique for feature extraction and machine learning for classi cation is employed. The accuracy achieved classi cation ranges from 97% to 98.5% with average time to extract features 1.98 ms, 8 sec and 34.25 sec per URL for the URL-Analyzer's three developed modes: (i) o ine, (ii) online and (iii) partially online respectively.