Abstract:
The dramatic growth in encrypted traffic changes the security landscape. As more organizations are becoming conscious of the need to protect their data, more devices, services and applications use encryption as the fundamental way to secure information. As organizations progressively use encryption to keep confidential their network information, attackers use the technology to camouflage their activities. In other words, encryption, which is essential for the protection of sensitive information such as online transactions, e-mails and smartphone applications, can make it possible for malware that hides within that encrypted traffic to move through the security system of an organization uninspected. Encrypting malware payload prevent malware analyst, to reverse engineering of malicious code and identifying malware developer’s intension. This thesis evaluates malware encryption scheme based on ElGmal cryptosystem as a proof of concept. The thesis also present the novel scheme for malware encryption, propagation and re-randomization using environmental keys, based on Paillier cryptosystem. Furthermore, the thesis includes the review of existing methods for encrypted malware traffic analysis. Keywords: malicious cryptography, homomorphic encryption, re-encryption, ElGamal, Paillier cryptosystem.
