sidHadoop; Secure Inter-Domain Hadoop

Abstract

Hadoop is a big-data processing framework which is widely used for data storage and processing. Now-a-days security is one of the major concerns in the digital world. Any system is only considered reliable when it provides proper measures to secure the valuable data of an organization. Due to the vast popularity and success of Hadoop framework, its use cases started to evolve from an in-house deployment to grid, cloud and other heterogeneous environments. Researchers have provided some solutions to access geographically distant resources for Hadoop computation and storage, utilizing different techniques and frameworks. Due to security and design issues in those frameworks, we proposed to deploy Hadoop in inter-domain environment. Inter domain communication can help in collaboration without actually sharing the large amounts of data between independent Hadoop clusters. If the need to scale resources is temporary and or the resources are geographically distributed then sidHadoop can help to securely share resources of Hadoop clusters. One Hadoop cluster cannot communicate with another Hadoop cluster in the current out-of-the box setup. The proposed solution is working to achieve secure communication between two independent Hadoop clusters. For abstraction and security purpose the resources are not delegated to foreign cluster instead the master nodes communicate over WAN and post jobs for each other. The jobs are run within a cluster just like a single independent Hadoop setup. This way, the Hadoop core features are not disturbed and the benefits of Hadoop are still achieved. Our solution helps in the collaboration among different Hadoop clusters. It has use cases in academia and business world. It can ease the collaboration of resources of organization with multiple Hadoop deployments that are geographically distributed. It can help to utilize/control/manage all these deployments from one single location. Similarly, different educational institutions having their Hadoop clusters and collaboration agreement with other institutions will be able make use of data and/or resources of inter-institute Hadoop clusters in a secure manner.