Microservice-Oriented Distributed framework for resource intensive workloads

Abstract

Fixed allocation of GPU resources to virtual machines increases idle time in utilization of GPU resources when the workloads are being executed on a different machine and increases the cost of hardware as it requires GPUs for every virtual machine. Recent solutions optimise scheduling algorithms in container orchestration environments to distribute workloads across machines having GPUs directly attached to them. However if the workloads are distributed across different machines but require GPU for short periods, GPU resources will stay idle on different machines for remaining time and that results in increased cost and under-utilization of the available resources. To address this under-utilization problem we present a framework to arrange available resources in a way that it allocates GPU to a machine only when required for processing and after processing that GPU can be shared with other machines for their workloads. The Key of our framework is to create a pool of all the available GPUs and then reserve a GPU for workload if it requests processing and add that GPU back to the pool of available resources once released from workload. Therefore, this framework assures the maximum utilization of GPUs with minimum available resources that results in significant decrease in cost as well. Furthermore this framework proposes integration container orchestration through kubernetes, provisioning resources and managing kubernetes clusters through Rancher. This provides an end to end infrastructure to deploy workloads in a containerized environment and improve utilization of available resources. Experiment results show that with our approach there’s very little overhead with time but we do not need to directly attach GPU on each virtual machine to execute workloads.