Seamless Background Replacement and Emotion Recognition Using Machine Learning

Abstract

Video conferencing has been on a gradual rise since the last decade but in the ongoing pandemic its usage has soared to new heights. People are holding meetings and discussions online as well as keeping in touch with their loved ones. One major issue in video conferencing is to have a clean and tidy background because a messy background can look unprofessional and cause distractions. To overcome this issue applications have released a feature where users can replace their backgrounds with any other image. The person in the video will be displayed as it is but the background will be intelligently replaced using AI. This project is an implementation of the same feature. We have used machine learning to identify humans and backgrounds in a video feed. The background can then be blurred or changed dynamically to any image or looping video. The background can also be replaced to reflect a user’s emotions. This project can also be helpful for professional video editors who use effects like chroma keying (commonly known as green screen) to remove background from videos. Our segmentation model has achieved 82% testing accuracy while the emotion recognition model has achieved 64% testing accuracy. Both models run on more than 20fps on even low end computers.