Emotion detection in videos using non sequential deep convolutional neural network

Abstract

Emotions are fundamental for humans. They affect perception and everyday activities such as communication, learning and decision-making. Facial expression and body language are the main sources of this information. The goal is to classify these emotions to improve human-computer interaction. In proposed method, a non-sequential deep convolutional neural network is presented. It consists of multiple networks which run in parallel. These parallel networks are then merged together followed by relu, max-pool, drop-out, dense and soft-max layers. In proposed model, we have used multiple networks to cover local and global feature. After feature extraction from CNN, they are fed to Recurrent Neural Network (RNN) using Long Short- Term Memory (LSTM) layer in which time dependency is included. Every current output is dependent on previous all outputs. This way a sequence is learned in complete video. After that score based voting system is used to finally assign emotion to video. The evaluation of proposed method is done by using Surrey AudioVisual Expressed Emotion (SAVEE) dataset containing four actors and Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) containing 24 actors, covering seven emotions in their videos. K fold testing is used for evaluation of our proposed model. Results obtained from each dataset were extremely positive and the recognition rates 99.64% on SAVEE and 87.49 on RAVDESS were among the highest ever achieved.