Pose-Based Seamless Video Stitching for Real World Applications

Abstract

Combining videos of humans performing different gestures in a smooth way can potentially have many uses across a wide range of fields. These include entertainment, virtual reality, robotics, education, & communication. The goal of this research work is set in this context. This research focuses on developing a system that takes individual videos of humans performing motion gestures, and stitches them in a way that minimizes spatial discontinuities between upper torso joints, thus joining two or more human gestures into one seamless continuous motion. It begins by investigating & comparing current frameworks used to stitch individual human motion gestures and investigates the theoretical and mathematical approaches behind them, proceeding in a step-by-step way. First, it collects sign videos for most commonly used English sentences of lengths 2-8. Then, it preprocesses these videos to convert them into a standardized form. Following that, it extracts landmarks to prune unnecessary parts of videos. It then calculates human joint coordinates using pose estimation. After that it calculates link vectors and human shoulder, and elbow angles using linear algebra. Following that, the system interpolates joint coordinates at transitions between signs and uses them to calculate interpolated joint angles. Concurrently, actual joint coordinates are used to calculate actual joint angles, which are then used to calculate wrist poses using forward kinematics. These wrist poses are compared with the same obtained from feeding interpolated joint angles to forward kinematic models. An ablation study was then conducted that measured mean errors across different combinations of spline degree, percentage of knots, & length of sentences. LSQ Univariate Spline with degree 4, knots percentage of 90%, and sentence length of 4 produced least mean error. Transition errors (errors between sign transitions were also calculated & recorded for each of 100 sentences. In this way, smoothness of different interpolating functions was quantified.