Abstract:
Combining videos of humans performing different gestures in a smooth way can potentially
have many uses across a wide range of fields. These include entertainment, virtual reality,
robotics, education, & communication. The goal of this research work is set in this context.
This research focuses on developing a system that takes individual videos of humans
performing motion gestures, and stitches them in a way that minimizes spatial discontinuities
between upper torso joints, thus joining two or more human gestures into one seamless
continuous motion. It begins by investigating & comparing current frameworks used to stitch
individual human motion gestures and investigates the theoretical and mathematical
approaches behind them, proceeding in a step-by-step way. First, it collects sign videos for
most commonly used English sentences of lengths 2-8. Then, it preprocesses these videos to
convert them into a standardized form. Following that, it extracts landmarks to prune
unnecessary parts of videos. It then calculates human joint coordinates using pose estimation.
After that it calculates link vectors and human shoulder, and elbow angles using linear
algebra. Following that, the system interpolates joint coordinates at transitions between signs
and uses them to calculate interpolated joint angles. Concurrently, actual joint coordinates are
used to calculate actual joint angles, which are then used to calculate wrist poses using
forward kinematics. These wrist poses are compared with the same obtained from feeding
interpolated joint angles to forward kinematic models. An ablation study was then conducted
that measured mean errors across different combinations of spline degree, percentage of
knots, & length of sentences. LSQ Univariate Spline with degree 4, knots percentage of 90%,
and sentence length of 4 produced least mean error. Transition errors (errors between sign
transitions were also calculated & recorded for each of 100 sentences. In this way,
smoothness of different interpolating functions was quantified.
