Abstract:
In our final year project we have tried to address a very trivial issue the world faces since ages. In any community there exists such group of disable people who can’t speak or listen. The direct impact of hearing and speaking disability is that the person who suffers faces many problems when it comes to communication. Such people use various gestures and symbols to talk and convey/receive their messages and this mode of communication is called sign language. Yet the communication problem doesn’t end here, as natural language speakers don’t understand sign language so there exists a communication gap.
We have developed a software based solution using latest technologies from Microsoft (i.e. Kinect for windows V2) which acts as sign language interpreter and translator and provides a 2 way mode of communication between sign language speakers and natural language speakers.
As we have discussed earlier that our system provides 2 way mode of communication so main tasks it has to do are (i) Signs/Gesture to speech conversion (ii) Speech to sign language conversion. Subsequently we have categorized these two tasks into independent modules of the system.
In sign to speech module the person with disability comes in front of the device and performs gestures of sign language, the system receives those gestures using Kinect sensor and then comprehends those gestures by comparing them with pre stored gestures in gestures database. After a gesture is detected it is mapped to the keyword against the gesture. Which is then sent to speaking module which speaks the sentence from computer.
Apparently speech to sign module does the reverse as it translates spoken language to sign language. When normal person comes in front of the system and speaks in his native language (English for now) the system converts it into text using speech to text api. Then the keywords are mapped to their corresponding pre stored animations and then animations are played on screen in form of a sentence. So that disable person can see the translation in a 3d animated sign language.
