Sensor-Based Signer Independent Continuous Arabic Sign Language Recognition

A Master of Science thesis in Mechatronics Engineering by Mohamed Hassan entitled, "Sensor-Based Signer Independent Continuous Arabic Sign Language Recognition," submitted in May 2017. Thesis advisor is Dr. Khaled Assaleh and thesis co-advisor is Dr. Tamer Shanableh. Soft and hard copy available.

Abstract

The deaf community relies on sign language as the primary means of communication. For the millions of people around the world who suffer from hearing loss, interaction with hearing people is quite difficult. The main objective of Sign language recognition (SLR) is the development of automatic SLR systems to facilitate communication with the deaf community. SLR as a whole is considered a relatively new area. Arabic SLR (ArSLR) specifically did not receive much attention until recent years. This work presents a comprehensive comparison between two different recognition techniques for continuous ArSLR, namely a Modified k-Nearest Neighbor (MKNN) which is suitable for sequential data and Hidden Markov Models (HMMs) techniques based on two different toolkits. Additionally, in this thesis, two new ArSL datasets composed of 40 Arabic sentences are collected using Polhemus G4 motion tracker and a camera. An existing glove-based dataset is employed in this work as well. The three datasets are made publicly available to the research community. The advantages and disadvantages of each data acquisition approach and classification technique are discussed in this thesis. In the experimental results chapter, it has been shown that data acquisition using only the motion tracker results in accurate sentence recognition similar to that generated by the glove-based acquisition system. The modified KNN solution is inferior to HMMs in terms of the computational time required for classification. Moreover, the performance of Polhemus G4 and RASR on multiple users is examined and promising results have been achieved.