Handling Movement Epenthesis and Hand Segmentation Ambiguities in Continuous Sign Language Recognition Using Nested Dynamic Programming
Issue No. 03 - March (2010 vol. 32)
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TPAMI.2009.26
Barbara Loeding , University of South Florida Polytechnic, Lakeland
Sudeep Sarkar , University of South Florida, Tampa
Ruiduo Yang , University of South Florida, Tampa
We consider two crucial problems in continuous sign language recognition from unaided video sequences. At the sentence level, we consider the movement epenthesis (me) problem and at the feature level, we consider the problem of hand segmentation and grouping. We construct a framework that can handle both of these problems based on an enhanced, nested version of the dynamic programming approach. To address movement epenthesis, a dynamic programming (DP) process employs a virtual me option that does not need explicit models. We call this the enhanced level building (eLB) algorithm. This formulation also allows the incorporation of grammar models. Nested within this eLB is another DP that handles the problem of selecting among multiple hand candidates. We demonstrate our ideas on four American Sign Language data sets with simple background, with the signer wearing short sleeves, with complex background, and across signers. We compared the performance with Conditional Random Fields (CRF) and Latent Dynamic-CRF-based approaches. The experiments show more than 40 percent improvement over CRF or LDCRF approaches in terms of the frame labeling rate. We show the flexibility of our approach when handling a changing context. We also find a 70 percent improvement in sign recognition rate over the unenhanced DP matching algorithm that does not accommodate the me effect.
Sign language, movement epenthesis, continuous gesture, segmentation, level building.
Barbara Loeding, Sudeep Sarkar, Ruiduo Yang, "Handling Movement Epenthesis and Hand Segmentation Ambiguities in Continuous Sign Language Recognition Using Nested Dynamic Programming", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 32, no. , pp. 462-477, March 2010, doi:10.1109/TPAMI.2009.26