| | This Article | |
| |
| |
| | Share | |
| |
| |
| | Bibliographic References | |
| |
| |
| | Add to: | |
| |
 Digg
 Furl
 Spurl
 Blink
 Simpy
 Google
 Del.icio.us
 Y!MyWeb
| |
| | Search | |
| |
| |
| | |
Joint Optimization of Word Alignment and Epenthesis Generation for Chinese to Taiwanese Sign Synthesis
January 2007 (vol. 29 no. 1)
pp. 28-39
This work proposes a novel approach to translate Chinese to Taiwanese sign language and to synthesize sign videos. An aligned bilingual corpus of Chinese and Taiwanese Sign Language (TSL) with linguistic and signing information is also presented for sign language translation. A two-pass alignment in syntax level and phrase level is developed to obtain the optimal alignment between Chinese sentences and Taiwanese sign sequences. For sign video synthesis, a scoring function is presented to develop motion transition-balanced sign videos with rich combinations of intersign transitions. Finally, the maximum a posteriori (MAP) algorithm is employed for sign video synthesis based on joint optimization of two-pass word alignment and intersign epenthesis generation. Several experiments are conducted in an educational environment to evaluate the performance on the comprehension of sign expression. The proposed approach outperforms the IBM Model 2 in sign language translation. Moreover, deaf students perceived sign videos generated by the proposed method to be satisfactory.
[1] 28 S. Wilcox and P.P. Wilcox, Learning to See. Gallaudet Univ. Press, 1997.[2] F. Alonso, A. Antonio, J.L. Fuertes, and C. Montes, “Teaching Communication Skills to Hearing-Impaired Children,” IEEE Multimedia, pp. 55-67, 1995.[3] C. Brown, “Assistive Technology Computers and Persons with Disabilities,” Comm. ACM, vol. 5, pp. 36-46, 1992.[4] D.L. Speers, “Representation of American Sign Language for Machine Translation,” PhD dissertation, Graduate School of Arts and Sciences, Georgetown Univ., 2001.[5] L.L. Lloyd, D.R. Fuller, and H.H. Arvidson, Augmentative and Alternative Communication: A Handbook of Principles and Practices. Allyn and Bacon, Inc., 1997.[6] C. Vogler and D. Metaxas, “Toward Scalability in ASL Recognition: Breaking Down Signs into Phonemes,” Lecture Notes in Artificial Intelligence, vol. 1739, pp. 211-224, 1999.[7] C. Vogler and D. Metaxas, “A Framework for Recognizing the Simultaneous Aspects of American Sign Language,” Computer Vision and Image Understanding, no. 81, pp. 358-384, 2001.[8] T. Starner, J. Weaver, and A. Pentland, “Real-Time American Sign Language Recognition Using Desk and Wearable Computer-Based Video,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 12, pp. 1371-1375, Dec. 1998.[9] M.C. Su, Y.X. Zhao, H. Huang, and H.F. Chen, “A Fuzzy Rule-Based Approach to Recognizing 3-D Arm Movements,” IEEE Trans. Neural Systems and Rehabilitation Eng., vol. 9, no. 2, 2001.[10] R. Liang, “Continuous Gesture Recognition System for Taiwanese Sign Language,” PhD dissertation, Nat'l Taiwan Univ., 1997.[11] S.C.W. Ong and S. Ranganath, “Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 6, pp. 873-891, June 2005.[12] C.C. Manning and H. Schütze, Foundations of Statistical Natural Language Processing. MIT Press, 1999.[13] W. Chou and B.H. Juang, Pattern Recognition in Speech and Language Processing. CRC Press, 2003.[14] P.F. Brown, S.A. Della Pietra, V.J. Della Pietra, and R.L. Mercer, “The Mathematics of Statistical Machine Translation: Parameter Estimation,” Computational Linguistics, vol. 19, no. 2, pp. 263-311, 1993.[15] H. Ney, S. Niessen, F. Och, H. Sawaf, C. Tillmann, and S. Vogel, “Algorithms for Statistical Translation of Spoken Language,” IEEE Trans. Speech and Audio Processing, vol. 8, no. 1, pp. 24-36, 2000.[16] R. Kennaway, “Synthetic Animation of Deaf Signing Gestures,” Proc. Fourth Int'l Workshop Gesture and Sign Language Based Human-Computer Interaction, 2001.[17] A. Irving and R. Foulds, “A Parametric Approach to Sign Language Synthesis,” Proc. SIGACCESS, pp. 212-213, 2005.[18] Y. Chen, W. Gao, G. Fang, C. Yang, and Z. Wang, “CSLDS: Chinese Sign Language Dialog System,” Proc. IEEE Int'l Workshop Analysis and Modeling of Faces and Gestures, pp. 236-237, 2003.[19] A.B. Grieve-Smith, “SignSynth: A Sign Language Synthesis Application Using Web3D and Perl,” Proc. Gesture Workshop, pp.134-145, 2001.[20] E.J. Holden, J.C. Wong, and R. Owens, “An Effective Sign Language Display System,” Proc. Eighth Int'l Symp. Signal Processing and Its Applications, vol. 1, pp. 54-57, 2005.[21] O. Arikan and D.A. Forsyth, “Interactive Motion Generation from Examples,” Proc. 29th Ann. Conf. Computer Graphics and Interactive Techniques, pp. 483-490, 2002.[22] Y. Li, T. Wang, and H.Y. Shum, “Motion Texture: A Two-Level Statistical Model for Character Motion Synthesis,” ACM Trans. Graphics, vol. 21, no. 3, pp. 465-472, 2002.[23] L. Kovar, M. Gleicher, and F. Pighin, “Motion Graphs,” Proc. ACM SIGGRAPH, pp. 473-482, 2002.[24] J. Lee, J. Chai, P.S.A. Reitsma, J.K. Hodgins, and N.S. Pollard, “Interactive Control of Avatars Animated with Human Motion Data,” Proc. ACM SIGGRAPH, pp. 491-500, 2002.[25] S.W. Kim, Z.X. Li, and Y. Aoki, “On Intelligent Avatar Communication Using Korean, Chinese and Japanese Sign-Languages: An Overview,” Proc. Eighth Control, Automation, Robotics and Vision Conf., vol. 1, pp. 747-752, 2004.[26] Y. Cao, P. Faloutsos, E. Kohler, and F. Pighin, “Real-Time Speech Motion Synthesis from Recorded Motions,” Proc. ACM SIGGRAPH Eurographics Symp. Computer Animation, pp. 347-355, 2004.[27] T. Ezzat, G. Geiger, and T. Poggio, “Trainable Video-Realistic Speech Animation,” Proc. ACM SIGGRAPH, vol. 21, pp. 388-397, 2002.[28] C. Bregler, M. Covell, and M. Slaney, “Video Rewrite: Driving Visual Speech with Audio,” Proc. ACM SIGGRAPH, pp. 353-360, 1997.[29] F. Solina and S. Krapež, “Synthesis of the Sign Language of the Deaf from the Sign Video Clips,” Electrotechnical Rev., vol. 66, pp.260-265, 1999.[30] Ministry of Education, Division of Special Education, Changyong Cihui Shouyu Huace (Sign Album of Common Words), vol. 1. Taipei: Ministry of Education, 2000.[31] “The Chinese Knowledge Information Processing Group, Analysis of Chinese Part of Speech,” CKIP Technical Report, no. 93-05, Inst. of Information Science, Academic Sinica, Taipei, 1993 (in Chinese).[32] Z. Dong, The HowNet Web Site, http:/www.keenage.com, 1999.[33] Inst. of Linguistics, Nat'l Chung Cheng Univ., Chiayi, Taiwan, Proc. Int'l Symp. Taiwan Sign Language Linguistics, http://www.ccunix.ccu.edu.tw/~lngsigntsl-links-e.htm , 2003.[34] P.A. Lachenbruch and M.R. Mickey, “Estimation of Error Rate in Discriminant Analysis,” Technometrics, pp. 1-11, 1968.[35] S. Shott, Statistics for Health Professionals. W.B. Sauders, 1990.[36] C.H. Wu, Y.H. Chiu, and C.S. Guo, “Text Generation from Taiwanese Sign Language Using a PST-Based Language Model for Augmentative Communication,” IEEE Trans. Neural Systems and Rehabilitation Eng., vol. 12, no. 4, pp. 441-454, 2004.[37] C.H. Wu, Y.H. Chiu, and K.W. Cheng, “Error-Tolerant Sign Retrieval Using Visual Features and Maximum A Posteriori Estimation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 4, pp. 495-508, Apr. 2004.
Index Terms:
Taiwanese sign language, language translation, sign language synthesis, video concatenation.
Citation:
Yu-Hsien Chiu, Chung-Hsien Wu, Hung-Yu Su, Chih-Jen Cheng, "Joint Optimization of Word Alignment and Epenthesis Generation for Chinese to Taiwanese Sign Synthesis," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 1, pp. 28-39, Jan. 2007, doi:10.1109/TPAMI.2007.15
