Issue No.05 - May (2013 vol.19)
pp: 811-823
Yin Yang , Dept. of Comput. Sci., Univ. of Texas at Dallas, Richardson, TX, USA
Xiaohu Guo , Dept. of Comput. Sci., Univ. of Texas at Dallas, Richardson, TX, USA
J. Vick , Dept. of Psychological Sci., Case Western Reserve Univ., Cleveland, OH, USA
L. G. Torres , Dept. of Comput. Sci., Univ. of North Carolina at Chapel Hill, Chapel Hill, NC, USA
T. F. Campbell , Callier Center for Commun. Disorders, Univ. of Texas at Dallas, Richardson, TX, USA
In this paper, a physics-based framework is presented to visualize the human tongue deformation. The tongue is modeled with the Finite Element Method (FEM) and driven by the motion capture data gathered during speech production. Several novel deformation visualization techniques are presented for in-depth data analysis and exploration. To reveal the hidden semantic information of the tongue deformation, we present a novel physics-based volume segmentation algorithm. This is accomplished by decomposing the tongue model into segments based on its deformation pattern with the computation of deformation subspaces and fitting the target deformation locally at each segment. In addition, the strain energy is utilized to provide an intuitive low-dimensional visualization for the high-dimensional sequential motion. Energy-interpolation-based morphing is also equipped to effectively highlight the subtle differences of the 3D deformed shapes without any visual occlusion. Our experimental results and analysis demonstrate the effectiveness of this framework. The proposed methods, though originally designed for the exploration of the tongue deformation, are also valid for general deformation analysis of other shapes.
Tongue, Sensors, Speech, Production, Shape, Deformable models, Visualization,modal analysis, Deformable model, tongue, finite element method
Yin Yang, Xiaohu Guo, J. Vick, L. G. Torres, T. F. Campbell, "Physics-Based Deformable Tongue Visualization", IEEE Transactions on Visualization & Computer Graphics, vol.19, no. 5, pp. 811-823, May 2013, doi:10.1109/TVCG.2012.174
[1] M. Alexa, D. Cohen-Or, and D. Levin, "As-Rigid-as-Possible Shape Interpolation," Proc. SIGGRAPH Conf., pp. 157-164, 2000.
[2] P. Badin, G. Bailly, L. Revéret, M. Baciu, C. Segebarth, and C. Savariaux, "Three-Dimensional Linear Articulatory Modeling of Tongue, Lips and Face, Based on MRI and Video Images," J. Phonetics, vol. 30, no. 3, pp. 533-553, 2002.
[3] T. Baer, J. Gore, S. Boyce, and P. Nye, "Application of MRI to the Analysis of Speech Production," Magnetic Resonance Imaging, vol. 5, no. 1, pp. 1-7, 1987.
[4] Y. Bao, X. Guo, and H. Qin, "Physically Based Morphing of Point-Sampled Surfaces," Computer Animation and Virtual Worlds, vol. 16, pp. 509-518, July 2005.
[5] S. Buchaillard, P. Perrier, and Y. Payan, "A Biomechanical Model of Cardinal Vowel Production: Muscle Activations and the Impact of Gravity on Tongue Positioning," The J. the Acoustical Soc. of Am., vol. 126, no. 4, pp. 2033-2051, 2009.
[6] W. Chen, Z. Ding, S. Zhang, A. MacKay-Brandt, S. Correia, H. Qu, J.A. Crow, D.F. Tate, Z. Yan, and Q. Peng, "A Novel Interface for Interactive Exploration of DTI Fibers," IEEE Trans. Visualization and Computer Graphics, vol. 15, no. 6, pp. 1433-1440, Nov./Dec. 2009.
[7] M.G. Choi and H.-S. Ko, "Modal Warping: Real-Time Simulation of Large Rotational Deformation and Manipulation," IEEE Trans. Visualization and Computer Graphics, vol. 11, no. 1, pp. 91-101, Jan. 2005.
[8] R. Craig, Structural Dynamics: An Introduction to Computer Methods. Wiley, 1981.
[9] S.F. Els, F.L.V. Ogt, K.V.D. Doel, J.E. Lloyd, and O. Guenther, "Artisynth: An Extensible, Cross-Platform 3D Articulatory Speech Synthesizer," Proc. Conf. Auditory and Visual Speech Processing, 2005.
[10] Q. Fang, S. Fujita, X. Lu, and J. Dang, "A Model-Based Investigation of Activations of the Tongue Muscles in Vowel Production," Acoustical Science and Technology, vol. 30, no. 4, pp. 277-287, 2009.
[11] Y. Fung and P. Tong, Classical and Computational Solid Mechanics, Advanced Series in Engineering Science. World Scientific, 2001.
[12] J.-M. Gérard, J. Ohayon, V. Luboz, P. Perrier, and Y. Payan, "Indentation for Estimating the Human Tongue Soft Tissues Constitutive Law: Application to a 3D Biomechanical Model," Medical Simulation, vol. 3078, pp. 77-83, 2004.
[13] J.-M. Gérard, P. Perrier, and Y. Payan, "3D Biomechanical Tongue Modeling to Study Speech Production," Proc. Speech Production: Models, Phonetic Processes, and Techniques, pp. 85-102, 2006.
[14] J.-M. Gérard, R. Wilhelms Tricarico, P. Perrier, and Y. Payan, "A 3D Dynamical Biomechanical Tongue Model to Study Speech Motor Control," Research Developments in Biomechanics, vol. 1, pp. 49-64, 2003.
[15] J.R. Green, C.A. Moore, M. Higashikawa, and R.W. Steeve, "The Physiologic Development of Speech Motor Control: Lip and Jaw Coordination," J. Speech, Language, and Hearing Research, vol. 43, no. 1, pp. 239-255, 2000.
[16] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. Springer, 2003.
[17] S.-M. Hu, C.-F. Li, and H. Zhang, "Actual Morphing: A Physics-Based Approach to Blending," Proc. Ninth ACM Symp. Solid Modeling and Applications, pp. 309-314, 2004.
[18] J. Huang, Y. Tong, K. Zhou, H. Bao, and M. Desbrun, "Interactive Shape Interpolation through Controllable Dynamic Deformation," IEEE Trans. Visualization and Computer Graphics, vol. 17, no. 7, pp. 983-992, July 2011.
[19] Q.-X. Huang, M. Wicke, B. Adams, and L. Guibas, "Shape Decomposition Using Modal Analysis," Computer Graphics Forum, vol. 28, no. 2, pp. 407-416, 2009.
[20] T. Hughes, The Finite Element Method: Linear Static and Dynamic Finite Element Analysis. Dover Publications, 2000.
[21] D. Keefe, M. Ewert, W. Ribarsky, and R. Chang, "Interactive Coordinated Multiple-View Visualization of Biomechanical Motion Data," IEEE Trans. Visualization and Computer Graphics, vol. 15, no. 6, pp. 1383-1390, Nov./Dec. 2009.
[22] S.A. King and R.E. Parent, "A 3D Parametric Tongue Model for Animated Speech," The J. Visualization and Computer Animation, vol. 12, no. 3, pp. 107-115, 2001.
[23] S. Mcleod, "Speechclanguage Pathologists Knowledge of Tongue/Palate Contact for Consonants," Clinical Linguistics and Phonetics, vol. 25, nos. 11/12, pp. 1004-1013, 2011.
[24] W. Moss, M.C. Lin, and D. Manocha, "Constraint-Based Motion Synthesis for Deformable Models," Computer Animation and Virtual Worlds, vol. 19, nos. 3/4, pp. 421-431, 2008.
[25] M. Müller, J. Dorsey, L. McMillan, R. Jagnow, and B. Cutler, "Stable Real-Time Deformations," Proc. ACM SIGGRAPH/Eurographics Symp. Computer Animation, pp. 49-54, 2002.
[26] Northern Digital Inc., "Wave Speech Research System," maplambda.htm, 2012.
[27] D.J. Ostry, E. Vatikiotis-Bateson, and P.L. Gribble, "An Examination of the Degrees of Freedom of Human Jaw Motion in Speech and Mastication," J. Speech, Language, and Hearing Research, vol. 40, no. 6, pp. 1341-1351, 1997.
[28] A. Pentland and J. Williams, "Good Vibrations: Modal Dynamics for Graphics and Animation," Computer Graphics, vol. 23, no. 3, pp. 207-214, 1989.
[29] E. Pernkopf, Atlas of Topographical and Applied Human Anatomy: Head and Neck, third ed. Williams Wilkins, Dec. 1989.
[30] G. Robertson, R. Fernandez, D. Fisher, B. Lee, and J. Stasko, "Effectiveness of Animation in Trend Visualization," IEEE Trans. Visualization and Computer Graphics, vol. 14, no. 6, pp. 1325-1332, Nov. 2008.
[31] A. Shamir, "A Survey on Mesh Segmentation Techniques," Computer Graphics Forum, vol. 27, no. 6, pp. 1539-1556, 2008.
[32] E. Sifakis, I. Neverov, and R. Fedkiw, "Automatic Determination of Facial Muscle Activations from Sparse Motion Capture Marker Data," ACM Trans. Graphics, vol. 24, no. 3, pp. 417-425, July 2005.
[33] B.C. Sonies, T.H. Shawker, T.E. Hall, L.H. Gerber, and S.B. Leighton, "Ultrasonic Visualization of Tongue Motion During Speech," The J. the Acoustical Soc. of Am., vol. 70, no. 3, pp. 683-686, 1981.
[34] I. Stavness, A.G. Hannam, J.E. Lloyd, and S. Fels, "Predicting Muscle Patterns for Hemimandibulectomy Models," Computer Methods in Biomechanics and Biomedical Eng., vol. 13, no. 4, pp. 483-491, 2010.
[35] I. Stavness, J.E. Lloyd, Y. Payan, and S. Fels, "Coupled Hard-Soft Tissue Simulation with Contact and Constraints Applied to Jaw-Tongue-Hyoid Dynamics," Int'l J. for Numerical Methods in Biomedical Eng., vol. 27, no. 3, pp. 367-390, 2011.
[36] M. Stone, "A Three-Dimensional Model of Tongue Movement Based on Ultrasound and X-Ray Microbeam Data," The J. the Acoustical Soc. of Am., vol. 87, no. 5, pp. 2207-2217, 1990.
[37] M. Stone, E.P. Davis, A.S. Douglas, M.N. Aiver, R. Gullapalli, W.S. Levine, and A.J. Lundberg, "Modeling Tongue Surface Contours from Cine-MRI Images," J. Speech, Language, and Hearing Research, vol. 44, no. 5, pp. 1026-1040, 2001.
[38] M. Stone and A. Lundberg, "Three-Dimensional Tongue Surface Shapes of English Consonants and Vowels," The J. the Acoustical Soc. of Am., vol. 99, no. 6, pp. 3728-3737, 1996.
[39] R.W. Sumner, M. Zwicker, C. Gotsman, and J. Popović, "Mesh-Based Inverse Kinematics," Proc. SIGGRAPH '05, pp. 488-495, 2005.
[40] H. Takemoto, "Morphological Analyses of the Human Tongue Musculature for Three-Dimensional Modeling," J. Speech, Language, and Hearing Research, vol. 44, no. 1, pp. 95-107, 2001.
[41] F. Vogt, J. Lloyd, S. Buchaillard, P. Perrier, M. Chabanas, Y. Payan, and S. Fels, "Efficient 3D Finite Element Modeling of a Muscle-Activated Tongue," Biomedical Simulation, vol. 4072, pp. 19-28, 2006.
[42] R. Wilhelms-Tricarico, "Physiological Modeling of Speech Production: Methods for Modeling Soft-Tissue Articulators," The J. the Acoustical Soc. of Am., vol. 97, no. 5, pp. 3085-3098, 1995.
[43] A. Witkin and W. Welch, "Fast Animation and Control of Nonrigid Structures," Proc. SIGGRAPH '90, pp. 243-252, 1990.
[44] H.-B. Yan, S.-M. Hu, and R. Martin, "Morphing Based on Strain Field Interpolation," Computer Animation and Virtual Worlds, vol. 15, nos. 3/4, pp. 443-452, July 2004.
[45] Y. Yang, G. Rong, L. Torres, and X. Guo, "Real-Time Hybrid Solid Simulation: Spectral Unification of Deformable and Rigid Materials," Computer Animation and Virtual Worlds, vol. 21, nos. 3/4, pp. 151-159, 2010.
[46] H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, "Quantitative Association of Vocal-Tract and Facial Behavior," Speech Comm., vol. 26, pp. 23-43, Oct. 1998.