Issue No. 09 - September (2009 vol. 31)
Kate Saenko , MIT, Cambridge
Trevor Darrell , MIT, Cambridge
Karen Livescu , Toyota Technological Institute, Chicago
James Glass , MIT, Cambridge
We study the problem of automatic visual speech recognition (VSR) using dynamic Bayesian network (DBN)-based models consisting of multiple sequences of hidden states, each corresponding to an articulatory feature (AF) such as lip opening (LO) or lip rounding (LR). A bank of discriminative articulatory feature classifiers provides input to the DBN, in the form of either virtual evidence (VE) (scaled likelihoods) or raw classifier margin outputs. We present experiments on two tasks, a medium-vocabulary word-ranking task and a small-vocabulary phrase recognition task. We show that articulatory feature-based models outperform baseline models, and we study several aspects of the models, such as the effects of allowing articulatory asynchrony, of using dictionary-based versus whole-word models, and of incorporating classifier outputs via virtual evidence versus alternative observation models.
Visual speech recognition, articulatory features, dynamic Bayesian networks, support vector machines.
Kate Saenko, Trevor Darrell, Karen Livescu, James Glass, "Multistream Articulatory Feature-Based Models for Visual Speech Recognition", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 31, no. , pp. 1700-1707, September 2009, doi:10.1109/TPAMI.2008.303