Abstract: We present a method to track and recognize shape-changing hand gestures simultaneously. The switching linear model using active contour model well corresponds to temporal shapes and motions of hands. Inference in the switching linear model is computationally intractable, and therefore the learning process cannot be performed via the exact EM(Expectation Maximization) algorithm. However, we present an approximate EM algorithm using a collapsing method in which some Gaussians are merged into a single Gaussian. Tracking is performed through the forward algorithm based on Kalman filtering and the collapsing method. We also present the regularized smoothing, which plays a role of reducing jump changes between the training sequences of state vectors to cope with complex- variable hand shapes. The recognition process is performed by the selection of a model with the maximum likelihood from some learned models while tracking is being performed. Experiments for several shape-changing hand gestures are demonstrated.