Pages: pp. 1729-1732
An exciting development over the last decade has been the gradually widespread adoption of probabilistic graphical models (PGMs) in many areas of computer vision and pattern recognition. Representing an integration of graph and probability theories, a number of families of graphical models have been studied in the statistics and machine learning literatures. Examples of directed graphical models include Bayesian networks (BNs) and hidden Markov models (HMMs). Alternatively, undirected graphical models such as Markov random fields (MRFs) and conditional random fields (CRFs) are widely used to model spatial dependencies. Graphical models lead to powerful computational methods for representation, learning, and inference in computer vision. In particular, PGMs provide a unified framework for representing observations and domain-specific contextual knowledge, and for performing recognition and classification through rigorous probabilistic inference.
The history of PGMs in computer vision closely follows that of graphical models in general. Research in the late 1980s by Judea Pearl and Steffen Lauritzen, among others, played a seminal role in introducing this formalism to areas of AI and statistics. Not long after, the formalism spread to other fields, including systems engineering, information theory, pattern recognition, and computer vision. One of the earliest applications of graphical models in computer vision was work by Binford et al. [ 1], in which Bayesian inference in a hierarchical probability model was used to match 3D object models to groupings of curves in a single image. The following year marked the publication of Pearl's influential book Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference [ 2].
Since then, hundreds of technical papers have been published that address different aspects and continuing applications of PGMs in computer vision. Many classical tools from the vision literature, including MRFs, HMMs, and the Kalman Filter, are unified and generalized by PGMs. For example, MRFs have become a widely used, standard framework for both image segmentation and stereo reconstruction. More recently, discriminatively trained CRFs have increasingly been used to incorporate rich, nonlocal features in image labeling tasks. For motion analysis, event modeling, and activity recognition, HMMs have become the standard tool. Markov networks have similarly led to progress in multitarget tracking, where the relationships of multiple objects can be intuitively captured via probabilistic graphs.
Variants of HMMs (such as hierarchical HMMs and coupled HMMs) have recently been introduced to model nonlocal dependencies and complex dynamic events. In particular, dynamic Bayesian networks (DBNs) are increasingly used to capture complex spatio-temporal patterns for object tracking and for modeling and recognizing complex dynamic activities such as human behaviors and facial expressions. More generally, PGMs excel as a natural framework for integrating top-down and bottom-up visual processes. Image segmentation, object detection, and object recognition have been shown to not only benefit from one another, but also be computationally feasible due to efficient PGM inference and learning algorithms.
We believe the powerful capability of graphical models to encode and integrate information at different levels, from complex high-level domain knowledge to low-level image properties, will continue to produce extremely effective models of visual data. In spite of recent successes, however, PGMs' use in computer vision still has tremendous room to expand in scope, depth, and rigor. The main objective of this special section is to review the latest developments in applications of PGMs to various computer vision tasks, identify important modeling and computational issues, and point out future research directions.
For this special section, we solicited original contributions describing applications of PGMs in all areas of computer vision, including (but not limited to):
We were, in particular, interested in contributions that demonstrate innovative applications to computer vision tasks and that result in significant performance improvements over alternative methods. We also sought to identify new theoretical developments in PGM learning and inference algorithms, as driven by computer vision applications. To reach a wide audience, we published the special section's call for papers through a number of channels, including mailing lists, posters at major computer vision conferences, and advertisement in journals. The response was overwhelming: We received a total of 68 submissions from all over the world, covering almost every aspect of computer vision from shape matching, feature extraction, superresolution, and tracking to recognition, 3D reconstruction, and activity recognition. The theoretical contributions are also diverse, ranging from innovative new models tailored to particular applications, to new algorithms for inference and learning.
To ensure a careful and fair review process, we recruited more than 200 expert reviewers from the computer vision and machine learning communities. Each paper received at least three reviews, with additional reviews and comments by the guest editors. Some papers underwent two or three rounds of reviews before a final decision was made. After much discussion and deliberation, and accounting for the special section's tight publication schedule, we accepted the 10 outstanding papers contained in this section. We are very impressed by the diversity and quality of these contributions and hope this special section will have a long lasting positive effect on the community.
The 10 accepted papers can be coarsely categorized based on their specific computer vision application domains: four explore frameworks for object or activity recognition, two describe methods for detecting objects or visual patterns, and four consider visual tracking or motion analysis. In the paragraphs below, we briefly summarize the papers in each category.
Scene text recognition deals with the problem of recognizing text in a natural environment, such as on traffic signs and store fronts. In contrast to standard document analysis, this problem is complicated by extreme font variability, uncontrolled viewing conditions, and minimal language context. In their paper titled "Scene Text Recognition Using Similarity and a Lexicon with Sparse Belief Propagation," Weinman, Learned-Miller, and Hanson propose a solution to this problem, where a discriminative model is developed that combines character appearance, character bigrams, character similarity, and a lexicon. Using an efficient sparse belief propagation algorithm, inference is simultaneously performed on factor graphs built from various cues, avoiding the error accumulation that often occurs with sequential processing.
The paper titled "Unsupervised Learning of Probabilistic Object Models (POMs) for Object Classification, Segmentation, and Recognition Using Knowledge," by Chen, Zhu, Yuille, and Zhang, addresses the difficult problem of learning probabilistic object models with minimal supervision for object classification, segmentation, and recognition. The authors formulate the task of learning POMs which model object categories as a structure induction problem. Then, they use the strategy of knowledge propagation to enable POMs to provide information to other POMs. Extensive experimental results, including analysis of robustness and comparisons to state-of-the-art methods, demonstrate the power of this formulation.
Collections of discriminative local features, when combined with models originally used to learn topics underlying document collections, have recently shown promise for modeling visual object categories. In their paper "Human Action Recognition by Semilatent Topic Models," Yang and Mori show that related approaches also provide effective models of human actions in video. Mapping quantized motion descriptors to "visual words" and topics to action classes, they use variational inference methods to accurately cluster test video frames into semantic activities.
The paper titled "Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition" by Gupta, Kembhavi, and Davis, introduces an integrated system for recognizing actions and objects. By solving these two recognition tasks in an integrated system, the authors show that the system can recognize objects and actions even when appearances are not discriminative enough. For instance, when only analyzing hand trajectories, differentiating among actions might be hard, but discrimination can become easier by integrating information about the objects involved in the action. Likewise, recognition of similar objects might be easier if they are involved on easy to discriminate actions.
In the paper titled "A Probabilistic Framework for 3D Visual Object Representation," authors Detry, Pugeault, and Piater tackle the difficult task of detection and localization of objects within highly cluttered scenes by developing a generative 3D visual representation. They encode the 3D geometry and visual appearance of an object into a part-based model, and develop mechanisms for autonomous learning and probabilistic inference in that model. A Markov network is used to combine local appearance and 3D spatial relationships through a hierarchy of increasingly expressive features. The authors demonstrate efficiency and robustness to input noise, viewpoint changes, and occlusions.
Park, Brocklehurst, Collins, and Liu, in their paper "Deformed Lattice Detection in Real-World Images Using Mean-Shift Belief Propagation," address the issue of detecting repetitive patterns within deformed 2D lattices. The lattice detection problem is formulated as a spatial, multitarget tracking problem, solved within the MRF framework using a novel mean-shift belief propagation method. The model is evaluated in a wide range of images that contain near-regular textures (such as windows in buildings, fruits, wiry structures, etc.)
An elastic motion is a nonrigid motion constrained by some degrees of smoothness and continuity. In their paper titled "A Mixture of Transformed Hidden Markov Models for Elastic Motion Estimation," authors Di, Tao, and Xu address the problem of tracking deformable objects, such as faces and human figures, by assuming an elastic deformation. Their method extracts edge points from each frame and formulates motion estimation as a shape registration task that also incorporates temporal consistency constraints. A mixture of transformed hidden Markov models is used to account for both spatial smoothness and temporal continuity. The end result is a more globally coherent interpretation of elastic motion from local edge features, which are often subject to ambiguities, incompleteness, and clutter.
Tracking multiple people from a mobile platform is challenging due to camera motion, occlusion, and dynamic backgrounds. In their paper titled "Robust Multiperson Tracking from a Mobile Platform," authors Ess, Leibe, Schindler, and van Gool propose an integrated approach which uses a graphical model to systematically capture dependencies between different components (camera position, depth, object detection, tracking) of a multiperson tracking system. With such a model, they can perform joint camera calibration, 3D reconstruction, object detection, and tracking in a principled manner. The evaluation of the system on busy inner city streets demonstrates its robust performance under realistic scenarios.
Linear dynamical systems are widely used in the analysis of video sequences and motion data. In the paper "Discriminative Learning for Dynamic State Prediction," Kim and Pavlovic propose a pair of novel, conditional models for continuous time series. These discriminative objective functions lead to efficient algorithms for selecting and incorporating sophisticated, potentially nonlocal features. In experiments, these approaches lead to more accurate predictions of articulated motion than standard generative models.
So-called dynamic texture models describe repetitive motion patterns via the evolution of a latent dynamical system. In their paper, Chan and Vasconcelos have proposed a family of "Layered Dynamic Textures" which allow videos to be segmented into multiple, independently evolving motions. Spatial continuity of segments is encouraged via a Markov random field prior and inference performed via either variational or Monte Carlo methods. Their approach is validated on a range of synthetic and real-world video sequences.
We would like to thank many people for their contributions to this special section, which would not have been possible without their efforts. First and foremost, we thank the authors for their enthusiastic response and innovative contributions. Second, we would like to express our great appreciation to the more than 200 reviewers for their timely, detailed, and thoughtful reviews. Their work was essential to ensure the quality of this special section. Third, we would are thankful for the support we have received from the TPAMI editorial office, including past Editor-in-Chief Professor David Kriegman, current Editor-in-Chief Professor Ramin Zabih, and editorial assistant Elaine Stephenson. In particular, we thank Elaine for her timely answers to our inquiries, and for actively keeping the special section on schedule.
Thomas S. Huang
Erik B. Sudderth