This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Image Segmentation with a Unified Graphical Model
August 2010 (vol. 32 no. 8)
pp. 1406-1425
Lei Zhang, Rensselaer Polytechnic Institute, Troy
Qiang Ji, Rensselaer Polytechnic Institute, Troy
We propose a unified graphical model that can represent both the causal and noncausal relationships among random variables and apply it to the image segmentation problem. Specifically, we first propose to employ Conditional Random Field (CRF) to model the spatial relationships among image superpixel regions and their measurements. We then introduce a multilayer Bayesian Network (BN) to model the causal dependencies that naturally exist among different image entities, including image regions, edges, and vertices. The CRF model and the BN model are then systematically and seamlessly combined through the theories of Factor Graph to form a unified probabilistic graphical model that captures the complex relationships among different image entities. Using the unified graphical model, image segmentation can be performed through a principled probabilistic inference. Experimental results on the Weizmann horse data set, on the VOC2006 cow data set, and on the MSRC2 multiclass data set demonstrate that our approach achieves favorable results compared to state-of-the-art approaches as well as those that use either the BN model or CRF model alone.

[1] D. Comaniciu and P. Meer, "Mean Shift: A Robust Approach Toward Feature Space Analysis," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, May 2002.
[2] M. Kass, A. Witkin, and D. Terzopoulos, "Snakes: Active Contour Models," Int'l J. Computer Vision, vol. 1, pp. 321-331, 1988.
[3] J. Shi and J. Malik, "Normalized Cuts and Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.
[4] T. Chan and L. Vese, "Active Contours without Edges," IEEE Trans. Image Processing, vol. 10, no. 2, pp. 266-277, Feb. 2001.
[5] S. Geman and D. Geman, "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 6, no. 6, pp. 721-741, Nov. 1984.
[6] C. Bouman and B. Liu, "Multiple Resolution Segmentation of Textured Images," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 2, pp. 99-113, Feb. 1991.
[7] D. Melas and S. Wilson, "Double Markov Random Fields and Bayesian Image Segmentation," IEEE Trans. Signal Processing, vol. 50, no. 2, pp. 357-365, Feb. 2002.
[8] W. Pieczynski and A. Tebbache, "Pairwise Markov Random Fields and Segmentation of Textured Images," Machine Graphics and Vision, vol. 9, pp. 705-718, 2000.
[9] C. D'Elia, G. Poggi, and G. Scarpa, "A Tree-Structured Markov Random Field Model for Bayesian Image Segmentation," IEEE Trans. Image Processing, vol. 12, no. 10 pp. 1259-1273, Oct. 2003.
[10] X. He, R. Zemel, and M. Carreira Perpinan, "Multiscale Conditional Random Fields for Image Labeling," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 695-702, 2004.
[11] X. Ren, C.C. Fowlkes, and J. Malik, "Cue Integration in Figure/Ground Labeling," Advances in Neural Information Processing Systems, pp. 1121-1128, MIT Press, 2005.
[12] P. Awasthi, A. Gagrani, and B. Ravindran, "Image Modeling Using Tree Structured Conditional Random Fields," Proc. Int'l Joint Conf. Artificial Intelligence, pp. 2054-2059, 2007.
[13] X. Feng, C. Williams, and S. Felderhof, "Combining Belief Networks and Neural Networks for Scene Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 4, pp. 467-483, Apr. 2002.
[14] E.N. Mortensen and J. Jia, "Real-Time Semi-Automatic Segmentation Using a Bayesian Network," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 1007-1014, 2006.
[15] P. Alvarado, A. Berner, and S. Akyol, "Combination of High-Level Cues in Unsupervised Single Image Segmentation Using Bayesian Belief Networks," Proc. Int'l Conf. Imaging Science, Systems, and Technology, vol. 2, pp. 675-681, 2002.
[16] S. Zheng, Z. Tu, and A. Yuille, "Detecting Object Boundaries Using Low-, Mid-, and High-Level Information," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007.
[17] B. Leibe, A. Leonardis, and B. Schiele, "Combined Object Categorization and Segmentation with an Implicit Shape Model," Proc. European Conf. Computer Vision Workshop Statistical Learning in Computer Vision, pp. 17-32, 2004.
[18] J. Borges, J. Bioucas Dias, and A. Marcal, "Bayesian Hyperspectral Image Segmentation with Discriminative Class Learning," Proc. Third Iberian Conf. Pattern Recognition and Image Analysis, pp. I: 22-29, 2007.
[19] T. Hosaka, T. Kobayashi, and N. Otsu, "Image Segmentation Using MAP-MRF Estimation and Support Vector Machine," Interdisciplinary Information Sciences, vol. 13, no. 1, pp. 33-42, 2007.
[20] J. Winn and N. Jojic, "Locus: Learning Object Classes with Unsupervised Segmentation," Proc. IEEE Int'l Conf. Computer Vision, pp. 756-763, 2005.
[21] S. Khan and M. Shah, "Object Based Segmentation of Video Using Color, Motion and Spatial Information," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 746-751, 2001.
[22] Y.-W. Tai, J. Jia, and C.-K. Tang, "Soft Color Segmentation and Its Applications," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 9, pp. 1520-1537, Sept. 2007.
[23] E. Borenstein and J. Malik, "Shape Guided Object Segmentation," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 969-976, 2006.
[24] Z. Tu, X. Chen, A.L. Yuille, and S.-C. Zhu, "Image Parsing: Unifying Segmentation, Detection, and Recognition," Proc. IEEE Int'l Conf. Computer Vision, pp. 18-25, 2003.
[25] Z. Tu, C. Narr, P. Dollar, I. Dinov, P. Thompson, and A. Toga, "Brain Anatomical Structure Segmentation by Hybrid Discriminative/Generative Models," IEEE Trans. Medical Imaging, vol. 27, no. 4, pp. 495-508, Apr. 2008.
[26] S.Z. Li, Markov Random Field Modeling in Image Analysis. Springer, 2001.
[27] J. Lafferty, A. McCallum, and F. Pereira, "Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data," Proc. Int'l Conf. Machine Learning, pp. 282-289, 2001.
[28] S.L. Lauritzen, Graphical Models. Oxford Univ. Press, 1996.
[29] F.V. Jensen, Bayesian Networks and Decision Graphs. Springer-Verlag, 2001.
[30] J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan-Kaufmann Publishers, 1988.
[31] C.M. Bishop, Pattern Recognition and Machine Learning. Springer, 2006.
[32] E. Borenstein, E. Sharon, and S. Ullman, "Combining Top-Down and Bottom-Up Segmentation," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition Workshop Perceptual Organization in Computer Vision, p. 46, 2004.
[33] J. Shotton, J. Winn, C. Rother, and A. Criminisi, "Textonboost: Joint Appearance, Shape and Context Modeling for Multi-Class Object Recognition and Segmentation," Proc. European Conf. Computer Vision, pp. 1-15, 2006.
[34] M. Everingham, "The VOC2006 Database," Univ. of Oxford, http://www.pascal-network.org/challenges/ VOCdatabases. html#VOC2006, 2010.
[35] C. Bouman and M. Shapiro, "A Multiscale Random Field Model for Bayesian Image Segmentation," IEEE Trans. Image Processing, vol. 3, no. 2, pp. 162-177, Mar. 1994.
[36] H. Cheng and C. Bouman, "Multiscale Bayesian Segmentation Using a Trainable Context Model," IEEE Trans. Image Processing, vol. 10, no. 4, pp. 511-525, Apr. 2001.
[37] R. Wilson and C. Li, "A Class of Discrete Multiresolution Random Fields and Its Application to Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 1, pp. 42-56, Jan. 2003.
[38] W. Irving, P. Fieguth, and A. Willsky, "An Overlapping Tree Approach to Multiscale Stochastic Modeling and Estimation," IEEE Trans. Image Processing, vol. 6, no. 11, pp. 1517-1529, Nov. 1997.
[39] S. Kumar and M. Hebert, "Discriminative Random Fields," Int'l J. Computer Vision, vol. 68, no. 2, pp. 179-201, 2006.
[40] S. Kumar and M. Hebert, "A Hierarchical Field Framework for Unified Context-Based Classification," Proc. IEEE Int'l Conf. Computer Vision, pp. 1284-1291, 2005.
[41] A. Kapoor and J. Winn, "Located Hidden Random Fields: Learning Discriminative Parts for Object Detection," Proc. European Conf. Computer Vision, pp. 302-315, 2006.
[42] J. Winn and J. Shotton, "The Layout Consistent Random Field for Recognizing and Segmenting Partially Occluded Objects," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 37-44, 2006.
[43] D. Hoiem, C. Rother, and J. Winn, "3D LayoutCRF for Multi-View Object Class Recognition and Segmentation," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007.
[44] S. Sarkar and K.L. Boyer, "Integration, Inference, and Management of Spatial Information Using Bayesian Networks: Perceptual Organization," IEEE Trans. Pattern Analysis and Machine Intelligence, special section on probabilistic reasoning, vol. 15, no. 3, pp. 256-274, Mar. 1993.
[45] S.C. Zhu and A. Yuille, "Region Competition: Unifying Snake/Balloon, Region Growing and Bayes/MDL/Energy for Multiband Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 9, pp. 884-900, Sept. 1996.
[46] S. Todorovic and M. Nechyba, "Dynamic Trees for Unsupervised Segmentation and Matching of Image Regions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 11, pp. 1762-1777, Nov. 2005.
[47] R. Huang, V. Pavlovic, and D. Metaxas, "A Graphical Model Framework for Coupling MRFs and Deformable Models," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 739-746, 2004.
[48] C.H. Lee, M. Schmidt, A. Murtha, A. Bistritz, J. Sander, and R. Greiner, "Segmenting Brain Tumors with Conditional Random Fields and Support Vector Machines," Proc. Computer Vision for Biomedical Image Applications: Current Techniques and Future Trends (within Int'l Conf. Computer Vision), pp. 469-478, 2005.
[49] F. Liu, D. Xu, C. Yuan, and W. Kerwin, "Image Segmentation Based on Bayesian Network-Markov Random Field Model and its Application on in vivo Plaque Composition," Proc. IEEE Int'l Symp. Biomedical Imaging, pp. 141-144, 2006.
[50] V. Murino, C.S. Regazzoni, and G. Vernazza, "Distributed Propagation of A-Priori Constraints in a Bayesian Network of Markov Random Fields," IEE Proc. I Comm., Speech and Vision, vol. 140, no. 1, pp. 46-55, Feb. 1993.
[51] M.P. Kumar, P.H.S. Torr, and A. Zisserman, "OBJ CUT," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 18-25, 2005.
[52] G.E. Hinton, S. Osindero, and K. Bao, "Learning Causally Linked Markov Random Fields," Proc. 10th Int'l Workshop Artificial Intelligence and Statistics, 2005.
[53] B. Sumengen and B.S. Manjunath, "Edgeflow-Driven Variational Image Segmentation: Theory and Performance Evaluation," technical report, Univ. of California, Santa Barbara, http://barissumengen.comseg/, 2005.
[54] S. Kumar and M. Hebert, "Discriminative Random Fields: A Discriminative Framework for Contextual Interaction in Classification," Proc. IEEE Int'l Conf. Computer Vision, pp. 1150-1157, 2003.
[55] S.V.N. Vishwanathan, N.N. Schraudolph, M.W. Schmidt, and K.P. Murphy, "Accelerated Training of Conditional Random Fields with Stochastic Gradient Methods," Proc. Int'l Conf. Machine Learning, vol. 148, pp. 969-976, 2006.
[56] J. Marroquin, S. Mitter, and T. Poggio, "Probabilistic Solution of Ill-Posed Problems in Computational Vision," J. Am. Statistical Assoc., vol. 82, pp. 76-89, 1987.
[57] J. Yedidia, W. Freeman, and Y. Weiss, "Understanding Belief Propagation and Its Generalizations," Proc. Int'l Joint Conf. Artificial Intelligence, 2001.
[58] C. Harris and M. Stephens, "A Combined Corner and Edge Detector," Proc. Fourth Alvey Vision Conf., pp. 147-152, 1988.
[59] M. Pradhan, M. Henrion, G.M. Provan, B.D. Favero, and K. Huang, "The Sensitivity of Belief Networks to Imprecise Probabilities: An Experimental Investigation," Artificial Intelligence, vol. 85, nos. 1/2, pp. 363-397, 1996.
[60] F. Kschischang, B.J. Frey, and H.-A. Loeliger, "Factor Graphs and the Sum-Product Algorithm," IEEE Trans. Information Theory, vol. 47, no. 2, pp. 498-519, Feb. 2001.
[61] B.J. Frey, "Extending Factor Graphs so as to Unify Directed and Undirected Graphical Models," Proc. 19th Conf. Uncertainty in Artificial Intelligence, pp. 257-264, 2003.
[62] J. Park, "Using Weighted Max-Sat Engines to Solve MPE," Proc. 18th Nat'l Conf. Artificial Intelligence, pp. 682-687, 2002.
[63] F. Hutter, H.H. Hoos, and T. Stutzle, "Efficient Stochastic Local Search for MPE Solving," Proc. Int'l Joint Conf. Artificial Intelligence, pp. 169-174, 2005.
[64] T. Cour and J. Shi, "Recognizing Objects by Piecing Together the Segmentation Puzzle," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007.
[65] A. Levin and Y. Weiss, "Learning to Combine Bottom-Up and Top-Down Segmentation," Proc. European Conf. Computer Vision, pp. 581-594, 2006.
[66] L. Zhu, C. Lin, H. Huang, Y. Chen, and A.L. Yuille, "Unsupervised Structure Learning: Hierarchical Recursive Composition, Suspicious Coincidence and Competitive Exclusion," Proc. European Conf. Computer Vision, pp. 759-773, 2008.
[67] E. Borenstein and S. Ullman, "Learning to Segment," Proc. European Conf. Computer Vision, pp. 1-8, 2004.
[68] L. Yang, P. Meer, and D.J. Foran, "Multiple Class Segmentation Using a Unified Framework over Mean-Shift Patches," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007.
[69] M. Varma and A. Zisserman, "A Statistical Approach to Texture Classification from Single Images," Int'l J. Computer Vision, vol. 62, nos. 1/2, pp. 61-81, 2005.
[70] Z. Tu, "Auto-Context and Its Application to High-Level Vision Tasks," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2008.

Index Terms:
Image segmentation, probabilistic graphical model, Conditional Random Field, Bayesian Network, factor graph.
Citation:
Lei Zhang, Qiang Ji, "Image Segmentation with a Unified Graphical Model," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 8, pp. 1406-1425, Aug. 2010, doi:10.1109/TPAMI.2009.145
Usage of this product signifies your acceptance of the Terms of Use.