This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Contour Detection and Hierarchical Image Segmentation
May 2011 (vol. 33 no. 5)
pp. 898-916
ASCII Text
x 
 
Pablo Arbeláez, Michael Maire, Charless Fowlkes, Jitendra Malik, "Contour Detection and Hierarchical Image Segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898-916, May, 2011.
 
 
BibTex
x
 
@article{ 10.1109/TPAMI.2010.161,
author = {Pablo Arbeláez and Michael Maire and Charless Fowlkes and Jitendra Malik},
title = {Contour Detection and Hierarchical Image Segmentation},
journal ={IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume = {33},
number = {5},
issn = {0162-8828},
year = {2011},
pages = {898-916},
doi = {http://doi.ieeecomputersociety.org/10.1109/TPAMI.2010.161},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
TI - Contour Detection and Hierarchical Image Segmentation
IS - 5
SN - 0162-8828
SP898
EP916
EPD - 898-916
A1 - Pablo Arbeláez,
A1 - Michael Maire,
A1 - Charless Fowlkes,
A1 - Jitendra Malik,
PY - 2011
KW - Contour detection
KW - image segmentation
KW - computer vision.
VL - 33
JA - IEEE Transactions on Pattern Analysis and Machine Intelligence
ER -
 
Pablo Arbeláez, University of California at Berkeley, Berkeley
Michael Maire, California Institute of Technology, Pasadena
Charless Fowlkes, University of California at Irvine, Irvine
Jitendra Malik, University of California at Berkeley, Berkeley
This paper investigates two fundamental problems in computer vision: contour detection and image segmentation. We present state-of-the-art algorithms for both of these tasks. Our contour detector combines multiple local cues into a globalization framework based on spectral clustering. Our segmentation algorithm consists of generic machinery for transforming the output of any contour detector into a hierarchical region tree. In this manner, we reduce the problem of image segmentation to that of contour detection. Extensive experimental evaluation demonstrates that both our contour detection and segmentation methods significantly outperform competing algorithms. The automatically generated hierarchical segmentations can be interactively refined by user-specified annotations. Computation at multiple image resolutions provides a means of coupling our system to recognition applications.

[1] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics,” Proc. Eighth IEEE Int'l Conf. Computer Vision, 2001.
[2] D.R. Martin, C.C. Fowlkes, and J. Malik, “Learning to Detect Natural Image Boundaries Using Local Brightness, Color and Texture Cues,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 5, pp. 530-549, May 2004.
[3] M. Maire, P. Arbeláez, C. Fowlkes, and J. Malik, “Using Contours to Detect and Localize Junctions in Natural Images,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[4] P. Arbeláez, M. Maire, C. Fowlkes, and J. Malik, “From Contours to Regions: An Empirical Evaluation,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[5] R. Unnikrishnan, C. Pantofaru, and M. Hebert, “Toward Objective Evaluation of Image Segmentation Algorithms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. 929-944, June 2007.
[6] M. Meila, “Comparing Clusterings: An Axiomatic View,” Proc. 22nd Int'l Conf. Machine Learning, 2005.
[7] A.Y. Yang, J. Wright, Y. Ma, and S.S. Sastry, “Unsupervised Segmentation of Natural Images via Lossy Data Compression,” Computer Vision and Image Understanding, vol. 110, pp. 212-225, 2008.
[8] M. Everingham, L. van Gool, C. Williams, J. Winn, and A. Zisserman “PASCAL 2008 Results,” http://www.pascal-network. org/challenges/ VOC/voc2008/workshopindex.html, 2008.
[9] D. Hoiem, A.A. Efros, and M. Hebert, “Geometric Context from a Single Image,” Proc. 10th IEEE Int'l Conf. Computer Vision, 2005.
[10] A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora, and S. Belongie, “Objects in Context,” Proc. 11th IEEE Int'l Conf. Computer Vision, 2007.
[11] T. Malisiewicz and A.A. Efros, “Improving Spatial Support for Objects via Multiple Segmentations,” Proc. British Machine Vision Conf., 2007.
[12] N. Ahuja and S. Todorovic, “Connected Segmentation Tree: A Joint Representation of Region Layout and Hierarchy,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[13] A. Saxena, S.H. Chung, and A.Y. Ng, “3-D Depth Reconstruction from a Single Still Image,” Int'l J. Computer Vision, vol. 76, pp. 53-69, 2008.
[14] T. Brox, C. Bregler, and J. Malik, “Large Displacement Optical Flow,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[15] C. Gu, J. Lim, P. Arbeláez, and J. Malik, “Recognition Using Regions,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[16] J. Lim, P. Arbeláez, C. Gu, and J. Malik, “Context by Region Ancestry,” Proc. 12th IEEE Int'l Conf. Computer Vision, 2009.
[17] L.G. Roberts, “Machine Perception of Three-Dimensional Solids,” Optical and Electro-Optical Information Processing, J.T. Tippett, et al., eds., MIT Press, 1965.
[18] R.O. Duda and P.E. Hart, Pattern Classification and Scene Analysis. Wiley, 1973.
[19] J.M.S. Prewitt, “Object Enhancement and Extraction,” Picture Processing and Psychopictorics, B. Lipkin and A. Rosenfeld, eds., Academic Press, 1970.
[20] D.C. Marr and E. Hildreth, “Theory of Edge Detection,” Proc. Royal Soc. of London, 1980.
[21] P. Perona and J. Malik, “Detecting and Localizing Edges Composed of Steps, Peaks and Roofs,” Proc. Third IEEE Int'l Conf. Computer Vision, 1990.
[22] J. Canny, “A Computational Approach to Edge Detection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp. 679-698, Nov. 1986.
[23] X. Ren, C. Fowlkes, and J. Malik, “Scale-Invariant Contour Completion Using Conditional Random Fields,” Proc. 10th IEEE Int'l Conf. Computer Vision, 2005.
[24] Q. Zhu, G. Song, and J. Shi, “Untangling Cycles for Contour Grouping,” Proc. 11th IEEE Int'l Conf. Computer Vision, 2007.
[25] P. Felzenszwalb and D. McAllester, “A Min-Cover Approach for Finding Salient Curves,” Proc. Fifth IEEE CS Workshop Perceptual Organization in Computer Vision, 2006.
[26] J. Mairal, M. Leordeanu, F. Bach, M. Hebert, and J. Ponce, “Discriminative Sparse Image Models for Class-Specific Edge Detection and Image Interpretation,” Proc. 10th European Conf. Computer Vision, 2008.
[27] P. Dollar, Z. Tu, and S. Belongie, “Supervised Learning of Edges and Object Boundaries,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2006.
[28] X. Ren, “Multi-Scale Improves Boundary Detection in Natural Images,” Proc. 10th European Conf. Computer Vision, 2008.
[29] M. Donoser, M. Urschler, M. Hirzer, and H. Bischof, “Saliency Driven Total Variational Segmentation,” Proc. 12th IEEE Int'l Conf. Computer Vision, 2009.
[30] L. Bertelli, B. Sumengen, B. Manjunath, and F. Gibou, “A Variational Framework for Multi-Region Pairwise Similarity-Based Image Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 8, pp. 1400-1414, Aug. 2008.
[31] E. Sharon, M. Galun, D. Sharon, R. Basri, and A. Brandt, “Hierarchy and Adaptivity in Segmenting Visual Scenes,” Nature, vol. 442, pp. 810-813, 2006.
[32] P.F. Felzenszwalb and D.P. Huttenlocher, “Efficient Graph-Based Image Segmentation,” Int'l J. Computer Vision, vol. 59, pp. 167-181, 2004.
[33] T. Cour, F. Benezit, and J. Shi, “Spectral Segmentation with Multiscale Graph Decomposition,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2005.
[34] D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach Toward Feature Space Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, May 2002.
[35] P. Arbeláez, “Boundary Extraction in Natural Images Using Ultrametric Contour Maps,” Proc. Fifth IEEE CS Workshop Perceptual Organization in Computer Vision, 2006.
[36] M.C. Morrone and R.A. Owens, “Feature Detection from Local Energy,” Pattern Recognition Letters, vol. 6, pp. 303-313, 1987.
[37] W.T. Freeman and E.H. Adelson, “The Design and Use of Steerable Filters,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 9, pp. 891-906, Sept. 1991.
[38] T. Lindeberg, “Edge Detection and Ridge Detection with Automatic Scale Selection,” Int'l J. Computer Vision, vol. 30, pp. 117-156, 1998.
[39] Z. Tu, “Probabilistic Boosting-Tree: Learning Discriminative Models for Classification, Recognition, and Clustering,” Proc. 10th IEEE Int'l Conf. Computer Vision, 2005.
[40] P. Parent and S.W. Zucker, “Trace Inference, Curvature Consistency, and Curve Detection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 11, no. 8, pp. 823-839, Aug. 1989.
[41] L.R. Williams and D.W. Jacobs, “Stochastic Completion Fields: A Neural Model of Illusory Contour Shape and Salience,” Proc. Fifth IEEE Int'l Conf. Computer Vision, 1995.
[42] J.H. Elder and S.W. Zucker, “Computing Contour Closures,” Proc. Fourth European Conf. Computer Vision, 1996.
[43] Y. Weiss, “Correctness of Local Probability Propagation in Graphical Models with Loops,” Neural Computation, vol. 12, pp. 1-41, 2000.
[44] S. Belongie, C. Carson, H. Greenspan, and J. Malik, “Color and Texture-Based Image Segmentation Using EM and Its Application to Content-Based Image Retrieval,” Proc. Sixth IEEE Int'l Conf. Computer Vision, pp. 675-682, 1998.
[45] J. Shi and J. Malik, “Normalized Cuts and Image Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.
[46] J. Malik, S. Belongie, T. Leung, and J. Shi, “Contour and Texture Analysis for Image Segmentation,” Int'l J. Computer Vision, vol. 43, pp. 7-27, 2001.
[47] D. Tolliver and G.L. Miller, “Graph Partitioning by Spectral Rounding: Applications in Image Segmentation and Clustering,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2006.
[48] F.R.K. Chung, Spectral Graph Theory. Am. Math. Soc., 1997.
[49] C. Fowlkes and J. Malik, “How Much Does Globalization Help Segmentation?” Technical Report CSD-04-1340, Univ. of California, Berkeley, 2004.
[50] S. Wang, T. Kubota, J.M. Siskind, and J. Wang, “Salient Closed Boundary Extraction with Ratio Contour,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 4, pp. 546-561, Apr. 2005.
[51] S.X. Yu, “Segmentation Induced by Scale Invariance,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2005.
[52] S. Alpert, M. Galun, R. Basri, and A. Brandt, “Image Segmentation by Probabilistic Bottom-Up Aggregation and Cue Integration,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[53] D. Mumford and J. Shah, “Optimal Approximations by Piecewise Smooth Functions, and Associated Variational Problems,” Comm. Pure and Applied Math., vol. 42, pp. 577-684, 1989.
[54] J.M. Morel and S. Solimini, Variational Methods in Image Segmentation. Birkhäuser, 1995.
[55] G. Koepfler, C. Lopez, and J.M. Morel, “A Multiscale Algorithm for Image Segmentation by Variational Method,” SIAM J. Numerical Analysis, vol. 31, pp. 282-299, 1994.
[56] T. Chan and L. Vese, “Active Contours without Edges,” IEEE Trans. Image Processing, vol. 10, no. 2, pp. 266-277, Feb. 2001.
[57] L. Vese and T. Chan, “A Multiphase Level Set Framework for Image Segmentation Using the Mumford and Shah Model,” Int'l J. Computer Vision, vol. 50, pp. 271-293, 2002.
[58] S. Osher and J.A. Sethian, “Fronts Propagation with Curvature Dependent Speed: Algorithms Based on Hamilton-Jacobi Formulations,” J. Computational Physics, vol. 79, pp. 12-49, 1988.
[59] J.A. Sethian, Level Set Methods and Fast Marching Methods. Cambridge Univ. Press, 1999.
[60] T. Pock, D. Cremers, H. Bischof, and A. Chambolle, “An Algorithm for Minimizing the Piecewise Smooth Mumford-Shah Functional,” Proc. 12th IEEE Int'l Conf. Computer Vision, 2009.
[61] F.J. Estrada and A.D. Jepson, “Benchmarking Image Segmentation Algorithms,” Int'l J. Computer Vision, vol. 85, pp. 167-181, 2009.
[62] W.M. Rand, “Objective Criteria for the Evaluation of Clustering Methods,” J. Am. Statistical Assoc., vol. 66, pp. 846-850, 1971.
[63] A. Savitzky and M.J.E. Golay, “Smoothing and Differentiation of Data by Simplified Least Squares Procedures,” Analytical Chemistry, vol. 36, pp. 1627-1639, 1964.
[64] C. Fowlkes, D. Martin, and J. Malik, “Learning Affinity Functions for Image Segmentation: Combining Patch-Based and Gradient-Based Approaches,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2003.
[65] T. Leung and J. Malik, “Contour Continuity in Region-Based Image Segmentation,” Proc. Fifth European Conf. Computer Vision, 1998.
[66] S. Belongie and J. Malik, “Finding Boundaries in Natural Images: A New Method Using Point Descriptors and Area Completion,” Proc. Fifth European Conf. Computer Vision, 1998.
[67] X. Ren and J. Malik, “Learning a Classification Model for Segmentation,” Proc. Ninth IEEE Int'l Conf. Computer Vision, 2003.
[68] S. Beucher and F. Meyer, Mathematical Morphology in Image Processing. Marcel Dekker, 1992.
[69] L. Najman and M. Schmitt, “Geodesic Saliency of Watershed Contours and Hierarchical Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 12, pp. 1163-1173, Dec. 1996.
[70] S. Rao, H. Mobahi, A. Yang, S. Sastry, and Y. Ma, “Natural Image Segmentation with Adaptive Texture and Boundary Encoding,” Proc. Asian Conf. Computer Vision, 2009.
[71] J. Shotton, J. Winn, C. Rother, and A. Criminisi, “Textonboost: Joint Appearance, Shape and Context Modeling for Multi-Class Object Recognition and Segmentation,” Proc. European Conf. Computer Vision, 2006.
[72] Y.Y. Boykov and M.-P. Jolly, “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images.” Proc. Eighth IEEE Int'l Conf. Computer Vision, 2001.
[73] C. Rother, V. Kolmogorov, and A. Blake, “'Grabcut': Interactive Foreground Extraction Using Iterated Graph Cuts,” ACM Trans. Graphics, vol. 23, pp. 309-314, 2004.
[74] Y. Li, J. Sun, C.-K. Tang, and H.-Y. Shum, “Lazy Snapping,” ACM Trans. Graphics, vol. 23, pp. 303-308, 2004.
[75] S. Bagon, O. Boiman, and M. Irani, “What Is a Good Image Segment? A Unified Approach to Segment Extraction,” Proc. 10th European Conf. Computer Vision, 2008.
[76] P. Arbeláez and L. Cohen, “Constrained Image Segmentation from Hierarchical Boundaries,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[77] B. Catanzaro, B.-Y. Su, N. Sundaram, Y. Lee, M. Murphy, and K. Keutzer, “Efficient, High-Quality Image Contour Detection,” Proc. 12th IEEE Int'l Conf. Computer Vision, 2009.

Index Terms:
Contour detection, image segmentation, computer vision.
Citation:
Pablo Arbeláez, Michael Maire, Charless Fowlkes, Jitendra Malik, "Contour Detection and Hierarchical Image Segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898-916, May 2011, doi:10.1109/TPAMI.2010.161
Usage of this product signifies your acceptance of the Terms of Use.