This Article 
 Bibliographic References 
 Add to: 
Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues
May 2004 (vol. 26 no. 5)
pp. 530-549

Abstract—The goal of this work is to accurately detect and localize boundaries in natural scenes using local image measurements. We formulate features that respond to characteristic changes in brightness, color, and texture associated with natural boundaries. In order to combine the information from these features in an optimal way, we train a classifier using human labeled images as ground truth. The output of this classifier provides the posterior probability of a boundary at each image location and orientation. We present precision-recall curves showing that the resulting detector significantly outperforms existing approaches. Our two main results are 1) that cue combination can be performed adequately with a simple linear model and 2) that a proper, explicit treatment of texture is required to detect boundaries in natural images.

[1] Z. Tu, S. Zhu, and H. Shum, Image Segmentation by Data Driven Markov Chain Monte Carlo Proc. Int'l Conf. Computer Vision, vol. 2, pp. 131-138 July 2001.
[2] J. Malik, S. Belongie, T. Leung, and J. Shi, Contour and Texture Analysis for Image Segmentation Int'l J. Computer Vision, vol. 43, no. 1, pp. 7-27, June 2001.
[3] L. Williams and D. Jacobs, Stochastic Completion Fields: A Neural Model of Illusory Contour Shape and Salience Proc. Fifth Int'l Conf. Computer Vision, pp. 408-415, 1995.
[4] X. Ren and J. Malik, A Probabilistic Multi-Scale Model for Contour Completion Based on Image Statistics Proc. Seventh European Conf. Computer Vision, 2002.
[5] J. Canny, A Computational Approach to Edge Detection IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 8, pp. 679-698, 1986.
[6] Y. Rubner and C. Tomasi, Coalescing Texture Descriptors Proc. ARPA Image Understanding Workshop, 1996.
[7] S. Will, L. Hermes, J.M. Buhmann, and J. Puzicha, On Learning Texture Edge Detectors Proc. Int'l Conf. Image Processing, pp. 877-880, 2000.
[8] C. Carson, S. Belongie, H. Greenspan, and J. Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 8, pp. 1026-1038, Aug. 2002.
[9] L. Hermes, T. Zoller, and J. Buhmann, Parametric Distributional Clustering for Image Segmentation Proc. European Conf. Computer Vision, 2002.
[10] J. Rivest and P. Cavanagh, Localizing Contours Defined by More Than One Attribute Vision Research, vol. 36, no. 1, pp. 53-66, 1996.
[11] D. Martin, C. Fowlkes, D. Tal, and J. Malik, A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics Proc. Eighth Int'l Conf. Computer Vision, pp. 416-424, 2001.
[12] S. Konishi, A.L. Yuille, J. Coughlan, and S.C. Zhu, Fundamental Bounds on Edge Detection: An Information Theoretic Evaluation of Different Edge Cues Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 573-579, 1999.
[13] M. Morrone and D. Burr, Feature Detection in Human Vision: A Phase Dependent Energy Model Proc. Royal Soc. of London B, vol. 235, pp. 221-245, 1988.
[14] P. Perona and J. Malik, "Detecting and Localizing Edges Composed of Steps, Peaks and Roofs," Proc. Third Int'l Conf. Computer Vision,Osaka, Japan, pp. 52-57, 1990.
[15] W. Niblack et al., The QBIC Project: Querying Image by Content Using Color, Texture, and Shape Proc. SPIE, vol. 1908, 1993.
[16] M. Ruzon and C. Tomasi, Color Edge Detection with the Compass Operator Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
[17] M. Ruzon and C. Tomasi, Corner Detection in Textured Color Images Proc. Int'l Conf. Computer Vision, pp. 1039-1045, 1999.
[18] I. Fogel and D. Sagi, Gabor Filters as Texture Discriminator Biological Cybernetics, vol. 61, pp. 103-113, 1989.
[19] J. Malik and P. Perona, Preattentive Texture Discrimination with Early Vision Mechanisms J. Optical Soc. Am., vol. 7, no. 2, pp. 923-932, May 1990.
[20] D.J. Heeger and J.R. Bergen, "Pyramid-Based Texture Analysis/Synthesis," Computer Graphics(Proc. Siggraph 95), ACM Press, New York, 1995, pp. 229-238.
[21] J. Puzicha, T. Hoffman, and J.M. Buhmann, “Non-Parametric Similarity Measures for Unsupervised Texture Segmentation and Image Retrieval,” Proc. Computer Vision and Pattern Recognition, 1997.
[22] J. Puzicha, Y. Rubner, C. Tomasi, and J. Buhmann, Empirical Evaluation of Dissimilarity Measures for Color and Texture Proc. Int'l Conf. Computer Vision, 1999.
[23] C. Mallows, A Note on Asymptotic Joint Normality Annals of Math. Statistics, vol. 43, no. 2, pp. 508-515, 1972.
[24] E. Levina and P. Bickel, The Earth Mover's Distance is the Mallows Distance: Some Insights From Statistics Proc. Int'l Conf. Computer Vision, vol. 2, pp. 251-256, 2001.
[25] S. Palmer, Vision Science. MIT Press, 1999.
[26] E. Levina, Statistical Issues in Texture Analysis PhD thesis, Univ. of California, Berkeley, 2002.
[27] Berkeley Segmentation and Boundary Detection Benchmark and Dataset 2003, groupingsegbench.
[28] C. Van Rijsbergen, Information Retrieval, second ed. Dept. of Computer Science, Univ. of Glasgow, 1979.
[29] I. Abdou and W. Pratt, Quantitative Design and Evaluation of Enhancement/Thresholding Edge Detectors Proc. IEEE, vol. 67, no. 5, pp. 753-763, May 1979.
[30] K. Bowyer, C. Kranenburg, and S. Dougherty, Edge Detector Evaluation Using Empirical ROC Curves Proc. Conf. Computer Vision and Pattern Recognition, 1999.
[31] D. Martin, C. Fowlkes, and J. Malik, Learning to Detect Natural Image Boundaries Using Brightness and Texture Neural Information Processing Systems, 2002.
[32] M. Meila and J. Shi, Learning Segmentation by Random Walks Neural Information Processing Systems, 2001.
[33] R.E. Schapire and Y. Singer, Improved Boosting Algorithms Using Confidence-Rated Predictions Machine Learning, vol. 37, no. 3, pp. 297-336, 1999.
[34] M.I. Jordan and R.A. Jacobs, Hierarchical Mixtures of Experts and the EM Algorithm Neural Computation, vol. 6, pp. 181-214, 1994.
[35] C. Chang and C. Lin, LIBSVM: A Library for Support Vector Machines, 2001, available at
[36] C. Harris and M.J. Stephens, A Combined Corner and Edge Detector Proc. Fourth Alvey Vision Conf., pp. 147-151. 1988,
[37] W. Förstner and E. Gulch, A Fast Operator for Detection and Precise Locations of Distinct Points, Corners, and Centres of Circular Features Proc. Intercommission Conf. Fast Processing of Photogrammetric Data, pp. 281-305, 1987.
[38] M. Nitzberg, D. Mumford, and T. Shiota, Filtering, Segmentation, and Depth. Springer-Verlag, 1993
[39] D. Martin, C. Fowlkes, and J. Malik, Local Boundary Detection in Natural Images: Matching Human and Machine Performance Proc. European Conf. Visual Perception, 2003.
[40] R. Jonker and A. Volgenant, A Shortest Augmenting Path Algorithm for Dense and Sparse Linear Assignment Problems Computing, vol. 38, pp. 325-340, 1987.
[41] G. Carpaneto, S. Martello, and P Toth, Algorithms and Codes for the Assignment Problem Annals of Operations Research, vol. 13, pp. 193-223, 1988.
[42] A. Goldberg and R. Kennedy, An Efficient Cost Scaling Algorithm for the Assignment Problem SIAM J. Discrete Math., 1993.
[43] B.V. Cherkassky and A.V. Goldberg, On Implementing Push-Relabel Method for the Maximum Flow Problem Proc. Fourth Integer Programming and Combinatorial Optimization Conf., pp. 157-171, May 1995.

Index Terms:
Texture, supervised learning, cue combination, natural images, ground truth segmentation data set, boundary detection, boundary localization.
David R. Martin, Charless C. Fowlkes, Jitendra Malik, "Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 5, pp. 530-549, May 2004, doi:10.1109/TPAMI.2004.1273918
Usage of this product signifies your acceptance of the Terms of Use.