The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.07 - July (2012 vol.34)
pp: 1312-1328
João Carreira , University of Bonn, Bonn
ABSTRACT
We present a novel framework to generate and rank plausible hypotheses for the spatial extent of objects in images using bottom-up computational processes and mid-level selection cues. The object hypotheses are represented as figure-ground segmentations, and are extracted automatically, without prior knowledge of the properties of individual object classes, by solving a sequence of Constrained Parametric Min-Cut problems (CPMC) on a regular image grid. In a subsequent step, we learn to rank the corresponding segments by training a continuous model to predict how likely they are to exhibit real-world regularities (expressed as putative overlap with ground truth) based on their mid-level region properties, then diversify the estimated overlap score using maximum marginal relevance measures. We show that this algorithm significantly outperforms the state of the art for low-level segmentation in the VOC 2009 and 2010 data sets. In our companion papers [1], [2], we show that the algorithm can be used, successfully, in a segmentation-based visual object category recognition pipeline. This architecture ranked first in the VOC2009 and VOC2010 image segmentation and labeling challenges.
INDEX TERMS
Image segmentation, figure-ground segmentation, learning.
CITATION
João Carreira, "CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.34, no. 7, pp. 1312-1328, July 2012, doi:10.1109/TPAMI.2011.231
REFERENCES
[1] F. Li, J. Carreira, and C. Sminchisescu, "Object Recognition as Ranking Holistic Figure-Ground Hypotheses," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, June 2010.
[2] J. Carreira, F. Li, and C. Sminchisescu, "Object Recognition by Sequential Figure-Ground Ranking," Int'l J. Computer Vision, 2012.
[3] J. Carreira, A. Ion, and C. Sminchisescu, "Image Segmentation by Discounted Cumulative Ranking on Maximal Cliques," Technical Report 06-2010, Computer Vision and Machine Learning Group, Inst. of Numerical Simulation, Univ. of Bonn, June 2010.
[4] A. Ion, J. Carreira, and C. Sminchisescu, "Image Segmentation by Figure-Ground Composition into Maximal Cliques," Proc. IEEE Int'l Conf. Computer Vision, Nov. 2011.
[5] J. Shotton, J.M. Winn, C. Rother, and A. Criminisi, "Textonboost: Joint Appearance, Shape and Context Modeling for Multi-Class Object Recognition and Segmentation," Proc. European Conf. Computer Vision, pp. 1-15, May 2006.
[6] S. Alpert, M. Galun, R. Basri, and A. Brandt, "Image Segmentation by Probabilistic Bottom-Up Aggregation and Cue Integration," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007.
[7] M. Everingham, L.V. Gool, C.K.I. Williams, J. Winn, and A. Zisserman, "The PASCAL Visual Object Classes Challenge 2009 (VOC2009) Results," http://www.pascal-network.org/ challenges/ VOC/voc2009/workshopindex.html , 2012.
[8] M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn, and A. Zisserman, "The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results," http://www.pascal-network.org/ challenges/ VOC/voc2010/workshopindex.html , 2012.
[9] P.F. Felzenszwalb, R.B. Girshick, D. McAllester, and D. Ramanan, "Object Detection with Discriminatively Trained Part-Based Models," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 9, pp. 1627-1645, Sept. 2010.
[10] A. Vedaldi, V. Gulshan, M. Varma, and A. Zisserman, "Multiple Kernels for Object Detection," Proc. IEEE 12th Int'l Conf. Computer Vision, Sept. 2009.
[11] S. Gould, T. Gao, and D. Koller, "Region-Based Segmentation and Object Detection," Proc. Advances in Neural Information Processing Systems, pp. 655-663, Dec. 2009.
[12] Y. Yang, S. Hallman, D. Ramanan, and C. Fowlkes, "Layered Object Detection for Multi-Class Segmentation," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, June 2010.
[13] L. Ladicky, P. Sturgess, K. Alaharia, C. Russel, and P.H. Torr, "What, Where and How Many? Combining Object Detectors and CRFs," Proc. 11th European Conf. Computer Vision, Sept. 2010.
[14] J.M. Gonfaus, X. Boix, J. van de Weijer, A.D. Bagdanov, J. Serrat, and J. Gonzàlez, "Harmony Potentials for Joint Classification and Segmentation," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 1-8, June 2010.
[15] B. Russell, W. Freeman, A. Efros, J. Sivic, and A. Zisserman, "Using Multiple Segmentations to Discover Objects and Their Extent in Image Collections," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 1605-1614, June 2006.
[16] J. Carreira and C. Sminchisescu, "Constrained Parametric Min-Cuts for Automatic Object Segmentation, Release 1," http://sminchisescu.ins.uni-bonn.de/code cpmc/, 2012.
[17] J. Muerle and D. Allen, "Experimental Evaluation of Techniques for Automatic Segmentation of Objects in a Complex Scene," Proc. Pictorial Pattern Recognition, pp. 3-13, 1968.
[18] R.M. Haralick and L.G. Shapiro, "Image Segmentation Techniques," Computer Vision, Graphics, and Image Processing, vol. 29, no. 1, pp. 100-132, 1985.
[19] S. Zhu, T. Lee, and A. Yuille, "Region Competition: Unifying Snakes, Region Growing, Energy/Bayes/MDL for Multi-Band Image Segmentation," Proc. Fifth Int'l Conf. Computer Vision, pp. 416-423, June 1995.
[20] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, "Contour Detection and Hierarchical Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898-916, May 2011.
[21] J.L. Mundy, "Object Recognition in the Geometric Era: A Retrospective," Proc. Toward Category-Level Object Recognition, pp. 3-28, 2006.
[22] P. Viola and M. Jones, "Rapid Object Detection Using a Boosted Cascade of Simple Features," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, Dec. 2001.
[23] D. Martin, C. Fowlkes, D. Tal, and J. Malik, "A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics," Proc. Eighth IEEE Int'l Conf. Computer Vision, vol. 2, pp. 416-423, July 2001.
[24] R. Unnikrishnan, C. Pantofaru, and M. Hebert, "Toward Objective Evaluation of Image Segmentation Algorithms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. 929-944, June 2007.
[25] D. Hoiem, A.A. Efros, and M. Hebert, "Geometric Context from a Single Image," Proc. IEEE 10th Int'l Conf. Computer Vision, vol. 1, pp. 654-661, Oct. 2005.
[26] T. Malisiewicz and A. Efros, "Improving Spatial Support for Objects via Multiple Segmentations," Proc. British Machine Vision Conf., Sept. 2007.
[27] D. Comaniciu and P. Meer, "Mean Shift: A Robust Approach toward Feature Space Analysis," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, May 2002.
[28] J. Shi and J. Malik, "Normalized Cuts and Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.
[29] P.F. Felzenszwalb and D.P. Huttenlocher, "Efficient Graph-Based Image Segmentation," Int'l J. Computer Vision, vol. 59, no. 2, pp. 167-181, Sept. 2004.
[30] T.S.A. Stein and M. Hebert, "Towards Unsupervised Whole-Object Segmentation: Combining Automated Matting with Boundary Detection," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, June 2008.
[31] A. Rabinovich, S. Belongie, T. Lange, and J.M. Buhmann, "Model Order Selection and Cue Combination for Image Segmentation," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 1130-1137, June 2006.
[32] E. Sharon, M. Galun, D. Sharon, R. Basri, and A. Brandt, "Hierarchy and Adaptivity in Segmenting Visual Scenes," Nature, vol. 442, no. 7104, pp. 719-846, June 2006.
[33] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, "From Contours to Regions: An Empirical Evaluation," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 2294-2301, June 2009.
[34] M. Maire, P. Arbelaez, C. Fowlkes, and J. Malik, "Using Contours to Detect and Localize Junctions in Natural Images," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 1-8, June 2008.
[35] G. Gallo, M.D. Grigoriadis, and R.E. Tarjan, "A Fast Parametric Maximum Flow Algorithm and Applications," SIAM J. Computing, vol. 18, no. 1, pp. 30-55, Feb. 1989.
[36] Y. Boykov and G. Funka-Lea, "Graph Cuts and Efficient n-d Image Segmentation," Int'l J. Computer Vision, vol. 70, no. 2, pp. 109-131, 2006.
[37] C. Rother, V. Kolmogorov, and A. Blake, "'GrabCut': Interactive Foreground Extraction Using Iterated Graph Cuts," ACM Trans. Graphics, vol. 23, no. 3, pp. 309-314, 2004.
[38] S. Bagon, O. Boiman, and M. Irani, "What Is a Good Image Segment? A Unified Approach to Segment Extraction," Proc. 10th European Conf. Computer Vision, pp. 30-44, Oct. 2008.
[39] T. Schoenemann, F. Kahl, and D. Cremers, "Curvature Regularity for Region-Based Image Segmentation and Inpainting: A Linear Programming Relaxation," Proc. IEEE 12th Int'l Conf. Computer Vision, 2009.
[40] S. Vicente, V. Kolmogorov, and C. Rother, "Graph Cut Based Image Segmentation with Connectivity Priors," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[41] C. Fowlkes, D. Martin, and J. Malik, "Learning Affinity Functions for Image Segmentation: Combining Patch-Based and Gradient-Based Approaches," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 54-61, June 2003.
[42] P. Dollár, Z. Tu, and S. Belongie, "Supervised Learning of Edges and Object Boundaries," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, June 2006.
[43] J. Kaufhold and A. Hoogs, "Learning to Segment Images Using Region-Based Perceptual Features," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 954-961, June 2004.
[44] X. Ren and J. Malik, "Learning a Classification Model for Segmentation," Proc. IEEE Ninth Int'l Conf. Computer Vision, vol. 1, pp. 10-17, Oct. 2003.
[45] B. Peng and O. Veksler, "Parameter Selection for Graph Cut Based Image Segmentation," Proc. British Machine Vision Conf., Sept. 2008.
[46] V. Lempitsky, A. Blake, and C. Rother, "Image Segmentation by Branch-and-Mincut," Proc. 10th European Conf. Computer Vision, pp. 15-29, Oct. 2008.
[47] D. Cremers, F.R. Schmidt, and F. Barthel, "Shape Priors in Variational Image Segmentation: Convexity, Lipschitz Continuity and Globally Optimal Solutions," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-6, June 2008.
[48] T. Schoenemann and D. Cremers, "Globally Optimal Image Segmentation with an Elastic Shape Prior," Proc. IEEE Int'l Conf. Computer Vision, pp. 1-6, Oct. 2007.
[49] E. Borenstein, E. Sharon, and S. Ullman, "Combining Top-Down and Bottom-Up Segmentation," Proc. Computer Vision and Pattern Recognition Workshop, p. 46, June 2004.
[50] B. Leibe, A. Leonardis, and B. Schiele, "Robust Object Detection with Interleaved Categorization and Segmentation," Int'l J. Computer Vision, vol. 77, nos. 1-3, pp. 259-289, 2008.
[51] A. Levin and Y. Weiss, "Learning to Combine Bottom-Up and Top-Down Segmentation," Int'l J. Computer Vision, vol. 81, no. 1, pp. 105-118, 2009.
[52] M.P. Kumar, P. Torr, and A. Zisserman, "Objcut: Efficient Segmentation Using Top-Down and Bottom-Up Cues," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 3, pp. 530-545, Mar. 2010.
[53] J. Carreira, F. Li, and C. Sminchisescu, "Ranking Figure-Ground Hypotheses for Object Segmentation," oral presentation at the PASCAL VOC 2009 Workshop, http://pascallin.ecs.soton.ac.uk/challenges/ VOC/voc2009workshop/, 2012.
[54] J. Carreira and C. Sminchisescu, "Constrained Parametric Min-Cuts for Automatic Object Segmentation," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, June 2010.
[55] I. Endres and A. Hoiem, "Category Independent Object Proposals," Proc. 11th European Conf. Computer Vision, Sept. 2010.
[56] A. Levinshtein, C. Sminchisescu, and S. Dickinson, "Optimal Contour Closure by Superpixel Grouping," Proc. 11th European Conf. Computer Vision, Sept. 2010.
[57] B. Alexe, T. Deselaers, and V. Ferrari, "What Is an Object?" Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, June 2010.
[58] T. Liu, J. Sun, N.-N. Zheng, X. Tang, and H.-Y. Shum, "Learning to Detect a Salient Object," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, June 2007.
[59] V. Kolmogorov, Y. Boykov, and C. Rother, "Applications of Parametric Maxflow in Computer Vision," Proc. 11th IEEE Int'l Conf. Computer Vision, pp. 1-8, Oct. 2007.
[60] D.S. Hochbaum, "The Pseudoflow Algorithm: A New Algorithm for the Maximum-Flow Problem," Operations Research, vol. 56, pp. 992-1009, July 2008.
[61] M.A. Babenko, J. Derryberry, A.V. Goldberg, R.E. Tarjan, and Y. Zhou, "Experimental Evaluation of Parametric Max-Flow Algorithms," Proc. Sixth Int'l Conf. Experimental Algorithms, pp. 256-269, 2007.
[62] S. Wang and J.M. Siskind, "Image Segmentation with Ratio Cut," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 6, pp. 675-690, June 2003.
[63] M. Wertheimer, "Laws of Organization in Perceptual Forms (Partial Translation)," Proc. A Sourcebook of Gestalt Psychology, pp. 71-88, 1938.
[64] S.E. Palmer, Vision Science: Photons to Phenomenology. The MIT Press, 1999.
[65] S.C. Zhu and D. Mumford, "Learning Generic Prior Models for Visual Computation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 11, pp. 1236-1250, Nov. 1997.
[66] S.-C. Zhu, "Embedding Gestalt Laws in Markov Random Fields," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 11, pp. 1170-1187, Nov. 1999.
[67] Z. Wu and R. Leahy, "An Optimal Graph Theoretic Approach to Data Clustering: Theory and Its Application to Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1101-1113, Nov. 1993.
[68] O. Chapelle, P. Haffner, and V. Vapnik, "Support Vector Machines for Histogram-Based Image Classification," IEEE Trans. Neural Networks, vol. 10, no. 5, pp. 1055-1064, Sept. 1999.
[69] A.M. Bruckstein, R.J. Holt, and A.N. Netravali, "Discrete Elastica," Proc. Sixth Int'l Workshop Discrete Geometry for Computer Imagery, pp. 59-72, 1996.
[70] L. Breiman, "Random Forests," Machine Learning, vol. 45, no. 1, pp. 5-32, 2001.
[71] A. Jaiantilal, "Classification and Regression by Randomforest-Matlab," http://code.google.com/prandomforest-matlab , 2009.
[72] J. Carbonell and J. Goldstein, "The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries," Proc. 21st Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 335-336, Aug. 1998.
[73] D.G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," Int'l J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
[74] S. Belongie, J. Malik, and J. Puzicha, "Shape Context: A New Descriptor for Shape Matching and Object Recognition," Proc. Advances in Neural Information Processing Systems, pp. 831-837, Nov. 2000.
[75] A. Bosch, A. Zisserman, and X. Munoz, "Representing Shape with a Spatial Pyramid Kernel," Proc. Sixth ACM Int'l Conf. Image and Video Retrieval, pp. 401-408, July 2007.
[76] P.F. Felzenszwalb, R.B. Girshick, and D. McAllester, "Discriminatively Trained Deformable Part Models, Release 4," http://people.cs.uchicago.edu/pff/latent-release4 , 2012.
[77] A. Rahimi and B. Recht, "Random Features for Large-Scale Kernel Machines," Proc. Advances in Neural Information Processing Systems, Dec. 2007.
[78] F. Li, C. Ionescu, and C. Sminchisescu, "Random Fourier Approximations for Skewed Multiplicative Histogram Kernels," Proc. 32nd DAGM Conf. Pattern Recognition, Sept. 2010.
[79] A. Ion, J. Carreira, and C. Sminchisescu, "Probabilistic Joint Image Segmentation and Labeling," Proc. Advances in Neural Information Processing Systems, Dec. 2011.
[80] S. Vicente, C. Rother, and V. Kolmogorov, "Object Cosegmentation," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2217-2224, June 2011.
[81] Z. Tu and S.-C. Zhu, "Parsing Images into Regions, Curves, and Curve Groups," Int'l J. Computer Vision, vol. 69, pp. 223-249, Aug. 2006.
[82] V. Kolmogorov and R. Zabin, "What Energy Functions Can Be Minimized via Graph Cuts?" IEEE Trans Pattern Analysis and Machine Intelligence, vol. 26, no. 2, pp. 147-159, Feb. 2004.
[83] Toward Category-Level Object Recognition, J. Ponce, M. Hebert, C. Schmid, and A. Zisserman, eds. Springer, 2006.
[84] S. Dickinson, A. Leonardis, B. Schiele, and M. Tarr, Object Categorization: Computer and Human Vision Perspectives. Cambridge Univ. Press, 2009.
[85] A. Vedaldi and B. Fulkerson, "VLFeat: An Open and Portable Library of Computer Vision Algorithms," http:/www.vlfeat. org/, 2008.
38 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool