The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.01 - Jan. (2014 vol.36)
pp: 99-112
Ming-Yu Liu , Mitsubishi Electr. Res. Labs. (MERL), Mitsubishi Electr. Corp., Cambridge, MA, USA
Oncel Tuzel , Mitsubishi Electr. Res. Labs. (MERL), Mitsubishi Electr. Corp., Cambridge, MA, USA
Srikumar Ramalingam , Mitsubishi Electr. Res. Labs. (MERL), Mitsubishi Electr. Corp., Cambridge, MA, USA
Rama Chellappa , Univ. of Maryland, College Park, MD, USA
ABSTRACT
We propose a new objective function for clustering. This objective function consists of two components: the entropy rate of a random walk on a graph and a balancing term. The entropy rate favors formation of compact and homogeneous clusters, while the balancing function encourages clusters with similar sizes and penalizes larger clusters that aggressively group samples. We present a novel graph construction for the graph associated with the data and show that this construction induces a matroid--a combinatorial structure that generalizes the concept of linear independence in vector spaces. The clustering result is given by the graph topology that maximizes the objective function under the matroid constraint. By exploiting the submodular and monotonic properties of the objective function, we develop an efficient greedy algorithm. Furthermore, we prove an approximation bound of 1/2 for the optimality of the greedy solution. We validate the proposed algorithm on various benchmarks and show its competitive performances with respect to popular clustering algorithms. We further apply it for the task of superpixel segmentation. Experiments on the Berkeley segmentation data set reveal its superior performances over the state-of-the-art superpixel segmentation algorithms in all the standard evaluation metrics.
INDEX TERMS
Entropy, Linear programming, Image segmentation, Clustering algorithms, Algorithm design and analysis, Topology, Uncertainty,discrete optimization, Clustering, superpixel segmentation, graph theory, information theory, submodular function
CITATION
Ming-Yu Liu, Oncel Tuzel, Srikumar Ramalingam, Rama Chellappa, "Entropy-Rate Clustering: Cluster Analysis via Maximizing a Submodular Function Subject to a Matroid Constraint", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.36, no. 1, pp. 99-112, Jan. 2014, doi:10.1109/TPAMI.2013.107
REFERENCES
[1] E.T. Jaynes, "Information Theory and Statistical Mechanics," Physical Rev., vol. 106, no. 4, pp. 620-630, 1957.
[2] L. Lovász, "Submodular Functions and Convexity," Math. Programming—State of the Art, pp. 235-257, Springer-Verlag, 1983.
[3] C. Guestrin, A. Krause, and A.P. Singh, "Near-Optimal Sensor Placements in Gaussian Processes: Theory, Efficient Algorithms and Empirical Studies," J. Machine Learning Research, vol. 9, pp. 235-284, 2008.
[4] J. Leskovec, A. Krause, C. Guestrin, C. Faloutsos, J. VanBriesen, and N. Glance, "Cost-Effective Outbreak Detection in Networks," Proc. 13th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, pp. 420-429. 2007.
[5] H. Lin and J. Bilmes, "Word Alignment via Submodular Maximization over Matroids," Proc. 49th Ann. Meeting Assoc. Computational Linguistics: Human Language Technologies—Short Papers, vol. 2, pp. 170-175, 2011.
[6] X. Ren and J. Malik, "Learning a Classification Model for Segmentation," Proc. Ninth IEEE Int'l Conf. Computer Vision, 2003.
[7] G. Mori, X. Ren, A.A. Efros, and J. Malik, "Recovering Human Body Configurations: Combining Segmentation and Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2004.
[8] P. Kohli, L. Ladicky, and P. Torr, "Robust Higher Order Potentials for Enforcing Label Consistency," Int'l J. Computer Vision, vol. 82, pp. 302-324, 2009.
[9] D. Hoiem, A.N. Stein, A.A. Efros, and M. Hebert, "Recovering Occlusion Boundaries from a Single Image," Proc. IEEE Int'l Conf. Computer Vision, 2007.
[10] S. Ramalingam, P. Kohli, K. Alahari, and P.H.S. Torr, "Exact Inference in Multi-Label CRFs with Higher Order Cliques," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[11] A.K. Jain, M.N. Murty, and P.J. Flynn, "Data Clustering: A Review," ACM Computing Surveys, vol. 31, no. 3, pp. 264-323, 1999.
[12] P. Berkhin, "Survey of Clustering Data Mining Techniques," technical report, 2002.
[13] R. Xu and D. WunschII, "Survey of Clustering Algorithms," IEEE Trans. Neural Networks, vol. 16, no. 3, pp. 645-678, May 2005.
[14] M. Filippone, F. Camastra, F. Masulli, and S. Rovetta, "A Survey of Kernel and Spectral Methods for Clustering," Pattern Recognition, vol. 41, pp. 176-190, Jan. 2008.
[15] C.T. Zahn, "Graph-Theoretical Methods for Detecting and Describing Gestalt Clusters," IEEE Trans. Computers, vol. 20, no. 1, pp. 68-86, Jan. 1971.
[16] Z. Wu and R. Leahy, "An Optimal Graph Theoretic Approach to Data Clustering: Theory and Its Application to Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1101-1113, Nov. 1993.
[17] J. Shi and J. Malik, "Normalized Cuts and Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.
[18] N. Bansal, A. Blum, and S. Chawla, "Correlation Clustering," Machine Learning, vol. 56, no. 1-3, pp. 89-113, 2004.
[19] A. Ng, M. Jordan, and Y. Weiss, "On Spectral Clustering: Analysis and an Algorithm," Proc. Advances in Neural Information Processing Systems Foundation, pp. 849-856, 2001.
[20] D. Yan, L. Huang, and M.I. Jordan, "Fast Approximate Spectral Clustering," Proc. 15th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, pp. 907-916, 2009.
[21] R. Krauthgamer, J. Naor, and R. Schwartz, "Partitioning Graphs into Balanced Components," Proc. 20th Ann. ACM-SIAM Symp. Discrete Algorithms, 2009.
[22] M. Meilă and J. Shi, "A Random Walks View of Spectral Segmentation," Proc. IEEE Int'l Conf. Artificial Intelligence and Statistics, 2001.
[23] D. Harel and Y. Koren, "On Clustering Using Random Walks," Proc. 21st Conf. Foundations of Software Technology and Theoretical Computer Science, vol. 2245, pp. 18-41, 2001.
[24] L. Yen, D. Vanvyve, F. Wouters, F. Fouss, M. Verleysen, and M. Saerens, "Clustering Using a Random-Walk Based Distance Measure," Proc. European Symp. Artificial Neural Networks, 2005.
[25] L. Grady, "Random Walks for Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 11, pp. 1768-1783, Nov. 2006.
[26] A. Banerjee, S. Merugu, I.S. Dhillon, and J. Ghosh, "Clustering with Bregman Divergences," J. Machine Learning Research, vol. 6, pp. 1705-1749, 2005.
[27] A.C. Müller, S. Nowozin, and C.H. Lampert, "Information Theoretic Clustering Using Minimum Spanning Trees," Proc. Ann. Symp. German Assoc. Pattern Recognition Symp., pp. 205-215, 2012.
[28] M. Narasimhan, N. Jojic, and J. Bilmes, "Q-Clustering," Proc. Advances in Neural Information Processing Systems Foundation, pp. 979-986, 2006.
[29] K. Nagano, Y. Kawahara, and S. Iwata, "Minimum Average Cost Clustering," Proc. Advances in Neural Information Processing Systems Foundation, 2010.
[30] S. Jegelka and J. Bilmes, "Submodularity beyond Submodular Energies: Coupling Edges in Graph Cuts," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2011.
[31] Y. Boykov and V. Kolmogorov, "An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 9, pp. 1124-1137, Sept. 2004.
[32] P.F. Felzenszwalb and D.P. Huttenlocher, "Efficient Graph-Based Image Segmentation," Int'l J. Computer Vision, vol. 59, no. 2, pp. 167-181, 2004.
[33] D. Comaniciu and P. Meer, "Mean Shift: A Robust Approach Toward Feature Space Analysis," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, May 2002.
[34] L. Vincent and P. Soille, "Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 6, pp. 583-598, June 1991.
[35] A. Levinshtein, A. Stere, K.N. Kutulakos, D.J. Fleet, and S.J. Dickinson, "Fast Superpixels Using Geometric Flows," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 12, pp. 2290-2297, Dec. 2009.
[36] O. Veksler and Y. Boykov, "Superpixels and Supervoxels in an Energy Optimization Framework," Proc. European Conf. Computer Vision, 2010.
[37] Y. Boykov, O. Veksler, and R. Zabih, "Efficient Approximate Energy Minimization via Graph Cut," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 12, pp. 1222-1239, Dec. 2001.
[38] A.P. Moore, S.J.D. Prince, J. Warrell, U. Mohammed, and G. Jones, "Superpixel Lattices," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[39] A.P. Moore, S.J.D. Prince, and J. Warrell, "'Lattice Cut'— Constructing Superpixels Using Layer Constraints," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[40] Y. Taguchi, B. Wilburn, and C.L. Zitnick, "Stereo Reconstruction with Mixed Pixels Using Adaptive Over-Segmentation," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[41] M. Bleyer, C. Rother, P. Kohli, D. Scharstein, and S. Sinha, "Object Stereo-Joint Stereo Matching and Object Segmentation," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2011.
[42] M.-Y. Liu, O. Tuzel, S. Ramalingam, and R. Chellappa, "Entropy Rate Superpixel Segmentation," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2011.
[43] T.M. Cover and J.A. Thomas, Elements of Information Theory, second ed. John Wiley & Sons, 1991.
[44] G.L. Nemhauser, L.A. Wolsey, and M.L. Fisher, "An Analysis of the Approximations for Maximizing Submodular Set Functions," Math. Programming, vol. 14, pp. 265-294, 1978.
[45] J. Oxley, Matroid Theory. Oxford Univ. Press, 1992.
[46] M.Á Carreira-Perpiñán and R.S. Zemel, "Proximity Graphs for Clustering and Manifold Learning," Proc. Advances in Neural Information Processing Systems Foundation, pp. 225-232, 2004.
[47] M.L. Fisher, G.L. Nemhauser, and L.A. Wolsey, "An Analysis of the Approximations for Maximizing Submodular Set Functions— ii," Math. Programming, vol. 8, pp. 73-87, 1978.
[48] A. Oliva and A. Torralba, "Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope," Int'l J. Computer Vision, vol. 42, no. 3, pp. 145-175, 2001.
[49] S. Jeannin and M. Bober, "Description of Core Experiments for MPEG-7 Motion/Shape," Technical Report ISO/IEC JTC 1/SC29/WG 11 MPEG99/N2690, MPEG-7, 1999.
[50] H. Ling and D.W. Jacobs, "Shape Classification Using the Inner-Distance," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 2, pp. 286-299, Feb. 2007.
[51] B.J. Frey and D. Dueck, "Clustering by Passing Messages between Data Points," Science, vol. 315, no. 5814, pp. 972-976, 2007.
[52] F. Wang, B. Zhao, and C. Zhang, "Linear Time Maximum Margin Clustering," IEEE Trans. Neural Networks, vol. 21, no. 2, pp. 319-332, Feb. 2010.
[53] D. Martin, C. Fowlkes, D. Tal, and J. Malik, "A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics," Proc. IEEE Int'l Conf. Computer Vision, 2001.
[54] S. Nowozin, P. Gehler, and C. Lampert, "On Parameter Learning in CRF-Based Approaches to Object Class Image Segmentation," Proc. European Conf. Computer Vision: Part VI, 2010.
102 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool