This Article 
 Bibliographic References 
 Add to: 
Evaluating Stability and Comparing Output of Feature Selectors that Optimize Feature Subset Cardinality
November 2010 (vol. 32 no. 11)
pp. 1921-1939
Petr Somol, Institute of Information Theory and Automation of the Czech Academy of Sciences, Prague
Jana Novovičová, Institute of Information Theory and Automation of the Czech Academy of Sciences, Prague
Stability (robustness) of feature selection methods is a topic of recent interest, yet often neglected importance, with direct impact on the reliability of machine learning systems. We investigate the problem of evaluating the stability of feature selection processes yielding subsets of varying size. We introduce several novel feature selection stability measures and adjust some existing measures in a unifying framework that offers broad insight into the stability problem. We study in detail the properties of considered measures and demonstrate on various examples what information about the feature selection process can be gained. We also introduce an alternative approach to feature selection evaluation in the form of measures that enable comparing the similarity of two feature selection processes. These measures enable comparing, e.g., the output of two feature selection methods or two runs of one method with different parameters. The information obtained using the considered stability and similarity measures is shown to be usable for assessing feature selection methods (or criteria) as such.

[1] K. Dunne, P. Cunningham, and F. Azuaje, "Solutions to Instability Problems with Sequential Wrapper-Based Approaches to Feature Selection," Technical Report TCD-CD-2002-28, Dept. of Computer Science, Trinity College, 2002.
[2] A. Kalousis, J. Prados, and M. Hilario, "Stability of Feature Selection Algorithms," Proc. Fifth IEEE Int'l Conf. Data Mining, pp. 218-225, 2005.
[3] L.I. Kuncheva, "A Stability Index for Feature Selection," Proc. 25th IASTED Int'l Multi-Conf. Artificial Intelligence and Applications, pp. 421-427, 2007.
[4] P. Křížek, J. Kittler, and V. Hlaváč, "Improving Stability of Feature Selection Methods," Proc. 12th Int'l Conf. Computer Analysis of Images and Patterns, pp. 929-936, 2007.
[5] A. Kalousis, J. Prados, and M. Hilario, "Stability of Feature Selection Algorithms: A Study on High-Dimensional Spaces," Knowledge and Information Systems, vol. 12, no. 1, pp. 95-116, 2007.
[6] P. Somol and J. Novovičová, "Evaluating the Stability of Feature Selectors that Optimize Feature Subset Cardinality," Proc. Joint IAPR Int'l Workshop Structural, Syntactic, and Statistical Pattern Recognition, pp. 956-966, 2008.
[7] S. Loscalzo, L. Yu, and C.H.Q. Ding, "Consensus Group Stable Feature Selection," Proc. 15th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, J.F. Elder, IV, F. Fogelman-Soulié, P.A. Flach, and M.J. Zaki, eds., pp. 567-576, http://doi.acm. org/10.11451557019.1557084 , 2009.
[8] Y. Saeys, T. Abeel, and Y.V. de Peer, "Towards Robust Feature Selection Techniques," Proc. Belgian-Dutch Conf. Machine Learning, pp. 45-46, 2008.
[9] S. Raudys, "Feature Over-Selection," Lecture Notes in Computer Science, vol. 4109, pp. 622-631, Springer, 2006.
[10] H. Vafaie and K.D. Jong, "Genetic Algorithms as a Tool for Feature Selection in Machine Learning," Proc. 1992 IEEE Int'l Conf. Tools with AI, pp. 200-204, 1992.
[11] F. Hussein, R. Ward, and N. Kharma, "Genetic Algorithms for Feature Selection and Weighting, A Review and Study," Proc. Int'l Conf. Document Analysis and Recognition, p. 1240, 2001.
[12] P. Somol, J. Novovičová, P. Pudil, and J. Grim, "Dynamic Oscillating Search Algorithm for Feature Selection," Proc. 19th Int'l Conf. Pattern Recognition, Dec. 2008.
[13] R. Duda, P. Hart, and D. Stork, Pattern Classification and Scene Analysis. J. Wiley, 2001.
[14] R.E. Bellman, Adaptive Control Processes. Princeton Univ. Press, 1961.
[15] A. Asuncion and D. Newman, "UCI Machine Learning Repository," , 2007.
[16] F. Sebastiani, "Machine Learning in Automated Text Categorization," ACM Computing Surveys, vol. 34, no. 1, pp. 1-47, Mar. 2002.
[17] T. Cover, "The Best Two Independent Measurements Are Not the Two Best," IEEE Trans. Systems, Man, and Cybernetics, vol. 4, no. 1, pp. 116-117, Jan. 1974.
[18] A.K. Jain, R.P.W. Duin, and J. Mao, "Statistical Pattern Recognition: A Review," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[19] A.W. Whitney, "A Direct Method of Nonparametric Measurement Selection," IEEE Trans. Computers, vol. 20, no. 9, pp. 1100-1103, Sept. 1971.
[20] P. Pudil, J. Novovičová, and J. Kittler, "Floating Search Methods in Feature Selection," Pattern Recognition Letters, vol. 15, pp. 1119-1125, 1994.
[21] P.A. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach. Prentice-Hall Int'l, 1982.
[22] J. Novovičová, P. Somol, and P. Pudil, "Oscillating Feature Subset Search Algorithm for Text Categorization," Lecture Notes in Computer Science, vol. 4225, pp. 572-587, Springer, 2006.
[23] R. Kohavi and G. John, "Wrappers for Feature Subset Selection," Artificial Intelligence, vol. 97, pp. 273-324, 1997.
[24] C.-C. Chang and C.-J. Lin, LIBSVM: A Library for Support Vector Machines,, 2001.
[25] A. McCallum and K. Nigam, "A Comparison of Event Models for Naive Bayes Text Classification," Proc. AAAI-98 Workshop Learning for Text Categorization, pp. 41-48, 1998.
[26] G. Forman, "An Experimental Study of Feature Selection Metrics for Text Categorization." J. Machine Learning Research, vol. 3, pp. 1289-1305, 2003.
[27] Y. Yang and J.O. Pedersen, "A Comparative Study on Feature Selection in Text Categorization," Proc. 14th Int'l Conf. Machine Learning, pp. 412-420, html , 1997.

Index Terms:
Feature selection, feature stability, stability measures, similarity measures, sequential search, individual ranking, feature subset-size optimization, high dimensionality, small sample size.
Petr Somol, Jana Novovičová, "Evaluating Stability and Comparing Output of Feature Selectors that Optimize Feature Subset Cardinality," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 11, pp. 1921-1939, Nov. 2010, doi:10.1109/TPAMI.2010.34
Usage of this product signifies your acceptance of the Terms of Use.