This Article 
 Bibliographic References 
 Add to: 
On the Selection and Classification of Independent Features
October 2003 (vol. 25 no. 10)
pp. 1312-1317

Abstract—This paper is focused on the problems of feature selection and classification when classes are modeled by statistically independent features. We show that, under the assumption of class-conditional independence, the class separability measure of divergence is greatly simplified, becoming a sum of unidimensional divergences, providing a feature selection criterion where no exhaustive search is required. Since the hypothesis of independence is infrequently met in practice, we also provide a framework making use of class-conditional Independent Component Analyzers where this assumption can be held on stronger grounds. Divergence and the Bayes decision scheme are adapted to this class-conditional representation. An algorithm that integrates the proposed representation, feature selection technique, and classifier is presented. Experiments on artificial, benchmark, and real-world data illustrate our technique and evaluate its performance.

[1] A. Bell and T. Sejnowski, The 'Independent Components' of Natural Scenes Are Edge Filters Neural Computation, vol. 11, pp. 1739-1768, 1999.
[2] C. Blake and C. Merz, UCI Repository of Machine Learning Databases 1998.
[3] M. Bressan, D. Guillamet, and J. Vitria, Using an ICA Representation of High Dimensional Data for Object Recognition and Classification Proc. IEEE Computer Soc. Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 1004-1009, 2001.
[4] J. Cardoso and P. Comon, Independent Component Analysis, a Survey of Some Algebraic Methods Proc. Int'l Symp. Circuits and Systems (ISCAS '96), vol. 2, pp. 93-96, 1996.
[5] S. Choi, A. Cichocki, and S. Amari, Flexible Independent Component Analysis J. VLSI Signal Processing, vol. 26, nos. 1/2, pp. 25-38, Aug. 2000.
[6] P. Comon, Independent Component Analysis a New Concept? Signal Processing, vol. 36, pp. 287-314, 1994.
[7] Corel Stock Photo Library, Corel Corp., Ontario, Canada, 1990.
[8] A.P. Dawid, Conditional Independence in Statistical Theory (with Discussion) J. Royal Statistical Soc., Ser. B, vol. 41, pp. 1-31, 1979.
[9] H. Decell and J. Quirein, An Iterative Approach to the Feature Selection Problem Proc. Purdue Univ. Conf. Machine Processing of Remotely Sensed Data, vol. 1, pp. 3B1-3B12, 1972.
[10] R. Duda, P. Hart, and D. Stork, Pattern Classication, second ed. John Wiley&Sons, 2001.
[11] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Academic Press, 1990.
[12] A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis. John Wiley&Sons, 2001.
[13] A. Jain and D. Zongker, Feature Selection: Evaluation, Application, and Small Sample Performance IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 2, pp. 153-158, Feb. 1997.
[14] T. Kailath, The Divergence and Bhattacharyya Distance Measures in Signal Selection IEEE Trans. Comm. Technology, vol. 15, no. 1, pp. 52-60, Feb. 1967
[15] S. Kullback, Information Theory and Statistics. John Wiley&Sons, 1968.
[16] T. Lee, M. Lewicki, and T. Seynowski, A Mixture Models for Unsupervised Classification of Non-Gaussian Sources and Automatic Context Switching in Blind Signal Separation IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp. 1-12, Oct. 2000.
[17] D. Lewis, Naive Bayes at Forty: The Independence Assumption in Information Retrieval Proc. ECML-98, 10th European Conf. Machine Learning, C. N'edellec and C. Rouveirol, eds., pp. 4-15, 1998.
[18] T. Marill and D. Green, On the Effectiveness of Receptors in Recognition Systems IEEE Trans. Information Theory, vol. 9, pp. 1-17, 1963.
[19] J. Miskin, Ensemble Learning for Independent Component Analysis PhD thesis, Selwyn College, Cambridge, Dec. 2000.
[20] E. Simpson, The Interpretation of Interaction in Contingency Tables J. Royal Statistical Soc., Ser. B, vol. 13, pp. 238-241, 1951.
[21] G. Trunk, A Problem of Dimensionality: A Simple Example IEEE Trans. Pattern Analysis and Machine Intelligence vol. 1, no. 3, pp. 306-307, July 1979.
[22] Y. Yang, S. Slattery, and R. Ghani, A Study of Approaches to Hypertext Categorization J. Intelligent Information Systems, 2002.

Index Terms:
Feature selection, divergence, independent component analysis, naive Bayes.
Marco Bressan, Jordi Vitri?, "On the Selection and Classification of Independent Features," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1312-1317, Oct. 2003, doi:10.1109/TPAMI.2003.1233904
Usage of this product signifies your acceptance of the Terms of Use.