This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
On the Selection and Classification of Independent Features
October 2003 (vol. 25 no. 10)
pp. 1312-1317

Abstract—This paper is focused on the problems of feature selection and classification when classes are modeled by statistically independent features. We show that, under the assumption of class-conditional independence, the class separability measure of divergence is greatly simplified, becoming a sum of unidimensional divergences, providing a feature selection criterion where no exhaustive search is required. Since the hypothesis of independence is infrequently met in practice, we also provide a framework making use of class-conditional Independent Component Analyzers where this assumption can be held on stronger grounds. Divergence and the Bayes decision scheme are adapted to this class-conditional representation. An algorithm that integrates the proposed representation, feature selection technique, and classifier is presented. Experiments on artificial, benchmark, and real-world data illustrate our technique and evaluate its performance.

[1] A. Bell and T. Sejnowski, The 'Independent Components' of Natural Scenes Are Edge Filters Neural Computation, vol. 11, pp. 1739-1768, 1999.
[2] C. Blake and C. Merz, UCI Repository of Machine Learning Databases 1998.
[3] M. Bressan, D. Guillamet, and J. Vitria, Using an ICA Representation of High Dimensional Data for Object Recognition and Classification Proc. IEEE Computer Soc. Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 1004-1009, 2001.
[4] J. Cardoso and P. Comon, Independent Component Analysis, a Survey of Some Algebraic Methods Proc. Int'l Symp. Circuits and Systems (ISCAS '96), vol. 2, pp. 93-96, 1996.
[5] S. Choi, A. Cichocki, and S. Amari, Flexible Independent Component Analysis J. VLSI Signal Processing, vol. 26, nos. 1/2, pp. 25-38, Aug. 2000.
[6] P. Comon, Independent Component Analysis a New Concept? Signal Processing, vol. 36, pp. 287-314, 1994.
[7] Corel Stock Photo Library, Corel Corp., Ontario, Canada, 1990.
[8] A.P. Dawid, Conditional Independence in Statistical Theory (with Discussion) J. Royal Statistical Soc., Ser. B, vol. 41, pp. 1-31, 1979.
[9] H. Decell and J. Quirein, An Iterative Approach to the Feature Selection Problem Proc. Purdue Univ. Conf. Machine Processing of Remotely Sensed Data, vol. 1, pp. 3B1-3B12, 1972.
[10] R. Duda, P. Hart, and D. Stork, Pattern Classication, second ed. John Wiley&Sons, 2001.
[11] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Academic Press, 1990.
[12] A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis. John Wiley&Sons, 2001.
[13] A. Jain and D. Zongker, Feature Selection: Evaluation, Application, and Small Sample Performance IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 2, pp. 153-158, Feb. 1997.
[14] T. Kailath, The Divergence and Bhattacharyya Distance Measures in Signal Selection IEEE Trans. Comm. Technology, vol. 15, no. 1, pp. 52-60, Feb. 1967
[15] S. Kullback, Information Theory and Statistics. John Wiley&Sons, 1968.
[16] T. Lee, M. Lewicki, and T. Seynowski, A Mixture Models for Unsupervised Classification of Non-Gaussian Sources and Automatic Context Switching in Blind Signal Separation IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp. 1-12, Oct. 2000.
[17] D. Lewis, Naive Bayes at Forty: The Independence Assumption in Information Retrieval Proc. ECML-98, 10th European Conf. Machine Learning, C. N'edellec and C. Rouveirol, eds., pp. 4-15, 1998.
[18] T. Marill and D. Green, On the Effectiveness of Receptors in Recognition Systems IEEE Trans. Information Theory, vol. 9, pp. 1-17, 1963.
[19] J. Miskin, Ensemble Learning for Independent Component Analysis PhD thesis, Selwyn College, Cambridge, Dec. 2000.
[20] E. Simpson, The Interpretation of Interaction in Contingency Tables J. Royal Statistical Soc., Ser. B, vol. 13, pp. 238-241, 1951.
[21] G. Trunk, A Problem of Dimensionality: A Simple Example IEEE Trans. Pattern Analysis and Machine Intelligence vol. 1, no. 3, pp. 306-307, July 1979.
[22] Y. Yang, S. Slattery, and R. Ghani, A Study of Approaches to Hypertext Categorization J. Intelligent Information Systems, 2002.

Index Terms:
Feature selection, divergence, independent component analysis, naive Bayes.
Citation:
Marco Bressan, Jordi Vitri?, "On the Selection and Classification of Independent Features," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1312-1317, Oct. 2003, doi:10.1109/TPAMI.2003.1233904
Usage of this product signifies your acceptance of the Terms of Use.