The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - October (2009 vol.21)
pp: 1475-1488
Bo Chen , Xidian University , Xi'an
Hongwei Liu , Xidian University, Xian
Jing Chai , Xidian University, Xi'an
Zheng Bao , Xidian University, Xi'an
ABSTRACT
The problem of feature selection is a difficult combinatorial task in machine learning and of high practical relevance. In this paper, we consider feature selection method for multimodally distributed data, and present a large margin feature weighting method for k-nearest neighbor (kNN) classifiers. The method learns the feature weighting factors by minimizing a cost function, which aims at separating different classes by large local margins and pulling closer together points from the same class, based on using as few features as possible. The consequent optimization problem can be efficiently solved by Linear Programming. Finally, the proposed approach is assessed through a series of experiments with UCI and microarray data sets, as well as a more specific and challenging task, namely, radar high-resolution range profiles (HRRP) automatic target recognition (ATR). The experimental results demonstrate the effectiveness of the proposed algorithms.
INDEX TERMS
Feature selection, feature weighting, large margin, linear programming.
CITATION
Bo Chen, Hongwei Liu, Jing Chai, Zheng Bao, "Large Margin Feature Weighting Method via Linear Programming", IEEE Transactions on Knowledge & Data Engineering, vol.21, no. 10, pp. 1475-1488, October 2009, doi:10.1109/TKDE.2008.238
REFERENCES
[1] A.K. Jain , R.P.W. Duin , and J. Mao , “Statistical Pattern Recognition: A Review,” IEEE Trans. Pattern Analysis Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[2] E. Fix and J. Hodges , “Discriminatory Analysis Nonparametric Discrimination: Consistency Properties,” Technical Report 4, USAF School of Aviation Medicine, 1951.
[3] Y. Yang and J.O. Pedersen , “A Comparative Study on Feature Selection,” Proc. ACM Int'l Conf. Research and Development in Information Retrieval, pp. 42-49, 1999.
[4] J. Mao and A.K. Jain , “Artificial Neural Networks for Feature Selection and Multivariate Data Projection,” IEEE Trans. Neural Networks, vol. 6, no. 2, pp. 296-317, Mar. 1995.
[5] R. Kohavi and G.H. John , “Wrappers for Feature Subset Selection,” Artificial Intelligence, vol. 97, pp. 273-324, 1997.
[6] I. Guyon and A. Elisseeff , “An Introduction to Variable and Feature Selection,” J. Machine Learning Research, vol. 3, pp. 1157-1182, 2003.
[7] I. Kononenko , “Estimating Attributes: Analysis and Extensions of RELIEF,” Proc. European Conf. Machine Learning, pp. 171-182, 1994.
[8] R. Gilad-Bachrach , A. Navot , and N. Tishby , “Margin Based Feature Selection—Theory and Algorithms,” Proc. 21st Int'l Conf. Machine Learning (ICML '04), pp. 43-50, 2004.
[9] Y. Sun , “Iterative RELIEF for Feeature Weighting: Algorithms, Theories, and Applications,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. 1-17, June 2007.
[10] K. Kira and L.A. Rendell , “A Practical Approach to Feature Selection,” Proc. Ninth Int'l Conf. Machine Learning (ICML '92), pp.249-256, 1992.
[11] C.J. Veenman and D.M.J. Tax , “LESS: A Model-Based Classifier for Sparse Subspaces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 9, pp. 1496-1500, Sept. 2005.
[12] K.Q. Weinberger , J. Blitzer , and L.K. Saul , “Distance Metric Learning for Large Margin Nearest Neighbor Classification,” Proc. Neural Information Processing Systems (NIPS), pp. 1473-1480, 2006.
[13] L. Vandenberghe and S.P. Boyd , “Semidefinite Programming,” SIAM Rev., vol. 38, no. 1, pp. 49-95, Mar. 1996.
[14] C. Cortes and V. Vapnik , “Support-Vector Networks,” Machine Learning, vol. 20, pp. 273-297, 1995.
[15] C.J.C. Burges , “A Tutorial on Support Vector Machines for Pattern Recognition,” Data Mining and Knowledge Discovery, vol. 2, pp. 121-167, 1998.
[16] L. Torresani and K.-C. Lee , “Large Margin Component Analysis,” Proc. Neural Information Processing Systems (NIPS), pp. 1385-1392, 2007.
[17] C. Blake , E. Keogh , and C.J. Merz , UCI Repository of Machine Learning Databases, Dept. of Information and Computer Science, Univ. of California, http://www.ics.uci.edu~mlearn, 2009.
[18] J. Weston , S. Mukherjee , O. Chapelle , M. Pontil , T. Poggio , and V. Vapnik , “Feature Selection for SVMs,” Neural Information Processing Systems, MIT Press, 2001.
[19] E.-J. Yeoh , M.E. Ross , S.A. Shurtleff , W.K. Williams , D. Patel , R. Mahrouz , F.G. Behm , S.C. Raimondi , M.V. Relling , A. Patel , C. Cheng , D. Campana , D. Wilkins , X. Zhou , J. Li , H. Liu , C.-H. Pui , W.E. Evans , C. Naeve , L. Wong , and J.R. Downing , “Classification, Subtype Discovery, and Prediction of Outcome in Pediatric Lymphoblastic Leukemia by Gene Expression Profiling,” Cancer Cell, vol. 1, pp. 133-143, 2002.
[20] T.R. Golub , D.K. Slonim , P. Tamayo , C. Huard , M. Gassenbeek , J.P. Mesirov , H. Coller , M.L. Loh , J.R. Downing , M.A. Caligiuri , C.D. Bloomfield , and E.S. Lander , “Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring,” Science, vol. 286, pp. 531-537, Oct. 1999.
[21] A.A. Alizadeh , “Distinct Types of Diffuse Large B-Cell Lymphoma Identified by Gene Expression Profiling,” Nature, vol. 403, pp.503-511, 2000.
[22] S.L. Pomeroy , P. Tamayo , M. Gaasenbeek , L.M. Sturla , M. Angelo , M.E. McLaughlin , J.Y.H. Kim , L.C. Goumnerova , P.M. Black , C. Lau , J.C. Allen , D. Zagzag , J.M. Olson , T. Curran , C. Wetmore , J.A. Biegel , T. Poggio , S. Mukherjee , R. Rifkin , A. Califano , G. Stolovitzky , D.N. Louis , J.P. Mesirov , E.S. Lander , and T.R. Golub , “Prediction of Central Nervous System Embryonal Tumor Outcome Based on Gene Expression,” Nature, vol. 415, pp. 436-442, 2002.
[23] U. Alon , N. Barkai , D. Notterman , K. Gish , S. Ybarra , D. Mack , and A. Levine , “Broad Patterns of Gene Expression Revealed by Clustering Analysis of Tumor and Normal Colon Cancer Tissues Probed by Oligonucleotide Arrays,” Cell Biology, vol. 96, pp. 6745-6750, 1999.
[24] L.J. van't Veer , H. Dai , M.J. van de Vijver , Y.D. He , A.A.M. Hart , M. Mao , H.L. Peterse , K. Van Der Kooy , M.J. Marton , A.T. Witteveen , G.J. Schreiber , R.M. Kerkhoven , C. Roberts , P.S. Linsley , R. Bernards , and S.H. Friend , “Gene Expression Profiling Predicts Clinical Outcome of Breast Cancer,” Nature, vol. 415, pp.530-536, Jan. 2002.
[25] D. Singh , P.G. Febbo , K. Ross , D.G. Jackson , J. Manola , C. Ladd , P. Tamayo , A.A. Renshaw , A.V. D'Amico , J.P. Richie , E.S. Lander , M. Loda , P.W. Kantoff , T.R. Golub , and W.R. Sellers , “Gene Expression Correlations of Clinical Prostate Cancer Behavior,” Cancer Cell, vol. 1, pp. 203-209, 2004.
[26] S.A. Armstrong , J.E. Staunton , L.B. Silverman , R. Pieters , M.L. den Boer , M.D. Minden , S.E. Sallan , E.S. Lander , T.R. Golub , and S.J. Korsmeyer , “MLL Translocations Specify a Distinct Gene Expression Profile That Distigushes a Unique Leukemia,” Nature Genetics, vol. 30, pp. 41-47, 2001.
[27] J. Weston , A. Elisseeff , B. Scholkopf , and M. Tipping , “Use of the Zero-Norm with Linear Models and Kernel Methods,” J. Machine Learning Research, vol. 3, pp. 1439-1461, 2003.
[28] G.J. Gordon , R.V. Jenson , L.-L. Hsiao , S.R. Gullans , J.E. Blumenstock , S. Ramaswamy , W.G. Richards , D.J. Sugarbaker , and R. Bueno , “Translation of Microarray Data into Clinically Relevant Cancer Diagnostic Tests Using Gene Expression Ratios in Lung Cancer and Mesothelima,” Cancer Research, vol. 62, pp. 4936-4967, 2002.
[29] S. Dudoit , J. Fridlyand , and T.P. Speed , “Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data,” J. Am. Statistical Assoc., vol. 97, pp. 77-87, 2002.
[30] D. Wettschereck , D.W. Aha , and T. Mohri , “A Review and Empirical Evaluation of Feature Weighting Methods for a Class of Lazy Learning Algorithms,” Artificial Intelligence Rev., vol. 11, nos.1-5, pp. 273-314, 2005.
[31] L. Du , H. Liu , Z. Bao , and M. Xing , “Radar HRRP Target Recognition Based on Higher Order Spectra,” IEEE Trans. Signal Processing, vol. 53, no. 7, pp. 2359-2368, July 2005.
[32] L. Du , H. Liu , Z. Bao , and J. Zhang , “A Two-Distribution Compounded Statistical Model for Radar HRRP Target Recognition,” IEEE Trans. Signal Processing, vol. 54, no. 6, pp. 2226-2238, June 2006.
[33] B. Pei and Z. Bao , “Bispectrum Based Approach to High Radar Range Profile for Automatic Target Recognition,” Pattern Recognition, vol. 35, no. 11, pp. 2643-2651, 2002.
[34] B. Chen , H. Liu , and Z. Bao , “A Kernel Optimization Method Based on the Localized Kernel Fisher,” Pattern Recognition, vol. 41, no. 3, pp. 1098-1109, 2008.
[35] B. Chen , H. Liu , and Z. Bao , “Optimizing the Data-Dependent Kernel under a Unified Kernel Optimization Framework,” Pattern Recognition, vol. 41, no. 6, pp. 2107-2119, 2008.
[36] Z. Bao , M. Xing , and T. Wang , Radar Imaging Technique. Publishing House of Electronics Industry, 2005.
[37] S. Mallat , A Wavelet Tour of Signal Processing, second ed. Academic Press, 1998.
[38] I. Daubechies , Ten Lectures on Wavelets. SIAM, 1992.
[39] J. Dai , S. Yan , X. Tang , and J.T. Kwok , “Locally Adaptive Classification Piloted by Uncertainty,” Proc. Int'l Conf. Machine Learning, 2006.
46 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool