CSDL Home IEEE Transactions on Pattern Analysis & Machine Intelligence 2011 vol.33 Issue No.10 - October

Subscribe

Issue No.10 - October (2011 vol.33)

pp: 2039-2050

Peter Wittek , National University of Singapore, Singapore

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TPAMI.2011.28

ABSTRACT

Wavelet kernels have been introduced for both support vector regression and classification. Most of these wavelet kernels do not use the inner product of the embedding space, but use wavelets in a similar fashion to radial basis function kernels. Wavelet analysis is typically carried out on data with a temporal or spatial relation between consecutive data points. We argue that it is possible to order the features of a general data set so that consecutive features are statistically related to each other, thus enabling us to interpret the vector representation of an object as a series of equally or randomly spaced observations of a hypothetical continuous signal. By approximating the signal with compactly supported basis functions and employing the inner product of the embedding L_2 space, we gain a new family of wavelet kernels. Empirical results show a clear advantage in favor of these kernels.

INDEX TERMS

Wavelet kernels, feature engineering, feature correlation, semantic kernels.

CITATION

Peter Wittek, "Compactly Supported Basis Functions as Support Vector Kernels for Classification",

*IEEE Transactions on Pattern Analysis & Machine Intelligence*, vol.33, no. 10, pp. 2039-2050, October 2011, doi:10.1109/TPAMI.2011.28REFERENCES

- [1] H. Liu and H. Motoda,
Computational Methods of Feature Selection. Chapman & Hall/CRC, 2008.- [2] V. Vapnik,
The Nature of Statistical Learning Theory. Springer, 1995.- [3] L. Zhang, W. Zhou, and L. Jiao, "Wavelet Support Vector Machine,"
IEEE Trans. Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 34, no. 1, pp. 34-39, Feb. 2004.- [4] E. Fonseca, R. Guido, A. Silvestre, and J. Pereira, "Discrete Wavelet Transform and Support Vector Machine Applied to Pathological Voice Signals Identification,"
Proc. Seventh IEEE Int'l Symp. Multimedia, pp. 785-789, Dec. 2005.- [5] T. Alexandrov, J. Decker, B. Mertens, A. Deelder, R. Tollenaar, P. Maass, and H. Thiele, "Biomarker Discovery in MALDI-TOF Serum Protein Profiles Using Discrete Wavelet Transformation,"
Bioinformatics, vol. 25, no. 5, pp. 643-649, 2009.- [6] E. Hoenkamp, "Unitary Operators on the Document Space,"
J. Am. Soc. Information Science and Technology, vol. 54, no. 4, pp. 314-320, 2003.- [7] A. Smola, B. Schölkopf, and K. Müller, "The Connection between Regularization Operators and Support Vector Kernels,"
Neural Networks, vol. 11, no. 4, pp. 637-649, 1998.- [8] T. Li, Q. Li, S. Zhu, and M. Ogihara, "A Survey on Wavelet Applications in Data Mining,"
ACM SIGKDD Explorations Newsletter, vol. 4, no. 2, pp. 49-68, 2002.- [9] H. Szu, B. Telfer, and S. Kadambe, "Neural Network Adaptive Wavelets for Signal Representation and Classification,"
Optical Eng., vol. 31, pp. 1907-1916, 1992.- [10] G. Sheikholeslami, S. Chatterjee, and A. Zhang, "Wavecluster: A Multi-Resolution Clustering Approach for Very Large Spatial Databases,"
Proc. 24th Int'l Conf. Very Large Data Bases, pp. 428-439, Aug. 1998.- [11] L. Zhang, W. Zhou, and L. Jiao, "Support Vector Machines Based on the Orthogonal Projection Kernel of Father Wavelet,"
Int'l J. Computational Intelligence and Applications, vol. 5, no. 3, pp. 283-303, 2005.- [12] F. Schleif, M. Lindemann, M. Diaz, P. Maaß, J. Decker, T. Elssner, M. Kuhn, and H. Thiele, "Support Vector Classification of Proteomic Profile Spectra Based on Feature Extraction with the Bi-Orthogonal Discrete Wavelet Transform,"
Computing and Visualization in Science, vol. 12, no. 4, pp. 1-11, 2009.- [13] E. Fonseca, R. Guido, P. Scalassara, C. Maciel, and J. Pereira, "Wavelet Time-Frequency Analysis and Least Squares Support Vector Machines for the Identification of Voice Disorders,"
Computers in Biology and Medicine, vol. 37, no. 4, pp. 571-578, 2007.- [14] S. Tuntisak and S. Premrudeepreechacharn, "Harmonic Detection in Distribution Systems Using Wavelet Transform and Support Vector Machine,"
Proc. Conf. IEEE Power Eng. Soc., pp. 1540-1545, July 2007.- [15] P. Hosseini, F. Almasganj, T. Emami, R. Behroozmand, S. Gharibzade, and F. Torabinezhad, "Local Discriminant Wavelet Packet Basis for Voice Pathology Classification,"
Proc. Second Int'l Conf. Bioinformatics and Biomedical Eng., pp. 2052-2055, May 2008.- [16] M. Ankerst, M. Breunig, H. Kriegel, and J. Sander, "OPTICS: Ordering Points to Identify the Clustering Structure,"
Proc. ACM SIGMOD Int'l Conf. Management of Data, pp. 49-60, 1999.- [17] M. Ester, H. Kriegel, J. Sander, and X. Xu, "A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise,"
Proc. Second Int'l Conf. Knowledge Discovery and Data Mining, vol. 96, pp. 226-231, Aug. 1996.- [18] P. Wittek, C. Tan, and S. Darányi, "An Ordering of Terms Based on Semantic Relatedness,"
Proc. Eighth Int'l Conf. Computational Semantics, H. Bunt, ed., Jan. 2009.- [19] T. Cormen, C. Leiserson, and R. Rivest,
Introduction to Algorithms. MIT Press, 2001.- [20] N. Beckmann, H. Kriegel, R. Schneider, and B. Seeger, "The R∗-Tree: An Efficient and Robust Access Method for Points and Rectangles,"
ACM SIGMOD Record, vol. 19, no. 2, pp. 322-331, 1990.- [21] S. Berchtold, D. Keim, and H. Kriegel, "The X-Tree: An Index Structure for High-Dimensional Data,"
Readings in Multimedia Computing and Networking, K. Jeffay and H. Zhang, eds., Morgan Kaufmann, p. 451, 2001.- [22] P. Ciaccia, M. Patella, and P. Zezula, "M-Tree: An Efficient Access Method for Similarity Search in Metric Spaces,"
Proc. 23rd Int'l Conf. Very Large Data Bases, pp. 426-435, Aug. 1997.- [23] H. Weaver,
Theory of Discrete and Continuous Fourier Analysis. John Wiley & Sons, 1988.- [24] M. Unser, "Ten Good Reasons for Using Spline Wavelets,"
Proc. SPIE, Wavelet Applications in Signal and Image Processing V, pp. 422-431, 1997.- [25] M. Unser and A. Aldroubi, "Polynomial Splines and Wavelets: A Signal Processing Perspective,"
Wavelet Analysis and Its Applications, pp. 91-122, Academic Press, 1993.- [26] P. Wittek and C. Tan, "A Kernel-Based Feature Weighting for Text Classification,"
Proc. IEEE Int'l Joint Conf. Neural Networks, pp. 3373-3379, June 2009.- [27] A. Asuncion and D. Newman, "UCI Machine Learning Repository," Dept. of Information and Computer Science, Univ. of California, 2007.
- [28] C.-C. Chang and C.-J. Lin,
LIBSVM: A Library for Support Vector Machines, http://www.csie.ntu.edu.tw/~cjlinlibsvm, 2001.- [29] D. Nazareth, E. Soofi, and H. Zhao, "Visualizing Attribute Interdependencies Using Mutual Information, Hierarchical Clustering, Multidimensional Scaling, and Self-Organizing Maps,"
Proc. 40th Ann. Hawaii Int'l Conf. System Sciences, vol. 40, no. 2, pp. 907-917, Jan. 2007.- [30] A. Kraskov, H. Stoegbauer, R. Andrzejak, and P. Grassberger, "Hierarchical Clustering Using Mutual Information,"
Europhysics Letters, vol. 70, no. 2, pp. 278-284, 2005.- [31] C. Hsu and C. Lin, "A Comparison of Methods for Multiclass Support Vector Machines,"
IEEE Trans. Neural Networks, vol. 13, no. 2, pp. 415-425, Mar. 2002.- [32] S. Deerwester, S. Dumais, G. Furnas, T. Landauer, and R. Harshman, "Indexing by Latent Semantic Analysis,"
J. Am. Soc. Information Science, vol. 41, no. 6, pp. 391-407, 1990.- [33] N. Cristianini, J. Shawe-Taylor, and H. Lodhi, "Latent Semantic Kernels,"
J. Intelligent Information Systems, vol. 18, no. 2, pp. 127-152, 2002.- [34] G. Siolas and F. d'Alché Buc, "Support Vector Machines Based on a Semantic Kernel for Text Categorization,"
Proc. IEEE Int'l Joint Conf. Neural Networks, 2000.- [35] S. Bloehdorn, R. Basili, M. Cammisa, and A. Moschitti, "Semantic Kernels for Text Classification Based on Topological Measures of Feature Similarity,"
Proc. Sixth IEEE Int'l Conf. Data Mining, Dec. 2006.- [36] D. Mavroeidis, G. Tsatsaronis, M. Vazirgiannis, M. Theobald, and G. Weikum, "Word Sense Disambiguation for Exploiting Hierarchical Thesauri in Text Classification,"
Proc. Ninth European Conf. Principles and Practice of Knowledge Discovery in Databases, pp. 181-192, Oct. 2005.- [37] R. Basili, M. Cammisa, and A. Moschitti, "Effective Use of WordNet Semantics via Kernel-Based Learning,"
Proc. Ninth Conf. Computational Natural Language Learning, pp. 1-8, June 2005.- [38] A. Budanitsky and G. Hirst, "Evaluating WordNet-Based Measures of Lexical Semantic Relatedness,"
Computational Linguistics, vol. 32, no. 1, pp. 13-47, 2006.- [39] S. Mohammad and G. Hirst, "Distributional Measures as Proxies for Semantic Relatedness," submitted for publication, 2005.
- [40] J. Lyons,
Semantics. Cambridge Univ. Press, 1977.- [41] Z. Harris, "Distributional Structure,"
Papers in Structural and Transformational Linguistics, Z. Harris, ed., pp. 775-794, Humanities Press, 1970.- [42] J. Karlgren and M. Sahlgren, "From Words to Understanding,"
Proc. Foundations of Real-World Intelligence, Y. Uesaka, P. Kanerva, and H. Asoh, eds., pp. 294-308, 2001.- [43] L. Wittgenstein,
Philosophical Investigations. Blackwell Publishing, 1967.- [44] Y. Wilks, D. Fass, C. Guo, J. McDonald, T. Plate, and B. Slator, "Providing Machine Tractable Dictionary Tools,"
Machine Translation, vol. 5, no. 2, pp. 99-154, 1990.- [45] S.I. Gallant, "A Practical Approach for Representing Context and for Performing Word Sense Disambiguation Using Neural Networks,"
Neural Computation, vol. 3, pp. 293-309, 1991.- [46] G. Grefenstette, "Use of Syntactic Context to Produce Term Association Lists for Text Retrieval,"
Proc. 15th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 89-97, June 1992.- [47] H. Schütze and T. Pedersen, "A Co-Occurrence-Based Thesaurus and Two Applications to Information Retrieval,"
Information Processing and Management, vol. 3, no. 33, pp. 307-318, 1997.- [48] A. Kontostathis and W. Pottenger, "A Framework for Understanding Latent Semantic Indexing (LSI) Performance,"
Information Processing and Management, vol. 42, no. 1, pp. 56-73, 2006.- [49] J. Jiang and D. Conrath, "Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy,"
Proc. Int'l Conf. Research in Computational Linguistics, pp. 19-33, 1997.- [50] R. Rada, H. Mili, E. Bicknell, and M. Blettner, "Development and Application of a Metric on Semantic Nets,"
IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 1, pp. 17-30, Jan./Feb. 1989.- [51] A. Dawson and A. Slevin, "Repository Case History: University of Strathclyde Strathprints," http://www.rsp.ac.uk/repos/ casestudies/ pdfsstrathclyde.pdf, 2008.
- [52] F. Sebastiani, "Machine Learning in Automated Text Categorization,"
ACM Computing Surveys, vol. 34, no. 1, pp. 1-47, 2002.- [53] Y. Yang, "An Evaluation of Statistical Approaches to Text Categorization,"
Information Retrieval, vol. 1, no. 1, pp. 69-90, 1999. |