Issue No. 10 - October (2010 vol. 32)
JooSeuk Kim , University of Michigan, Ann Arbor
Clayton D. Scott , University of Michigan, Ann Arbor
Nonparametric kernel methods are widely used and proven to be successful in many statistical learning problems. Well--known examples include the kernel density estimate (KDE) for density estimation and the support vector machine (SVM) for classification. We propose a kernel classifier that optimizes the L_2 or integrated squared error (ISE) of a “difference of densities.” We focus on the Gaussian kernel, although the method applies to other kernels suitable for density estimation. Like a support vector machine (SVM), the classifier is sparse and results from solving a quadratic program. We provide statistical performance guarantees for the proposed L_2 kernel classifier in the form of a finite sample oracle inequality and strong consistency in the sense of both ISE and probability of error. A special case of our analysis applies to a previously introduced ISE-based method for kernel density estimation. For dimensionality greater than 15, the basic L_2 kernel classifier performs poorly in practice. Thus, we extend the method through the introduction of a natural regularization parameter, which allows it to remain competitive with the SVM in high dimensions. Simulation results for both synthetic and real-world data are presented.
Kernel methods, sparse classifiers, integrated squared error, difference of densities, SMO algorithm.
J. Kim and C. D. Scott, "L₂ Kernel Classification," in IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 32, no. , pp. 1822-1831, 2009.