Publication 2005 Issue No. 5 - May Abstract - epsilon-SSVR: A Smooth Support Vector Machine for epsilon-Insensitive Regression
 This Article Share Bibliographic References Add to: Digg Furl Spurl Blink Simpy Google Del.icio.us Y!MyWeb Search Similar Articles Articles by Yuh-Jye Lee Articles by Wen-Feng Hsieh Articles by Chien-Ming Huang
epsilon-SSVR: A Smooth Support Vector Machine for epsilon-Insensitive Regression
May 2005 (vol. 17 no. 5)
pp. 678-685
 ASCII Text x Yuh-Jye Lee, Wen-Feng Hsieh, Chien-Ming Huang, "epsilon-SSVR: A Smooth Support Vector Machine for epsilon-Insensitive Regression," IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 5, pp. 678-685, May, 2005.
 BibTex x @article{ 10.1109/TKDE.2005.77,author = {Yuh-Jye Lee and Wen-Feng Hsieh and Chien-Ming Huang},title = {epsilon-SSVR: A Smooth Support Vector Machine for epsilon-Insensitive Regression},journal ={IEEE Transactions on Knowledge and Data Engineering},volume = {17},number = {5},issn = {1041-4347},year = {2005},pages = {678-685},doi = {http://doi.ieeecomputersociety.org/10.1109/TKDE.2005.77},publisher = {IEEE Computer Society},address = {Los Alamitos, CA, USA},}
 RefWorks Procite/RefMan/Endnote x TY - JOURJO - IEEE Transactions on Knowledge and Data EngineeringTI - epsilon-SSVR: A Smooth Support Vector Machine for epsilon-Insensitive RegressionIS - 5SN - 1041-4347SP678EP685EPD - 678-685A1 - Yuh-Jye Lee, A1 - Wen-Feng Hsieh, A1 - Chien-Ming Huang, PY - 2005KW - \epsilon{\hbox{-}}{\rm{insensitive}} loss functionKW - \epsilon{\hbox{-}}{\rm{smooth}} support vector regressionKW - kernel methodKW - Newton-Armijo algorithmKW - support vector machine.VL - 17JA - IEEE Transactions on Knowledge and Data EngineeringER -
A new smoothing strategy for solving \epsilon{\hbox{-}}{\rm{support}} vector regression (\epsilon{\hbox{-}}{\rm{SVR}}), tolerating a small error in fitting a given data set linearly or nonlinearly, is proposed in this paper. Conventionally, \epsilon{\hbox{-}}{\rm{SVR}} is formulated as a constrained minimization problem, namely, a convex quadratic programming problem. We apply the smoothing techniques that have been used for solving the support vector machine for classification, to replace the \epsilon{\hbox{-}}{\rm{insensitive}} loss function by an accurate smooth approximation. This will allow us to solve \epsilon{\hbox{-}}{\rm{SVR}} as an unconstrained minimization problem directly. We term this reformulated problem as \epsilon{\hbox{-}}{\rm{smooth}} support vector regression (\epsilon{\hbox{-}}{\rm{SSVR}}). We also prescribe a Newton-Armijo algorithm that has been shown to be convergent globally and quadratically to solve our \epsilon{\hbox{-}}{\rm{SSVR}}. In order to handle the case of nonlinear regression with a massive data set, we also introduce the reduced kernel technique in this paper to avoid the computational difficulties in dealing with a huge and fully dense kernel matrix. Numerical results and comparisons are given to demonstrate the effectiveness and speed of the algorithm.

[1] D.P. Bertsekas, Nonlinear Programming. Belmont, Mass.: Athena Scientific, 1995.
[2] C.J.C. Burges, “A Tutorial on Support Vector Machines for Pattern Recognition,” Data Mining and Knowledge Discovery, vol. 2, no. 2, pp. 121-167, 1998.
[3] C.-C. Chang and C.-J. Lin, LIBSVM: A Library for Support Vector Machines, 2001, software available at http://www.csie.ntu. edu.tw/~cjlinlibsvm .
[4] B. Chen, and P.T. Harker, “Smooth Approximations to Nonlinear Complementarity Problems,” SIAM J. Optimization, vol. 7, pp. 403-420, 1997.
[5] C. Chen and O.L. Mangasarian, “Smoothing Methods for Convex Inequalities and Linear Complementarity Problems,” Math. Programming, vol. 71, no. 1, pp. 51-69, 1995.
[6] C. Chen and O.L. Mangasarian, “A Class of Smoothing Functions for Nonlinear and Mixed Complementarity Problems,” Computational Optimization and Applications, vol. 5, no. 2, pp. 97-138, 1996.
[7] X. Chen, L. Qi, and D. Sun, “Global and Superlinear Convergence of the Smoothing Newton Method and Its Application to General Box Constrained Variational Inequalities,” Math. of Computation, vol. 67, pp. 519-540, 1998.
[8] X. Chen and Y. Ye, “On Homotopy-Smoothing Methods for Variational Inequalities,” SIAM J. Control and Optimization, vol. 37, pp. 589-616, 1999.
[9] P.W. Christensen and J.-S. Pang, “Frictional Contact Algorithms Based on Semismooth Newton Methods,” Reformulation: Nonsmooth, Piecewise Smooth, Semismooth and Smoothing Methods, M. Fukushima and L. Qi, eds., pp. 81-116, Kluwer Academic Publishers, 1999.
[10] N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines. Cambridge: Cambridge Univ. Press, 2000.
[11] DELVE, Data for Evaluating Learning in Valid Experiments, Comp-Activ Dataset, http://www.cs.toronto.edu/~delve/data/comp-activ desc.html, 2005.
[12] DELVE, Data for Evaluating Learning in Valid Experiments, Kin-family Dataset, http://www.cs.toronto.edu/~delve/data/kin desc.html, 2005.
[13] J.E. Dennis and R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Englewood Cliffs, N.J.: Prentice-Hall, 1983.
[14] H. Drucker, C.J.C. Burges, L. Kaufman, A. Smola, and V. Vapnik, “Support Vector Regression Machines,” Advances in Neural Information Processing Systems -9-, M.C. Mozer, M.I. Jordan, and T. Petsche, eds., pp. 155-161, Cambridge, Mass.: MIT Press, 1997.
[15] M. Fukushima and L. Qi, Reformulation: Nonsmooth, Piecewise Smooth, Semismooth, and Smoothing Methods. Dordrecht, The Netherlands: Kluwer Academic Publishers, 1999.
[16] G. Fung and O.L. Mangasarian, “Proximal Support Vector Machine Classifiers,” Proc. KDD-2001: Knowledge Discovery and Data Mining, F. Provost and R. Srikant, eds., pp. 77-86, 2001, ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports 01-02.ps.
[17] S.-Y. Huang and Y.-J. Lee, “Reduced Support Vector Machines: A Statistical Theory,” Preprint, Inst. of Statistical Science, Academia Sinica, 2004, http://www.stat.sinica.edu.twsyhuang/.
[18] T. Joachims, ${\rm{SVM}}^{light}$ , 2002, http:/svmlight.joachims.org.
[19] Y.-J. Lee and O.L. Mangasarian, “RSVM: Reduced Support Vector Machines,” Technical Report 00-07, Data Mining Inst., Computer Sciences Dept., Univ. of Wisconsin, Madison, Wisconsin, July 2000, also Proc. First SIAM Int'l Conf. Data Mining, 2001, ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports 00-07.ps.
[20] Y.-J. Lee and O.L. Mangasarian, “SSVM: A Smooth Support Vector Machine,” Computational Optimization and Applications, vol. 20, pp. 5-22, 2001, also Data Mining Inst., Univ. of Wisconsin, Technical Report 99-03, ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports 99-03.ps.
[21] K.-M. Lin and C.-J. Lin, “A Study on Reduced Support Vector Machines,” IEEE Trans. Neural Networks, vol. 14, no. 6, pp. 1449-1459, 2003.
[22] O.L. Mangasarian, “Generalized Support Vector Machines,” Advances in Large Margin Classifiers, A. Smola, P. Bartlett, B. Schölkopf, and D. Schuurmans, eds., pp. 135-146, Cambridge, Mass: MIT Press, 2000, ftp://ftp.cs.wisc.edu/math-prog/tech-reports 98-14.ps.
[23] O.L. Mangasarian and D.R. Musicant, “Robust Linear and Support Vector Regression,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 9, pp. 950-955, 2000, .
[24] O.L. Mangasarian and D.R. Musicant, “Large Scale Kernel Regression via Linear Programming,” Machine Learning, vol. 46, pp. 255-269, 2002, ftp://ftp.cs.wisc. edu/pub/dmi/tech-reports/ 99-09.psftp://ftp.cs.wisc.edu/pub/dmi/ tech-reports99-02.ps.
[25] MATLAB, User's Guide. Natick, Mass.: The MathWorks, Inc., 1994-2001, http:/www.mathworks.com.
[26] C.L. Blake and C.J. Merz UCI Repository of Machine Learning Databases, 1998, http://www.ics.uci.edu/~mlearnMLRepository.htm .
[27] D.R. Musicant, and A. Feinberg, “Active Set Support Vector Regression,” IEEE Trans. Neural Networks, vol. 15, no. 2, pp. 268-275, 2004.
[28] M. Stone, “Cross-Validatory Choice and Assessment of Statistical Predictions,” J. Royal Statistical Soc., vol. 36, pp. 111-147, 1974.
[29] P. Tseng, “Analysis of a Non-Interior Continuation Method Based on Chen-Mangasarian Smoothing Functions for Complementarity Problems,” Reformulation: Nonsmooth, Piecewise Smooth, Semismooth, and Smoothing Methods, M. Fukushima and L. Qi, eds., pp. 381-404, Dordrecht, Netherlands: Kluwer Academic Publishers, 1999.
[30] V.N. Vapnik, The Nature of Statistical Learning Theory. New York: Springer, 1995.
[31] R.C. Whaley, A. Petitet, and J.J. Dongarra, “Automated Empirical Optimization of Software and the ATLAS Project,” Parallel Computing, vol. 27, nos. 1-2, pp. 3-35, 2001, also available as Univ. of Tennessee LAPACK Working Note #147, UT-CS-00-448, www.netlib.org/lapack/lawnslawn147.ps, 2000.
[32] I.H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. San Francisco: Morgan Kaufmann, 1999.

Index Terms:
\epsilon{\hbox{-}}{\rm{insensitive}} loss function, \epsilon{\hbox{-}}{\rm{smooth}} support vector regression, kernel method, Newton-Armijo algorithm, support vector machine.
Citation:
Yuh-Jye Lee, Wen-Feng Hsieh, Chien-Ming Huang, "epsilon-SSVR: A Smooth Support Vector Machine for epsilon-Insensitive Regression," IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 5, pp. 678-685, May 2005, doi:10.1109/TKDE.2005.77