Subscribe
Issue No.05 - May (2008 vol.30)
pp: 810-822
ABSTRACT
ROC analysis has become a standard tool in the design and evaluation of 2-class classification problems. It allows for an analysis that incorporates all possible priors, costs, and operating points, which is important in many real problems, where conditions are often nonideal. Extending this to the multiclass case is attractive, conferring the benefits of ROC analysis to a multitude of new problems. Even though ROC analysis does extend theoretically to the multiclass case, the exponential computational complexity as a function of the number of classes is restrictive. In this paper we show that the multiclass ROC can often be simplified considerably because some ROC dimensions are independent of each other. We present an algorithm that analyses interactions between various ROC dimensions, identifying independent classes, and groups of interacting classes, allowing the ROC to be decomposed. The resultant decomposed ROC hypersurface can be interrogated in a similar fashion to the ideal case, allowing for approaches such as cost-sensitive and Neyman-Pearson optimisation, as well as the volume under the ROC. An extensive bouquet of examples and experiments demonstrates the potential of this methodology.
INDEX TERMS
T.C.W. Landgrebe and R.P.W. Duin are in the Information and Communication Theory Group, Delft University of Technology Mekelweg 4, 2628 CD Delft, The Netherlands
CITATION
Thomas C.W. Landgrebe, Robert P.W. Duin, "Efficient Multiclass ROC Approximation by Decomposition via Confusion Matrix Perturbation Analysis", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.30, no. 5, pp. 810-822, May 2008, doi:10.1109/TPAMI.2007.70740
REFERENCES
 [1] C. Metz, “Basic Principles of ROC Analysis,” Seminars in Nuclear Medicine, vol. 3, no. 4, 1978. [2] T. Fawcett, “An Introduction to ROC analysis,” Pattern Recognition Letters, special issue on ROC analysis, vol. 27, pp. 861-874, 2005. [3] F. Provost and T. Fawcett, “Robust Classification for Imprecise Environments,” Machine Learning, vol. 42, pp. 203-231, 2001. [4] R. Duda, P. Hart, and D. Stork, Pattern Classification, second ed. Wiley-Interscience, 2001. [5] A. Bradley, “The Use of the Area under the ROC Curve in the Evaluation of Machine Learning Algorithms,” Pattern Recognition, vol. 30, no. 7, pp. 1145-1159, 1997. [6] T. Landgrebe and R. Duin, “Combining Accuracy and Prior Sensitivity for Classifier Design under Prior Uncertainty,” Proc. IAPR Int'l Workshops Structural and Syntactic Pattern Recognition, pp. 512-521, Aug. 2006. [7] D. Mossman, “Three-Way ROC's,” Medical Decision Making, vol. 19, pp. 78-89, 1999. [8] S. Dreisetl, S. Ohno-Machado, and M. Binder, “Comparing Trichotomous Tests by Three-Way ROC Analysis,” Medical Decision Making, vol. 20, no. 3, pp. 323-331, 2000. [9] T. Landgrebe and R. Duin, “Approximating the Multiclass ROC by Pairwise Analysis,” Pattern Recognition Letters, 2007. [10] A. Srinivasan, “Note on the Location of Optimal Classifiers in $N$ -Dimensional ROC Space,” Technical Report PRG-TR-2-99, Computing Laboratory, Oxford Univ., Oct. 1999. [11] F. Provost and T. Fawcett, “Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions,” Proc. Third Int'l Conf. Knowledge Discovery and Data Mining, pp. 43-48, 2001. [12] C. Ferri, J. Hernandez-Orallo, and M. Salido, “Volume Under the ROC Surface for Multi-Class Problems,” Proc. 14th European Conf. Machine Learning, pp. 108-120, 2003. [13] D. Edwards, C. Metz, and R. Nishikawa, “The Hypervolume under the ROC Hypersurface of ‘Near-Guessing’ and ‘Near-Perfect’ Observers in $N$ -Class Classification Tasks,” IEEE Trans. Medical Imaging, vol. 24, no. 3, pp. 293-299, Mar. 2005. [14] T. Landgrebe and R. Duin, “A Simplified Extension of the Area under the ROC to the Multiclass Domain,” Proc. 17th Ann. Symp. Pattern Recognition Assoc. South Africa, Nov. 2006. [15] F. Provost and P. Domingos, Well Trained PETs: Improving Probability Estimation Trees, CeDER Working Paper IS-00-04, Stern School of Business, New York Univ., 2001. [16] D. Hand and R. Till, “A Simple Generalisation of the Area under the ROC Curve for Multiple Class Classification Problems,” Machine Learning, vol. 45, pp. 171-186, 2001. [17] N. Lachiche and P. Flach, “Improving Accuracy and Cost of Two-Class and Multi-Class Probabilistic Classifiers Using ROC Curves,” Proc. 20th Int'l Conf. Machine Learning, pp. 416-423, 2003. [18] D. O'Brien and R. Gray, “Improving Classification Performance by Exploring the Role of Cost Matrices in Partitioning the Estimated Class Probability Space,” Proc. 22nd Int'l Conf. Machine Learning Workshop ROC Analysis in Machine Learning, 2005. [19] R. Everson and J. Fieldsend, “Multi-Class ROC Analysis from a Multi-Objective Optimisation Perspective,” Pattern Recognition Letters, special issue on ROC analysis, vol. 27, 2005. [20] P. Domingos, “MetaCost: A General Method for Making Classifiers Cost-Sensitive,” Proc. Fifth ACM Int'l Conf. Knowledge Discovery and Data Mining, pp. 155-164, 1999. [21] N. Abe, B. Zadrozny, and J. Langford, “An Iterative Method for Multi-Class Cost-Sensitive Learning,” Proc. 10th ACM Int'l Conf. Knowledge Discovery and Data Mining, pp. 3-11, Aug. 2004. [22] D. Edwards, C. Metz, and M. Kupinski, “Ideal Observers and Optimal ROC Hypersurfaces in $N$ -Class Classification,” IEEE Trans. Medical Imaging, vol. 23, no. 7, pp. 891-895, July 2004. [23] T. Landgrebe and R. Duin, “On Neyman-Pearson Optimisation for Multiclass Classifiers,” Proc. 16th Ann. Symp. Pattern Recognition Assoc. South Africa, Nov. 2005. [24] M. Li and I. Sethi, “Confidence-Based Classifier Design,” Pattern Recognition, vol. 39, pp. 1230-1240, 2006. [25] P. Murphy and D. Aha, “CI Repository of Machine Learning Databases,” Dept. of Information and Computer Science, Univ. of California, ftp://ftp.ics.uci.edu/pubmachine-learning-data bases , 1992. [26] M. Loog, R. Duin, and H.-U.R., “Multiclass Linear Dimension Reduction by Weighted Pairwise Fisher Criteria,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 7, pp. 762-766, July 2001. [27] “Delve Datasets, Image Segmentation,” Univ. of Toronto, http:/www.cs.toronto.edu, 2008. [28] ELENA Project, “European ESPRIT 5516 Project,” Satimage Dataset, ftp://ftp.dice.ucl.ac.be/pub/neural-nets/ ELENAdata bases, 2004. [29] C. Wilson and M.M.D. Garris, “Handprinted Character Database3,” Advanced Systems Division, Nat'l Inst. of Standards and Tech nology, 1992. [30] R. McDonald, “The Mean Subjective Utility Score, a Novel Metric for Cost-Sensitive Classifier Evaluation,” Pattern Recognition Letters, vol. 27, no. 13, pp. 1472-1477, Oct. 2006.