This Article 
 Bibliographic References 
 Add to: 
Assessing Classifiers from Two Independent Data Sets Using ROC Analysis: A Nonparametric Approach
November 2006 (vol. 28 no. 11)
pp. 1809-1817
W.A. Yousef, Center for Devices & Radiol. Health, Food & Drug Adm., Rockville, MD
R.F. Wagner, Center for Devices & Radiol. Health, Food & Drug Adm., Rockville, MD
This paper considers binary classification. We assess a classifier in terms of the area under the ROC curve (AUC). We estimate three important parameters, the conditional AUC (conditional on a particular training set) and the mean and variance of this AUC. We derive, as well, a closed form expression of the variance of the estimator of the AUG. This expression exhibits several components of variance that facilitate an understanding for the sources of uncertainty of that estimate. In addition, we estimate this variance, i.e., the variance of the conditional AUC estimator. Our approach is nonparametric and based on general methods from U-statistics; it addresses the case where the data distribution is neither known nor modeled and where there are only two available data sets, the training and testing sets. Finally, we illustrate some simulation results for these estimators

[1] W.A. Yousef, R.F. Wagner, and M.H. Loew, “Comparison of Non-Parametric Methods for Assessing Classifier Performance in Terms of ROC Parameters,” Proc. 33rd IEEE CS Applied Imagery Pattern Recognition Workshop, pp. 190-195, 2004.
[2] W.A. Yousef, R.F. Wagner, and M.H. Loew, “Estimating the Uncertainty in the Estimated Mean Area under the ROC Curve of a Classifier,” Pattern Recognition Letters, vol. 26, no. 16, pp. 2600-2610, 2005.
[3] T. Hastie, R. Tibshirani, and J.H. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2001.
[4] S. Raudys, Statistical and Neural Classifiers: An Integrated Approach to Design. Springer, 2001.
[5] D. Bamber, “The Area above the Ordinal Dominance Graph and the Area below the Receiver Operating Characteristic Curve,” J.Math. Psychology, vol. 12, pp. 387-415, 1975.
[6] H.H. Barrett and K.J. Myers, Foundations of Image Science. Wiley, 2003.
[7] W.A. Yousef, R.F. Wagner, and M.H. Loew, “The Partial Area under the ROC Curve: Its Properties and Nonparametric Estimation for Assessing Classifier Performance,” Pattern Recognition, submitted.
[8] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Academic Press, 1990.
[9] M. Stone, “Cross-Validatory Choice and Assessment of Statistical Predictions,” J. Royal Statistical Soc., Series B (Methodological), vol. 36, no. 2, pp. 111-147, 1974.
[10] B. Efron, “Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation,” J. Am. Statistical Assoc., vol. 78, no. 382, pp. 316-331, 1983.
[11] B. Efron and R. Tibshirani, “Improvements on Cross-Validation: The $.632+$ Bootstrap Method,” J. Am. Statistical Assoc., vol. 92, no. 438, pp. 548-560, 1997.
[12] R.H. Randles and D.A. Wolfe, Introduction to the Theory of Nonparametric Statistics. Wiley, 1979.
[13] E.L. Lehmann and J.P. Romano, Testing Statistical Hypotheses, third ed. Springer, 2005.
[14] E.L. Lehmann and G. Casella, Theory of Point Estimation, second ed. Springer, 1998.
[15] G. Casella and R.L. Berger, Statistical Inference, second ed. Pacific Grove, Calif.: Duxbury/Thomson Learning, 2002.
[16] A.J. Lee, U-Statistics: Theory and Practice. New York: M. Dekker, 1990.
[17] C.A. Roe and C.E. Metz, “Dorfman-Berbaum-Metz Method for Statistical Analysis of Multireader, Multimodality Receiver Operating Characteristic Data: Validation with Computer Simulation,” Academic Radiology, vol. 4, no. 4, pp. 298-303, 1997.
[18] C.A. Roe and C.E. Metz, “Variance-Component Modeling in the Analysis of Receiver Operating Characteristic Index Estimates,” Academic Radiology, vol. 4, no. 8, pp. 587-600, 1997.
[19] S.V. Beiden, M.A. Maloof, and R.F. Wagner, “A General Model for Finite-Sample Effects in Training and Testing of Competing Classifiers,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 12, p. 1569, Dec. 2003.
[20] H.H. Barrett, M.A. Kupinski, and E. Clarkson, “Probabilistic Foundations of the MRMC Method,” Proc. Medical Imaging Conf. 2005: Image Perception, Observer Performance, and Technology Assessment, p. 21, 2005.
[21] B.D. Gallas, “One-Shot Estimate of MRMC Variance: AUC,” Academic Radiology, vol. 13, no. 3, p. 353, 2006.
[22] E.R. DeLong, D.M. DeLong, and D.L. Clarke-Pearson, “Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach,” Biometrics, vol. 44, no. 3, pp. 837-845, 1988.
[23] G.E. Noether, Elements of Nonparametric Statistics. Wiley, 1967.
[24] G. Campbell, M.A. Douglas, and J.J. Bailey, “Nonparametric Comparison of Two Tests of Cardiac Function on the Same Patient Population Using the Entire ROC Curve,” Proc. IEEE Conf. Computers in Cardiology, p. 267, 1988.
[25] W.A. Yousef, “Assessment of Statistical Classification Rules: Implications for Computational Intelligence,” DSc dissertation, The George Washington Univ., 2006.

Index Terms:
Training data,Parameter estimation,Uncertainty,Testing,Statistical analysis,Statistical distributions,Decision theory,Probability density function,Random variables,Medical diagnosis,ROC analysis.,Classification,nonparametric statistics
W.A. Yousef, R.F. Wagner, M.H. Loew, "Assessing Classifiers from Two Independent Data Sets Using ROC Analysis: A Nonparametric Approach," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 11, pp. 1809-1817, Nov. 2006, doi:10.1109/TPAMI.2006.218
Usage of this product signifies your acceptance of the Terms of Use.