Issue No. 11 - November (2006 vol. 28)
W.A. Yousef , Food and Drug Administration (FDA), Center for Devices and Radiological Health (CDRH), Rockville, MD
R.F. Wagner , Food and Drug Administration (FDA), Center for Devices and Radiological Health (CDRH), Rockville, MD
M.H. Loew , George Washington University, Phillips Hall, Washington DC
This paper considers binary classification. We assess a classifier in terms of the area under the ROC curve (AUC). We estimate three important parameters, the conditional AUC (conditional on a particular training set) and the mean and variance of this AUC. We derive, as well, a closed form expression of the variance of the estimator of the AUG. This expression exhibits several components of variance that facilitate an understanding for the sources of uncertainty of that estimate. In addition, we estimate this variance, i.e., the variance of the conditional AUC estimator. Our approach is nonparametric and based on general methods from U-statistics; it addresses the case where the data distribution is neither known nor modeled and where there are only two available data sets, the training and testing sets. Finally, we illustrate some simulation results for these estimators
Training data, Parameter estimation, Uncertainty, Testing, Statistical analysis, Statistical distributions, Decision theory, Probability density function, Random variables, Medical diagnosis
W. Yousef, R. Wagner and M. Loew, "Assessing Classifiers from Two Independent Data Sets Using ROC Analysis: A Nonparametric Approach," in IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 28, no. 11, pp. 1809-1817, 2009.