This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Performance Assessment Through Bootstrap
November 1997 (vol. 19 no. 11)
pp. 1185-1198

Abstract—A new performance evaluation paradigm for computer vision systems is proposed. In real situation, the complexity of the input data and/or of the computational procedure can make traditional error propagation methods infeasible. The new approach exploits a resampling technique recently introduced in statistics, the bootstrap. Distributions for the output variables are obtained by perturbing the nuisance properties of the input, i.e., properties with no relevance for the output under ideal conditions. From these bootstrap distributions, the confidence in the adequacy of the assumptions embedded into the computational procedure for the given input is derived. As an example, the new paradigm is applied to the task of edge detection. The performance of several edge detection methods is compared both for synthetic data and real images. The confidence in the output can be used to obtain an edgemap independent of the gradient magnitude.

[1] K.E. Abdou and W.K. Pratt, "Quantitative Design and Evaluation of Enhancement/Thresholding Edge Detectors," Proc. IEEE, vol. 67, pp. 753-763, May 1979.
[2] K.L. Boyer and S. Sarkar, "On the Localization Performance Measure of Optimal Edge Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 6, pp. 106-108, 1994.
[3] J. Canny, "A Computational Approach to Edge Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 6, pp. 679-698, 1986.
[4] N.A.C. Cressie, Statistics for Spatial Data. Wiley-Interscience, 1991.
[5] "Dialogue: Performance Characterization in Computer Vision. With Contributions From R.M. Haralick; L. Cinque, C. Guerra, S. Levialdi; J. Weng, T.S. Huang; P. Meer; Y. Shirai; B.A. Draper, and J.R. Beveridge," CVGIP: Image Understanding, vol. 60, pp. 245-265, 1994.
[6] B. Efron and R.J. Tibshirani, An Introduction to the Bootstrap.London: Chapman&Hall, 1993.
[7] M.M. Fleck, “Some Defects in Finite-Difference Edge Finders,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, no. 3, pp. 337-345, Mar. 1992.
[8] R.M. Haralick, "Performance Assessment of Near-Perfect Machines," Machine Vision and Applications, vol. 2, pp. 1-16, 1989.
[9] R.M. Haralick and L.G. Shapiro, Computer and Robot Vision, vol. 2. Addison-Wesley, 1992.
[10] R.M. Haralick, "Overview: Computer Vision Performance Characterization," Proc. 1994 ARPA Image Understanding Workshop, pp. 663-665,Monterey, Calif., 1994.
[11] R.M. Haralick, "Performance Characterization Protocol in Computer Vision," Proc. 1994 ARPA Image Understanding Workshop, pp. 667-673,Monterey, Calif., 1994.
[12] M.Y. Jaisimha,R.M. Haralick,, and D. Dori,“Quantitative performance evaluation of thinning algorithms under noisy conditions,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 678-683,Seattle, Wash., June 1994.
[13] L. Kitchen and A. Rosenfeld, "Edge Evaluation Using Local Edge Coherence," IEEE Trans. Systems, Man, and Cybernetics, vol. 1, pp. 597-605, 1981.
[14] L.J. Kitchen and J.A. Malin, "The Effect of Spatial Discretization on the Magnitude and Direction Response of Simple Differential Edge Operators on a Step Edge," Computer Vision, Graphics, and Image Processing, vol. 7, pp. 243-258, 1989.
[15] K. Cho, "Bootstrap Based Cooperative Processes in Computer Vision," PhD Thesis, Dept. Electrical Eng., Rutgers Univ., Oct. 1995.
[16] V. Lacroix, “A Three-Module Strategy for Edge Detection,” IEEE Trans. Pattern Analysis Machine Intelligence, vol. 10, pp. 803-810, 1988.
[17] K.V. Mardia, Statistics of Directional Data.New York: Academic Press, 1972.
[18] P. Meer and I. Weiss, "Smoothed Differentiation Filters for Images," J. Communication and Image Representation, pp. 58-72, 1992.
[19] R. Nevatia and K.R. Babu, "Linear Feature Extraction and Description," Computer Graphics and Image Processing, vol. 13, pp. 257-269 1980.
[20] D.J. Park, K.M. Nam, and R. Park, "Multiresolution Edge Detection Techniques," Pattern Recognition, vol. 28, pp. 211-229, 1995.
[21] M. Petrou and J. Kittler, "Optimal Edge Detection for Ramp Edges," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, pp. 483-490, 1991.
[22] V. Ramesh and R.M. Haralick, "Random Perturbation Models&Performance Characterization in Computer Vision," Proc. IEEE Conf. Computer Vision and Pattern Recognition '92, pp. 521-527, 1992.
[23] V. Ramesh, R.M. Haralick, X. Zhang, D.C. Nadadur, and K. Thornton, "Automatic Selection of Tuning Parameters for Feature Extraction Sequences," Proc. IEEE Conf. Computer Vision and Pattern Recognition '94, pp. 672-677,Seattle, Wash., June 1994.
[24] V. Ramesh and R.M. Haralick, "A Methodology for Automatic Selection of IU Algorithm Tuning Parameters," Proc. 1994 ARPA Image Understanding Workshop, pp. 675-687,Monterey, Calif., 1994.
[25] V. Ramesh and R.M. Haralick, "An Intergrated Gradient Edge Detector—Theory and Performance Evaluation," Proc. 1994 ARPA Image Understanding Workshop, pp. 689-702,Monterey, Calif., 1994.
[26] K.R. Rao and J. Ben-Arie, "Optimal Edge Detection Using Expansion Matching and Restoration," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 12, pp. 1,169-1,182, Dec. 1994.
[27] S. Sarkar and K. L. Boyer, "On Optimal Infinite Impulse Response Edge Detection Filters," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 11, pp. 1,154-1,171, 1991.
[28] H.D. Tagare and R. deFigueiredo, "On the Localization Performance Measure and Optimal Edge Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 12, pp. 1,186-1,190, Dec. 1990.
[29] H.D. Tagare and R.J.P. deFigueiredo, "Reply to 'On the Localization Performance Measure of Optimal Edge Detection,'" IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 6, pp. 108-110, 1994.
[30] S. Venkatesh and P.L. Rosin, "Dynamic Threshold Determination by Local and Global Edge Evaluation," Graphical Model and Image Processing, vol. 7, pp. 146-160, 1995.
[31] S. Wang and T. Binford, "Local Step Edge Estimation: A New Algorithm, Statistical Model and Performance Evaluation," Proc. 1993 ARPA Image Understanding Workshop, pp. 1,063-1,170,Washington, D.C., Apr. 1993.
[32] S. Yi, R.M. Haralick, and L.G. Shapiro, "Error Propagation in Machine Vision," Machine Vision and Applications, pp. 93-114, 1994.

Index Terms:
Performance evaluation, edge detection, bootstrap.
Citation:
Kyujin Cho, Peter Meer, Javier Cabrera, "Performance Assessment Through Bootstrap," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 11, pp. 1185-1198, Nov. 1997, doi:10.1109/34.632979
Usage of this product signifies your acceptance of the Terms of Use.