This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Empirical Performance Evaluation Methodology and Its Application to Page Segmentation Algorithms
March 2001 (vol. 23 no. 3)
pp. 242-256

Abstract—While numerous page segmentation algorithms have been proposed in the literature, there is lack of comparative evaluation—empirical or theoretical—of these algorithms. In the existing performance evaluation methods, two crucial components are usually missing: 1) automatic training of algorithms with free parameters and 2) statistical and error analysis of experimental results. In this paper, we use the following five-step methodology to quantitatively compare the performance of page segmentation algorithms: 1) First, we create mutually exclusive training and test data sets with groundtruth, 2) we then select a meaningful and computable performance metric, 3) an optimization procedure is then used to search automatically for the optimal parameter values of the segmentation algorithms on the training data set, 4) the segmentation algorithms are then evaluated on the test data set, and, finally, 5) a statistical and error analysis is performed to give the statistical significance of the experimental results. In particular, instead of the ad hoc and manual approach typically used in the literature for training algorithms, we pose the automatic training of algorithms as an optimization problem and use the Simplex algorithm to search for the optimal parameter value. A paired-model statistical analysis and an error analysis are then conducted to provide confidence intervals for the experimental results of the algorithms. This methodology is applied to the evaluation of five page segmentation algorithms of which, three are representative research algorithms and the other two are well-known commercial products, on 978 images from the University of Washington III data set. It is found that the performance indices (average textline accuracy) of the Voronoi, Docstrum, and Caere segmentation algorithms are not significantly different from each other, but they are significantly better than that of ScanSoft's segmentation algorithm, which, in turn, is significantly better than that of X-Y cut.

[1] H.S. Baird, S.E. Jones, and S.J. Fortune, Image Segmentation by Shape-Directed Covers Proc. Int'l Conf. Pattern Recognition, pp. 820-825, 1990.
[2] Empirical Evaluation Techniques in Computer Vision, K.W. Bowyer and P.J. Phillips, eds., Santa Barbara, Calif. June 1998.
[3] Document Layout Interpretation and Its Applications, T. Breuel and M. Worring, eds., Bangalore, India, Sept. 1999.
[4] Caere Developer's Kit 2000. Caere Co.http:/www.caere.com/ 1998.
[5] L.A. Fletcher and R. Kasturi, “A Robust Algorithm for Text String Separation from Mixed Text/Graphics Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 10, pp. 910-918, Nov. 1988.
[6] P.E. Gill, W. Murray, and M.H. Wright, Practical Optimization, chapter 4, London and New York: Academic Press, 1993.
[7] D.E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning. Reading, Mass.: Addison-Wesley, 1989.
[8] Performance versus Methodology in Computer Vision, R.M. Haralick and P. Meer, eds., Seattle, June 1994.
[9] R.M. Haralick and L.G. Shapiro, Computer and Robot Vision. New York: Addison-Wesley, 1993.
[10] R.M. Haralick,S.R. Sternberg,, and X. Zhuang,“Image analysis using mathematical morphology,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 9, no. 4, pp. 532-550, July 1988.
[11] J.K. Hollingsworth, E. Guven, and C. Akinlar, “Benchmarking a Network of PCs Running Parallel Applications,” Proc. Int'l Performance, Computing, and Communications Conf., pp. 447-453, Feb. 1998.
[12] A. Hoover, G. Jean-Baptiste, X. Jiang, P.J. Flynn, H. Bunke, D. Goldgof, K. Bowyer, D. Eggert, A. Fitzgibbon, and R. Fisher, “An Experimental Comparison of Range Segmentation Algorithms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol 18, no. 7, pp. 673-689, July 1996.
[13] A.K. Jain and B. Yu, “Document Representation and Its Application to Page Decomposition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 294-308, Mar. 1998.
[14] J. Kanai, S.V. Rice, T.A. Nartker, and G. Nagy, “Automated Evaluation of OCR Zoning” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 1, pp. 86-89, Jan. 1995.
[15] T. Kanungo, M.Y. Jaisimha, and R.M. Haralick, "A Methodology for Quantitative Performance Evaluation of Detection Algorithms," IEEE Trans. Image Processing, vol. 4, pp. 1,667-1,674, Dec. 1995.
[16] T. Kanungo, G.A. Marton, and O. Bulbul, “OmniPage vs. Sakhr: Paired Model Evaluation of Two Arabic OCR Products,” Proc. SPIE Conf. Document Recognition and Retrieval VI, vol. 3651, pp. 109-120, Jan. 1999.
[17] K. Kise, A. Sato, and M. Iwata, “Segmentation of Page Images Using the Area Voronoi Diagram,” Computer Vision and Image Understanding, vol. 70, pp. 370-382, 1998.
[18] P.J.M. Laarhoven and E.H.L. Aarts, Simulated Annealing: Theory and Applications. D. Reidel Publishing, 1987.
[19] R.M. Lewis, V. Torczon, and M.W. Trosset, “Why Pattern Search Works,” OPTIMA, vol. 59, pp. 1-7, 1998.
[20] J. Liang, I.T. Phillips, and R.M. Haralick, “Performance Evaluation of Document Layout Analysis Algorithms on the UW Data Set,” Proc. SPIE Conf. Document Recognition IV, vol. 3027, pp. 149-160, Feb. 1997.
[21] S. Mao and T. Kanungo, “A Methodology for Empirical Performance Evaluation of Page Segmentation Algorithms,” Technical Report CAR-TR-933, Univ. of Maryland, College Park, Dec. 1999. http://www.cfar.umd.edu/~kangungo/pubstrsegeval.ps .
[22] S. Mao and T. Kanungo, “Automatic Training of Page Segmentation Algorithms: An Optimization Approach,” Proc. Int'l Conf. Pattern Recognition, pp. 531-534, Sept. 2000.
[23] S. Mao and T. Kanungo, “Empirical Performance Evaluation of Page Segmentation Algorithms,” Proc. SPIE Conf. Document Recognition and Retrieval VII, vol. 3967, pp. 303-314, Jan. 2000.
[24] S. Mao and T. Kanungo, “PSET: A Page Segmentation Evaluation Toolkit,” Proc. Fourth IAPR Int'l Workshop Document Analysis Systems, pp. 451-462, Dec. 2000.
[25] S. Mao and T. Kanungo, “Software Architecture of PSET: A Page Segmentation Evaluation Toolkit,” Technical Report CAR-TR-955, Univ. of Maryland, College Park, Sept. 2000. http://www.cfar.umd.edu/~kanungo/pubstrpset.ps . Software is available athttp://www.cfar.umd.edu/~kanungo/software software.html.
[26] G. Nagy, S. Seth, and M. Viswanathan, “A Prototype Document Image Analysis System for Technical Journals,” Computer, vol. 25, no. 7, pp. 10-22, July 1992.
[27] J. Nelder and R. Mead, “A Simplex Method for Function Minimization,” Computer J., vol. 7, pp. 308-313, 1965.
[28] L. O'Gorman, “The Document Spectrum for Page Layout Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1,162-1,173, Nov. 1993.
[29] L. O'Gorman and R. Kasturi, Document Image Analysis. IEEE CS Press, 1995.
[30] T. Pavlidis and J. Zhou, “Page Segmentation and Classification,” CVGIP: Graphical Models and Image Processing, vol. 54, no. 6, pp. 484-496, Nov. 1992.
[31] I. Phillips, User's Reference Manual, CD-ROM, UW-III Document Image Database-III, July 1996.
[32] I. Phillips and A. Chhabra, “Empirical Performance Evaluation of Graphics Recognition Systems,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 9, pp. 849-870, Sept. 1999.
[33] P.J. Phillips, H. Moon, and S.A. Rozvi, The FERET Evaluation Methodolody for Face Recognition Algorithms IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp. 1090-1104, Oct. 2000.
[34] M.J.D. Powell, “Direct Search Algorithms for Optimization Calculations,” Acta Numerica, vol. 7, pp. 287-336, 1998.
[35] S. Randriamasy, L. Vincent, and B. Wittner, “An Automatic Benchmarking Scheme for Page Segmentation,” Proc. SPIE Conf. Document Recognition, vol. 2181, pp. 217-230, Feb. 1994.
[36] Application Programmer's Interface, ScanSoft Co., Dec. 1997. http:/www.scansoft.com.
[37] F. Wahl, K. Wong, and R. Casey, “Block Segmentation and Text Extraction in Mixed Text/Image Documents,” Graphical Models and Image Processing, vol. 20, pp. 375-390, 1982.
[38] M.H. Wright, “Direct Search Methods: Once Scorned, Now Respectable,” Numerical Analysis 1995, pp. 191-208, D.F. Griffiths and G.A. Watson, eds., Addison Wesley, Longman (Harlow), 1996.

Index Terms:
Document page segmentation, OCR, performance evaluation, performance metric, statistical significance, paired model, direct search, simplex method.
Citation:
Song Mao, Tapas Kanungo, "Empirical Performance Evaluation Methodology and Its Application to Page Segmentation Algorithms," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 3, pp. 242-256, March 2001, doi:10.1109/34.910877
Usage of this product signifies your acceptance of the Terms of Use.