This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Performance Evaluation and Benchmarking of Six-Page Segmentation Algorithms
June 2008 (vol. 30 no. 6)
pp. 941-954
ASCII Text
x 
 
Faisal Shafait, Daniel Keysers, Thomas Breuel, "Performance Evaluation and Benchmarking of Six-Page Segmentation Algorithms," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 6, pp. 941-954, June, 2008.
 
 
BibTex
x
 
@article{ 10.1109/TPAMI.2007.70837,
author = {Faisal Shafait and Daniel Keysers and Thomas Breuel},
title = {Performance Evaluation and Benchmarking of Six-Page Segmentation Algorithms},
journal ={IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume = {30},
number = {6},
issn = {0162-8828},
year = {2008},
pages = {941-954},
doi = {http://doi.ieeecomputersociety.org/10.1109/TPAMI.2007.70837},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
TI - Performance Evaluation and Benchmarking of Six-Page Segmentation Algorithms
IS - 6
SN - 0162-8828
SP941
EP954
EPD - 941-954
A1 - Faisal Shafait,
A1 - Daniel Keysers,
A1 - Thomas Breuel,
PY - 2008
KW - Document analysis
KW - Optical character recognition
VL - 30
JA - IEEE Transactions on Pattern Analysis and Machine Intelligence
ER -
 
Informative benchmarks are crucial for optimizing the page segmentation step of an OCR system, frequently the performance limiting step for overall OCR system performance. We show that current evaluation scores are insufficient for diagnosing specific errors in page segmentation and fail to identify some classes of serious segmentation errors altogether. This paper introduces a vectorial score that is sensitive to, and identifies, the most important classes of segmentation errors (over-, under-, and mis-segmentation) and what page components (lines, blocks, etc.) are affected. Unlike previous schemes, our evaluation method has a canonical representation of ground truth data and guarantees pixel-accurate evaluation results for arbitrary region shapes. We present the results of evaluating widely used segmentation algorithms (x-y cut, smearing, whitespace analysis, constrained text-line finding, docstrum, and Voronoi) on the UW-III database and demonstrate that the new evaluation scheme permits the identification of several specific flaws in individual segmentation methods.

[1] R. Cattoni, T. Coianiz, S. Messelodi, and C.M. Modena, “Geometric Layout Analysis Techniques for Document Image Understanding: A Review,” IRST Technical Report 9703-09, 1998.
[2] G. Nagy, “Twenty Years of Document Image Analysis in PAMI,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 38-62, Jan. 2000.
[3] S. Mao, A. Rosenfeld, and T. Kanungo, “Document Structure Analysis Algorithms: A Literature Survey,” Proc. SPIE Electronic Imaging, vol. 5010, pp. 197-207, Jan. 2003.
[4] S. Mao and T. Kanungo, “Empirical Performance Evaluation Methodology and Its Application to Page Segmentation Algorithms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 3, pp. 242-256, Mar. 2001.
[5] F. Lotti, P. Heroux, S. Adam, G. Sanchez, E. Valveny, P. Dosch, and J. Llados, “Performance Analysis and Evaluation Working Group Report,” Document Analysis Systems, http://www.dsi.unifi.it/DAS04DASPerfEv.pdf , Sept. 2004.
[6] A. Antonacopoulos, D. Karatzas, and D. Bridson, “Ground Truth for Layout Analysis Performance Evaluation,” Document Analysis Systems, pp. 302-311, Feb. 2006.
[7] A. Antonacopoulos, B. Gatos, and D. Karatzas, “ICDAR 2003 Page Segmentation Competition,” Proc. Seventh Int'l Conf. Document Analysis and Recognition, pp. 688-692, 2003.
[8] A. Antonacopoulos, B. Gatos, and D. Bridson, “ICDAR 2005 Page Segmentation Competition,” Proc. Eighth Int'l Conf. Document Analysis and Recognition, pp. 75-80, Aug. 2005.
[9] J. Kanai, T.A. Nartker, S.V. Rice, and G. Nagy, “Performance Metrics for Document Understanding Systems,” Proc. Second Int'l Conf. Document Analysis and Recognition, pp. 424-427, Oct. 1993.
[10] B.A. Yanikoglu and L. Vincent, “Ground-Truthing and Benchmarking Document Page Segmentation,” Proc. Third Int'l Conf. Document Analysis and Recognition, pp. 601-604, Aug. 1995.
[11] J. Liang, I.T. Phillips, and R.M. Haralick, “Performance Evaluation of Document Structure Extraction Algorithms,” Computer Vision and Image Understanding, vol. 84, pp. 144-159, 2001.
[12] A.K. Das, S.K. Saha, and B. Chanda, “An Empirical Measure of the Performance of a Document Image Segmentation Algorithm,” Int'l J. Document Analysis and Recognition, vol. 4, no. 3, pp. 183-190, 2002.
[13] A. Hoover, G. Jean-Baptiste, X. Jiang, P.J. Flynn, H. Bunke, D.B. Goldgof, K. Bowyer, D.W. Eggert, A. Fitzgibbon, and R.B. Fisher, “An Experimental Comparison of Range Image Segmentation Algorithms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 7, pp. 673-689, July 1996.
[14] X. Jiang, C. Marti, C. Irniger, and H. Bunke, “Distance Measures for Image Segmentation Evaluation,” EURASIP J. Applied Signal Processing, vol. 2006, Article ID 35 909, 2006.
[15] G. Nagy, S. Seth, and M. Viswanathan, “A Prototype Document Image Analysis System for Technical Journals,” Computer, vol. 25, no. 7, pp. 10-22, July 1992.
[16] L. O'Gorman, “The Document Spectrum for Page Layout Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1162-1173, Nov. 1993.
[17] K. Kise, A. Sato, and M. Iwata, “Segmentation of Page Images Using the Area Voronoi Diagram,” Computer Vision and Image Understanding, vol. 70, no. 3, pp. 370-382, June 1998.
[18] K.Y. Wong, R.G. Casey, and F.M. Wahl, “Document Analysis System,” IBM J. Research and Development, vol. 26, no. 6, pp. 647-656, 1982.
[19] H.S. Baird, “Background Structure in Document Images,” Document Image Analysis, H. Bunke, P. Wang, and H.S. Baird, eds., pp.17-34, World Scientific, 1994.
[20] T.M. Breuel, “Two Geometric Algorithms for Layout Analysis,” Document Analysis Systems, pp. 188-199, Aug. 2002.
[21] I. Guyon, R.M. Haralick, J.J. Hull, and I.T. Phillips, “Data Sets for OCR and Document Image Understanding Research,” Handbook of Character Recognition and Document Image Analysis, H. Bunke and P. Wang, eds., pp. 779-799, World Scientific, 1997.
[22] Y. Wang, R. Haralick, and I. Phillips, “Document Zone Content Classification and Its Performance Evaluation,” Pattern Recognition, vol. 39, no. 1, pp. 57-73, Jan. 2006.
[23] T.M. Breuel, “High Performance Document Layout Analysis,” Proc. Symp. Document Image Understanding Technology, Apr. 2003.
[24] S. Mandal, S. Chowdhury, A. Das, and B. Chanda, “A Simple and Effective Table Detection System from Document Images,” Int'l J. Document Analysis and Recognition, vol. 8, nos. 2-3, pp. 172-182, June 2006.
[25] C. Shin and D. Doermann, “Classification of Document Page Images,” Proc. Symp. Document Image Understanding Technology, pp. 166-175, Apr. 1999.
[26] F. Shafait, D. Keysers, and T.M. Breuel, “Performance Comparison of Six Algorithms for Page Segmentation,” Proc. Seventh IAPR Workshop Document Analysis Systems, pp. 368-379, Feb. 2006.
[27] F. Shafait, D. Keysers, and T.M. Breuel, “Pixel-Accurate Representation and Evaluation of Page Segmentation in Document Images,” Proc. 18th Int'l Conf. Pattern Recognition, pp. 872-875, Aug. 2006.
[28] D. Dori, D. Doermann, C. Shin, R. Haralick, I. Phillips, M. Buchman, and D. Ross, “The Representation of Document Structure: A Generic Object-Process Analysis,” Handbook of Character Recognition and Document Image Analysis, H. Bunke and P. Wang, eds., pp. 421-456, World Scientific, 1997.
[29] G. Ford and D. Thoma, “Ground Truth Data for Document Image Analysis,” Proc. Symp. Document Image Understanding and Technology, pp. 199-205, Apr. 2003.
[30] F. Shafait and T.M. Breuel, “Document Image Dewarping Contest,” Proc. Second Int'l Workshop Camera-Based Document Analysis and Recognition, pp. 181-188, Sept. 2007.
[31] T.M. Breuel, “Representations and Metrics for Off-Line Handwriting Segmentation,” Proc. Eighth Int'l Workshop Frontiers in Handwriting Recognition, pp. 428-433, Aug. 2002.
[32] T.M. Breuel, “Robust Least Square Baseline Finding Using a Branch and Bound Algorithm,” Proc. Document Recognition and Retrieval VIII, 2002.
[33] L. Cinque, S. Levialdi, L. Lombardi, and S. Tanimoto, “Segmentation of Page Images Having Artifacts of Photocopying and Scanning,” Pattern Recognition, vol. 35, pp. 1167-1177, 2002.
[34] F. Shafait, J. van Beusekom, D. Keysers, and T.M. Breuel, “Page Frame Detection for Marginal Noise Removal from Scanned Documents,” Proc. 15th Scandinavian Conf. Image Analysis, pp.651-660, June 2007.
[35] O. Okun, M. Pietikainen, and J. Sauvola, “Robust Skew Estimation on Low-Resolution Document Images,” Proc. Fifth Int'l Conf. Document Analysis and Recognition, pp. 621-624, Sept. 1999.
[36] D. Keysers, F. Shafait, and T.M. Breuel, “Document Image Zone Classification—A Simple High-Performance Approach,” Proc. Second Int'l Conf. Computer Vision Theory and Applications, pp. 44-51, Mar. 2007.
[37] S. Marinai, E. Marino, and G. Soda, “Layout Based Document Image Retrieval by Means of XY Tree Reduction,” Proc. Eighth Int'l Conf. Document Analysis and Recognition, pp. 432-436, Aug. 2005.
[38] S. Mao and T. Kanungo, “Software Architecture of PSET: A Page Segmentation Evaluation Toolkit,” Int'l J. Document Analysis and Recognition, vol. 4, no. 3, pp. 205-217, 2002.
[39] L. Vincent, “Google Book Search: Document Understanding on a Massive Scale,” Proc. Ninth Int'l Conf. Document Analysis and Recognition, pp. 819-823, Sept. 2007.

Index Terms:
Document analysis, Optical character recognition
Citation:
Faisal Shafait, Daniel Keysers, Thomas Breuel, "Performance Evaluation and Benchmarking of Six-Page Segmentation Algorithms," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 6, pp. 941-954, June 2008, doi:10.1109/TPAMI.2007.70837
Usage of this product signifies your acceptance of the Terms of Use.