This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Empirical Performance Evaluation of Graphics Recognition Systems
September 1999 (vol. 21 no. 9)
pp. 849-870

Abstract—This paper presents a methodology for evaluating graphics recognition systems operating on images that contain straight lines, circles, circular arcs, and text blocks. It enables an empirical comparison of vectorization software packages and uses practical performance evaluation methods that can be applied to complete vectorization systems. The methodology includes a set of matching criteria for pairs of graphical entities, a set of performance evaluation metrics, and a benchmark for the evaluation of graphics recognition systems. The benchmark was tested on three systems. The results are reported and analyzed in this paper.

[1] B. Kong, I. Phillips, R. Haralick, A. Prasad, and R. Kasturi, “A Benchmark: Performance Evaluation of Dashed Line Detection Algorithms,” Graphics Recognition: Methods and Applications, First Int'l Workshop, Selected Papers, R. Kasturi and K. Tombre, eds., pp. 270-285, Lecture Notes in Computer Science, vol. 1072.Berlin: Springer-Verlag, 1996.
[2] O. Hori and S. Doermann, “Quantitative Measurement of the Performance of Raster-to-Vector Conversion Algorithms,” Graphics Recognition: Methods and Applications, First Int'l Workshop, Selected Papers, R. Kasturi and K. Tombre, eds., pp. 57-68, Lecture Notes in Computer Science, vol. 1072.Berlin: Springer-Verlag, 1996.
[3] W. Liu and D. Dori, “A Protocol for Performance Evaluation of Line Detection Algorithms,” Machine Vision and Application, vol. 9, no. 5, pp. 240-250, 1997.
[4] W. Liu and D. Dori, “A Proposed Scheme for Performance Evaluation of Graphics/Text Separation Algorithms,” Graphics Recognition—Algorithms and Systems, K. Tombre and A.K. Chhabra, eds., pp. 359-371, 1998.
[5] R. Kasturi and I. Phillips, “The First International Graphics Recognition Contest—Dashed-Line Recognition Competition,” Graphics Recognition: Methods and Applications, First Int'l Workshop, Selected Papers, R. Kasturi and K. Tombre, eds., Lecture Notes in Computer Science, vol. 1072.Berlin: Springer-Verlag, 1996.
[6] A.K. Chhabra and I.T. Phillips, “The Second International Graphics Recognition Contest—Raster to Vector Conversion: A Report,” Graphics Recognition—Algorithms and Systems, K. Tombre and A.K. Chhabra, eds., pp. 390-410, 1998.
[7] I/Vector (Vectory) ver. 3.8 Raster to Vector Conversion Software, Graphikon, Berlin, Germany, and IDEAL Scanners&Systems, Rockville, Md.,http:/www. graphikon.comandhttp:/www. ideal.com.
[8] VPstudio ver. 6 rev. 2 Raster to Vector Conversion Software, Softelec, Munich, Germany, and Austin, Tex.,http:/www. softelec.comandhttp:/www. hybridcad.com.
[9] L. Wenyin and D. Dori, “Automated CAD Conversion with the Machine Drawing Understanding System,” Proc. IAPR Workshop Document Analysis Systems, pp. 241-259, Malvern, Pa., Oct. 1996, ftp://ftp. technion.ac. il/pub/supported/ ie/dori/MDUSsunmdus.gz.
[10] ECVNet, “Benchmarking and Performance Evaluation Web Site,” http://pandora.imag.fr/ECVNetbenchmarking.html .
[11] R. Haralick, “Performance Characterization in Image Analysis: Thinning, a Case in Point,” Pattern Recognition Letters, vol. 13, pp. 5-12, 1992.
[12] D. Dori, “Object-Process Analysis Maintaining the Balance between System Structure and Behavior,” Logic and Computation, vol. 5, no. 2, pp. 227-249, 1995.
[13] D. Dori,“Representing pattern recognition-embedded systems through object-process diagrams: The case of the Machine Drawing UnderstandingSystem,” Pattern Recognition Letters, vol. 16, no. 4, pp. 377-384, 1995.
[14] J. O'Rourke, Computational Geometry in C. Cambridge Univ. Press, 1993.
[15] VOGLE, a public domain device portable graphics library,ftp://munnari.oz.au/pub/graphicsvogle.tar.gz .
[16] S. Leffler and Silicon Graphics, Inc. “TIFF Software Distribution,” ftp://ftp. sgi.com/graphics/tifftiff-v3.4beta036-tar.gz .
[17] D. Knuth, The Art of Computer Programming, Vol. 2, Addison-Wesley, Reading, Mass., 1998.
[18] A. Chhabra and I. Phillips, “A Benchmark for Graphics Recognition Systems,” Empirical Evaluation Techniques in Computer Vision, K. Bowyer and P.J. Phillips, eds., pp. 28-44, 1998.
[19] I. Phillips, “Users' Reference Manual,” CD-ROM, UW-III Document Image Database-III.
[20] Autodesk, Inc., AutoCAD Release 13 Customization Guide, 1995.
[21] H. Baird, “Document Defect Models,” Proc. IAPR Workshop Syntactic and Structural Pattern Recognition (SSPR '90), pp. 38-46, Murray Hill, N.J., June 1990.
[22] T. Kanungo, R.M. Haralick, and I.T. Phillips, “Nonlinear Local and Global Document Degradation Models,” Int'l J. Imaging Systems and Technology, vol. 5, no. 4, pp. 220-230, Fall 1994.
[23] T. Kanungo, “Document Degradation Models and a Methodology for Degradation Model Validation,” PhD thesis, Univ. of Washington, Seattle, 1996. .
[24] A.K. Chhabra and I.T. Phillips, “The Second International Graphics Recognition Contest—Raster to Vector Conversion: A Report,” Graphics Recognition—Algorithms and Systems, K. Tombre and A.K. Chhabra, eds., pp. 390-410, 1998.
[25] I. Phillips, J. Liang, A. Chhabra, and R. Haralick, “A Performance Evaluation Protocol for Graphics Recognition Systems,” Graphics Recognition: Algorithms and Systems, Second Int'l Workshop, GREC '97, Selected Papers, pp. 372-389, Lecture Notes in Computer Science, vol. 1389.Berlin: Springer-Verlag, 1998.
[26] D. Dori, Vector-Based Arc Segmentation in the Machine Drawing Understanding System Environment IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 11, pp. 1057-1068, Nov. 1995.
[27] D. Dori and L. Wenyin, “Stepwise Recovery of Arc Segmentation in Complex Line Environents,” Int'l J. Document Analysis and Recognition, vol. 1, no. 1, pp. 62-71, 1998.
[28] J.P. Egan, Signal Detection Theory and ROC Analysis. New York: Academic Press, 1975.

Index Terms:
Empirical evaluation, benchmark, graphics recognition, engineering-drawing.
Citation:
Ihsin T. Phillips, Atul K. Chhabra, "Empirical Performance Evaluation of Graphics Recognition Systems," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 9, pp. 849-870, Sept. 1999, doi:10.1109/34.790427
Usage of this product signifies your acceptance of the Terms of Use.