This Article 
 Bibliographic References 
 Add to: 
Performance Characterization of Optimizing Compilers
July 1995 (vol. 21 no. 7)
pp. 615-628
Optimizing compilers have become an essential component in achieving high levels of performance. Various simple and sophisticated optimizations are implemented at different stages of compilation to yield significant improvements, but little work has been done in characterizing the effectiveness of optimizers, or in understanding where most of this improvement comes from. In this paper we study the performance impact of optimization in the context of our methodology for CPU performance characterization based on the abstract machine model. The model considers all machines to be different implementations of the same high level language abstract machine; in previous research, the model has been used as a basis to analyze machine and benchmark performance. In this paper, we show that our model can be extended to characterize the performance improvement provided by optimizers and to predict the run time of optimized programs, and measure the effectiveness of several compilers in implementing different optimization techniques.

[1] A.V. Aho, R. Sethi, and J.D. Ullman, Compilers, Principles, Techniques and Tools.New York: Addison-Wesley, 1985.
[2] H.E. Bal and A.S. Tanenbaum,“Language- and machine-independent global optimization on intermediate code,” Computer Languages, vol. 11, no. 2, 1986, pp. 105-121.
[3] R.N. Braswell and M.S. Keech,“An evaluation of vector FORTRAN 200 generated by CYBER 205 and ETA-10 pre-compilation tools,” Proc. Supercomputing’88 Conf.,Orlando, Fla., Nov.14-18, 1988, pp. 106-113.
[4] D. Callahan,J. Dongarra,, and D. Levine,“Vectorizing compilers: A test suite and results,” Proc. Supercomputing’88 Conf.,Orlando, Fla., Nov.14-18 1988, pp. 98-105.
[5] F. Chow,A Portable Machine-Independent Global Optimizer, PhD dissertation and Technical Report No. 83-254,Computer Systems Laboratory, Stanford Univ., Dec. 1983.
[6] F. Chow,M. Himelstein,E. Killian,, and L. Weber,“Engineering a RISC Compiler System,” Proc. Compcon’86 Conf.,San Francisco, Calif., Mar.4-6, 1986, pp. 132-137.
[7] R. Cmelik, S. Kong, D. Ditzel, and E. Kelly, "An Analysis of MIPS and SPARC Instruction Set Utilization on the SPEC Benchmarks," Proc. Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 290-302, 1991.
[8] J. Cocke and P. Markstein,“Measurement of program improvement algorithms,” Technical Report No. RC-8111 (#35193), IBM, Feb.7, 1980.
[9] G. Cybenko,L. Kipp,L. Pointer,, and D. Kuck,Supercomputer Performance Evaluation and the Perfect Benchmarks, Univ. of Illinois Center for Supercomputing R&D Technical Report 965, Mar. 1990.
[10] M. Jazayeri and M. Haden,“Optimizing compilers are here (mostly),” SIGPLAN Notices, vol. 21, no. 5, May 1986, pp. 61-63.
[11] M.S. Johnson and T.C. Miller,“Effectiveness of a machine-level, global optimizer,” Proc. SIGPLAN’86 Symp. on Compiler Construction,Palo Alto, Calif., June25-27, 1986, pp. 99-108.
[12] D.E. Knuth,“An empirical study of Fortran programs,” Software—Practice and Experience, vol. 1, 1971, pp. 105-133.
[13] D.S. Lindsay and T.E. Bell,“Directed benchmarks for CPU architecture evaluation,” Proc. CMG’86 Conf.,Las Vegas, Nev., Dec.9-12, 1986, pp. 379-385.
[14] S.S. Muchnick,“Here are (some of) the optimizing compilers,” SIGPLAN Notices, vol. 21, no. 2, Feb. 1986, pp. 11-15.
[15] K. O’Brien,B. Hay,J. Minisk,H. Schaffer,B. Schloss,A. Shepherd,, and M. Zaleski,“Advanced compiler technology for the RISC system/6000architecture,” IBM RISC System/6000 Technology, SA23-2619, IBM Corp., 1990, pp. 154-161.
[16] D.A. Padua and M.J. Wolfe, "Advanced Compiler Optimizations for Supercomputers," Comm. ACM, vol. 29, Dec. 1986.
[17] B.L. Peuto and L.J. Shustek,“An instruction timing model of CPU performance,” 4th Ann. Symp. on Computer Architecture, vol. 5, no. 7, Mar. 1977, pp. 165-178.
[18] C.G. Ponder,“An analytical look at linear performance models,” Lawrence Livermore National Laboratory, Technical Report UCRL-JC-106105, Sept. 1990.
[19] S. Richardson and M. Ganapathi,“Interprocedural optimization: Experimental results,” Software—Practice and Experience, vol. 19, no. 2, Feb. 1989, pp. 149-170.
[20] R.H. Saavedra-Barrera, A.J. Smith, and E. Miya, “Machine Characterization Based on an Abstract High-Level Language Machine,” IEEE Trans. Computers, vol. 38, no. 12, pp. 1,659-1,679, Dec. 1989.
[21] R.H. Saavedra-Barrera,CPU Performance Evaluation and Execution Time Prediction Using Narrow Spectrum Benchmarking, PhD thesis, Univ. of California, Berkeley, Technical Report No. UCB/CSD 92/684, Feb. 1992.
[22] R.H. Saavedra and A.J. Smith,“Analysis of benchmark characteristics and benchmark performance prediction,” paper in preparation, 1992.
[23] R.H. Saavedra and A.J. Smith,Performance Characterization of Optimizing Compilers, Univ. of Southern California Technical Report No. USC-CS-92-525, Aug. 1992.
[24] L.J. Shustek,Analysis and Performance of Instruction Sets, PhD dissertation, Stanford Univ., May 1978.
[25] J.P. Singh and J.L. Hennessy,“An empirical investigation of the effectiveness and limitations of automatic parallelization,” Proc. Int’l Symp. Shared Memory Multiprocessing,Tokyo, Japan, Apr. 1991, pp. 25-36.
[26] “SPEC,” SPEC Newsletter: Benchmark Results, vol. 1, no. 1, Fall 1989.
[27] M. Wolfe and T. Macke,“Where are the optimizing compilers,” SIGPLAN Notices, vol. 20, no. 11, Nov. 1985, pp. 64-77.

Index Terms:
Performance evaluation, optimizing compilers, benchmarking, execution time prediction, CPU performance characterization.
Rafael H. Saavedra, Alan Jay Smith, "Performance Characterization of Optimizing Compilers," IEEE Transactions on Software Engineering, vol. 21, no. 7, pp. 615-628, July 1995, doi:10.1109/32.392982
Usage of this product signifies your acceptance of the Terms of Use.