• Publication
  • 2000
  • Issue No. 5 - May
  • Abstract - Quantitative Characterization and Analysis of the I/O Behavior of a Commercial Distributed-Shared-Memory Machine
 This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Quantitative Characterization and Analysis of the I/O Behavior of a Commercial Distributed-Shared-Memory Machine
May 2000 (vol. 11 no. 5)
pp. 509-526

Abstract—This paper presents a unified evaluation of the I/O behavior of a commercial clustered DSM machine, the HP Exemplar. Our study has the following objectives: 1) To evaluate the impact of different interacting system components, namely, architecture, operating system, and programming model, on the overall I/O behavior and identify possible performance bottlenecks, and 2) To provide hints to the users for achieving high out-of-box I/O throughput. We find that for the DSM machines that are built as a cluster of SMP nodes, integrated clustering of computing and I/O resources, both hardware and software, is not advantageous for two reasons. First, within an SMP node, the I/O bandwidth is often restricted by the performance of the peripheral components and cannot match the memory bandwidth. Second, since the I/O resources are shared as a global resource, the file-access costs become nonuniform and the I/O behavior of the entire system, in terms of both scalability and balance, degrades.

We observe that the buffered I/O performance is determined not only by the I/O subsystem, but also by the programming model, global-shared memory subsystem, and data-communication mechanism. Moreover, programming-model support can be used effectively to overcome the performance constraints created by the architecture and operating system. For example, on the HP Exemplar, users can achieve high I/O throughput by using features of the programming model that balance the sharing and locality of the user buffers and file systems. Finally, we believe that at present, the I/O subsystems are being designed in isolation, and there is a need for mending the traditional memory-oriented design approach to address this problem.

[1] G. Astfalk and T. Brewer, An Overview of the HP/Convex Exemplar Hardware. June 1997.
[2] G.A. Abandah and E.S. Davidson, “Effects of Architectural and Technological Advances on the HP/Convex Exemplar's Memory and Communication Performance,” Proc. 25th Int'l Symp. Computer Architecture, June 1998.
[3] A.C. Arpaci-Dusseau, R.H. Arpaci-Dusseau, D.E. Culler, J.M. Hellerstein, and D.A. Patterson, “High–Performance Sorting on Networks of Workstations,” Proc. ACM SIGMOD '97 May 1997.
[4] T. Anderson, H. Levy, B. Bershad, and E. Lazowska, “The Interaction of Architecture and Operating System Design,” Proc. Fourth Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 108-120, Apr. 1991.
[5] R. Bordawekar, “A Case for Compositional File Systems (Extended Abstract),” SIGOPS Operating Systems Review, vol. 32,no. (3), pp. 72–80, July 1998.
[6] W.J. Bolosky, M.L. Scott, R.P. Fitzgerald, R.J. Fowler, and A.L. Cox, “NUMA Policies and Their Relation to Memory Architecture,” Proc. Fourth Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 212–221, Apr. 1991.
[7] P.E. Crandall, R.A. Aydt, A.A. Chien, and D.A. Reed, “Input/Output Characteristics of Scalable Parallel Applications,” Proc. Supercomputing, Dec. 1995.
[8] J.B. Chen and B.N. Bershad, “The Impact of Operating System Structure on Memory System Performance,” Proc. 14th ACM Symp. Operating System Principles, pp. 120-133, Dec. 1993.
[9] S. Chandra, M. Dahlin, B. Richards, R.Y. Wang, T.E. Anderson, and J.R. Larus, “Experience with a Language for Writing Coherence Protocols,” Proc. USENIX Conf. Domain-Specific Languages, Oct. 1997.
[10] A.L. Chervenak and R.H. Katz, “Performance of a Disk Array Prototype,” Proc. 1991 ACM Sigmetrics Conf. Measurement and Modeling of Computer Systems, pp. 188–197, 1991.
[11] OpenMP Consortium, “OpenMP: A Proposed Industry Standard API for Shared Memory Programming,” Oct. 1997.
[12] Transaction Processing Performance Council, “Transaction Processing Performance Council Benchmark Descriptions.”
[13] M. Dahlin, R. Wang, T. Anderson, and D. Patterson, “Cooperative Caching: Using Remote Client Memory to Improve File System Performance,” Proc. First Symp. Operating System Design and Implementation (OSDI), pp. 267–280, Nov. 1994.
[14] G. Ganger and Y. Patt, "The Process-Flow Model: Examing I/O Performance from the System's Point of View," Proc. ACM Sigmetrics Conf., pp. 86-97, May 1993.
[15] Hewlett-Packard Company, Exemplar Programming Guide. Rich–ardson, Tex.: first ed., Jan. 1997.
[16] C. Holt, J.P. Singh, and J. Hennessy, “Application and Architectural Bottlenecks in Large Scale Distributed Shared Memory Machines,” Proc. 23rd Ann. Int'l Symp. Computer Architecture, pp. 134–145, May 1996.
[17] J. Gray, Sort Benchmark Home Page,http:/www.benchmarkresources.com.
[18] E.J. Koldinger, J.S. Chase, and S.J. Eggers, “Architectural Support for Single Address Space Operating Systems,” Proc. Fifth Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 175-186, Oct. 1992.
[19] D. Kotz and N. Nieuwejaar, “File-System Workload on a Scientific Multiprocessor,” IEEE Parallel and Distributed Technology, pp. 51-60, Spring 1995.
[20] K. Keeton and D.A. Patterson, “Importance of Proper Configuration in Architectural Evaluations Using Database Workloads,” Proc. First Workshop Computer Architecture Evaluation Using Commercial Workloads, Feb. 1998.
[21] O. Krieger, “HFS: A Flexible File System for Shared-Memory Multiprocessors,” PhD thesis, Dept. Electrical and Computer Eng., Univ. of Toronto, 1994.
[22] T. Lovett and R. Clapp, “STiNG: A CC-NUMA Computer System for the Commercial Marketplace,” Proc. 23rd Ann. Int'l Symp. Computer Architecture, pp. 308-317, May 1996.
[23] J. Laudon and D. Lenoski, "The SGI Origin: A cc-NUMA Highly Scalable Server," Proc. 24th Ann. Int'l Symp. Computer Architecture, May 1997.
[24] Message Passing Interface Forum, “MPI-2: Extensions to the Message-Passing Interface,” Apr. 1997.
[25] A. Reddy and P. Banerjee, “Evaluation of Multiple-Disk I/O Systems,” IEEE Trans. Computers, vol. 38, pp. 1,680–1,690, Dec. 1989.
[26] M. Rosenblum, E. Bugnion, S.A. Herrod, E. Witchel, and A. Gupta, “The Impact of Architectural Trends on Operating System Performance,” Proc. 15th ACM Symp. Operating System Principles, Dec. 1995.
[27] K. Ramakrishnan, P. Biswas, and R. Karedla, “Analysis of File I/O Traces in Commercial Computing Environments,” Performance Evaluation Rev., Vol. 20, No. 1, June 1992, pp. 78-90.
[28] E. Riedel, C. van Ingen, and J. Gray, “Sequential I/O on Windows NT 4.0 - Achieving Top Performance,” technical report, Microsoft Bay Area Research Center, Nov. 1997.
[29] D. Scales, K. Gharachorloo, and A. Aggarwal, “Fine-Grain Software Distributed Shared Memory on SMP Clusters,” Proc. Fourth Int'l Symp. High-Performance Computer Architecture, pp. 125–137, Feb. 1998.
[30] T. Sterling, P. Merkey, and D. Savarese, “Improving Application Performance on the HP/Convex Exemplar,” Computer, vol. 29, no. 12, pp. 50–57 Dec. 1996.
[31] M. Stonebraker, “Operating System Support for Database Management,” Comm. ACM, vol. 24, no. 7, pp. 412–418, July 1981.
[32] R. Thekkath, A.P. Singh, J.P. Singh, S. John, and J. Hennessy, “An Evaluation of a Commercial CC-NUMA Architecture—the Convex Exemplar SPP1200,” Proc. Int'l Parallel Processing Symp., Apr. 1997.
[33] B. Verghese, S. Devine, A. Gupta, and M. Rosenblum, "Operating System Support for Improving Data Locality on cc-NUMA Compute Servers," Proc. Seventh Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 279-289,Cambridge, Mass., Oct. 1996.
[34] S.C. Woo et al., "The SPLASH-2 Programs: Characterization and Methodological Considerations," Proc. 22nd Annual Int'l Symp. Computer Architecture, IEEE CS Press, Los Alamitos, Calif., June 1995, pp. 24-36.
[35] Z. Zhang and S. Sarukkai, “Commercial Applications on Shared-Memory Multiprocessors,” Proc. First Workshop Computer Architecture Evaluation Using Commercial Workloads, Feb. 1998.

Index Terms:
Input-output, operating/file systems, distributed-shared-memory architecture, clustered computing, performance evaluation, shared-memory programming.
Citation:
Rajesh R. Bordawekar, "Quantitative Characterization and Analysis of the I/O Behavior of a Commercial Distributed-Shared-Memory Machine," IEEE Transactions on Parallel and Distributed Systems, vol. 11, no. 5, pp. 509-526, May 2000, doi:10.1109/71.852403
Usage of this product signifies your acceptance of the Terms of Use.