This Article 
 Bibliographic References 
 Add to: 
Orthogonal Striping and Mirroring in Distributed RAID for I/O-Centric Cluster Computing
January 2002 (vol. 13 no. 1)
pp. 26-44

This paper presents a new distributed disk-array architecture for achieving high I/O performance in scalable cluster computing. In a serverless cluster of computers, all distributed local disks can be integrated as a distributed-software redundant array of independent disks (ds-RAID) with a single I/O space. We report the new RAID-x design and its benchmark performance results. The advantage of RAID-x comes mainly from its orthogonal striping and mirroring (OSM) architecture. The bandwidth is enhanced with distributed striping across local and remote disks, while the reliability comes from orthogonal mirroring on local disks at the background. Our RAID-x design is experimentally compared with the RAID-5, RAID-10, and chained-declustering RAID through benchmarking on a research Linux cluster at USC. Andrew and Bonnie benchmark results are reported on all four disk-array architectures. Cooperative disk drivers and Linux extensions are developed to enable not only the single I/O space, but also the shared virtual memory and global file hierarchy. We reveal the effects of traffic rate and stripe unit size on I/O performance. Through scalability and overhead analysis, we find the strength of RAID-x in three areas: 1) improved aggregate I/O bandwidth especially for parallel writes, 2) orthogonal mirroring with low software overhead, and 3) enhanced scalability in cluster I/O processing. Architectural strengths and weakness of all four ds-RAID architectures are evaluated comparatively. The optimal choice among them depends on parallel read/write performance desired, the level of fault tolerance required, and the cost-effectiveness in specific I/O processing applications.

[1] T. Anderson, M. Dahlin, J. Neefe, D. Patterson, D. Roselli, and R. Wang, “Serverless Network File Systems,” ACM Trans. Computer Systems, pp. 41-79, Jan. 1996.
[2] M. Arunachalam, A. Choudhary, and B. Rullman, “Implementation and Evaluation of Prefetching in the Intel Paragon Parallel File System,” Proc. 10th Int'l Parallel Processing Symp. (IPPS '96), pp. 554-559, Apr. 1996.
[3] S. Asami, N. Talagala, and D.A. Patterson, “Designing a Self-Maintaining Storage System,” Proc. 16th IEEE Symp. Mass Storage Systems, pp. 222-233, Mar. 1999.
[4] L.F. Cabrera and D.E. Long, “Swift: Using Distributed Disk Striping to Provide High I/O Data Rates,” Proc. USENIX Computing Systems, pp. 405-433, Fall 1991.
[5] P. Cao, S.B. Lim, S. Venkataraman, and J. Wilkes, “The TickerTAIP Parallel RAID Architecture,” ACM Trans. Computer System, vol. 12, no. 3, pp. 236-269, Aug. 1994.
[6] P.H. Carnset et al., “PVFS: A Parallel File System for Linux Clusters,” Proc. Extreme Linux Track: Fourth Ann. Linux Showcase and Conf., Oct. 2000.
[7] P.M. Chen, E.K. Lee, G.A. Gibson, R.H. Katz, and D.A. Patterson, "RAID: High-Performance Reliable Secondary Storage," ACM Computing Surveys, vol. 36, no. 3, pp. 145-185, Aug. 1994.
[8] P.F. Corbett et al., “Parallel Access to Files in the Vesta File System,” Proc. Supercomputing’93, CS Press, 1993, pp. 472–481.
[9] T. Cortes, “Software RAID and Parallel Filesystems,” High Performance Cluster Computing, R. Buyya, ed., Prentice Hall PTR, pp. 463-496, 1999.
[10] M. Dahlin, R. Wang, T. Anderson, and D. Patterson, “Cooperative Caching: Using Remote Client Memory to Improve File System Performance,” Proc. Operating System Design and Implementation, 1994.
[11] I. Foster, D. Kohr Jr., R. Krishnaiyer, and J. Mogill, “Remote I/O: Fast Access to Distant Storage,” Proc. Fifth Workshop I/O in Parallel and Distributed Systems, pp. 14-25, Nov. 1997.
[12] G. Gibson et al. “A Cost-effective, High-bandwidth Storage Architecture,” Proc. Eighth Conf. Architectural Support for Programming Languages and Operating Systems, 1998.
[13] M. Harry, J.M. Del Rosario, and A. Choudhary, “VIP-FS: A VIrtual, Parallel File System for High Performance Parallel and Distributed Computing,” Proc. Ninth Int'l Parallel Processing Symp. (IPPS '95), pp. 159-164, Apr. 1995.
[14] J.H. Hartman, I. Murdock, and T. Spalink, “The Swarm Scalable Storage System,” Proc. 19th IEEE Int'l Conf. Distributed Computer Systems (ICDCS '99), June 1999.
[15] R. Ho, K. Hwang, and H. Jin, “Design and Analysis of Clusters with Single I/O Space,” Proc. 20th Int'l Conf. Distributed Computing Systems (ICDCS 2000), pp. 120-127, Apr. 2000.
[16] J.H. Howard, M.L. Kazar, S.G. Menees, D.A. Nichols, M. Satyanarayanan, R.N. Sidebotham, and M.J. West, "Scale and performance in a distributed file system," ACM Trans. Comp. Sys., vol. 6, no. 1, Feb. 1988.
[17] H. Hsiao and D.J. DeWitt, “Chained Declustering: A New Availability Strategy for Multiprocessor Database Machines,” Proc. Data Eng., pp. 456–465, 1990.
[18] Y. Hu, Q. Yang, and T. Nightingale, “RAPID-Cache—A Reliable and Inexpensive Write Cache for Disk I/O Systems,” Proc. Fifth Int'l Symp. High Performance Computer Architecture (HPCA-5), pp. 204-213, Jan. 1999.
[19] J. Huber, C.L. Elford, D.A. Reed, A.A. Chien, and D.S. Blumenthal, “PPFS: A High Performance Portable Parallel File System,” Proc. Ninth ACM Int'l Conf. Supercomputing, pp. 385–394, July 1995.
[20] K. Hwang, H. Jin, E. Chow, C.-L. Wang, Z. Xu, “Designing SSI Clusters with Hierarchical Checkpointing and Single I/O Space,” IEEE Concurrency, vol. 7, no. 1, pp. 60-69, 1999.
[21] K. Hwang and Z.W. Xu, Scalable Parallel Computing: Technology, Architecture, Programming. McGraw-Hill, 1998.
[22] K. Hwang, H. Jin, and R. Ho, “RAID-x: A New Distributed Disk Array for I/O-Centric Cluster Computing,” Proc. Nineth IEEE Int'l Symp. High-Performance Distributed Computing (HPDC-9), pp. 279-286, Aug. 2000.
[23] H. Jin and K. Hwang, “Adaptive Sector Grouping to Reduce False Sharing of Distributed RAID,” Cluster Computing J., vol. 4, no. 2, pp. 133-143, Apr. 2001.
[24] J.H. Kim, S.W. Eom, S.H. Noh, and Y.H. Won, “Striping and Buffer Caching for Software RAID File Systems in Workstation Clusters,” Proc. 19th IEEE Int'l Conf. Distributed Computing Systems, pp. 544-551, 1999.
[25] E.K. Lee and C.A. Thekkath, “Petal: Distributed Virtual Disks,” Proc. Seventh Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 84-92, Oct. 1996.
[26] C.S. Li, M.-S. Chen, P.S. Yu, and H.I. Hsiao, “Combining Replication and Parity Approaches for Fault-Tolerant Disk Arrays,” Proc. IEEE Symp. Parallel and Distributed Processing, pp. 360-367, Oct. 1994.
[27] R.P. Martin et al., "Effects of Communication Latency, Overhead and Bandwidth in a Cluster Architecture," Computer Architecture News, May 1997, pp. 85-97.
[28] G.F. Pfister, “The Varieties of Single System Image,” Proc. IEEE Workshop Advances in Parallel and Distributed System, pp. 59-63, 1993.
[29] R. Sandberg, D. Goldberg, S. Kleiman, D. Walsh, and B. Lyon, “Design and Implementation of the Sun Network Filesystem,” Proc. USENIX Conf., pp. 119-130, June 1985.
[30] M. Stonebraker and G.A. Schloss, “Distributed RAID—A New Multiple Copy Algorithm,” Proc. Sixth Int'l Conf. Data Eng., pp. 430-437, Feb. 1990.
[31] N. Talagala, S. Asami, D. Patterson, and K. Lutz, “Tertiary Disk: Large Scale Distributed Storage,” Technical Report UCB//CSD-98-989, Univ. of California, Berkeley, 1998.
[32] C. Thekkath, T. Mann, and E. Lee, "Frangipani: A Scalable Distributed File System," Proc. 16th ACM Symp. Operating System Principles, ACM Press, New York, 1997, pp. 224-237.
[33] P.J. Varman and R.M. Verma, “Tight Bounds for Prefetching and Buffer Management Algorithms for Parallel I/O Systems,” IEEE Trans. Parallel and Distributed Systems, pp. 1262-1275, Dec. 1999.
[34] R.W. Watson and R.A. Coyne, “Parallel I/O Architecture of the High Performance Storage System,” Proc. 14th IEEE Symp. Mass Storage Systems, pp. 27-44, Sept. 1995.
[35] J. Wilkes, R. Golding, C. Staelin, and T. Sullivan, The HP Auto RAID Hierarchical Storage System ACM Trans. Computer Systems, vol. 14, pp. 108-136, Feb. 1996.

Index Terms:
Distributed computing, parallel I/O, software RAID, single I/O space, Linux clusters, fault tolerance, Andrew and Bonnie benchmarks, network file servers, scalability and overhead analysis.
Kai Hwang, Hai Jin, Roy S.C. Ho, "Orthogonal Striping and Mirroring in Distributed RAID for I/O-Centric Cluster Computing," IEEE Transactions on Parallel and Distributed Systems, vol. 13, no. 1, pp. 26-44, Jan. 2002, doi:10.1109/71.980025
Usage of this product signifies your acceptance of the Terms of Use.