This Article 
 Bibliographic References 
 Add to: 
NFS-CD: Write-Enabled Cooperative Caching in NFS
March 2008 (vol. 19 no. 3)
pp. 323-333
We present NFS-CD, an enhancement to the NFSv4 file system that reduces server load and increases the scalability of distributed file systems in computing clusters. The cluster delegation feature of NFS-CD allows data sharing among clients by extending the NFSv4 delegation model so that multiple clients manage a single file without interacting with the server. Based on cluster delegation, we implement a fast commit primitive, cooperative caching, and the ability to recover the uncommitted updates of a failed computer. NFS-CD supports both read and write operations in the cooperative cache without degrading the consistency model of NFSv4. We have implemented NFSCD by modifying the Linux NFSv4 client only. Because the server remains unchanged, NFS-CD preserves the simple administration model of NFSv4 and interoperates with standard NFS clients. NFS-CD offers improved performance when compared to NFSv4 at the expense of slightly weaker reliability guarantees.An experimental evaluation of our implementation, using industry standard benchmarks and application workloads, reveals that NFS-CD reduces server load by more than half. It also demonstrates that under most workloads file systems must support writes to the cooperative cache to achieve scale.

[1] A.D. Birrell, A. Hisgen, C. Jerian, T. Mann, and G. Swart, “The Echo Distributed File System,” Technical Report 111, Hewlett-Packard Labs, Palo Alto, Calif., Oct. 1993.
[2] L. Alvisi and K. Marzullo, “Message Logging: Pessimistic, Optimistic, and Causal,” Proc. 15th Int'l Conf. Distributed Computing Systems (ICDCS), 1995.
[3] T. Anderson, M. Dahlin, J. Neefe, D. Patterson, D. Roselli, and R. Wang, “Serverless Network File Systems,” Proc. 15th ACM Symp. Operating System Principles (SOSP), 1995.
[4] “Advanced Simulation and Computing, Purple,” Statement of Work,, 2007.
[5] B. Pawlowski, C. Juszczak, P. Staubach, C. Smith, D. Lebel, and D. Hitz, “NFS Version 3: Design and Implementation,” Proc. Usenix Summer Technical Conf., 1994.
[6] C. Lesniewski-Laas and M.F. Kaashoek, “SSL Splitting: Securely Serving Data from Untrusted Caches,” Proc. 12th Usenix Security Symp., 2003.
[7] P.M. Chen, E.K. Lee, G.A. Gibson, R.H. Katz, and D.A. Patterson, “RAID: High-Performance, Reliable Secondary Storage,” ACM Computing Surveys, vol. 26, no. 2, pp. 145-185, 1994.
[8] D. Noveck and R. Burnett, Implementation Guide for Referrals in NFSv4, RFC, 2005.
[9] F. Dabek, M.F. Kaashoek, D. Karger, R. Morris, and I. Stoica, “Wide-Area Cooperative Storage with CFS,” Proc. 18th ACM Symp. Operating Systems Principles (SOSP), 2001.
[10] E.W. Felten and J. Zahorjan, “Issues in the Implementation of a Remote Paging System,” Technical Report 91-03-09, Dept. Computer Science, Univ. of Washington, 1991.
[11] D. Ellard and M. Seltzer, “NFS Tricks and Benchmarking Traps,” Proc. FREENIX Technical Conf., 2003.
[12] B. Pawlowski et al., “The NFS Version 4 Protocol,” Proc. Second Int'l System Administration and Networking Conf. (SANE), 2000.
[13] I. Foster et al., “The Grid2003 Production Grid: Principles and Practice,” Proc. 13th IEEE Int'l Symp. High Performance Distributed Computing (HPDC-13), 2004.
[14] G.A. Gibson and D.A. Patterson, “Designing Disk Arrays for High Data Reliability,” J. Parallel and Distributed Computing, vol. 17, pp.4-27, Jan./Feb. 1993.
[15] G. Gibson, “Keynote Speech,” Proc. Third Ann. Usenix File and Storage Technologies Conf. (FAST), 2004.
[16] A. Haeberlen, A. Mislove, and P. Druschel, “Glacier: Highly Durable, Decentralized Storage despite Massive Correlated Failures,” Proc. Second Usenix Symp. Networked Systems Design and Implementation (NSDI), 2005.
[17] J.H. Howard, “An Overview of the Andrew File System,” Proc. Winter Usenix Conf., 1988.
[18] The IOzone Filesystem Benchmark, http:/, 2007.
[19] J. Griffioen, R. Vingralek, T.A. Anderson, and Y. Breitbart, “DERBY: A Memory Management System for Distributed Main Memory Databases,” Proc. Sixth Int'l Workshop Research Issues in Data Eng. (RIDE), 1996.
[20] K. Keeton and A. Merchant, “A Framework for Evaluating Storage System Dependability—Extended Version,” Technical Report HPL-2004-53, Hewlett-Packard Laboratories, 2004.
[21] K. Skadron and D.W. Clark, “Design Issues and Tradeoffs for Write Buffers,” Proc. Third Int'l Symp. High-Performance Computer Architecture (HPCA), 1997.
[22] M. Blaze and R. Alonso, “Dynamic Hierarchical Caching in Large-Scale Distributed File Systems,” Proc. 12th Int'l Conf. Distributed Computing Systems (ICDCS), 1992.
[23] M. Castro and B. Liskov, “Practical Byzantine Fault Tolerance,” Proc. Third Usenix Symp. Operating Systems Design and Implementation (OSDI), 1999.
[24] M.D. Schroeder, D.K. Gifford, and R.M. Needham, “A Caching File System for a Programmer's Workstation,” Proc. 10th ACM Symp. Operating Systems Principles (SOSP), 1985.
[25] M. Dahlin, C. Mather, R. Wang, T.E. Anderson, and D.A. Patterson, “A Quantitative Analysis of Cache Policies for Scalable Network File Systems,” Proc. ACM SIGMETRICS, 1994.
[26] M. Dahlin, R. Wang, T.E. Anderson, and D.A. Patterson, “Cooperative Caching: Using Remote Client Memory to Improve File System Performance,” Proc. First Usenix Symp. Operating Systems Design and Implementation (OSDI), 1994.
[27] M.J. Feeley, W.E. Morgan, F.H. Pighin, A.R. Karlin, H.M. Levy, and C.A. Thekkath, “Implementing Global Memory Management in a Workstation Cluster,” Proc. 15th Symp. Operating Systems Principles (SOSP), 1995.
[28] M. Nelson, B. Welch, and J. Ousterhout, “Caching in the Sprite Network File System,” ACM Trans. Computer Systems, vol. 6, no. 1, pp. 134-154, Feb. 1988.
[29] S. Narasimhan, S. Sohoni, and Y. Hu, “A Log-Based Write-Back Mechanism for Cooperative Caching,” Proc. 17th Int'l Parallel and Distributed Processing Symp. (IPDPS), 2003.
[30] O. Rodeh, U. Schonfeld, and A. Teperman, “zFS—A Scalable Distributed File System Using Object Disks,” Proc. 20th IEEE/11th NASA Goddard Conf. Mass Storage Systems and Technologies (MSST), 2003.
[31] OSDL, Linux IP Routing Utilities, http:/, 2007.
[32] P. Sarkar and J. Hartman, “Efficient Cooperative Caching Using Hints,” Proc. Second Usenix Symp. Operating Systems Design and Implementation (OSDI), 1996.
[33] N. Peyrouze and G. Muller, “FT-NFS: An Efficient Fault-Tolerant NFS Server Designed for Off-the-Shelf Workstations,” Proc. 26th Ann. Int'l Symp. Fault-Tolerant Computing (FTCS), 1996.
[34] R.Y. Wang, T.E. Anderson, and M.D. Dahlin, “Experience with a Distributed File System Implementation with Adaptive Methods Global Address-Based Communication in Large-Scale Parallel Machines,” Technical Report CSD-98-986, Univ. of Texas at Austin, 1998.
[35] M.J. Freedman, S. Annapureddy, and D. Mazieres, “SHARK: Scaling File Servers via Cooperative Caching,” Proc. Second Usenix/ACM Symp. Networked Systems Design and Implementation (NSDI), 2005.
[36] S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, and D. Noveck, NFSv4, RFC 3530, 2003.
[37] R. Sandberg, D. Goldberg, S. Kleiman, D. Walsh, and B. Lyon, “Design and Implementation of the Sun Network Filesystem,” Proc. Summer Usenix Conf., 1985.
[38] S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, and D. Noveck, NFSv4, RFC 3530, 2003.
[39] T.A. Anderson and J. Griffioen, “An Application-Aware Data Storage Model,” Proc. Usenix Ann. Technical Conf., 1999.
[40] T.M. Wong and J. Wilkes, “My Cache or Yours? Making Storage More Exclusive,” Proc. Usenix Ann. Technical Conf., 2002.
[41] VMware Inc., The VMware GSX Server, http:/www.vmware. com, 2007.
[42] Y. Xu and B.D. Fleisch, “NFS-cc: Tuning NFS for Concurrent Read Sharing,” Int'l J. High Performance Computing and Networking, vol. 3, 2004.

Alexandros Batsakis, Randal Burns, "NFS-CD: Write-Enabled Cooperative Caching in NFS," IEEE Transactions on Parallel and Distributed Systems, vol. 19, no. 3, pp. 323-333, March 2008, doi:10.1109/TPDS.2008.14
Usage of this product signifies your acceptance of the Terms of Use.