This Article 
 Bibliographic References 
 Add to: 
Scalable Consistency Protocols for Distributed Services
September 1999 (vol. 10 no. 9)
pp. 888-903

Abstract—A common way to address scalability requirements of distributed services is to employ server replication and client caching of objects that encapsulate the service state. The performance of such a system could depend very much on the protocol implemented by the system to maintain consistency among object copies. We explore scalable consistency protocols that never require synchronization and communication between all nodes that have copies of related objects. We achieve this by developing a novel approach called local consistency (LC). LC based protocols can provide increased flexibility and efficiency by allowing nodes control over how and when they become aware of updates to cached objects. We develop two protocols for implementing strong consistency using this approach and demonstrate that they scale better than a traditional invalidation based consistency protocol along the system load and geographic distribution dimensions of scale.

[1] S.V. Adve and M.H. Hill, “Implementing Sequential Consistency in Cache-Based Systems,” Proc. Int'l Conf. Parallel Processing, 1990.
[2] M. Ahamad, G. Neiger, P. Hutto, P. Kohli, and J. Burns, “Causal Memory: Definitions and Implementations,” Distributed Computing, 1995.
[3] M. Baker, J.H. Hartman, M.D. Kupfer, K.W. Shirriff, and J. Ousterhout, "Measurements of a Distributed File System," Proc. 13th ACM Symp. Operating Systems Principles, pp. 198-211, Oct. 1991.
[4] M.A. Blaze, "Caching in Large-Scale Distributed File Systems," PhD thesis, Princeton Univ., Jan. 1993.
[5] R.R. Bodnarchuk and F.B. Bunt, “A Synthetic Workload Model for a Distributed System File Server,” Proc. ACM SIGMETRICS, 1991.
[6] V. Cate, “Alex—A Global File System,” Proc. 1992 USENIX File System Workshop, 1992.
[7] M.D. Dahlin, R.Y. Wang, T.E. Anderson, and D.A. Patterson, “Cooperative Caching: Using Remote Client Memory to Improve File System Performance,” Proc. Operating Systems Design and Implementation, 1994.
[8] M.D. Dahlin, C.J. Mather, R.Y. Wang, T.E. Anderson, and D.A. Patterson, "A quantitative analysis of cache policies for scalable network file systems," Proc. ACM SIGMETRICS,Nashville, Tenn., pp. 150-160, May 1994.
[9] K.P. Eswaran, J.N. Gray, R.A. Lorie, and I.L. Traiger, "The Notions of Consistency and Predicate Locks in a Relational Database System," Comm. ACM, vol. 8, no. 11, pp. 624-633, 1976.
[10] C. Fidge, “Timestamps in Message-Passing Systems that Preserve the Partial Ordering, Proc. Australian Computer Science Conf., 1988.
[11] K. Gharachorloo, D. Lenoski, J. Laudon, P. Gibbons, A. Gupta, and J. Hennessy, “Memory Consistency and Event Ordering in Scalable Shared-Memory Multiprocessors,” Proc. 17th Ann. Int'l Symp. Computer Architecture, 1990.
[12] D.S. Gill, S. Zhou, and H.S. Sandhu, “A Case Study of File System Workload in a Large-Scale Distributed Environment,” Univ. of Toronto, technical report, 1994.
[13] D.S. Gill, S. Zhou, and H.S. Sandhu, "A Case Study of File System Workload in a Large-Scale Distributed Environment," Proc. ACM SIGMETRICS Conf. Measurement and Modelling of Computer Systems, ACM Press, New York, 1994, pp. 276-277.
[14] C. Gray and D. Cheriton, "Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency," Proc. 12th Int'l Symp. Operating System Principles, 1989.
[15] J. Gwerzman and M. Seltzer, “World Wide Web Cache Consistency,” Proc. 1996 Usenix Technical Conf., Jan. 1996.
[16] M. Herlihy and J. Wing, “Linearizability: A Correctness Condition for Concurrent Objects,” ACM Trans. Programming Languages and Systems, vol. 12, no. 3,pp. 463-492, 1990.
[17] J.H. Howard, M.L. Kazar, S.G. Menees, D.A. Nichols, M. Satyanarayanan, R.N. Sidebotham, and M.J. West, "Scale and performance in a distributed file system," ACM Trans. Comp. Sys., vol. 6, no. 1, Feb. 1988.
[18] A.D. Joseph, A.F. de Lespinasse, J.A. Tauber, D.K. Gifford, and M.F. Kaashoek, “Rover: A Toolkit for Mobile Information Access,” Proc. ACM Symp. Operating Systems Principles, 1995.
[19] P. Keleher, A.L. Cox, and W. Zwaenepoel, “Lazy Release Consistency for Software Distributed Shared Memory,” Proc. 19th Ann. Int'l Symp. Computer Architecture, pp. 13-21, May 1992.
[20] J. Kistler and M. Satyanarayanan, "Disconnected Operation in the Coda File System," Proc. 13th Int'l Symp. Operating System Principles,Pacific Grove, Calif., Oct. 1991.
[21] R. Kordale, “System Support for Scalable Services,” PhD thesis,Georgia Inst. of Tech nology, 1997.
[22] R. Kordale and M. Ahamad, “Object Caching in a CORBA Compliant System,” Usenix Systems J., 1996.
[23] L. Lamport, “How to Make a Multiprocessor Computer that Correctly Executes Multiprocess Programs,” IEEE Trans. Computers, Sept. 1979.
[24] S. Landis and S. Maffeis, “Building Reliable Distributed Systems with CORBA,” Proc. USENIX Conf. Object-Oriented Technologies and Systems (COOTS), 1995.
[25] A.R. Lebeck and D.A. Wood, “Dynamic Self-Invalidation: Reducing Coherence Overhead in Shared-Memory Multiprocessors,” Proc. 22nd Int'l Symp. Computer Architecture, pp. 48-59, New York, 22-24 June 1995.
[26] B. Liskov, A. Adya, M. Castro, M. Day, S. Ghemawat, R. Gruber, U. Maheshwari, A. Myers, and L. Shrira, “Safe and Efficient Sharing of Persistent Objects in Thor,” Proc. SIGMOD '96, pp. 318–329, June 1996.
[27] C. Liu and P. Cao, "Maintaining Strong Cache Consistency in the World-Wide Web," Proc. Int'l Conf. Distributed Computing Systems, 1997.
[28] J. Misra, “Axioms for Memory Access in Asynchronous Hardware Systems,” ACM Trans. Programming Languages and Systems, vol. 8, no. 1, pp. 142–153, Jan. 1986.
[29] M. Mizuno, M. Raynal, and J. Zhou, Sequential Consistency in Distributed Systems Proc. Int'l Workshop Theory and Practice in Distributed Systems, pp. 224-241, 1994.
[30] F. Mattern, “Time and Global States of Distributed Systems,” Proc. Int'l Workshop Parallel and Distributed Algorithms, 1989.
[31] M.N. Nelson, M. Linton, and S. Owicki, “A Highly Available, Scalable ITV System,” Proc. 15th ACM Symp. Operating Systems Principles, pp. 54-67, 1995.
[32] M. Raynal and A. Schiper, “From Causal Consistency to Sequential Consistency in Shared Memory Systems,” Technical Report no. 926, IRISA, France.
[33] D. Terry, A. Demers, K. Petersen, M. Spreitzer, M. Theimer, and B. Welch, Session Guarantees for Weakly Consistent Replicated Data Proc. Int'l Conf. Parallel and Distributed Information Systems, pp. 140-149, Sept. 1994.
[34] D.B. Terry, M.M. Theimer, K. Petersen, A.J. Demers, M.J. Spreitzer, and C.H. Hauser, Managing Update Conflicts in a Weakly Connected Replicated Storage System Proc. ACM Symp. Operating Systems Principles, 1995.
[35] F.J. Torres-Rojas and M. Ahamad, “Plausible Clocks: Constant Size Logical Clocks for Distributed Systems,” Proc. 10th Workshop Distributed Algorithms (WDAG), Italy, Oct. 1996.
[36] R.Y. Wang and T.E. Anderson, “xFS: A Wide Area Mass Storage File System,” Proc. Fourth Workshop Workstation Operating Systems, Oct. 1993.
[37] K. Worrell, “Invalidation in Large Scale Network Object Caches,” masters thesis, Univ. of Colorado, 1994.

Index Terms:
Scalable services, distributed objects, replication, caching, consistency protocols.
Mustaque Ahamad, Rammohan Kordale, "Scalable Consistency Protocols for Distributed Services," IEEE Transactions on Parallel and Distributed Systems, vol. 10, no. 9, pp. 888-903, Sept. 1999, doi:10.1109/71.798314
Usage of this product signifies your acceptance of the Terms of Use.