The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.03 - July-September (2008 vol.1)
pp: 141-154
Youssef Achbany , Université Catholique de Louvain, Louvain-la-Neuve
Ivan J. Jureta , PReCISE, University of Namur, Namur
Stephane Faulkner , PReCISE, University of Namur, Namur
Francois Fouss , Facultés Univeristaires Catholiques de Mons, Namur
ABSTRACT
Open service-oriented systems which autonomously and continually satisfy users' service requests to optimal levels are an appropriate response to the need for increased automation of information systems. Given a service request, an open service-oriented system interprets the functional and nonfunctional requirements laid out in the request and identifies the optimal selection of services—that is, identifies services. These services' coordinated execution optimally satisfies the requirements in the request. When selecting services, it is relevant to: (1) revise selections as new services appear and others become unavailable; (2) use multiple criteria, including nonfunctional ones to choose among competing services; (3) base the comparisons of services on observed, instead of advertised performance; and (4) allow for uncertainty in the outcome of service executions. To address issues (1)–(4), we propose the Multi-Criteria Randomized Reinforcement Learning (MCRRL) service selection approach. MCRRL learns and revises service selections using a novel multicriteria-driven (including quality of service parameters, deadline, reputation, cost, and preferences) reinforcement learning algorithm, which integrates the exploitation of data about individual services' past performance with optimal, undirected, continual exploration of new selections that involve services whose behavior has not been observed. The experiments indicate the algorithm behaves as expected and outperforms two standard approaches.
INDEX TERMS
Learning, Optimization of Services Systems
CITATION
Youssef Achbany, Ivan J. Jureta, Stephane Faulkner, Francois Fouss, "Continually Learning Optimal Allocations of Services to Tasks", IEEE Transactions on Services Computing, vol.1, no. 3, pp. 141-154, July-September 2008, doi:10.1109/TSC.2008.12
REFERENCES
[1] S. Abdallah and V. Lesser, “Modeling Task Allocation Using a Decision Theoretic Model,” Proc. Fourth Int'l Joint Conf. Autonomous Agents and Multi-Agent Systems (AAMAS '05), 2005.
[2] S. Abdallah and V. Lesser, “Learning the Task Allocation Game,” Proc. Fifth Int'l Joint Conf. Autonomous Agents and Multi-Agent Systems (AAMAS '06), 2006.
[3] Y. Achbany, F. Fouss, L. Yen, A. Pirotte, and M. Saerens, “Tuning Continual Exploration in Reinforcement Learning,” Neurocomputing, accepted.
[4] Amazon S3 Developer Guide, Amazon Web Services LLC, 2007.
[5] O. Babaoglu, H. Meling, and A. Montresor, “Anthill: A Framework for the Development of Agent-Based Peer-to-Peer Systems,” Proc. 22nd Int'l Conf. Distributed Computing Systems (ICDCS '02), 2002.
[6] J. Bather, Decision Theory: An Introduction to Dynamic Programming and Sequential Decisions. Wiley, 2000.
[7] Distributed and Parallel Database, B. Benatallah and F. Casati, eds., special issue on web services, Springer-Verlag, 2002.
[8] B. Benatallah, M.-S. Hacid, A. Leger, C. Rey, and F. Toumani, “On Automating Web Services Discovery,” The VLDB J., vol. 14, pp. 84-96, 2005.
[9] T. Berners-Lee, J. Hendler, and O. Lassila, “The Semantic Web,” Scientific Am., pp. 7-15, 2001.
[10] D.P. Bersekas, Dynamic Programming and Optimal Control. Athena Scientific, 2000.
[11] “Object Management Group/Business Process Management Initiative,” Business Process Modeling Notation Specification, Final Adopted Specification dtc/06-02-01, 2006.
[12] M. Brambilla, S. Ceri, P. Fraternali, and I. Manolescu, “Process Modeling in Web Applications,” ACM Trans. Software Eng. and Methodology, vol. 15, no. 4, pp. 360-409, 2006.
[13] G. Canfora, M. Di Penta, R. Esposito, and M.L. Villani, “ALightweight Approach for QoS-Aware Service Composition,” The VLDB J., Proc. Second Int'l Conf. Service Oriented Computing (ICSOC '04), F. Casati, M.-C. Shan, and D. Georgakopoulos, eds., vol. 10, no. 1, 2001.
[14] N. Christofides, Graph Theory: An Algorithmic Approach. Academic Press, 1975.
[15] T.M. Cover and J.A. Thomas, Elements of Information Theory. John Wiley & Sons, 1991.
[16] A. D'Ambrogio, “A Model-Driven WSDL Extension for Describing the QoS of Web Services,” Proc. Int'l Conf. Web Services (ICWS), 2006.
[17] S. Shaheen Fatima and M. Wooldridge, “Adaptive Task and Resource Allocation in Multi-Agent Systems,” Proc. Fifth Int'l Conf. Autonomous Agents, 2001.
[18] Google's Developer Network, Google Inc., 2007.
[19] H. Hannah and A.-I. Mouaddib, “Task Selection Problem under Uncertainty as Decision-Making,” Proc. First Int'l Joint Conf. Autonomous Agents and Multi-Agent Systems (AAMAS '02), 2002.
[20] “International Business Machines. Service-Oriented Architecture,” IBM Systems J., vol. 44, no. 4, 2005.
[21] N.R. Jennings, “On Agent-Based Software Engineering,” Artificial Intelligence, vol. 117, pp. 277-296, 2000.
[22] J.N. Kapur and H.K. Kesavan, Entropy Optimization Principles with Applications. Academic Press, 1992.
[23] A. Keller and H. Ludwig, “The WSLA Framework: Specifying and Monitoring Service Level Agreements for Web Services,” J. Network and Systems Management, vol. 11, no. 1, 2003.
[24] J.O. Kephart and D.M. Chess, “The Vision of Autonomic Computing,” Computer, vol. 36, no. 1, pp. 41-52, Jan. 2003.
[25] F. Klein and M. Tichy, “Building Reliable Systems Based on Self-Organizing Multi-Agent Systems,” Proc. Fifth Int'l Workshop Software Eng. for Large-Scale Multi-Agent Systems (SELMAS '06), pp. 51-57, 2006.
[26] L. Li and I. Horrocks, “A Software Framework for Matchmaking Based on Semantic Web Technology,” Proc. 12th Int'l World Wide Web Conf. (WWW '03), pp. 331-339, May 2003.
[27] E.M. Maximilien and M.P. Singh, “Toward Autonomic Web Services Trust and Selection,” Proc. Second Int'l Conf. Service Oriented Computing (ICSOC '04), 2004.
[28] E.M. Maximilien and M.P. Singh, “Multiagent System for Dynamic Web Services Selection,” Proc. Fourth Int'l Joint Conf. Autonomous Agents and Multi-Agent Systems (AAMAS '05), 2005.
[29] S.A. McIlraith, T.C. Son, and H. Zeng, “Semantic Web Services,” IEEE Intelligence System, vol. 16, no. 2, pp. 46-53, 2001.
[30] S.A. McIlraith and T.C. Son, “Adapting Golog for Composition of Semantic Web Services,” Proc. Eighth Int'l Conf. Knowledge Representation and Reasoning (KR '02), 2002.
[31] B. Medjahed, A. Bougettaya, and A.K. Elmagarmid, “Composing Web Services on the Semantic Web,” The VLDB J., vol. 12, pp. 333-351, 2003.
[32] T.M. Mitchell, Machine Learning. McGraw-Hill, 1997.
[33] D. Martin, M. Burstein, J. Hobbs, O. Lassila, D. McDermott, S. McIlraith, S. Narayanan, M. Paolucci, B. Parsia, T. Payne, E. Sirin, N. Srinivasan, and K. Sycara, OWL-S: Semantic Markup for Web Services, W3C Subm., Nov. 2004.
[34] M. Paolucci, T. Kawamura, T.R. Payne, and K. Sycara, “Semantic Matching of Web Services Capabilities,” Proc. First Int'l Semantic Web Conf. Semantic Web (ISWC '02), pp. 318-332, June 2002.
[35] M.P. Papazoglou and D. Georgakopoulos, “Service-Oriented Computing,” Comm. ACM, vol. 46, no. 10, pp. 25-28, 2003.
[36] K. Rohanimanesh and S. Mahadevan, “Decision Theoretic Planning with Concurrent Temporally Extended Actions,” Proc. 17th Conf. Uncertainty in Artificial Intelligence (UAI), 2001.
[37] K. Rohanimanesh and S. Mahadevan, “Learning to Take Concurrent Actions,” Proc. 16th Int'l Conf. Neural Information Processing Systems (NIPS), 2003.
[38] Y. Achbany, F. Fouss, L. Yen, A. Pirotte, and M. Saerens, “Optimal Tuning of Continual Online Exploration in Reinforcement Learning,” Proc. 16th Int'l Conf. Artificial Neural Networks (ICANN '06), 2006.
[39] R.S. Sutton and A.G. Barto, Reinforcement Learning: An Introduction. MIT Press, 1998.
[40] R.S. Sutton, D. Precup, and S.P. Singh, “Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning,” Artificial Intelligence, vol. 112, nos. 1/2, 1999.
[41] K. Sycara, M. Klush, and S. Widoff, “Dynamic Service Matchmaking among Agents in Open Information Environments,” ACM SIGMOD Record, vol. 28, no. 1, pp. 47-53, 1999.
[42] G. Tesauro, D.M. Chess, W.E. Walsh, R. Das, A. Segal, I. Whalley, J.O. Kephart, and S.R. White, “A Multi-Agent Systems Approach to Autonomic Computing,” Proc. Third Int'l Joint Conf. Autonomous Agents and Multi-Agent Systems (AAMAS '04), 2004.
[43] D. Tennenhouse, “Proactive Computing,” Comm. ACM, vol. 42, no. 5, 2000.
[44] S. Thrun, “Efficient Exploration in Reinforcement Learning,” technical report, School of Computer Science, Carnegie Mellon Univ., 1992.
[45] S. Thrun, “The Role of Exploration in Learning Control,” Handbook for Intelligent Control: Neural, Fuzzy and Adaptive Approaches, D.White and D. Sofge, eds., Van Nostrand Reinhold, 1992.
[46] S. Thrun, W. Burgard, and D. Fox, Probabilistic Robotics. MIT Press, 2005.
[47] B. Urgaonkar, G. Pacifici, P. Shenoy, M. Spreitzer, and A. Tantawi, “An Analytical Model for Multi-Tier Internet Services and Its Applications,” Proc. SIGMETRICS, 2005.
[48] K. Verbeeck, “Coordinated Exploration in Multi-Agent Reinforcement Learning,” PhD thesis, Vrije Universiteit Brussel, 2004.
[49] P. Vincke, Multicriteria Decision-Aid. Wiley, 1992.
[50] G. Weikum, ed., “Special Issue on Organizing and Discovering the Semantic Web,” IEEE Data Eng. Bull., vol. 25, no. 1, pp. 1-58, 2002.
[51] W3C, Simple Object Access Protocol (SOAP), 2003.
[52] W3C, Universal Description, Discovery, and Integration (UDDI), 2003.
[53] W3C, Web Services Description Language (WSDL), 2003.
[54] L. Zeng, B. Benatallah, M. Dumas, J. Kalagnanam, and Q.Z. Sheng, “Quality Driven Web Services Composition,” Proc. 12th Int'l World Wide Web Conf. (WWW '03), 2003.
[55] C. Zhou, L.-T. Chia, and B.-S. Lee, “DAML-QoS Ontology for Web Services,” Proc. Int'l Conf. Web Services (ICWS '04), 2004.
[56] A. Ankolenkar, M. Burstein, J.R. Hobbs, O. Lassila, D.L. Martin, D. McDermott, S.A. McIlraith, S. Narayanan, M. Paolucci, T.R. Payne, and K. Sycara, OWL-S: Semantic Markup for Web Services 1.1. DAML Services Coalition, 2004.
[57] G. Antoniou and F. van Harmelen, “Web Ontology Language: Owl,” Handbook on Ontologies in Information Systems, S. Staab and R. Studer, eds., pp. 67-92, Springer-Verlag, 2003.
[58] S. Battle, A. Bernstein, H. Boley, B. Grosof, M. Gruninger, R. Hull, M. Kifer, D. Martin, S. McIlraith, D. McGuinness, J. Su, and S. Tabet, Proc. W3C Workshop Frameworks for Semantics in Web Services (SWSF '05), 2005.
[59] S. Bechhofer, F. van Harmelen, J. Hendler, I. Horrocks, D.L. McGuiness, P.F. Patel-Schneider, and L.A. Stein, “OWL Web Ontology Language,” World Wide Web Consortium, 2004.
[60] B. Benatallah, M.-S. Hacid, A. Leger, C. Rey, and F. Toumani, “OnAutomating Web Services Discovery,” The VLDB J., vol. 14, 2005.
[61] B. Benatallah and F. Casati, Guest editorial, Distributed and Parallel Databases, vol. 12, nos. 2/3, pp. 115-116, 2002.
[62] D. Berardi, M. Gruninger, R. Hull, and S. McIlraith, “Towards a First-Order Ontology for Semantic Web Services,” Proc. W3C Workshop Constraints and Capabilities for Web Services, 2005.
[63] T. Berners-Lee, J. Hendler, and O. Lassila, “The Semantic Web,” Scientific Am., May 2001.
[64] F. Casati, M.-C. Shan, and D. Georgakopoulos, The VLDB J., guest editorial, vol. 10, no. 1, p. 1, 2001.
[65] E. Christensen, F. Curbera, G. Meredith, and S. Weerawarana, Web Services Description Language (WSDL 1.1), 2001.
[66] I. Horrocks, “DAML$+$ OIL: A Description Logic for the Semantic Web,” IEEE Data Eng. Bull., vol. 25, no. 1, pp. 4-9, 2002.
[67] I.J. Jureta, S. Faulkner, Y. Achbany, and M. Saerens, “Dynamic Web Service Composition within a Service-Oriented Architecture,” Proc. Int'l Conf. Web Services (ICWS '07), 2007.
[68] I.J. Jureta, S. Faulkner, Y. Achbany, and M. Saerens, “Dynamic Task Allocation within an Open Service-Oriented MAS Architecture,” Proc. Sixth Int'l Joint Conf. Autonomous Agents and Multi-Agent Systems (AAMAS '07), 2007.
[69] J.O. Kephart and D.M. Chess, “The Vision of Autonomic Computing,” Computer, vol. 36, no. 1, pp. 41-50, Jan. 2003.
[70] S. McIlraith and T. Cao Son, “Adapting Golog for Composition of Semantic Web Services,” Proc. Eighth Int'l Conf. Knowledge Representation and Reasoning (KR '02), 2002.
[71] S.A. McIlraith and D.L. Martin, “Bringing Semantics to Web Services,” IEEE Intelligent Systems, vol. 18, no. 1, pp. 90-93, Jan./Feb. 2003.
[72] S.A. McIlraith, T.C. Son, and H. Zeng, “Semantic Web Services,” IEEE Intelligent Systems, vol. 16, no. 2, Mar./Apr. 2001.
[73] B. Medjahed, A. Bougettaya, and A.K. Elmagarmid, “Composing Web Services on the Semantic Web,” The VLDB J., vol. 12, 2003.
[74] S. Narayanan and S.A. McIlraith, “Simulation, Verification and Automated Composition of Web Services,” Proc. 11th Int'l World Wide Web Conf. (WWW '02), 2002.
[75] M.P. Papazoglou and D. Georgakopoulos, “Service-Oriented Computing,” Comm. ACM, vol. 46, no. 10, pp. 24-28, 2003.
[76] N. Shadbolt, T. Berners-Lee, and W. Hall, “The Semantic Web Revisited,” IEEE Intelligent Systems, vol. 21, no. 3, May/June 2006.
[77] “Handbook on Ontologies,” Int'l Handbooks on Information Systems, S. Staab and R. Studer, eds., Springer, 2004.
[78] D. Tennenhouse, “Proactive Computing,” Comm. ACM, vol. 43, no. 5, pp. 43-50, 2000.
[79] W3C, Simple Object Access Protocol (SOAP), 2003.
[80] W3C, Universal Description, Discovery, and Integration (UDDI), 2003.
[81] G. Zacharia and P. Maes, “Trust Management through Reputation Mechanisms,” Applied Artificial Intelligence, vol. 14, 2000.
[82] “Foundation for Intelligent Physical Agents,” FIPA Quality of Service Ontology Specification, Doc. SC00094A, 2002.
[83] Y. Liu, A.H. Ngu, and L.Z. Zeng, “QoS Computation and Policing in Dynamic Web Service Selection,” Proc. 13th Int'l World Wide Web Conf. (WWW '04), pp. 66-73, 2004.
[84] F. Naumann, U. Leser, and J.C. Freytag, “Quality-Driven Integration of Heterogenous Information Systems,” Proc. Int'l Conf. Very Large Data Bases (VLDB '99), pp. 447-458, 1999.
[85] H. Tong and S. Zhang, “A Fuzzy Multi-Attribute Decision Making Algorithm for Web Services Selection Based on QoS,” Proc. IEEE Asia-Pacific Conf. Services Computing (APSCC '06), pp. 51-57, 2006.
[86] T. Kawamura, J.A. De Blasio, T. Hasegawa, M. Paolucci, and K. Sycara, “Public Deployment of Semantic Service Matchmaker with UDDI Business Registry,” Proc. Eighth IEEE Int'l Symp. Wearable Computers (ISWC '04), 2004.
[87] R. Akkiraju, R. Goodwin, P. Doshi, and S. Roeder, “A Method for Semantically Enhancing the Service Discovery Capabilities of UDDI,” Proc. Workshop Information Integration on the Web (IIWeb), 2003.
[88] K. Verma, K. Sivashanmugam, A. Sheth, A. Patil, S. Oundhakar, and J. Mille, “METEOR-S WSDI: A Scalable Infrastructure of Registries for Semantic Publication and Discovery of Web Services,” J. Information Technology and Management, special issue on universal global integration, vol. 6, no. 1, pp. 17-39, 2005.
[89] S. Casare and J. Sichman, “Towards a Functional Ontology of Reputation,” Proc. Fourth Int'l Joint Conf. Autonomous Agents and Multi-Agent Systems (AAMAS '05), 2005.
[90] E.M. Maximilien and M.P. Singh, “Conceptual Model of Web Service Reputation,” ACM SIGMOD Record, vol. 31, no. 4, 2002.
[91] I.J. Jureta, C. Herssens, and S. Faulkner, “A Comprehensive Quality Model for Service-Oriented Systems,” Software Quality J., accepted.
[92] C. Herssens, I.J. Jureta, and S. Faulkner, “Capturing and Using QoS Relationships to Improve Service Selection,” Proc. 20th Int'l Conf. Advanced Information Systems Eng. (CAiSE '08), 2008.
[93] F. Casati, M. Castellanos, U. Dayal, and M.-C. Shan, “Probabilistic, Context-Sensitive, and Goal-Oriented Service Selection,” Proc. Second Int'l Conf. Service Oriented Computing (ICSOC '04), 2004.
[94] H. Tong and S. Zhang, “A Fuzzy Multi-Attribute Decision Making Algorithm for Web Services Selection Based on QoS,” Proc. IEEE Asia-Pacific Conf. Services Computing (APSCC '06), pp. 51-57, 2006.
[95] Y. Liu, A.H. Ngu, and L.Z. Zeng, “QoS Computation and Policing in Dynamic Web Service Selection,” Proc. 13th Int'l World Wide Web Conf. (WWW '04), 2004.
[96] L.-J. Zhang and B. Li, “Requirements Driven Dynamic Services Composition for Web Services and Grid Solutions,” J. Grid Computing, vol. 2, no. 2, pp. 121-140, 2004.
16 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool