This Article 
 Bibliographic References 
 Add to: 
QUEM: An Achievement Test for Knowledge-Based Systems
November-December 1997 (vol. 9 no. 6)
pp. 838-847

Abstract—This paper describes the QUality and Experience Metric (QUEM), a method for estimating the skill level of a knowledge-based system based on the quality of the solutions it produces. It allows one to assess how many years of experience the system would be judged to have if it were a human by providing a quantitative measure of the system's overall competence. QUEM can be viewed as a type of achievement or job-placement test administered to knowledge-based systems to help system designers determine how the system should be used and by what level of user. To apply QUEM, a set of subjects, experienced judges, and problems must be identified. The subjects should have a broad range of experience levels. Subjects and the knowledge-based system are asked to solve the problems; and judges are asked to rank order all solutions—from worst quality to best. The data from the subjects is used to construct a skill-function relating experience to solution quality, and confidence bands showing the variability in performance. The system's quality ranking is then plugged into the skill function to produce an estimate of the system's experience level. QUEM can be used to gauge the experience level of an individual system, to compare two systems, or to compare a system to its intended users. This represents an important advance in providing quantitative measures of overall performance that can be applied to a broad range of systems.

[1] J.S. Aikins, "Representation of Control Knowledge in Expert Systems," Proc. First AAAI, pp. 121-123,Stanford, Calif., 1981.
[2] J.R. Anderson, "Development of Expertise," Readings in Knowledge Acquisition," B.G. Buchanan et al., eds., Morgan Kaufmann, San Mateo, Calif., pp. 61-77, 1993.
[3] C. Baykan and M. Fox, "WRIGHT: A Constraint-Based Spatial Layout System," C. Tong and D. Sriram, eds., Artificial Intelligence in Eng. Design, vol. 1, ch. 13, pp. 395-432, Academic Press, San Diego, 1992.
[4] D.N. Card, "What Makes for Effective Measurement?" IEEE Software, No. 5, Nov. 1993, pp. 94-95.
[5] The Nature of Expertise, M. Chi, R. Glasser, and M.J. Farr., eds. Hillsdale, N.J.: Lawrence Erlbaum Associates, 1988.
[6] W.J. Clancey, "Acquiring, Representing, and Evaluating a Competence Model of Diagnostic Strategy," B.G. Buchanan et al., eds. Readings in Knowledge Acquisition, pp. 178-215, Morgan Kaufmann, San Mateo, Calif., 1993.
[7] P. Cohen, T. Dean, Y. Gil, M. Ginsberg, and L. Hoebel, "Handbook of Evaluation for the ARPA/Rome Lab Planning Initiative," Proc. Workshop ARPA/Rome Laboratory Knowledge-Based Planning and Scheduling Initiative,Tucson, Ariz., Morgan Kaufmann, Feb. 1994.
[8] J.R. Dixon, A. Howe, P.R. Cohen, and M.K. Simmons, "Dominic I: Progress toward Domain Independence in Design by Iterative Redesign," Eng. with Computers, vol. 2, pp. 137-145, 1987.
[9] C. Fiebig and C.C. Hayes, "Modeling the Development from Novice to Expert Human Planners," Proc. IEEE Conf. Systems, Man, and Cybernetics,Bejing, China, 1997.
[10] C.C. Hayes, "Observing Machinists' Planning Methods: Using Goal Interactions to Guide Search," Proc. Ninth Ann. Conf. Cognitive Science Soc., pp. 952-958,Seattle, 1987.
[11] C.C. Hayes, "Machine Planning: A Model of an Expert Level Planning Process," PhD thesis, Carnegie Mellon Univ., Pittsburgh, Penn., May 1990.
[12] C.C. Hayes and H. Sun, "Reasoning About Manufacturability Prior to Reaching the Shop Floor," Proc. SIGMAN Workshop Reasoning about the Shop Floor,Seattle, 1994.
[13] J.S. Lancaster and J.L. Kolodner, "Problem Solving in a Natural Task as a Function of Experience," Proc. Ninth Ann. Conf. Cognitive Science Soc., pp. 727-736,Seattle, 1987.
[14] N.E. Lane, "Global Issues in Evaluation of Expert Systems," Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics," pp. 121-125,Atlanta, 1986.
[15] J. Liebowitz, "Useful Approaches for Evaluating Expert Systems," Expert Systems, vol. 3, no. 2, pp. 86-96, 1986.
[16] J. MacMillan, E.B. Entin, and D. Serfaty, "Evaluating Expertise in a Complex Domain—Mesures Based on Theory," Proc. Human Factors and Ergonomics Soc.," pp. 1,152-1,155, 1993.
[17] J. Mostow, M. Barley, and T. Weinrich, "Automated Reuse of Design Plans in BOGART," C. Tong and D. Sriram, eds., Artificial Intelligence in Eng. Design, vol. 2, ch. 2, pp. 57-104, Academic Press, San Diego, 1992.
[18] D.L. Nazareth and M.H. Kennedy, "Knowledge-Based System Verification, Validation, and Testing," Int'l J. Expert Systems, vol. 6, no. 2, pp. 143-162, 1993.
[19] L.I. Steinberg, "Design as Top-Down Refinement Plus Constraint Propegation," C. Tong and D. Sriram, eds., Artificial Intelligence in Eng. Design, vol. 1, ch. 8, pp. 251-272, Academic Press, San Diego, 1992.

Index Terms:
Knowledged-based systems, expertise, performance measures, knowledge engineering, solution quality.
Caroline C. Hayes, Michael I. Parzen, "QUEM: An Achievement Test for Knowledge-Based Systems," IEEE Transactions on Knowledge and Data Engineering, vol. 9, no. 6, pp. 838-847, Nov.-Dec. 1997, doi:10.1109/69.649311
Usage of this product signifies your acceptance of the Terms of Use.