This Article 
 Bibliographic References 
 Add to: 
Towards Benchmarks for Knowledge Systems and Their Implications for Data Engineering
March 1989 (vol. 1 no. 1)
pp. 101-110

The author suggests a new focus on benchmarks for knowledge systems, following the lines of similar benchmarks in other computing fields. It is noted that knowledge systems differ from conventional systems in a key way, namely their ability to interpret and apply knowledge. This gives rise to a distinction between intrinsic measures concerned with engineering qualities and extrinsic measures relating to task productivity, and both warrant improved measurement techniques. Primary concerns within the extrinsic realm include advice quality, reasoning correctness, robustness, and solution efficiency. Intrinsic concerns, on the other hand, center on elegance of knowledge base design, modularity, and architecture. The author suggests criteria for good measures and benchmarks, and ways to satisfy these through the design of knowledge and key knowledge engineering costs and performance parameters. It is suggest that the focus on measuring knowledge systems should help clarify the technical relationships between knowledge engineering and data engineering.

[1] E. Feigenbaum, P. McCorduck, and H. P. Nii,The Rise of the Expert Company. New York: Times, 1988.
[2] F. Hayes-Roth, "Using proofs and refutations to learn from experience," inMachine Learning, R. S. Michalski, J. G. Carbonell, and T. M. Mitchell, Eds. Palo Alto, CA: Tioga, 1983, pp. 221-240.
[3] J.H. Fetzer, "Program Verification: The Very Idea,"Comm. ACM, Sept. 1988, pp. 1,048-1,063.
[4] F. Hayes-Roth and P. London, "Software tool speeds expert systems,"Syst. Software, vol. 71, pp. 71-75, Aug. 1985.
[5] B. Hayes-Roth, "A Blackboard Architecture for Control,"Artificial Intelligence, Vol. 26, No. 3, 1985, pp. 251-321.
[6] F. Hayes-Rothet al., Building Expert Systems. New York: Addison Wesley, 1983.

Index Terms:
benchmarks; knowledge systems; data engineering; interpret; apply; intrinsic measures; engineering qualities; extrinsic measures; task productivity; advice quality; reasoning correctness; robustness; solution efficiency; elegance; knowledge base design; modularity; architecture; knowledge engineering; knowledge based systems; knowledge engineering; performance evaluation; program testing
F. Hayes-Roth, "Towards Benchmarks for Knowledge Systems and Their Implications for Data Engineering," IEEE Transactions on Knowledge and Data Engineering, vol. 1, no. 1, pp. 101-110, March 1989, doi:10.1109/69.43407
Usage of this product signifies your acceptance of the Terms of Use.