This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
A Requirements Driven Framework for Benchmarking Semantic Web Knowledge Base Systems
February 2007 (vol. 19 no. 2)
pp. 297-309
A key challenge for the Semantic Web is to acquire the capability to effectively query large knowledge bases. As there will be several competing systems, we need benchmarks that will objectively evaluate these systems. Development of effective benchmarks in an emerging domain is a challenging endeavor. In this paper, we propose a requirements driven framework for developing benchmarks for Semantic Web Knowledge Base Systems (SW KBSs). In this paper, we make two major contributions. First, we provide a list of requirements for SW KBS benchmarks. This can serve as an unbiased guide to both the benchmark developers and personnel responsible for systems acquisition and benchmarking. Second, we provide an organized collection of techniques and tools needed to develop such benchmarks. In particular, the collection contains a detailed guide for generating benchmark workload, defining performance metrics, and interpreting experimental results.

[0]
[0]
[0]
[0]
[0]
[0]
[0]
[0]
[0]
[0]
[0]

Index Terms:
Semantic Web, knowledge base system, benchmark, evaluation, scalability, reasoning.
Citation:
Yuanbo Guo, Abir Qasem, Zhengxiang Pan, Jeff Heflin, "A Requirements Driven Framework for Benchmarking Semantic Web Knowledge Base Systems," IEEE Transactions on Knowledge and Data Engineering, vol. 19, no. 2, pp. 297-309, Feb. 2007, doi:10.1109/TKDE.2007.19
Usage of this product signifies your acceptance of the Terms of Use.