In the current age of information explosion, people need an effective way to help them find useful information in the midst of the vast data “ocean”. To address this issue, ontology-related techniques have been increasingly applied to information retrieval in recent years, expected to help computers understand the intentions of human being and thus provide better services. It is clear that, with the help of ontology, the performances of information retrieval systems can be improved. However, it is then necessary to propose a method indicating how to measure the improvement, and to demonstrate its benefit to the industry. This paper proposes a method to evaluate retrieval systems based on ontology vocabulary, which is different from traditional evaluation methods in its purposes and applications. Experimental results have also been provided to show the effectiveness of the proposed evaluation measure.