This Article 
 Bibliographic References 
 Add to: 
January-March 2008 (vol. 1 no. 1)
pp. 34-48
Xavier Ochoa, Escuela Superior Politécnica del Litoral, Guayaquil
Erik Duval, Katholieke Universiteit Leuven, Leuven
This paper develops the concept of relevance in the context of learning object search. It proposes a set of metrics to estimate the topical, personal and situational relevance dimensions. These metrics are derived mainly from usage and contextual information. An exploratory evaluation of the metrics shows that even the simplest ones provide statistically significant improvement in the ranking order over the most common algorithmic relevance metric. Moreover, the combination of the metrics through the RankNet learning sorts the result list 50% better than the base-line ranking. The paper also presents openquestions in the field of learning object relevance ranking that deserve further attention.

[1] R. McGreal, “Learning Objects: A Practical Definition,” Int'l J. Instructional Technology and Distance Learning, vol. 1, no. 9, p.9, 2004.
[2] F. Neven and E. Duval, “Reusable Learning Objects: A Survey of LOM-Based Repositories,” Proc. 10th ACM Int'l Conf. Multimedia (MULTIMEDIA '02), pp. 291-294, 2002.
[3] E. Duval, K. Warkentyne, F. Haenni, E. Forte, K. Cardinaels, B. Verhoeven, R. Van Durm, K. Hendrikx, M.W. Forte, N. Ebel, and M. Macowicz, “The Ariadne Knowledge Pool System,” Comm. ACM, vol. 44, no. 5, pp. 72-78, 2001.
[4] J. Najjar, J. Klerkx, R. Vuorikari, and E. Duval, “Finding Appropriate Learning Objects: An Empirical Evaluation,” Proc. Ninth European Conf. Research and Advanced Technology for Digital Libraries (ECDL '05), A. Rauber, S. Christodoulakis, and A.M. Tjoa, eds., pp. 323-335, 2005.
[5] J. Najjar, S. Ternier, and E. Duval, “User Behavior in Learning Objects Repositories: An Empirical Analysis,” Proc. World Conf. Educational Multimedia, Hypermedia and Telecomm. (ED-MEDIA '04), L.C. Cantoni and C. McLoughlin, eds., pp.4373-4378, 2004.
[6] L. Sokvitne, “An Evaluation of the Effectiveness of Current Dublin Core Metadata for Retrieval,” Proc. VALA (Libraries, Technology, and the Future) Biennial Conf., 2000.
[7] H. Chu and M. Rosenthal, “Search Engines for the World Wide Web: A Comparative Study and Evaluation Methodology,” Proc. 59th Ann. Meeting of the Am. Soc. Information Science, S. Hardin, ed., vol. 33, pp. 127-135, 1996.
[8] B. Simon, D. Massart, F. van Assche, S. Ternier, E. Duval, S. Brantner, D. Olmedilla, and Z. Miklos, “A Simple Query Interface for Interoperable Learning Repositories,” Proc. First Workshop Interoperability of Web-Based Educational Systems (WBES'05), N. Saito, D. Olmedilla, and B. Simon, eds., pp.11-18, May 2005.
[9] H. Van de Sompel, M. Nelson, C. Lagoze, and S. Warner, “Resource Harvesting within the OAI-PMH Framework,” D-Lib Magazine, vol. 10, no. 12, pp. 1082-9873, 2004.
[10] K. Verbert, J. Jovanovic, D. Gasevic, and E. Duval, “Repurposing Learning Object Components,” Proc. Move to Meaningful Internet Systems 2005: OTM Workshops, R. Meersman, Z. Tari, and P.Herrero, eds., pp. 1169-1178, 2005.
[11] X. Ochoa, K. Cardinaels, M. Meire, and E. Duval, “Frameworks for the Automatic Indexation of Learning Management Systems Content into Learning Object Repositories,” Proc. World Conf. Educational Multimedia, Hypermedia, and Telecomm. (ED-MEDIA '05), P. Kommers and G. Richards, eds., pp. 1407-1414, June 2005.
[12] E. Duval, Policy and Innovation in Education—Quality Criteria, European Schoolnet, chapter LearnRank: The Real Quality Measure for Learning Materials, pp. 457-463, 2005.
[13] S. Kirsch, “Infoseek's Experiences Searching the Internet,” SIGIR Forum, vol. 32, no. 2, pp. 3-7, 1998.
[14] L. Page, S. Brin, R. Motwani, and T. Winograd, “The Pagerank Citation Ranking: Bringing Order to the Web,” technical report, Stanford Digital Library Technologies Project, 1998.
[15] X. Ochoa and E. Duval, “Use of Contextualized Attention Metadata for Ranking and Recommending Learning Objects,” Proc. First Int'l Workshop Contextualized Attention Metadata (CAMA'06), pp. 9-16, 2006.
[16] J. Najjar, M. Wolpers, and E. Duval, “Attention Metadata: Collection and Management,” Proc. 15th Int'l Conf. World Wide Web Workshop Logging Traces of Web Activity (WWW '06), C. Goble and M. Dahlin, eds., p. 4, 2006.
[17] J. Nesbit, K. Belfer, and J. Vargo, “A Convergent Participation Model for Evaluation of Learning Objects,” Canadian J. Learning and Technology, vol. 28, no. 3, pp. 105-120, 2002.
[18] R. Zemsky and W. Massy, “Thwarted Innovation: What Happened to e-Learning and Why,” technical report, Univ. of Pennsylvania and Thomson Corporation, 2004.
[19] J. Vargo, J.C. Nesbit, K. Belfer, and A. Archambault, “Learning Object Evaluation: Computer-Mediated Collaboration and Inter-Rater Reliability,” Int'l J. Computers and Applications, vol. 25, pp. 198-205, 2003.
[20] S. Weibel, “Border Crossings: Reflections on a Decade of Metadata Consensus Building,” D-Lib Magazine, vol. 11, nos. 7/8, p. 6, 2005.
[21] A. Agogino, “Visions for a Digital Library for Science, Mathematics, Engineering and Technology Education (SMETE),” Proc. Fourth ACM Digital Libraries Conf. (DL '99), pp. 205-206, 1999.
[22] G. Salton and C. Buckley, “Term-Weighting Approaches in Automatic Text Retrieval,” Information Processing and Management, vol. 24, no. 5, pp. 513-523, 1988.
[23] G. Salton and M. McGill, Introduction to Modern Information Retrieval. McGraw-Hill, 1986.
[24] R. Stata, K. Bharat, and F. Maghoul, “The Term Vector Database: Fast Access to Indexing Terms for Web Pages,” Computer Networks, vol. 33, nos. 1-6, pp. 247-255, 2000.
[25] V. Chellappa, “Content-Based Searching with Relevance Ranking for Learning Objects,” PhD dissertation, Univ. of Kansas, 2004.
[26] T. Joachims and F. Radlinski, “Search Engines that Learn from Implicit Feedback,” Computer, vol. 40, no. 8, pp. 34-40, 2007.
[27] T. Landauer, P. Foltz, and D. Laham, “An Introduction to Latent Semantic Analysis,” Discourse Processes, vol. 25, nos. 2-3, pp. 259-284, 1998.
[28] D. Olmedilla, “Realizing Interoperability of e-Learning Repositories,” PhD dissertation, Universidad Autónoma de Madrid, May 2007.
[29] E.L.-C. Law, T. Klobucar, and M. Pipan, “User Effect in Evaluating Personalized Information Retrieval Systems,” Proc. First European Conf. Technology Enhanced Learning (EC-TEL '06), W. Nejdl and K. Tochterman, eds., pp. 257-271, 2006.
[30] E. Hatcher and O. Gospodnetic, Lucene in Action (in Action Series). Manning, 2004.
[31] P. Dolog, N. Henze, W. Nejdl, and M. Sintek, “Personalization in Distributed e-Learning Environments,” Proc. 13th Int'l World Wide Web Conf. (WWW '04), M. Najork and C. Wills, eds., pp. 170-179, 2004.
[32] K. Sugiyama, K. Hatano, and M. Yoshikawa, “Adaptive Web Search Based on User Profile Constructed without Any Effort from Users,” Proc. 13th Int'l Conf. World Wide Web (WWW '04), pp. 675-684, 2004.
[33] L.M. Quiroga and J. Mostafa, “Empirical Evaluation of Explicit versus Implicit Acquisition of User Profiles in Information Filtering Systems,” Proc. Fourth ACM Conf. Digital Libraries (DL'99), N. Rowe and E.A. Fox, eds., pp. 238-239, 1999.
[34] A. Gulli and A. Signorini, “The Indexable Web Is More than 11.5Billion Pages,” Proc. 14th Int'l Conf. World Wide Web (WWW'05), F. Douglis and P. Raghavan, eds., pp. 902-903, 2005.
[35] E. Garfield, “The Impact Factor,” Current Contents, vol. 25, no. 20, pp. 3-7, 1994.
[36] P. Borlund, “The Concept of Relevance in IR,” J. Am. Soc. Information Science and Technology, vol. 54, no. 10, pp. 913-925, May 2003.
[37] J. Broisin, P. Vidal, M. Meire, and E. Duval, “Bridging the Gap between Learning Management Systems and Learning Object Repositories: Exploiting Learning Context Information,” Proc. Advanced Industrial Conf. Telecomm./Service Assurance with Partial and Intermittent Resources Conf./E-Learning on Telecomm. Workshop (AICT-SAPIR-ELETE '05), pp. 478-483, 2005.
[38] P. Vandepitte, L. Van Rentergem, E. Duval, S. Ternier, and F. Neven, “Bridging an LCMS and an LMS: A Blackboard Building Block for the Ariadne Knowledge Pool System,” Proc. World Conf. Educational Multimedia, Hypermedia, and Telecomm. (ED-MEDIA '03), D.L.C. McNaught, ed., pp. 423-424, 2003.
[39] K. Verbert and E. Duval, “Evaluating the Alocom Approach for Scalable Content Repurposing,” Proc. Second European Conf. Technology Enhanced Learning (ECTEL '07), E. Duval, R. Klamma, and M. Wolpers, eds., vol. 4753, pp. 364-377, 2007.
[40] G. Linden, B. Smith, and J. York, “ Recommendations: Item-to-Item Collaborative Filtering,” IEEE Internet Computing, vol. 7, no. 1, pp. 76-80, Jan. 2003.
[41] A. Pigeau, G. Raschia, M. Gelgon, N. Mouaddib, and R. Saint-Paul, “A Fuzzy Linguistic Summarization Technique for TV Recommender Systems,” Proc. IEEE Int'l Conf. Fuzzy Systems (FUZZ-IEEE '03), O. Nasraoui, H. Frigui, and J.M. Keller, eds., vol.1, pp. 743-748, 2003.
[42] E.H. Chi, P. Pirolli, K. Chen, and J. Pitkow, “Using Information Scent to Model User Information Needs and Actions and the Web,” Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI '01), J. Jacko and A. Sears, eds., pp. 490-497, 2001.
[43] A. Budanitsky and G. Hirst, “Semantic Distance in Wordnet: An Experimental, Application-Oriented Evaluation of Five Measures,” Proc. Workshop WordNet and Other Lexical Resources, Second Meeting of the North Am. Chapter of the Assoc. Computational Linguistics, pp. 29-34, 2001.
[44] G. Jeh and J. Widom, “Simrank: A Measure of Structural-Context Similarity,” Proc. Eighth ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD '02), D. Hand, D. Keim, and R.Ng, eds., pp. 538-543, 2002.
[45] S. Downes, “Models for Sustainable Open Educational Resources,” Interdisciplinary J. Knowledge and Learning Objects, vol. 3, pp. 29-44, 2007.
[46] J.M. Kleinberg, “Authoritative Sources in a Hyperlinked Environment,” J. ACM, vol. 46, no. 5, pp. 604-632, 1999.
[47] A.M.A. Wasfi, “Collecting User Access Patterns for Building User Profiles and Collaborative Filtering,” Proc. Fourth Int'l Conf. Intelligent User Interfaces (IUI '99), M. Maybury, P. Szekely, and C.G. Thomas, eds., pp. 57-64, 1999.
[48] B. Mobasher, R. Cooley, and J. Srivastava, “Automatic Personalization Based on Web Usage Mining,” Comm. ACM, vol. 43, no. 8, pp. 142-151, 2000.
[49] E. Pampalk, T. Pohle, and G. Widmer, “Dynamic Playlist Generation Based on Skipping Behavior,” Proc. Sixth Int'l Conf. Music Information Retrieval (ISMIR '05), T. Crawford and M.Sandler, eds., pp. 634-637, 2005.
[50] O. Medelyan and I. Witten, “Thesaurus Based Automatic Keyphrase Indexing,” Proc. Sixth ACM/IEEE CS Joint Conf. Digital Libraries (JCDL '06), M.L. Nelson and C.C. Marshall, eds., pp. 296-297, 2006.
[51] M. Sicilia, E. Garcia, C. Pages, and J. Martinez, “Complete Metadata Records in Learning Object Repositories: Some Evidence and Requirements,” Int'l J. Learning Technology, vol. 1, no. 4, pp.411-424, 2005.
[52] S. Upendra, “Social Information Filtering for Music Recommendation,” master's thesis, Massachusetts Inst. of Technology, , 1994.
[53] A. Aizawa, “An Information-Theoretic Perspective of TF-IDF Measures,” Information Processing and Management, vol. 39, no. 1, pp. 45-65, Jan. 2003.
[54] R. Kraft, F. Maghoul, and C.C. Chang, “Y!Q: Contextual Search at the Point of Inspiration,” Proc. 14th ACM Int'l Conf. Information and Knowledge Management (CIKM '05), F. Douglis and P. Raghavan, eds., pp. 816-823, 2005.
[55] Proc. Learning to Rank Workshop, Neural Information Processing Systems Conf. (NIPS), S. Agarwal, C. Cortes, and R. Herbrich, eds., 2005.
[56] R. Herbrich, T. Graepel, and K. Obermayer, Large Margin Rank Boundaries for Ordinal Regression, chapter Large Margin Rank Boundaries for Ordinal Regression, pp. 115-132, MIT Press, 2000.
[57] M. Richardson, A. Prakash, and E. Brill, “Beyond Pagerank: Machine Learning for Static Ranking,” Proc. 15th Int'l Conf. World Wide Web (WWW '06), C. Goble and M. Dahlin, eds., pp.707-715, 2006.
[58] R. Yan and A. Hauptmann, “Efficient Margin-Based Rank Learning Algorithms for Information Retrieval,” Proc. Fifth Int'l Conf. Image and Video Retrieval (CIVR '06), W.-K. Leow, M.S.Lew, T.-S. Chua, W.-Y. Ma, L. Chaisorn, and E.M.Bakker, eds., pp.113-122, 2006.
[59] V. Raykar, R. Duraiswami, and B. Krishnapuram, “A Fast Algorithm for Learning Large Scale Preference Relations,” Proc. 11th Int'l Conf. Artificial Intelligence and Statistics (AISTATS '07), M.Meila and X. Shen, eds., vol. 2, pp. 388-395, 2007.
[60] S. Carson, “Program Evaluation Findings Report MIT Opencourseware,” technical report, Massachusetts Inst. of Tech nology, 2005.
[61] R. Fagin, R. Kumar, M. Mahdian, D. Sivakumar, and E. Vee, “Comparing and Aggregating Rankings with Ties,” Proc. 23rd ACM SIGMOD-SIGACT-SIGART Symp. Principles of Database Systems (PODS '04), C. Beeri, ed., pp. 47-58, 2004.
[62] R. Kraft, C.C. Chang, F. Maghoul, and R. Kumar, “Searching with Context,” Proc. 15th Int'l Conf. World Wide Web (WWW '06), C.Groble and M. Dahlin, eds., pp. 477-486, 2006.

Index Terms:
Digital Libraries, Systems issues, Search process, Metadata, Information filtering
Xavier Ochoa, Erik Duval, "Relevance Ranking Metrics for Learning Objects," IEEE Transactions on Learning Technologies, vol. 1, no. 1, pp. 34-48, Jan.-March 2008, doi:10.1109/TLT.2008.1
Usage of this product signifies your acceptance of the Terms of Use.