This Article 
 Bibliographic References 
 Add to: 
Mihaela Cocea, Dept. of Comput. Sci. & Inf. Syst., Univ. of London, London, UK
Stephan Weibelzahl, Sch. of Comput., Nat. Coll. of Ireland, Dublin, Ireland
Learning environments aim to deliver efficacious instruction, but rarely take into consideration the motivational factors involved in the learning process. However, motivational aspects like engagement play an important role in effective learning-engaged learners gain more. E-Learning systems could be improved by tracking students' disengagement that, in turn, would allow personalized interventions at appropriate times in order to reengage students. This idea has been exploited several times for Intelligent Tutoring Systems, but not yet in other types of learning environments that are less structured. To address this gap, our research looks at online learning-content-delivery systems using educational data mining techniques. Previously, several attributes relevant for disengagement prediction were identified by means of log-file analysis on HTML-Tutor, a web-based learning environment. In this paper, we investigate the extendibility of our approach to other systems by studying the relevance of these attributes for predicting disengagement in a different e-learning system. To this end, two validation studies were conducted indicating that the previously identified attributes are pertinent for disengagement prediction, and two new meta-attributes derived from log-data observations improve prediction and may potentially be used for automatic log-file annotation.

[1] R. Baker, A. Corbett, and K. Koedinger, "Detecting Student Misuse of Intelligent Tutoring Systems," Proc. Seventh Int'l Conf. Intelligent Tutoring Systems, pp. 531-540, 2004.
[2] T. Connolly and M. Stansfield, "Using Games-Based eLearning Technologies in Overcoming Difficulties in Teaching Information Systems," J. Information Technology Education, vol. 5, pp. 459-476, 2006.
[3] G.D. Chen, G.Y. Shen, K.L. Ou, and B. Liu, "Promoting Motivation and Eliminating Disorientation for Web Based Courses by a Multi-User Game," Proc. World Conf. Educational Multimedia and Hypermedia and World Conf. Educational Telecomm., June 1998.
[4] C.R. Beal, L. Qu, and H. Lee, "Classifying Learner Engagement through Integration of Multiple Data Sources," Proc. 21st Nat'l Conf. Artificial Intelligence, pp. 2-8, 2006.
[5] A. de Vicente and H. Pain, "Informing the Detection of the Students' Motivational State: An Empirical Study," Proc. Sixth Int'l Conf. Intelligent Tutoring Systems, S.A. Cerri et al., eds., pp. 933-943, 2002.
[6] J. Beck, "Engagement Tracing: Using Response Times to Model Student Disengagement," Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology, C. Looi et al., eds., pp. 88-95, IOS Press, 2005.
[7] I. Arroyo and B.P. Woolf, "Inferring Learning and Attitudes from a Bayesian Network of Log File Data," Artificial Intelligence in Education, Supporting Learning through Intelligent and Socially Informed Technology, C.K. Looi et al., eds., pp. 33-34, IOS Press, 2005.
[8] L. Qu, N. Wang, and W.L. Johnson, "Detecting the Learner's Motivational States in an Interactive Learning Environment," Artificial Intelligence in Education, C.-K. Looi et al., eds., pp. 547-554, IOS Press, 2005.
[9] J. Johns and B. Woolf, "A Dynamic Mixture Model to Detect Student Motivation and Proficiency," Proc. 21st Nat'l Conf. Artificial Intelligence (AAAI-06), 2006.
[10] J. Walonoski and N.T. Heffernan, "Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems," Proc. Eighth Int'l Conf. Intelligent Tutoring Systems, M. Ikeda, K. Ashley, and T.-W. Chan, eds., pp. 382-391, 2006.
[11] J. Walonoski and N.T. Heffernan, "Prevention of Off-Task Gaming Behaviour within Intelligent Tutoring Systems," Proc. Eighth Int'l Conf. Intelligent Tutoring Systems, M. Ikeda, K. Ashley, and T.-W. Chan, eds., pp. 722-724, 2006.
[12] M. Cocea and S. Weibelzahl, "Eliciting Motivation Knowledge from Log Files towards Motivation Diagnosis for Adaptive Systems," Proc. 11th Int'l Conf. User Modelling (UM '07), C. Conati, K. McCoy, and G. Paliouras, eds., pp. 197-206, 2007.
[13] P.R. Pintrich and D.H. Schunk, Motivation in Education: Theory, Research and Applications. Prentice Hall, 2002.
[14] J.M. Keller, "Development and Use of the ARCS Model of Instructional Design," J. Instructional Development, vol. 10, no. 3, pp. 2-10, 1987.
[15] W. Burleson and R.W. Picard, "Evidence for Gender Specific Approaches to the Development of Emotionally Intelligent Learning Companions," IEEE Intelligent Systems, Special Issue on Intelligent Educational Systems, vol. 22, no. 4, pp. 62-69, 2007.
[16] S. D'Mello, T. Jackson, S. Craig, B. Morgan, P. Chipman, H. White, N. Person, B. Kort, R. el Kaliouby, R.W. Picard, and A. Graesser, "AutoTutor Detects and Responds to Learners Affective and Cognitive States," Proc. Workshop Emotional and Cognitive Issues at the Int'l Conf. Intelligent Tutoring Systems, June 2008.
[17] B. Woolf, W. Burleson, I. Arroyo, T. Dragon, D. Cooper, and R. Picard, "Affect-Aware Tutors: Recognising and Responding to Student Affect," Int'l J. Learning Technology, vol. 4, nos. 3/4, pp. 129-163, 2009.
[18] I. Arroyo, K. Ferguson, J. Johns, T. Dragon, H. Meheranian, D. Fisher, A. Barto, S. Mahadevan, and B.P. Woolf, "Repairing Disengagement with Non-Invasive Interventions," Proc. 13th Int'l Conf. Artificial Intelligence in Education, pp. 195-202, 2007.
[19] M.M.T. Rodrigo, G. Rebolledo-Mendez, R.S.J.d. Baker, B. du Boulay, J.O. Sugay, S.A.L. Lim, M.B. Espejo-Lahoz, and R. Luckin, "The Effects of Motivational Modeling on Affect in an Intelligent Tutoring System," Proc. Int'l Conf. Computers in Education, 2008.
[20] R. Baker, S. D'Mello, M. Rodrigo, and A. Graesser, "Better to be Frustrated than Bored: The Incidence and Persistence of Affect during Interactions with Three Different Computer-Based Learning Environments," Int'l J. Human-Computer Studies, vol. 68, no. 4, pp. 223-241, 2010.
[21] C. Conati and H. Maclaren, "Empirically Building and Evaluating a Probabilistic Model of User Affect," User Modeling and User-Adapted Interaction, vol. 19, no. 3, pp. 267-303, 2009.
[22] M.M.T. Rodrigo, R. Baker, M.C. Jadud, A.C.M. Amarra, T. Dy, M.B.V. Espejo-Lahoz, S.A.L. Lim, S.A.M.S. Pascua, J.O. Sugay, and E.S. Tabanao, "Affective and Behavioral Predictors of Novice Programmer Achievement," Proc. Conf. Innovation and Technology in Computer Science Education (ITiCSE '09), pp. 156-160, 2009.
[23] M.M.T. Rodrigo, R.S.J.d. Baker, M.C.V. Lagud, S.A.L. Lim, A.F. Macapanpan, S.A.M.S. Pascua, J.Q. Santillano, L.R.S. Sevilla, J.O. Sugay, S. Tep, and N.J. B. Viehland, "Affect and Usage Choices in Simulation Problem-Solving Environments," Proc. Conf. Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work (AIED '07), pp. 145-152, 2007.
[24] S. D'Mello and A. Graesser, "Automatic Detection of Learners' Emotions from Gross Body Language," Applied Artificial Intelligence, vol. 23, no. 2, pp. 123-150, 2009.
[25] S. Lee, S.W. McQuiggan, and J.C. Lester, "Inducing User Affect Recognition Models for Task-Oriented Environments," Proc. Int'l Conf. User Modeling, pp. 380-384, 2007.
[26] J.P. Rowe, S.W. McQuiggan, J.L. Robison, and J.C. Lester, "Off-Task Behavior in Narrative-Centered Learning Environments," Proc. Conf. Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work (AIED '09), pp. 99-106, 2009.
[27] G. Weber, H.-C. Kuhl, and S. Weibelzahl, "Developing Adaptive Internet Based Courses with the Authoring System NetCoach2," Hypermedia: Openness, Structural Awareness, and Adaptivity, pp. 226-238, Springer, 2001.
[28] M. Cocea and S. Weibelzahl, "Can Log Files Analysis Estimate Learners' Level of Motivation?" Proc. 14th Workshop Adaptivity and User Modeling in Interactive Systems (ABIS '06), pp. 32-35, 2006.
[29] I.H. Witten and E. Frank, Data Mining. Practical Machine Learning Tools and Techniques, second ed. Morgan Kauffman/Elsevier, 2005.
[30] T.M. Mitchell, Machine Learning. McGraw Hill, 1997.
[31] J. Cohen, "A Coefficient of Agreement for Nominal Scales," Educational and Psychological Measurement, vol. 20, no. 1, pp. 37-46, 1960.
[32] K. Krippendorff, Content Analysis: An Introduction to Its Methodology. Sage, 2004.
[33] M. Lombard, J. Snyder-Duch, and C.C. Bracken, "Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research,", 2003.
[34] R. Rafter and B. Smyth, "Passive Profiling from Server Logs in an Online Recruitment Environment," Proc. IJCAI Workshop Intelligent Techniques for Web Personalization (ITWP '01), 2001.
[35] R. Farzan and P. Brusilovsky, "Social Navigation Support in E-Learning: What Are Real Footprints," Proc. Workshop Intelligent Techniques for Web Personalization (IJCAI '05), pp. 49-56, 2005.
[36] Speed Reading Test, http:/, 2007.
[37] TurboRead Speed Reading, http:/, 2007.
[38] M. Cocea, "Assessment of Motivation in Online Learning Environments," Proc. Fourth Int'l Conf. Adaptive Hypermedia and Adaptive Web-Based Systems, V. Wade et al., eds., pp. 414-418, 2006.
[39] T. Hurley, "Intervention Strategies to Increase Self-Efficacy and Self-Regulation in Adaptive OnLine Learning," Proc. Fourth Int'l Conf. Adaptive Hypermedia and Adaptive Web-Based Systems, V. Wade et al., eds., pp. 440-444, 2006.

Index Terms:
Internet,hypermedia markup languages,intelligent tutoring systems,Web-based learning environment,disengagement detection,online learning,efficacious instruction,effective learning,e-learning systems,intelligent tutoring systems,log-file analysis,HTML-Tutor,Decision support systems,log-file analysis.,e-Learning,educational data mining,disengagement prediction
Mihaela Cocea, Stephan Weibelzahl, "Disengagement Detection in Online Learning: Validation Studies and Perspectives," IEEE Transactions on Learning Technologies, vol. 4, no. 2, pp. 114-124, April-June 2011, doi:10.1109/TLT.2010.14
Usage of this product signifies your acceptance of the Terms of Use.