The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - April-June (2009 vol.2)
pp: 79-92
Mingyu Feng , Worcester Polytechnic Institute, Worcester
Neil T. Heffernan , Worcester Polytechnic Institute, Worcester
Cristina Heffernan , Worcester Polytechnic Institute, Worcester
Murali Mani , Worcester Polytechnic Institute, Worcester
ABSTRACT
Student modeling and cognitive diagnostic assessment are important issues that need to be addressed for the development and successful application of intelligent tutoring systems (ITS). ITS needs the construction of complex models to represent the skills that students are using and their knowledge states, and practitioners want cognitively diagnostic information at a finer grained level. Traditionally, most assessments treat all questions on the test as sampling a single underlying knowledge component. Can we have our cake and eat it, too? That is, can we have a good overall prediction of a high stakes test, while at the same time be able to tell teachers meaningful information about fine-grained knowledge components? In this paper, we introduce an online intelligent tutoring system that has been widely used. We then present some encouraging results about a fine-grained skill model with the system that is able to predict state test scores. This model allows the system track about 106 knowledge components for eighth grade math. In total, 921 eighth grade students were involved in the study. We show that our fine-grained model could improve prediction compared to other coarser grained models and an IRT-based model. We conclude that this intelligent tutoring system can be a good predictor of performance.
INDEX TERMS
Intelligent tutoring systems, cognitive diagnostic assessment, fine-grained skill model, statistical analysis of skill models, mixed-effects model.
CITATION
Mingyu Feng, Neil T. Heffernan, Cristina Heffernan, Murali Mani, "Using Mixed-Effects Modeling to Analyze Different Grain-Sized Skill Models in an Intelligent Tutoring System", IEEE Transactions on Learning Technologies, vol.2, no. 2, pp. 79-92, April-June 2009, doi:10.1109/TLT.2009.17
REFERENCES
[1] R.G. Almond, L.V. DiBello, B. Moulder, and D. Zapata-Rivera, “Modeling Diagnostic Assessment with Bayesian Networks,” J.Educational Measurement, vol. 44, no. 4, pp. 341-359, 2007.
[2] J.R. Anderson, Rules of Mind. Lawrence Erlbaum Assoc., 1993.
[3] J.R. Anderson and C. Lebiere, The Atomic Components of Thought. Lawrence Erlbaum Assoc., 1998.
[4] J.R. Anderson, A.T. Corbett, K.R. Koedinger, and R. Pelletier, “Cognitive Tutors: Lessons Learned,” J. Learning Sciences, vol. 4, no. 2, pp. 167-207, 1995.
[5] E. Ayers and B.W. Junker, “Do Skills Combine Additively to Predict Task Difficulty in Eighth Grade Mathematics?” Proc. Educational Data Mining: Papers from the Am. Assoc. for Artificial Intelligence (AAAI) Workshop, J. Beck, E. Aimeur, and T. Barnes, eds., pp. 14-20, 2006.
[6] T. Barnes, “Q-Matrix Method: Mining Student Response Data for Knowledge,” Educational Data Mining: Papers from the 2005 Am. Assoc. for Artificial Intelligence (AAAI) Workshop, J. Beck, ed., 2005.
[7] D. Bates, “Linear Mixed Model Implementation in lme4,” Manuscript, Univ. of Wisconsin, May 2007.
[8] R.D. Bock, R. Gibbons, and E.J. Muraki, “Full Information Item Factor Analysis,” Applied Psychological Measurement, vol. 12, pp.261-280, 1988.
[9] H. Cen, K. Koedinger, and B. Junker, “Automating Cognitive Model Improvement by A*Search and Logistic Regression,” Proc. Educational Data Mining: Papers from the 2005 Am. Assoc. for Artificial Intelligence (AAAI) Workshop, J. Beck, ed., 2005.
[10] J. Collins, J. Greer, and S. Huang, “Adaptive Assessment of Using Granularity Hierarchies and Bayesien Nets,” Proc. Int'l Conf. Intelligent Tutoring Systems, pp. 569-577, 1996.
[11] J. Confrey, A. Valenzuela, and A. Ortiz, “Recommendation to the Texas State Board of Education on the Setting of TAKS Standards: A Call to Responsible Action,” http://www.syrce.orgState_ Board.htm, 2002.
[12] A.T. Corbett, J.R. Anderson, and A.T. O'Brien, “Student Modeling in the ACT Programming Tutor,” Cognitively Diagnostic Assessment, P. Nichols, S. Chipman, and R. Brennan, eds., Lawrence Erlbaum Assoc., 1995.
[13] E. Croteau, N.T. Heffernan, and K.R. Koedinger, “Why Are Algebra Word Problems Difficult? Using Tutorial Log Files and the Power Law of Learning to Select the Best Fitting Cognitive Model,” Proc. Seventh Int'l Conf. Intelligent Tutoring Systems, pp.240-250, 2004.
[14] B. Daniel, D. Zapata-Rivera, and G. McCalla, “A Bayesian Belief Network Approach for Modeling Complex Domains,” Bayesian Network Technologies: Applications and Graphical Models, A. Mittal and A. Kassim, eds., pp. 13-41, IRM Press, 2007.
[15] K.L. Draney, P. Pirolli, and M. Wilson, “A Measurement Model for a Complex Cognitive Skill,” Cognitively Diagnostic Assessment, P. Nichols, S. Chipman, and R. Brennan, eds., Lawrence Erlbaum Assoc., 1995.
[16] S.E. Embretson, “Structured Rasch Models for Measuring Individual-Difference in Learning and Change,” Int'l J. Psychology, vol. 27, nos. 3/4, pp. 372-372, 1992.
[17] M. Feng, N.T. Heffernan, and K.R. Koedinger, “Predicting State Test Scores Better with Intelligent Tutoring Systems: Developing Metrics to Measure Assistance Required,” Proc. Eighth Int'l Conf. Intelligent Tutoring Systems, M. Ikeda, K.D. Ashley, and T.-W.Chan, eds., pp. 31-40, 2006.
[18] M. Feng, N.T. Heffernan, M. Murali, and C. Heffernan, “Using Mixed-Effects Modeling to Compare Different Grain-Sized Skill Models,” Proc. Educational Data Mining: Papers from the Am. Assoc. for Artificial Intelligence (AAAI) Workshop, J. Beck, E. Aimeur, and T. Barnes, eds., pp. 57-66, 2006.
[19] M. Feng and N. Heffernan, “Towards Live Informing and Automatic Analyzing of Student Learning: Reporting in ASSISTment System,” J. Interactive Learning Research, vol. 18, no. 2, pp.207-230, 2007.
[20] M. Feng, J. Beck, N. Heffernan, and K. Koedinger, “Can an Intelligent Tutoring System Predict Math Proficiency as well as a Standardized Test?” Proc. First Int'l Conf. Education Data Mining, R.S.J.d. Baker and J.E. Beck, eds., pp. 107-116, 2008.
[21] M. Feng, N. Heffernan, J. Beck, and K. Koedinger, “Can We Predict Which Groups of Questions Students Will Learn From?” Proc. First Int'l Conf. Education Data Mining, R.S.J.d. Baker and J.E.Beck, eds., pp. 218-225, 2008.
[22] M. Feng, N.T. Heffernan, and K.R. Koedinger, “Addressing the Assessment Challenge in an Online System That Tutors as It Assesses,” to be published in User Modeling and User-Adapted Interaction: The J. Personalization Research.
[23] K. Ferguson, I. Arroyo, S. Mahadevan, B. Woolf, and A. Barto, “Improving Intelligent Tutoring Systems: Using Expectation Maximization to Learn Student Skill Levels,” Proc. Eighth Int'l Conf. Intelligent Tutoring Systems, M. Ikeda, K.D. Ashley, and T.-W.Chan, eds., pp. 453-462, 2006.
[24] M.J. Gierl, C. Wang, and J. Zhou, “Using the Attribute Hierarchy Method to Make Diagnostic Inferences about Examinees' Cognitive Skills in Algebra on the SAT,” J. Technology, Learning, and Assessment, vol. 6, no. 6,http:/www.jtla.org, 2008.
[25] D. Hedeker and R.D. Gibbons, Longitudinal Data Analysis. JohnWiley & Sons, 2006.
[26] N.T. Heffernan, T.E. Turner, A.L.N. Lourenco, M.A. Macasek, G. Nuzzo-Jones, and K.R. Koedinger, “The ASSISTment Builder: Towards an Analysis of Cost Effectiveness of ITS Creation,” Proc. 19th Int'l Florida Artificial Intelligence Research Soc. Conf. (FLAIRS '06), G. Sutcliffe and R. Goebel, eds., pp. 515-520, 2006.
[27] B. Junker, “Using On-Line Tutoring Records to Predict End-of-Year Exam Scores: Experience with the ASSISTments Project and MCAS Eighth Grade Mathematics,” Assessing and Modeling Cognitive Development in School, R.W. Lissitz, ed., JAM Press, 2007.
[28] S. Katz, A. Lesgold, G. Eggan, and M. Gordin, “Modeling the Student in Sherlock II,” Int'l J. Artificial Intelligence in Education, vol. 3, no. 4, pp. 495-518, 1992.
[29] M. Militello, S. Sireci, and J. Schweid, “Intent, Purpose, and Fit: An Examination of Formative Assessment Systems in School Districts,” Proc. Am. Educational Research Assoc. Ann. Meeting, 2008.
[30] R. Mislevy, “Cognitive Psychology and Educational Assessment,” Educational Measurement, fourth ed., R.L. Brennan, ed., Am. Council on Education, 2006.
[31] M.J. Nathan and K.R. Koedinger, “An Investigation of Teachers' Beliefs of Students' Algebra Development,” Cognition and Instruction, vol. 18, no. 2, pp. 209-237, 2000.
[32] Cognitively Diagnostic Assessment, P.D. Nichols, S.F. Chipman, and R.L. Brennan, eds. Lawrence Erlbaum Assoc., 1995.
[33] L. Olson, “State Test Programs Mushroom as NCLB Mandate Kicks,” Education Week, pp. 10-14, Nov. 2004.
[34] Z. Pardos, M. Feng, N.T. Heffernan, and C. Heffernan-Lindquist, “Analyzing Fine-Grained Skill Models Using Bayesian and Mixed Effect Methods,” Proc. 13th Conf. Artificial Intelligence in Education, R. Luckin and K. Koedinger, eds., pp. 626-628, 2007.
[35] Z.A. Pardos, N.T. Heffernan, C. Ruiz, and J. Beck, “The Composition Effect: Conjunctive or Compensatory? An Analysis of Multi-Skill Math Questions in ITS,” Proc. First Int'l Conf. Educational Data Mining, R.S.J.d. Baker and J.E. Beck, eds., 2008.
[36] R Development Core Team “R: A Language and Environment for Statistical Computing,” R Foundation for Statistical Computing, http:/www.r-project.org, 2007.
[37] A. Raftery, “Bayesian Model Selection in Social Research,” Sociological Methodology, vol. 25, pp. 111-163, 1995.
[38] E. Ramírez and K. Clark, “What Arne Duncan Thinks of No Child Left Behind: The New Education Secretary Talks about the Controversial Law and Financial Aid Forms,” http://www. usnews.com/articles/education/ 2009/02/05what-arne-duncan-thinks-of-no-child-left-behind.html , 2009.
[39] L. Razzaq and N.T. Heffernan, “Scaffolding vs. Hints in the Assistment System,” Proc. Int'l Conf. Intelligent Tutoring Systems, M.Ikeda, K.D. Ashley, and T.-W. Chan, eds., pp. 635-644, 2006.
[40] L. Razzaq, N.T. Heffernan, and R.W. Lindeman, “What Level of Tutor Interaction Is Best?,” Proc. 13th Conf. Artificial Intelligence in Education, R. Luckin and K. Koedinger, eds., pp. 222-229, 2007.
[41] J.D. Singer and J.B. Willett, Applied Longitudinal Data Analysis: Modeling Change and Occurrence. Oxford Univ. Press, 2003.
[42] E.S. Tan, T. Imbos, and R.J.M. Does, “A Distribution-Free Approach to Comparing Growth of Knowledge,” J. Education Measurement, vol. 31, no. 1, pp. 51-65, 1994.
[43] K.K. Tatsuoka, “Toward an Integration of Item Response Theory and Cognitive Error Diagnosis,” Diagnostic Monitoring of Skill and Knowledge Acquisition, N. Frederiksen, R. Glaser, A. Lesgold, and M.G. Shafto, eds., pp. 453-488, Lawrence Erlbaum Assoc., 1990.
[44] Handbook of Modern Item Response Theory, W.J. Van Der Linden, and R.K. Hambleton eds. Springer Verlag, 1997.
[45] E.C. Wylie, and J. Ciofalo, “Supporting Teachers' Use of Individual Diagnostic Itemsm,” http://www.tcrecord.orgPrintContent. asp?ContentID=15363 , 2008.
[46] J.T. Yun, J. Willet, and R. Murnane, “Accountability-Based Reforms and Instruction: Testing Curricular Alignment for Instruction Using the Massachusetts Comprehensive Assessment System,” Proc. Ann. Am. Educational Research Assoc. Meeting, 2004.
5 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool