• Publication
  • 2009
  • Issue No. 2 - April-June
  • Abstract - Scaling Up Programming by Demonstration for Intelligent Tutoring Systems Development: An Open-Access Web Site for Middle School Mathematics Learning
 This Article 
 Bibliographic References 
 Add to: 
Vincent Aleven, Carnegie Mellon University, Pittsburgh
Bruce M. McLaren, Carnegie Mellon University, Pittsburgh
Jonathan Sewall, Carnegie Mellon University, Pittsburgh
Intelligent tutoring systems (ITSs), which provide step-by-step guidance to students in complex problem-solving activities, have been shown to enhance student learning in a range of domains. However, they tend to be difficult to build. Our project investigates whether the process of authoring an ITS can be simplified, while at the same time maintaining the characteristics that make ITS effective, and also maintaining the ability to support large-scale tutor development. Specifically, our project tests whether authoring tools based on programming-by-demonstration techniques (developed in prior research) can support the development of a large-scale, real-world tutor. We are creating an open-access Web site, called Mathtutor (http://webmathtutor.org), where middle school students can solve math problems with step-by-step guidance from ITS. The Mathtutor site fields example-tracing tutors, a novel type of ITS that are built "by demonstration,” without programming, using the Cognitive Tutor Authoring Tools (CTATs). The project's main contribution will be that it represents a stringent test of large-scale tutor authoring through programming by demonstration. A secondary contribution will be that it tests whether an open-access site (i.e., a site that is widely and freely available) with software tutors for math learning can attract and sustain user interest and learning on a large scale.

[1] V. Aleven, B.M. McLaren, J. Sewall, and K.R. Koedinger, “Example-Tracing Tutors: A New Paradigm for Intelligent Tutoring Systems,” Int'l J. Artificial Intelligence and Education, to appear.
[2] S.R. Alpert, M.K. Singley, and P.G. Fairweather, “Deploying Intelligent Tutors on the Web: An Architecture and an Example,” Int'l J. Artificial Intelligence in Education, vol. 10, no. 2, pp. 183-197, 1999.
[3] J.R. Anderson, Rules of the Mind. Erlbaum, 1993.
[4] J.R. Anderson, A.T. Corbett, K.R. Koedinger, and R. Pelletier, “Cognitive Tutors: Lessons Learned,” J. Learning Sciences, vol. 4, no. 2, pp. 167-207, 1995.
[5] J.R. Anderson and R. Pelletier, “A Development System for Model-Tracing Tutors,” Proc. Int'l Conf. Learning Sciences, pp. 1-8, 1991.
[6] R.K. Atkinson, S.J. Derry, A. Renkl, and D. Wortham, “Learning from Examples: Instructional Principles from the Worked Examples Research,” Rev. of the Educational Research, vol. 70, no. 2, pp.181-214, 2000.
[7] R.K. Atkinson, A. Renkl, and M.M. Merrill, “Transitioning from Studying Examples to Solving Problems: Combining Fading with Prompting Fosters Learning,” J. Educational Psychology, vol. 95, pp.774-783, 2003.
[8] I. Arroyo, B.P. Woolf, and C.R. Beal, “Addressing Cognitive Differences and Gender during Problem Solving,” Technology, Instruction, Cognition, and Learning, vol. 4, pp. 31-63, 2006.
[9] R.S. Baker, A.T. Corbett, and K.R. Koedinger, “Toward a Model of Learning Data Representations,” Proc. 23rd Ann. Conf. Cognitive Science Soc., pp. 45-50, 2001.
[10] R.S. Baker, A.T. Corbett, and K.R. Koedinger, “Learning to Distinguish between Representations of Data: A Cognitive Tutor That Uses Contrasting Cases,” Proc. Int'l Conf. Learning Sciences, pp. 58-65, 2004.
[11] R.S.J.d. Baker, A.T. Corbett, and K.R. Koedinger, “The Difficulty Factors Approach to the Design of Lessons in Intelligent Tutor Curricula,” Int'l J. Artificial Intelligence in Education, vol. 17, no. 4, pp. 341-369, 2007.
[12] R.S. Baker, A.T. Corbett, K.R. Koedinger, and M.P. Schneider, “A Formative Evaluation of a Tutor for Scatterplot Generation: Evidence on Difficulty Factors,” Proc. Conf. Artificial Intelligence in Education: Shaping the Future of Learning through Intelligent Technologies (AI-ED '03), U. Hoppe, F. Verdejo, and J. Kay, eds., pp. 107-114, 2003.
[13] C.R. Beal, R. Walles, I. Arroyo, and B.P. Woolf, “Online Tutoring for Math Achievement: A Controlled Evaluation,” J. Interactive Online Learning, vol. 6, pp. 43-55, 2007.
[14] P. Black and D. William, “Inside the Black Box: Raising Standards through Classroom Assessment,” Phi Delta Kappan, vol. 80, no. 2, pp. 139-148, http://www.pdkintl.org/kappankbla9810.htm , Oct. 1998.
[15] P. Black and D. William, “Assessment and Classroom Learning,” Assessment in Education, vol. 5, no. 1, pp. 7-74, Mar. 1998.
[16] Q. Brown, F.J. Lee, D.D. Salvucci, and V. Aleven, “Interface Challenges for Mobile Tutoring Systems,” Proc. Ninth Int'l Conf. Intelligent Tutoring Systems (ITS '08), B. Woolf, E. Aimeur, R.Nkambou, and S. Lajoie, eds., pp. 693-695, 2008.
[17] R. Catrambone, “The Subgoal Learning Model: Creating Better Examples so that Students Can Solve Novel Problems,” J.Experimental Psychology General, vol. 127, pp. 355-376, 1998.
[18] P.R. Cohen, C.R. Beale, and N.M. Adams, “The Design, Deployment and Evaluation of the AnimalWatch Intelligent Tutoring System,” Proc. 18th European Conf. Artificial Intelligence (ECAI '08), M. Ghallab, C.D. Spyropoulos, N. Fakotakis, and N. Avouris, eds., pp. 663-667, 2008.
[19] A.T. Corbett and J.R. Anderson, “Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge,” User Modeling and User-Adapted Interaction, vol. 4, pp. 253-278, 1995.
[20] A. Corbett, A. Wagner, and J. Raspat, “The Impact of Analyzing Example Solutions on Problem Solving in a Pre-Algebra Tutor,” Artificial Intelligence in Education, U. Hoppe, F. Verdejo and J. Kay, eds., Proc. 11th Int'l Conf. Artificial Intelligence and Education (AIED '03), pp. 133-140, 2003.
[21] N.T. Heffernan, T.E. Turner, A.L.N. Lourenco, M.A. Macasek, G. Nuzzo-Jones, and K.R. Koedinger, “The ASSISTment Builder: Towards an Analysis of Cost Effectiveness of ITS Creation,” Proc. Int'l Florida Artificial Intelligence Research Soc. Conf. (FLAIRS '06), 2006.
[22] K.R. Koedinger and V. Aleven, “Exploring the Assistance Dilemma in Experiments with Cognitive Tutors,” Educational Psychology Rev., vol. 19, no. 3, pp. 239-264, 2007.
[23] K.R. Koedinger, V. Aleven, N. Heffernan, B.M. McLaren, and M. Hockenberry, “Opening the Door to Non-Programmers: Authoring Intelligent Tutor Behavior by Demonstration,” Proc. Seventh Int'l Conf. Intelligent Tutoring Systems (ITS '04), J.C. Lester, R.M.Vicario, and F. Paraguaçu, eds., pp. 162-174, 2004.
[24] K.R. Koedinger, J.R. Anderson, W.H. Hadley, and M.A. Mark, “Intelligent Tutoring Goes to School in the Big City,” Int'l J. Artificial Intelligence in Education, vol. 8, pp. 30-43, 1997.
[25] K.R. Koedinger and A.T. Corbett, “Cognitive Tutors: Technology Bringing Learning Science to the Classroom,” The Cambridge Handbook of the Learning Sciences, K. Sawyer, ed., Cambridge Univ. Press, 2006.
[26] K. Koedinger, K. Cunningham, A. Skogsholm, and B. Leber, “An Open Repository and Analysis Tools for Fine-Grained, Longitudinal Learner Data,” Educational Data Mining 2008: Proc. First Int'l Conf. Educational Data Mining, R.S.J.d., Baker, T. Barnes, and J.E. Beck, eds., pp. 157-166, 2008.
[27] K.R. Koedinger and A. Terao, “A Cognitive Task Analysis of Using Pictures to Support Pre-Algebraic Reasoning,” Proc. 24th Ann. Conf. Cognitive Science Soc., C.D. Schunn and W. Gray, eds., pp. 542-547, 2002.
[28] B. Leber et al., “CTAT's Tool-Tutor Application Interface,” http://ctat.pact.cs.cmu.eduindex.php?id=tool-tutor , 2007.
[29] Your Wish Is my Command: Programming by Example, H. Lieberman ed. Morgan Kaufmann, 2001.
[30] M. Mavrikis and A. Maciocia, “WALLIS: A Web-Based ILE for Science and Engineering Students Studying Mathematics,” Proc. Workshop Advanced Technologies for Math. Education in 11th Int'l Conf. Artificial Intelligence in Education, http://www.maths.ed.ac. uk~wallis/, 2003.
[31] B.M. McLaren, S. Lim, and K.R. Koedinger, “When and How Often Should Worked Examples be Given to Students? New Results and a Summary of the Current State of Research,” Proc. 30th Ann. Conf. Cognitive Science Soc., B.C. Love, K. McRae, and V.M. Sloutsky, eds., pp. 2176-2181, 2008.
[32] E. Melis, E. Andrès, J. Büdenbender, A. Frischauf, G. Goguadze, P. Libbrecht, M. Pollet, and C. Ullrich, “ActiveMath: A Generic and Adaptive Web-Based Learning Environment,” Int'l J. Artificial Intelligence in Education, vol. 12, pp. 385-407, 2001.
[33] E. Melis, G. Goguadze, M. Homik, P. Libbrecht, C. Ullrich, and S. Winterstein, “Semantic-Aware Components and Services of ActiveMath,” British J. Educational Technology, vol. 37, no. 3, pp.405-423, 2006.
[34] A. Mitrovic, P. Suraweera, B. Martin, K. Zakharov, N. Milik, and J. Holland, “Authoring Constraint-Based Tutors in ASPIRE,” Proc. Eighth Int'l Conf. Intelligent Tutoring Systems (ITS '06), M. Ikeda, K.D. Ashley, and T.W. Chan, eds., pp. 41-50, 2006.
[35] A. Mitrovic, B. Martin, and P. Suraweera, “Intelligent Tutors for All: The Constraint-Based Approach,” IEEE Intelligent Systems, vol. 22, no. 4, pp. 38-45, July/Aug. 2007.
[36] P. Morgan and S. Ritter, “An Experimental Study of the Effects of Cognitive Tutor Algebra I on Student Knowledge and Attitude,” Carnegie Learning, Inc., http:/www.carnegielearning.com, 2002.
[37] T. Murray, “An Overview of Intelligent Tutoring System Authoring Tools: Updated Analysis of the State of the Art,” Authoring Tools for Advanced Learning Environments, T. Murray, S. Blessing, and S. Ainsworth, eds., chapter 17, pp. 491-544, Kluwer Academic Publishers, 2003.
[38] B.A. Myers, R.G. McDaniel, and D.S. Kosbie, “Marquise: Creating Complete User Interfaces by Demonstration,” Proc. Conf. Human Factors in Computing Systems (INTERCHI '93), 1993.
[39] NCTM, Curriculum Focal Points for Prekindergarten through Grade 8 Math. Nat'l Council of Teachers of Math. 2008.
[40] NCTM, Principles and Standards for School Math. Nat'l Council of Teachers of Math., 2000.
[41] J. Ong and S. Noneman, “Intelligent Tutoring Systems for Procedural Task Training of Remote Payload Operations at NASA,” Proc. Industry/Interservice, Training, Simulation and Education Conf. (I/ITSEC '00), 2000.
[42] F. Paas and J. Van Merriënboer, “Variability of Worked Examples and Transfer of Geometry Problem-Solving Skills: A Cognitive-Load Approach,” J. Educational Psychology, vol. 86, pp. 122-133, 1994.
[43] J. Patvarczki, S.F. Almeida, J.E. Beck, and N.T. Heffernan, “Lessons Learned from Scaling Up a Web-Based Intelligent Tutoring System,” Proc. Ninth Int'l Conf. Intelligent Tutoring Systems (ITS '08), B. Woolf, E. Aimeur, R. Nkambou, and S.Lajoie, eds., pp. 766-770, 2008.
[44] Pittsburgh Science of Learning Center, Guide to the Tutor Message Format, https://pslcdatashop.web.cmu.edu/dtd/guide tutor_ message_dtd_guide_v4.pdf, 2008.
[45] G.S. Plano, “The Effects of the Cognitive Tutor Algebra on Student Attitudes and Achievement in a Ninth Grade Algebra Course,” PhD dissertation, Seton Hall Univ., unpublished.
[46] L. Razzaq, M. Feng, G. Nuzzo-Jones, N.T. Heffernan, K.R. Koedinger, B. Junker, S. Ritter, A. Knight, C. Aniszczyk, S. Choksey, T. Livak, E. Mercado, T.E. Turner, R. Upalekar, J.A. Walonoski, M.A. Macasek, and K.P. Rasmussen, “The Assistment Project: Blending Assessment and Assisting,” Proc. 12th Int'l Conf. Artificial Intelligence in Education, C.K. Looi, G. McCalla, B.Bredeweg, and J.Breuker, eds., pp. 555-562, 2005.
[47] A. Renkl, “Worked-Out Examples: Instructional Explanations Support Learning by Self-Explanations,” Learning and Instruction, vol. 12, no. 5, pp. 529-556, 2002.
[48] A. Renkl, R.K. Atkinson, and C.S. Große, “How Fading Worked Solution Steps Works—A Cognitive Load Perspective,” Instructional Science, vol. 32, pp. 59-82, 2004.
[49] B. Rittle-Johnson and K.R. Koedinger, “Comparing Instructional Strategies for Integrating Conceptual and Procedural Knowledge,” Proc. Conf. Int'l Group for the Psychology of Math. Education, North Am. Chapter (PME-NA XXXIII), 2002.
[50] B. Rittle-Johnson and K.R. Koedinger, “Using Cognitive Models to Guide Instructional Design: The Case of Fraction Division,” Proc. 23rd Ann. Conf. Cognitive Science Soc., pp. 857-862, 2001.
[51] S. Ruby, D. Thomas, and D.H. Hansson, Agile Web Development with Rails, third ed. The Pragmatic Programmers LLC, 2009.
[52] R. Salden, V. Aleven, A. Renkl, and R. Schwonke, “Worked Examples and Tutored Problem Solving: Redundant or Synergistic forms of Support?” Proc. 30th Ann. Conf. Cognitive Science Soc., B.C. Love, K. McRae, and V.M. Sloutsky, eds., pp. 589-594, 2008.
[53] C. Sangwin and S. Hammond, “Enhancing Traditional Teaching through the STACK CAA System,” Proc. Fifth Workshop Joining Educational Math. (JEM), http:/www.stack.bham.ac.uk/, 2007.
[54] H. Sarkis, “Cognitive Tutor Algebra 1 Program Evaluation,” Miami-Dade County Public Schools, The Reliability Group, Carnegie Learning, Inc., http:/www.carnegielearning.com, 2004.
[55] R. Schwonke, A. Renkl, C. Krieg, J. Wittwer, V. Aleven, and R. Salden, “The Worked-Example Effect: Is It Just an Artefact of Lousy Control Conditions?” Computers in Human Behavior, to appear.
[56] J.G. Trafton and B.J. Reiser, “The Contributions of Studying Examples and Solving Problems to Skill Acquisition,” Proc. 15th Ann. Conf. Cognitive Science Soc., M. Polson, ed., pp. 1017-1022, 1993.
[57] K. VanLehn, “The Behavior of Tutoring Systems,” Int'l J. Artificial Intelligence in Education, vol. 16, no. 3, pp. 227-265, 2006.
[58] WebALT “State of the Art in Mathematical E-Learning,” EDC-22253-WEBALT, WebALT Consortium, WebALT Project Deliverable D1.1., http://webalt.math.helsinki.fi/content/e16/ e301/e304D1.1._State_of_the_Art_in_mathematical_e-learning_ eng.pdf , Apr. 2005.
[59] G. Weber and P. Brusilovsky, “ELM-ART: An Adaptive Versatile System for Web-Based Instruction,” Int'l J. Artificial Intelligence in Education, Special Issue on Adaptive and Intelligent Web-Based Educational Systems, vol. 12, no. 4, pp. 351-384, 2001.
[60] B.P. Woolf, Building Intelligent Interactive Tutors: Student-Centered Strategies for Revolutionizing e-Learning. Morgan Kaufmann, 2008.
[61] K. Yacef, “The Logic-ITA in the Classroom: A Medium Scale Experiment,” Int'l J. Artificial Intelligence in Education, vol. 15, pp.41-60, 2005.

Index Terms:
Homework support systems, adaptive educational systems, intelligent tutoring systems, authoring tools.
Vincent Aleven, Bruce M. McLaren, Jonathan Sewall, "Scaling Up Programming by Demonstration for Intelligent Tutoring Systems Development: An Open-Access Web Site for Middle School Mathematics Learning," IEEE Transactions on Learning Technologies, vol. 2, no. 2, pp. 64-78, April-June 2009, doi:10.1109/TLT.2009.22
Usage of this product signifies your acceptance of the Terms of Use.