# Lessons Learned from Comprehensive Deployments of Multiagent CSCL Applications I-MINDS and ClassroomWiki

Nobel Khandaker
Leen-Kiat Soh, IEEE
Lee Dee Miller
Hong Jiang, IEEE

Pages: pp. 47-58

Abstract—Recent years have seen a surge in the use of intelligent computer-supported collaborative learning (CSCL) tools for improving student learning in traditional classrooms. However, adopting such a CSCL tool in a classroom still requires the teacher to develop (or decide on which to adopt) the CSCL tool and the CSCL script, design the relevant pedagogical aspects (i.e., the learning objectives, assessment method, etc.) to overcome the associated challenges (e.g., free riding, student assessment, forming student groups that improve student learning, etc). We have used a multiagent-based system to develop a CSCL application and multiagent-frameworks to form student groups that improve student collaborative learning. In this paper, we describe the contexts of our three generations of CSCL applications (i.e., I-MINDS and ClassroomWiki) and provide a set of lessons learned from our deployments in terms of the script, tool, and pedagogical aspects of using CSCL. We believe that our lessons would allow 1) the instructors and students to use intelligent CSCL applications more effectively and efficiently, and help to improve the design of such systems, and 2)the researchers to gain additional insights into the impact of collaborative learning theories when they are applied to real-world classrooms.

Index Terms—Collaborative learning, education, multiagent systems.

## Introduction

Cscl systems have been gaining popularity in recent years as a method for improving classroom instructions [ 1], [ 2]. However, as noted by researchers [ 3], there are several challenges with using CSCL systems. First, it is difficult to find the appropriate composition of student groups participating in the CSCL system in a given environment. Second, accurate evaluation of individuals and groups is often difficult because of insufficient associated tracking and modeling of student activities in the environment. Third, determining the optimal script or mode of operations and building a platform/tool that is able to support that script where both components try to achieve objectives like: 1)enhancing collaboration, 2)capturing students' contributions in details. Since these objectives often cannot be met by the same script, determining the optimal script and the tool is often difficult. To summarize, these challenges, if not addressed well, may hurt the collaborative learning outcome of the participating students and discourage the teachers from adopting the CSCL techniques.

In this paper, we describe our efforts toward overcoming or addressing a variety of challenges in the CSCL environment using a multiagent-based CSCL tool and multiagent algorithms and frameworks for group formation. In our research, the CSCL systems that we have developed and evaluated include the Intelligent Multiagent Infrastructure for Distributed Systems in Education (I-MINDS) [ 4], [ 5], [ 6], [ 7], [ 8], [ 9], [ 10], [ 11], [ 12], [ 13], [ 14], [ 15], [ 16], [ 17], the ConferenceXP-powered I-MINDS [ 18], [ 19], and the ClassroomWiki [ 20], [ 21]. To be specific, we divide our experience in developing and deploying these CSCL systems into three categories: script, tool, and pedagogy. We describe the challenges we faced and the lessons we learned from the deployments.

Generally, a script in the CSCL environment describes how the students should collaborate and solve problems [ 22]. Under this category, we describe the scripts we have used in our CSCL deployments extracting valuable lessons useful for designing better scripts for future CSCL environments.

The tools used in the CSCL environment facilitate the students' interaction with the instructor and students' collaboration among themselves. Since the collaborative learning outcome depends upon these interactions, a user-friendly tool that facilitates and encourages the right type of interactions and collaborations is a critical component of the CSCL environment. Under this category, we describe our experience in developing three generations of multiagent-based CSCL tools and multiagent-based group formation techniques and discuss valuable insights that would allow future researchers to design and develop effective and efficient CSCL tools.

Finally, under pedagogy, we discuss the important lessons we have learned related to the pedagogical aspects in the CSCL classroom. To be specific, we discuss the impact of multiagent group formation techniques on the performance of the students, the necessity of detailed and accurate tracking and modeling, the impact of accurate assessment of individual student contributions to their groups, and the improvement in students' understanding and performance due to our use of CSCL tools.

The development and deployment of our CSCL tools occurred in three stages. In our initial prototype [ 11], [ 12], [ 14], [ 17], we tested the feasibility of using CSCL as a medium of interactions in the classroom. In our second generation CSCL tool [ 6], [ 7], [ 8], [ 15], [ 23], we developed a multiagent group formation algorithm that formed student groups by balancing the competence and compatibility of its members. We also extended our initial prototype on a different platform (ConferenceXP) [ 18] to test the feasibility of deploying CSCL in classrooms with audio and video capabilities. In our third generation CSCL tool [ 20], [ 21], [ 24], we moved toward asynchronous collaborative writing paradigm and developed a collaborative Wiki for instructional use with multiagent-based tracking, modeling, and group formation.

This paper is organized as follows: in Section 2, we motivate the need for discussing our experience in designing and developing the script, tool, and pedagogy aspects of a CSCL environment. In Section 3, we provide a brief overview of the history of our seven year development of CSCL tools with the details of the initial prototype, the second and the third generation of CSCL tools. Section 4 describes our experience regarding the development of the collaboration script, Section 5 summarizes our experience in designing and developing the tool for the CSCL environment, and Section 6 describes the issues we have encountered with respect to the student pedagogy in CSCL environments. Finally, Section 7 concludes and Section 8 presents our future work.

## Motivation and Overview

The discussions regarding the learned lessons in the CSCL development and deployment are useful from the perspective of collaborative learning and CSCL researchers and the teachers implementing CSCL in their classrooms. First, the lessons learned from the design of the CSCL tool, the script, and the student pedagogy would allow the collaborative learning researchers to evaluate their existing theories using the results of real-world CSCL applications. This evaluation can then lead to refinement of those existing theories and scripts or discovery of new theories. Second, any use of CSCL in improving classroom teaching requires the teacher to design the classroom's evaluation and learners' collaboration sequences and choose learning objectives. So, from the lessons regarding 1) the adoption process of the CSCL script, 2) the choice of evaluation mechanisms, and 3) the choice of the learning objectives, other teachers would be able to go through the decision making aspects of the process more effectively and efficiently. Finally, intelligent agents have been used as an underlying paradigm to improve the effectiveness of CSCL tools in improving classroom teaching. So, our lessons may improve the design and development of such intelligent tools. Here, we summarize our experience in designing intelligent CSCL tools over the last seven years.

Several researchers have discussed their learned lessons regarding the use of CSCL in the classroom and or intelligent support tools to enhance student learning. In [ 25], the researchers discuss their lessons learned regarding the I-HELP system that uses a multiagent system to find matching helpers for the participating students to support their learning. In [ 26], the researchers discuss their efforts toward implementing rapid collaborative knowledge building in traditional classrooms to enhance student learning. In [ 27], the researchers discuss their ideas regarding analyzing the interactions among the students in an asynchronous collaborative forum.

The key distinction between these discussions of the lessons learned in CSCL systems and our discussions is that we are able to combine our observations from the several design-and-deployment phases of CSCL tools, scripts, and pedagogy to provide new insights. For example, to the best of our knowledge, our work is unique with respect to the impact of the use of intelligent agent technology on modeling student activities and using that model to form student groups over several deployments. We also compare and contrast between the synchronous and asynchronous modes of collaborative interactions from our experience with the CSCL deployments.

We hope that these unique insights provided in this paper would 1) allow the CSCL researchers to better evaluate their theories from a practitioner's perspective, 2) enable the CSCL practitioners to develop their scripts, tools, and student pedagogy more effectively and efficiently, and 3) provide insights into the effectiveness of multiagent technologies in solving the problems (e.g., student evaluation, group formation) incurred in CSCL.

## History

In our effort to develop a CSCL application, we have gone through three generations of CSCL tool developments: prototype, second generation, and third generation. In the following sections, we briefly discuss the learning theories and the design and development of our CSCL applications through those three generations. Table 1 summarizes our deployments and findings.

Table 1. Deployments of Our CSCL Tools

### 3.1 Initial Prototype

Figure    Fig. 1. First generation I-MINDS student GUI.

To summarize, the initial version of I-MINDS was designed to create an interactive virtual environment for collaborative learning and contained intelligent agents to provide classroom support (chat and whiteboard-based collaboration) to the teacher and the students and carry out Jigsaw-based collaborative learning sessions.

Deployment 1. In the deployments of this version of I-MINDS, the teacher's goal was to investigate the impact of CSCL on improving the understanding of the students regarding a chosen topic. This study involved 19 undergraduate and graduate students. The instructor delivered two lectures on the geographic information systems (GIS). The study followed a control-treatment protocol and the assessment of the impact of students' collaborative learning was measured by comparing the pre- and post-test scores of the control and treatment group students. The students' activities primarily involved synchronous collaboration with the teacher (chat messages) and their peers using our CSCL tool interface. This initial study found that students were able to use I-MINDS positively toward their in-class, synchronous discussions [ 4]. After the initial study, additional teacher support and multimedia features were added to the system [ 14], [ 31], putting in place the foundation for the multiagent coalition formation component that we eventually have built into ClassroomWiki.

### 3.2 Second Generation I-MINDS with Multiagent Group Formation

Our I-MINDS work was further supported by an NSF SBIR grant DMI-0441249 to enhance the software with distributed computing infrastructure. Though that venture did not turn fruitful in terms of solving the distributedness problem with I-MINDS [ 5], it led us to further investigate the underlying communication and coordination infrastructure for supporting the student agents online through automated group formation. In [ 6] and [ 7], we described an innovative infrastructure to support student participation and collaboration and help the instructor manage large or distance classrooms using multiagent system intelligence. The upgraded I-MINDS contained a host of intelligent agents for each classroom: a teacher agent ranked and categorized real-time questions from the students and collected statistics on student participation, a number of group agents that each maintained a collaborative group and facilitated student discussions, and a student agent for each student that profiled a student and found other compatible students to form the student's “buddy group.” Each agent was capable of machine learning, thus improving its performance and services over time. This improved I-MINDS supported student participation and collaboration and helped the instructor manage large distance classrooms. We the developed a multiagent-based learning-enabled algorithm called VALCAM to form student groups in a structured cooperative learning setting. As reported by the collaborative learning researchers, two critical components that impact the students' learning and collaboration in a student group are their prior knowledge (i.e., competence) [ 32], [ 33] and their social relationship (e.g., compatibility) [ 34], [ 35]. So, we designed the VALCAM algorithm to form student groups by balancing the competence and compatibility of members.

Deployment 1. We deployed I-MINDS in an introductory computer science course (CS1) and conducted studies [ 16] to 1) compare I-MINDS-supported Jigsaw with simple Jigsaw in terms of student performance and 2) evaluate, how I-MINDS+Jigsaw supports structured cooperative learning. The teacher's goal for this deployment was to improve students' learning on particular topics in the subject matter through collaboration. I-MINDS was deployed in two lab sections of the CS1 course and each section had about 15 students. This study followed a control-treatment protocol—where the treatment is delivered through I-MINDS+Jigsaw—and the students' performance was evaluated by comparing their pre- and post-test scores. We found that without face-to-face interactions, students were able to make use of I-MINDS+Jigsaw to achieve performances comparable to the students using face-to-face interactions.

Understanding the usefulness of multiagent systems in tracking, modeling, forming, and scaffolding student groups, we further normalized this idea using a novel framework. In [ 23], we described a formal framework ( $i$ HUCOFS) that takes the dynamic nature of the human users and the complex interplay of human factors (e.g., comfort level, proficiency, changing behavior over time) into account and provides a set of guidelines for developing multiagent systems and algorithms for forming and scaffolding human groups. Our preliminary results suggested the effectiveness of $i$ HUCOFS framework for group formation and scaffolding.

To inventory all the I-MINDS features and design, we detailed the updated version of I-MINDS in [ 8] as a CSCL infrastructure and environment for learners in synchronous learning and classroom management applications for instructors, for large classroom or distance education situations. At this point, the I-MINDS system was able to provide classroom support to the instructor, e.g., Q&A session management, intelligent ranking of students' questions, quiz administration, grade book management, agent-based automatic group formation through VALCAM, individual, and group performance monitoring. I-MINDS also provided standard online collaborative features such as chat rooms and whiteboards and implemented Jigsaw.

Deployment 2. In [ 7], [ 8], we also provided new results of our two-semester-long deployments of I-MINDS in an introductory computer science course (CS1) where the study's objective was to further compare 1) the performance of conventional face-to-face teamwork with the teamwork in structured computer-supported collaborative learning, 2)structured CSCL environment's impact on students' learning and performance. The experiment setup was similar to that of Deployment 1 and the study was performed with two lab sections of an introductory computer science course where each section had 15-25 students. Our results showed that students in I-MINDS+Jigsaw with VALCAM-formed groups were able to perform better and rated their teams and peers better compared to the students in face-to-face teams.

### 3.3 Second Generation I-MINDS' Extension with ConferenceXP

To further advance our I-MINDS environment as a CSCL platform and explore new ways to support and promote group collaborative activities, we extended the I-MINDS platform to enhance an existing group communications platform in development at Microsoft Research: ConferenceXP (CXP) [ 18] (see Fig. 2). The marriage of CXP and I-MINDS was a good fit for several reasons: 1) CXP provided a foundation to build more advanced tools without concerning the developer with the underlying details. I-MINDS improved CXP by adding intelligent collaborative features such as user modeling and evaluation; fine-grained tracking, search, and recall of user activities; individual and group quizzes for student assessment; and question-answer interactions between students and instructors, which automatically learned and weighed keywords used in questions to help instructors pinpoint the “best” questions to answer first or identify key concepts in answers from students. Also, CXP offered multicast communication between multiple students and the instructor. However, this communication protocol [ 18] suffered from scalability and reliability concerns that motivated the inclusion of a more reliable traffic delivery system, i.e., the PGM protocol [ 36].

Figure    Fig. 2. I-MINDS Conference XP classroom statistics for teachers.

Deployment 1. We performed several successful studies [ 18]—including one involving the online Bellevue University in Omaha, NE—that demonstrated the ability of I-MINDS to support question-answer-based learning in a nontraditional classroom setting. We also experienced the challenges of deploying CSCL software to environments where administrative control is provided by an institution separate from the original developers. Specifically, issues involving database management and problems with network connectivity needed to be addressed and resolved between the original developers and systems managers. The teacher's goal for this deployment was to investigate the impact of the newly added student and teacher support tools (question ranking and classification, student contribution summary, etc.) on the collaboration and learning of the students. The assessment of the impact of students' collaborative learning was measured by interviews and surveys. The students' activities were mainly confined to synchronous chat-based communication with the teacher and their peers. The number of students in this study was 20.

Note that at the end of this research and development phase, we realized the need for a CSCL system that is more convenient to use in terms of user-friendliness, installation, maintenance, data tracking, reporting, and incorporation of algorithms. This directly gave birth to ClassroomWiki, a more focused system with features specific to collaborative learning activities, and completely Web based with a server-maintained database.

### 3.4 Third Generation Design: ClassroomWiki

In 2008, using our existing concepts of tracking, modeling, and group formation, we developed the Asynchronous I-MINDS [ 37]. In that version, we used our original group formation algorithm (VALCAM) but provided a user-friendly interface that is geared toward improving student and instructor access to the collaborative writing activities. This interface allowed the students to collaboratively write an essay on a topic with a set of discrete contributions (e.g., propose an idea, reject an idea, revise/extend an idea) and allowed the teachers to track the individual contributions of the students. The results of our semester-long experiments— in an advanced computer science course [ 37] suggested that our agents were able to track and model those students' collaborative actions to create more effective and efficient groups compared to randomly formed groups.

To make our collaborative writing environment more accessible, user friendly, and robust, we extended our asynchronous collaborative writing environment to develop ClassroomWiki [ 20], [ 21], [ 24], that contains:

• An intuitive, user friendly Wiki-like interface (see Fig. 3) that is accessible through a Web browser.
• Detailed tracking and modeling capabilities based on Web 2.0 technologies that are used by:

Our use of a Wiki for asynchronous collaborative writing was inspired by a set of collaborative learning theories [ 39] that use Piaget's model of equilibration [ 40], [ 41], [ 42] to describe how the cognitive conflicts generated by the heterogeneity of the participants of a student group motivates them to contribute to a Wiki and learn from their collaborations. Furthermore, our use of multiagent tracking and modeling models students' contributions and this use of these models in forming better performing students groups was driven by the common problems (e.g., free riding, student apathy [ 3]) stemming from inaccurate tracking and evaluation.

Figure    Fig. 3. ClassroomWiki's asynchronous revision GUI for students.

Notice that we have used a multiagent-based approach for ClassroomWiki for the following reasons. Today's collaborative learning theory provides us directions about what type of groups may improve the collaborative learning outcome (e.g., a group that fosters collaboration and knowledge exchange). However, finding the right combination of students with the characteristics appropriate for a given problem in an uncertain and dynamic environment is a computationally complex problem that requires; 1) modeling the impact of students' attributes to their performances as group members and 2) optimizing the distribution of the participating students into disjoint student groups so that each group is able to a) solve the current task well and b) encourage collaboration among its members to yield better collaborative learning. Research in multiagent systems has yielded coalition formation algorithms that have enabled us to solve this computationally complex problem with intelligent agents who are able to track and model their assigned students and use their learning abilities to form better student groups. In addition, one of our goals regarding our CSCL research is to provide automated intelligent support to the participating students when they are struggling to collaborate (see Section 8). The multiagent system provides agents that use automated reasoning to provide support to individual students.

To test the effectiveness of ClassroomWiki in addressing the group formation and student assessment issues, we employed ClassroomWiki in two deployments as described next. Our deployment results show that ClassroomWiki 1)formed student groups that yielded improved student performance, and 2) provided a detailed and accurate view of student activities that in turn allowed the course teacher to a) more accurately assess a student's contributions, and b) provide specific interventions when necessary, improving student learning.

Deployment 1. We deployed ClassroomWiki in an introductory history course (HIST202—America after 1877, Section 3) [ 21] where ClassroomWiki was used to conduct a collaborative Wiki-writing assignment. The teacher's goal for this deployment was to investigate the improvement of students'

1. understanding on specific topics,
2. general writing and teamwork,
3. research and
4. cross-referencing skills due to a) their participation in collaborative writing using our CSCL tool and b)our intelligent group formation. This study was performed using a control-treatment protocol where the control sets of student groups were formed randomly and the treatment set of student groups were formed using our intelligent group formation method. The evaluation of the impacts of the tool and our intelligent group formation was performed by comparing the collaboratively written essays of the control and treatment set students. The students' activities primarily involved 1) asynchronous collaborative writing, 2) communication in a threaded forum, and 3) doing research online.

In this study, the 145 participating students were divided into 28 groups and each group was assigned to write a Wiki on the topic “US as World Power.” Once assigned to their groups, each student collaborated with his or her group members to prepare a collaborative Wiki writing assignment on the topic “US as a super power” for three weeks. Then, the teacher reviewed each group's Wiki essay and scored each (0-100) and converted a group's Wiki grade to the student members' individual grades by amounts proportional to the modeled student contributions based on the tracked information (see [ 21]). Also, for this deployment, we obtained a pre-ClassroomWiki assignment that was done on Blackboard's ( www.blackboard.com) Wiki system.

Deployment 2. We deployed ClassroomWiki [ 24] in an advanced CS course (CSCE875—Multiagent Systems Section 1) for six Wiki assignments. The teacher's goal for this deployment was to investigate the improvement of students' 1) understanding on specific topics and 2) general writing and teamwork due to our intelligent group formation. This study was performed using a control-treatment protocol: controls set student groups were formed randomly and the treatment set student groups were formed using our intelligent group formation method. The measurement of the impacts of the tool and our intelligent group formation was performed by comparing their collaboratively written essays of the control and treatment sets of students. The students' activities primarily involved 1) asynchronous collaborative writing and 2)communication in a threaded forum.

In this deployment, the 17 participating students were divided into five groups for those six Wiki assignments. Each student then collaborated with his or her group members on their Wiki assignment on a particular Multiagent Systems topic. After due date, the teacher reviewed and scored (0-100) those essays. Then, the teacher converted those group scores to individual student grades using the tracked student contributions (see [ 24]).

## Script

As defined by [ 22], [ 43], collaboration scripts are scaffolds that aim to improve collaboration through structuring the interactive processes between two or more learning partners. Collaboration scripts generally consist of five components:

1. learning objectives,
2. type of activities,
3. sequencing,
4. role distribution, and
5. type of representation.

In this section, we describe our experiences in designing and deploying those components in the aforementioned three generations of CSCL systems.

### 4.1 Types of Activities

Synchronous versus Asynchronous. One key problem in our I-MINDS deployments (see Sections 3.1-3.3) was that due to the synchronous nature of the setting, some students were always absent in their classroom session. As a result, those absent students' groups had to collaborate and solve problems without them. This reduced the collaboration among those group members and hurt their learning. On the other hand, in our ClassroomWiki deployments (see Section 3.4), we were able to create an asynchronous environment that provides the freedom for students to collaborate from anywhere at their own times. So, asynchronous collaborative activities provided the most flexibility and ease of use which supports previously reported CSCL research findings [ 44].

Message-Based Collaboration versus Collaborative Writing. In our first and second generation I-MINDS (see Sections 3.1-3.3), the students' activities were mainly composed of reading/listening teacher-provided lecture/material and collaboration through text messages. In the third generation CSCL studies (see Section 3.4), the students' activities were mainly composed of collaborative writing and participating in a thread-based forum. In our experience, we have found it easier to track and monitor the students' progress in collaborative writing assignments than in the traditional message-based CSCL activities. That is, because due to the atomic nature of asynchronous collaborative writing interactions, it is possible to categorize and discretize the students' activities without any sophisticated methods. For example, by looking at a student's timeline of editions (revisions, additions, deletions), it is possible to have a rough estimate of his or her contributions to the group. On the other hand, analyzing the text of a chat log may require natural language processing and/or information retrieval techniques. Further, in synchronous message-based collaboration, we found that quite a significant amount of messages were off-task—messages exchanged among students that were not related to the lectures or the subject matter [ 4]. Therefore, we have found that collaborative writing

• Allows the teacher to better monitor the progress of the students.
• Creates a perception of accountability among the students.
• Motivates them to collaborate.

### 4.2 Overscripting in Collaborative Writing

In the asynchronous collaborative writing version of I-MINDS (see Section 3.2), to compensate for the lack of natural language processing and analysis tools that could discretize and categorize students' collaborative actions, we have tried to use our collaboration script to also provide a structure to track student contributions. According to our script, the students had to choose to: revise, propose, accept, reject, and extend for each of their writing-related collaboration. However, as mentioned by researchers [ 22], our effort to guide and track students' collaborations was not successful for several reasons. First, although their responses were kept anonymous, the students did not feel comfortable rejecting their group members' contributions/writing pieces due to their existing social relationships. Second, due to our choice of putting low emphasis (i.e., low contribution toward a student's final grade) on the accept collaborative action, the students chose not to accept their group members' written contributions. Finally, the students found it difficult to extend their group members' written topic summaries without changing it. As a result, our efforts regarding carefully designing the script as well as an evaluation scheme that reinforced that script did not yield the improvement in tracking of student activities, collaboration, and learning as initially expected. As a solution to this scripting issue, we have moved toward a more free-form collaboration environment with natural language processing and Web 2.0 technologies for capturing the quality and quantity of collaborative actions in subsequent deployments (see Section 3, Table 1).

### 4.3 Sequencing

In our second generation I-MINDS (see Sections 3.2 and 3.3), we have used the structured Jigsaw model of collaboration (see Section 3, Table 1). Later, in our third generation tools, we have adopted a nonstructured approach where we do not set up any sequence of student activities, and instead encourage the student groups to find the suitable collaboration sequence. In our experience, we have found that the structured collaboration scheme to be difficult to implement in the classroom-oriented synchronous CSCL setting for the following reasons. First, Jigsaw is more suitable for problems that are decomposable. However, not all classroom problems can be easily decomposed for the Jigsaw collaboration scenario. Second, students' expertise for a chosen topic varies for a given set of students, and thus, when a problem is decomposed and divided among the members, the expert members often are able to solve the whole problem by themselves and as a result, do not see the need to follow the sequence of collaboration steps prescribed in Jigsaw. Due to these problems, the students often refrain from collaborating with their group members degrading the quality of collaboration and learning in the CSCL setting.

Finding the design of the sequences of students' interactions in the traditional, synchronous, and classroom-oriented CSCL setting to be difficult to implement, we have moved toward a free-form asynchronous collaborative writing assignments in the third generation CSCL tools. Although, this move eliminated the issues associated with the synchronous-sequenced student interactions with Jigsaw, it created problems (see Table 1, Section 3, Deployment 2) regarding student coordination.

### 4.4 Role Distribution

In our first generation CSCL tool (see Section 3.1), the role of the leader was not specifically assigned. Instead, we have treated all students as equally capable peers who can help their group members. As a result, we have designed several “buddy group” formation algorithms that formed just-in-time student groups whenever the participating students were having difficulty solving the assigned problem. In our second (see Section 3.2) and third (see Section 3.4) generation CSCL tools, we have more closely adopted the idea of having a leader in a student group who is able to guide/teach the rest of the group. As a result, in the second generation, i.e., in I-MINDS, we used a group formation mechanism that formed groups that contained expert as well as nonexpert students. Furthermore, in ClassroomWiki, we chose to form heterogeneous student groups (see Section 3, Table 1). Our observations [ 21] suggest that in heterogeneous student groups, where competent students have an opportunity to take the lead of the group, they emerge as leaders providing: 1) explicit guidance, e.g., in terms of messages and 2) implicit guidance, e.g., written contributions to their groups' topic summary/essay. Although, we have not performed any specific study, our experience suggests that distribution of the heterogeneous students in groups provides an opportunity for their roles (leader/follower) to emerge through interactions.

## Tool

The tool used in a CSCL classroom is a critical component since the collaborative interactions among the students that yield student learning occur through the functionalities provided by the tool they use. As a result, the user-friendliness, the provided functionalities, and the overall design determine how well the students collaborate and consequently how well they are able to learn. Table 2 shows the functionalities provided by our three generations of CSCL tools. Briefly, we see several trends of tool development. First, the first generation (see Section 3.1) features were heavily motivated by our emphasis on synchronous collaboration and lecture delivery, leading to development of audio/visual mechanisms and even Mimio digital whiteboard interfaces. Then, for the second generation (see Sections 3.2 and 3.3) features, we focused on question analysis and ranking for, once again, synchronous use of I-MINDS. At the same time, we also ported I-MINDS to the Microsoft ConferenceXP platform, as discussed in Section 3.3. With this porting to enhance its adoption, we further improved its instructional support features including individual and collaborative quiz mechanism, student contribution charts, and automated question parsing and ranking and management. For the third generation, we shifted our focus to asynchronous collaboration in ClassroomWiki and also significantly simplified the interface by redesigning the features in the first and second generation systems.

Table 2. Functionalities Provided by Our CSCL Tools

In the following, we describe our lessons learned regarding the design, development, and deployments (see Section 3, Table 1) of our CSCL tools and the functionalities they provided to the teacher and the students.

### 5.1 Architecture and Delivery of Software Tool

The design of our first and second generation I-MINDS tool (see Sections 3.1-3.3) used the client-server technology where each heavyweight client, i.e., the agent driven I-MINDS interface, communicated with the server to create a virtual classroom session for its user. Our third generation (see Section 3.4) CSCL tool, i.e., ClassroomWiki also used the client-server technology but provided a lightweight client, i.e., a Javascript-enabled browser, for the students to log on to a server website that hosts the Wiki, the agents, and the repository. In our experience, we found a lightweight client-based architecture to be more advantageous:

• Since the students are not burdened with installing, updating, or troubleshooting the heavy weight client, a web-based design of CSCL tools is better in terms of reducing the learning curve for the students and improved accessibility across computers, operating systems, speed, and performance of hardware.
• A lightweight client-based architecture allowed us to more easily,

### 5.2 Open Source Technologies

Our use of a variety of open source technologies (see Table3) to develop our CSCL tools (see Sections 3.1-3.4) reduced our development, testing, and deployment time.

## Pedagogy

Here, we discuss our lessons regarding four pedagogical aspects of our agent-based CSCL classroom:

1. the need for accurate and detailed tracking and modeling of student behavior in an agent-based CSCL environment (see Section 6.1),
2. the impact of multiagent-based group formation on the individual and collaborative learning performance of the students (see Section 6.2),
3. the need for accurate assessment of individual contributions (see Section 6.3), and
4. the improvement in students' understanding and performance that results due to the use of CSCL in the classroom (see Section 6.4).

### 6.1 Accurate and Detailed Tracking and Modeling

Accurate and detailed tracking and modeling are an essential component of a CSCL tool because of a variety of reasons. First, such tracking and modeling allow the teacher to better understand the learning dynamics of the students in the CSCL environment. This insight is essential for any teacher who would like to improve the students' participation and collaboration by changing the 1) instruction delivery method, 2) collaboration script, or 3) the design of the CSCL tool. Furthermore, such tracking and modeling allow the teacher to provide scaffolding to the struggling students or student groups proactively and timely and discover hidden trends and patterns in the student behavior. Understanding this necessity from our first (see Section 3.1) and second generation (see Sections 3.2 and 3.3) CSCL deployments, we have utilized Web 2.0 technologies to track all user interactions in our third generation (see Section 3.4) CSCL tool.

### 6.2 Impact of Multiagent Group Formation

Researchers describe that due to the impact of group composition on the collaboration and learning, student group formation remains a challenge to be addressed [ 3]. In our I-MINDS deployments (see Sections 3.1 and 3.3), we have used intelligent group formation method that balanced the competence and compatibility of the students in a group. Furthermore, in our ClassroomWiki deployments (see Section 3.4), we have used a learning-enabled multiagent group formation algorithm that used the tracked student attributes to build a model of the participating students' contributions, and then, used that model to find the most appropriate composition of student groups. Table 4 summarizes the results regarding our use of group formation schemes, which shows the success of multiagent group formation schemes in forming student groups that improve student performance.

Table 3. Our Use of Open Source Technologies

Table 4. Results of Our Use of Intelligent Group formation Scheme

### 6.3 Assessment of Individual Contributions

Due to the complex nature of the students' interactions in a collaborative learning environment, accurately assessing the contributions of a student to his or her group is difficult [ 3]. However, student assessment is a critical part of the pedagogy in CSCL environments since the inferences and interpretations the students get from their evaluations guide and drives their learning [ 45], [ 46]. Furthermore, the assessment infuses the teacher's instruction with objective information to stimulate deeper knowledge and motivate personal goals in students and educators [ 47]. Furthermore, inaccurate assessment of the individual contributions of the students prevents the teacher's pedagogy with issues like: free riding, the sucker effect, etc., [ 3]. All these issues reduce the quality of the students' collaborations, and thereby reduce the learning benefits of the collaborating students. To overcome these problems, we have improved our third-generation (see Section 3.4) CSCL tool design by adding accurate and detailed student modeling.

One way to evaluate the contributions of a student toward his or her group is by tracking all students' interactions with their group members and with the system at a microlevel (i.e., in details), and then, using the quantity as well as the quality of the contributions of those interactions to create a model that represents a student's contributions toward his or her group. Such detailed assessment method would allow the teacher to evaluate the students according to their contributions. Furthermore, the detailed assessment method may alleviate the common issues like free riding, sucker effect, student apathy, etc. Furthermore, such tracking and modeling based assessment of students' contributions would allow the teacher to proactively intervene or scaffold the student groups or individuals who are not collaborating before the collaborative session is over.

Table 5 shows the correlations between the students' scores in the ClassroomWiki assignment and their scores in the other tests/assignments in Deployment 1 (HIST 202) (see Section 3, Table 1). Using these correlation values, we were able to conduct a baseline comparison of ClassroomWiki's performance with a Wiki that does not provide any tracking/modeling of student activities for individual assessment. This Wiki system provided by Blackboard had an interface and functionality similar to ClassroomWiki's but did not provide any tracking/modeling of students' interactions for assessing their contributions toward their groups. Table 5 indicates that the correlation between the students' exam scores in the class and their scores in Wiki were higher for ClassroomWiki than for Blackboard Wiki's. Since these exam scores represent the knowledge and understanding of the students gained in the class, the higher correlations suggest that the detailed microlevel tracking and modeling allowed ClassroomWiki to capture students' performances more accurately than Blackboard's Wiki.

Table 5. Correlation of Student Scores in ClassroomWiki Deployments

Furthermore, Table 5 shows that in Deployment 2 (see Section 3.2), the students' evaluations for the topic summary scores were highly correlated with their final exam evaluations. The values in Table 5 indicate that except for the first document analysis assignment, the scores the students received in the ClassroomWiki assignments were well correlated with their scores in the other assignments/exams. These moderately high-correlation values suggest that individual student scores that were calculated based on ClassroomWiki's student contribution summary (e.g., number of words added/deleted, number of forum messages posted, etc.) closely represented the actual performance of the students in the other tests in the class.

### 6.4 Student Learning

The ultimate aim of CSCL is to enhance student learning through collaboration. Although not statistically significant, the analysis of the results of our studies suggests that participation in CSCL activities actually helped the students learn to perform better in the classroom, e.g.,

• Better Understanding of Subject Matter. In our first generation deployment (see Section 3, Table 1) of the CSCL tool [ 11], [ 17], we have observed that the students who collaborated using our CSCL tool were able to learn/understand the chosen subject matter better. Although not statistically significant, this result suggested that the intelligent interface, the interactivity with the instructor, the communication, and the archival/retrieval capabilities helped the students learn and understand the subject matter better than the students working in the traditional classroom setting.
• Improvement in Students' Performances. In our third generation deployment (see Section 3, Table 1) of the CSCL tool (Deployment 1 [ 20]), we have also compared the students' performances in the classroom before and after their participation in using the CSCL tool. Our results show that after participating in one CSCL session, the performances in classroom tests that covered the topics of that CSCL session improved (improvement in mean: 2.71, median: 3.00 in a scale of 100). Although not statistically significant, this improvement in student performances does suggest positive impact of CSCL on students' learning.

## Conclusions

In this paper, we have summarized our experiences in developing and deploying three generations of CSCL tools (I-MINDS and ClassroomWiki) over the last seven years regarding the script, tool, and pedagogy in CSCL. By analyzing the collected deployment data and our interactions with the users (teachers and students), we were able to derive the following useful lessons:

Script.

• Asynchronous collaborative learning activities allow the students greater freedom in their collaborations with their peers and alleviate the student absentia related problems associated with synchronous activities.
• Collaborative writing-based interactions provide more opportunity for tracking, modeling, and categorizing students' interactions and contributions to the groups.
• Overscripting students' interactions to track and model their contributions often yields unexpected results and may not be helpful for the students.
• Even without explicit role distribution/assignment, expert and hardworking students often emerge as the leaders in heterogeneous student groups.

Tool.

• A lightweight client-based architecture may
• Use of open source technologies allows us to effectively and efficiently develop the functionalities and interfaces of CSCL tools.

Pedagogy.

• Accurate and detailed tracking and modeling of students' interactions may allow the teacher to 1)better understand the learning dynamics of the students in the CSCL environment, 2) provide scaffolding to the struggling students or student groups proactively and timely, and 3) discover hidden trends and patterns in the student behavior.
• Accurate individual contribution assessment may improve the free-riding and sucker-effect problems by improving student accountability regarding their contributions toward their groups.
• CSCL may improve students' understanding of the discussed subject matter and may improve their performance regarding solving problems related to that subject matter.

## Future Work

We are now planning several large-scale deployments of our third-generation CSCL tool (i.e., ClassroomWiki) in two different university level courses. In addition, we are now working on an automated intelligent support tool for the teacher and the students that would provide: 1) categorized and summarized alerts for the teachers so he or she can help the student groups that are having difficulty and 2) provide content-dependent and teamwork-related suggestions to the individuals and student groups that are having difficulty collaborating.

Some of the tools and techniques in our initial prototype of I-MINDS were not tested well since we had very few participants in the study. This lack of students sometimes yielded results that were not statistically significant. We are now analyzing the data from our just completed semester-long study to corroborate the findings from the deployments of our initial prototype. Furthermore, we are planning several large-scale deployments of ClassroomWiki to more comprehensively validate the lessons from our initial deployments for which we did not have statistically significant results.

Finally, in our first and second generation I-MINDS experiments, we have observed that due to easy collaborative problem assignment, the expert students expressed that the collaboration in the CSCL environment provides them no added benefits and some of them actually opted out from the experiment. We are now preparing a collaborative motivation scale and a set of collaborative problems with varying difficulty that would be used in our future deployments of ClassroomWiki to measure the impact of the difficulty of the collaborative problem on the students' motivation for collaboration.

## Acknowledgments

The I-MINDS project was seed-funded by a National Center for Information Technology in Education grant in 2002 and 2003. Further development on I-MINDS was subsequently funded with US National Science Foundation (NSF) grant DMI-0441249 in 2005. Extension to ConferenceXP was supported by two separate grants from Microsoft Research from 2005 to 2007, as well as three undergraduate research UCARE grants from UNL and Pepsi. ClassroomWiki was supported in part by NSF grant CNS-0829647. The authors thank I-MINDS and ClassroomWiki team members: Jameela Al-Jaroodi, Xuli Liu, Suresh Namala, Phanivas Vemuri, and Xuesong Zhang. They also thank instructors and faculty who adopted I-MINDS/ClassroomWiki or facilitated their deployments and studies: Charles Ansorge, faculty from Bellevue University, Will Thomas, and June Griffin.

## References

• 1. P.A. Kirschner, R.L. Martens, and J. Strijbos, “CSCL in Higher Education?: A Framework for Designing Multiple Collaborative Environments,” What We Know about CSCL and Implementing it in Higher Education, pp. 3-30, Kluwer Academic, 2004.
• 2. S. Lukosch, M. Blais, and M. Rasel, “CSCL, Anywhere and Anytime,” Proc. Int'l Workshop Groupware: Design, Implementation, and Use, pp. 326-340, 2006.
• 3. T.S. Roberts, and J.M. McInnerney, “Seven Problems of Online Group Learning (and Their Solutions),” Educational Technology and Soc., vol. 10, pp. 257-268, 2007.
• 4. L.-K. Soh, H. Jiang, and C. Ansorge, “Agent-Based Cooperative Learning: A Proof-of-Concept Experiment,” Proc. Technical Symp. Computer Science Education, pp. 368-372, 2004.
• 5. L.-K. Soh, and H. Jiang, “Commercializing a Multiagent-Supported Collaborative System,” Proc. Int'l Conf. Autonomous Agents and Multiagent Systems (AAMAS '06), pp. 1522-1529, 2006.
• 6. L.-K. Soh, N. Khandaker, and H. Jiang, “Multiagent Coalition Formation for Computer-Supported Cooperative Learning,” Proc. Innovative Applications of Artificial Intelligence Conf. (IAAI '06), pp.1844-1851, 2006.
• 7. N. Khandaker, L.-K. Soh, and H. Jiang, “Student Learning and Team Formation in a Structured CSCL Environment,” Proc. Int'l Conf. Computers in Education (ICCE '06), pp. 185-192, 2006.
• 8. L.-K. Soh, N. Khandaker, and H. Jiang, “I-MINDS: A Multiagent System for Intelligent Computer-Supported Collaborative Learning and Classroom Management,” Int'l J. Artificial Intelligence in Education, vol. 18, pp. 119-151, 2008.
• 9. C. Sombattheera, and A. Ghose, “A Best-First Anytime Algorithm for Computing Optimal Coalition Structures,” Proc. Int'l Conf. Autonomous Agents and Multiagent Systems (AAMAS '08), pp.1425-1428, 2008.
• 10. L.-K. Soh, X. Liu, X. Zhang, J. Al-Jaroodi, H. Jiang, and P. Vemuri, “I-MINDS: An Agent-Oriented Information System for Applications in Education,” Agent-Oriented Information Systems, pp. 16-31, Springer, 2004.
• 11. X. Liu, X. Zhang, J. Al-Jaroodi, P. Vemuri, H. Jiang, and L.-K. Soh, “I-MINDS: An Application of Multiagent System Intelligence to On-Line Education,” Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics (SMC '03), pp. 4864-4871, 2003.
• 12. X. Zhang, L.-K. Soh, X. Liu, and H. Jiang, “Intelligent Collaborating Agents to Support Teaching and Learning,” Proc. Int'l Electro-Information Technology Conf., pp. 22-25, 2005.
• 13. L.-K. Soh, and H. Jiang, “Intelligent Multiagent Cooperative Learning System,” Encyclopedia of Human Computer Interaction, C. Ghaoui ed., pp. 18-23, Idea Group Reference, 2005.
• 14. X. Zhang, L.-K. Soh, and H. Jiang, “Using Multiagent Intelligence to Support Synchronous and Asynchronous Learning,” Studies in Fuzziness and Soft Computing: Knowledge-Based Virtual Education, C. Chaoui, M. Jain, V. Bannore, and L.C. Jain, eds., pp.111-140, 2005.
• 15. L.-K. Soh, N. Khandaker, X. Liu, and H. Jiang, “A Computer-Supported Learning System with Multiagent Intelligence,” Proc. Int'l Conf. Autonomous Agents and Multiagent Systems (AAMAS '06), pp. 1556-1563, 2006.
• 16. L.-K. Soh, N. Khandaker, and H. Jiang, “Computer-Supported Structured Cooperative Learning,” Proc. Int'l Conf. Computers in Education (ICCE '07), pp. 428-435, 2005.
• 17. X. Liu, X. Zhang, L.-K. Soh, J. Al-Jaroodi, and H. Jiang, “A Distributed, Multiagent Infrastructure for Real-Time, Virtual Classrooms,” Proc. Int'l Conf. Computers in Education (ICCE '03), pp. 640-647, 2003.
• 18. L. Miller, A. Eck, L.-K. Soh, and H. Jiang, “Statistics and Analysis Tools for a Computer-Supported Collaborative Learning System,” Proc. Frontiers In Education Conf. (FIE '07), pp. F3J-1-F3J-6, 2007.
• 19. A. Eck, L.-K. Soh, H. Jiang, and T. Chou, “Testing Collaborative Traffic over Wireless Protocols,” Proc. Frontiers In Education Conf. (FIE '07), pp. F4J-11-F4J-16, 2007.
• 20. N. Khandaker, and L.-K. Soh, “ClassroomWiki: A Wiki for the Classroom with Multiagent Tracking, Modeling, and Group Formation,” Proc. Int'l Conf. Autonomous Agents and Multiagent Systems (AAMAS '10), pp. 1377-1378, 2010.
• 21. N. Khandaker, and L.-K. Soh, “ClassroomWiki: A Collaborative Wiki for Instructional Use with Multiagent Group Formation,” IEEE Trans. Learning Technologies, vol. 3, no. 3, pp. 190-202, July-Sept. 2010.
• 22. P. Dillenbourg, “Over-Scripting CSCL: The Risks of Blending Collaborative Learning with Instructional Design,” Three Worlds of CSCL. Can We Support CSCL, P.A. Kirschner, ed., pp. 61-91, Open Univ. Nederland, 2002.
• 23. N. Khandaker, and L.-K. Soh, “Formation and Scaffolding Human Coalitions in I-MINDS - A Computer-Supported Collaborative Learning Environment,” Proc. Int'l Conf. Autonomous Agents and Multiagent Systems (AAMAS '06), pp. 64-75, 2006.
• 24. N. Khandaker, and L.-K. Soh, “A Wiki with Multiagent Tracking, Modeling, and Coalition Formation,” Proc. Innovative Applications of Artificial Intelligence Conf. (IAAI '10), pp. 1799-1806, 2010.
• 25. J. Greer, G. McCalla, J. Vassileva, R. Deters, S. Bull, and L. Kettel, “Lessons Learned in Deploying a Multi-Agent Learning Support System: The I-Help Experience,” Proc. Int'l Conf. AI in Education, pp. 410-421, 2001.
• 26. F. Ng, C. Looi, and W. Chen, “Rapid Collaborative Knowledge Building: Lessons Learned from Two Primary Science Classrooms,” Proc. Int'l Conf. for the Learning Sciences, pp. 115-123, 2008.
• 27. T. Bratitsis, and A. Dimitracopoulou, “Interaction Analysis in Asynchronous Discussions: Lessons Learned on the Learners' Perspective, Using the DIAS System,” Proc. Int'l Conf. Computers in Education (ICCE '07), pp. 90-92, 2007.
• 28. D. Johnson, R. Johnson, and K. Smith, Cooperative Learning: Increasing College Faculty Instructional Productivity. George Washington Univ., 1991.
• 29. J. Clarke, “Pieces of the Puzzle: The Jigsaw Method,” Handbook of Cooperative Learning Methods, pp. 34-50, Greenwood Press, 1994.
• 30. E. Aronson, and N. Blaney, The Jigsaw Classroom. Sage, 1978.
• 31. L.-K. Soh, X. Zhang, and H. Jiang, “Intelligent Collaborating Agents to Support Teaching and Learning,” Technology-Based Education: Bringing Researchers and Practitioners Together, PytlikZillig, M. Bodvarsson, and R. Bruning eds., pp. 203-224, Information Age Publishing, 2005.
• 32. G. Stahl, “Building Collaborative Knowing: Elements of a Social Theory of CSCL,” What We Know about CSCL and Implementing It in Higher Education, pp. 53-85, Springer, 2004.
• 33. A. Inaba, T. Supnithi, M. Ikeda, R. Mizoguchi, and J. Toyoda, “How Can We Form Effective Collaborative Learning Groups?” Intelligent Tutoring Systems, pp. 282-291, Springer, 2000.
• 34. C. Chalmers, and R. Nason, “Group Metacognition in a Computer-Supported Collaborative Learning Environment,” Proc. Int'l Conf. Computers in Education (ICCE '05), pp. 35-41, 2005.
• 35. D. Johnson, and R. Johnson, Learning Together and Alone: Cooperative, Competitive, and Individualistic Learning. Allyn and Bacon, 1999.
• 36. M. Boyer, and M. Sighireanu, “Synthesis and Verification of Constraints in the PGM Protocol,” Proc. Int'l Symp. Formal Methods Europe (FME '03), pp. 264-281, 2003.
• 37. N. Khandaker, and L.-K. Soh, “Improving Group Selection and Assessment in an Asynchronous Collaborative Writing Application,” Int'l J. Artificial Intelligence in Education, 2010.
• 38. L.-K. Soh, and N. Khandaker, “Formation and Scaffolding Human Coalitions with a Multi-Agent Framework,” Proc. Int'l Conf. Autonomous Agents and Multiagent Systems (AAMAS '07), pp.394-396, 2007.
• 39. U. Cress, and J. Kimmerle, “A Systemic and Cognitive View on Collaborative Knowledge Building with Wikis,” Int'l J. Computer-Supported Collaborative Learning, vol. 3, pp. 105-122, 2008.
• 40. J. Piaget, “Piaget's Theory,” Carmichael's Manual of Child Psychology, pp. 703-732, Wiley, 1970.
• 41. J. Piaget, “Problems of Equilibration,” Topics in Cognitive Development, vol. 1, pp. 3-14, 1977.
• 42. J. Piaget, The Development of Thought: Equilibration of Cognitive Structures. The Viking Press, 1977.
• 43. I. Kollar, F. Fischer, and W. Friedrich, “Collaboration Scripts - A Conceptual Analysis,” Educational Psychology Rev., vol. 18, pp. 159-185, 2006.
• 44. P. Abrami, and E. Bures, “Computer-Supported Collaborative Learning and Distance Education,” Am. J. Distance Education, vol. 10, pp. 37-42, 1996.
• 45. B. Knight, and C. Knight, “Cognitive Theory and the Use of Computers in the Primary Classroom,” British J. Educational Technology, vol. 26, pp. 141-148, 1995.
• 46. J. Macdonald, “Assessing Online Collaborative Learning: Process and Product,” Computers and Education, vol. 40, pp. 377-391, 2003.
• 47. E.L. Baker, and H.F.J. O'Neil, “Measuring Problem Solving in Computer Environments: Current and Future States,” Computers in Human Behavior, vol. 18, pp. 609-622, 2002.