WIKIS today are gaining popularity as a tool for implementing collaborative learning for instructional uses. Recently published examples of such uses of Wiki are described in [ 1
], [ 2
], [ 3
], [ 4
], [ 5
], [ 6
], [ 7
], [ 8
], [ 9
], [ 10
], [ 11
], [ 12
]. Typical Wiki environments are designed to generate informative artifacts (e.g., webpages) through cooperation, where the quality of the generated content
is the focus. However, when used as a collaborative writing educational tool, the quality of the collaboration among the group members is as important, if not more, as the quality of the artifacts generated by the groups. This is because the improvement of students' knowledge and understanding due to learning largely depends on how well they collaborate to exchange knowledge and information with one another [ 13
]. One way to improve the collaboration, and thus, the collaborative learning outcome of the students in a Wiki is by addressing the important factors that impact the collaboration process of the students [ 14
], e.g., 1) group formation and 2) individual assessment of students.
The method used for forming student groups in a collaborative learning environment impacts the collaboration and the learning of the students because some groups of students are able to collaborate better than others. For example, researchers [ 14
] suggest that forming heterogeneous student groups that combine students with a variety of skills may help them collaborate better. Researchers [ 14
] explain that the improvement in collaboration and learning in heterogeneous student groups occur since the students with different perspectives
are able to exchange their ideas and skills with their group members. So, a Wiki that forms heterogeneous groups considering the knowledge and skills of the students would yield better collaborative learning outcomes (e.g., student performance, collaboration) than a Wiki that does not consider these factors. In addition, researchers [ 14
] indicate that accurate assessment of student contributions remains a difficult challenge to overcome in a collaborative learning environment. For example, typical collaborative learning environments often suffer from free riding phenomenon [ 14
], where some students do not contribute to their group's work but take credit as a group member. This free riding phenomenon, if left unchecked, may create student antipathy toward collaborative work and reduce collaboration in collaborative Wiki assignments [ 8
]. This means that a Wiki tool that allows the teacher to track the students' activities to better assess their individual contributions toward their groups would motivate the students to collaborate yielding better collaborative learning outcome. Such improved tracking would also allow the teacher to provide specific and precise interventions proactively, which may alleviate problems like free riding [ 14
However, the typical Wiki tools we have researched [ 1
], [ 2
], [ 3
], [ 4
], [ 5
], [ 6
], [ 7
], [ 8
], [ 9
], [ 10
], [ 11
], [ 12
] do not provide any group formation techniques and do not adequately address the assessment of student contributions toward the group. So, to improve the use of Wiki as an educational tool, in ClassroomWiki, we have combined a set of pedagogy theories that explains the collaborative learning process in Wikis with the tracking, modeling, and group formation capabilities of a multiagent framework to design and implement an improved Wiki-based collaborative learning tool. Our ClassroomWiki uses the Multiagent Human Coalition Formation (MHCF) framework (based on the principles described in [ 15
]) to form heterogeneous student groups using the data tracked in ClassroomWiki environment. The novelty of this MHCF framework is in its group formation process, which, due to its design and implementation, 1) adapts to the changing behavior of the students and 2) balances the heterogeneity of the members so that a student group contains students of all levels of performances. None of these two aspects is accommodated by the typical Wiki tools (e.g., [ 1
], [ 2
], [ 3
], [ 4
], [ 5
], [ 6
], [ 7
]). However, recently published learning theories [ 13
] that describe the collaborative learning process in Wikis suggest that heterogeneity in the performance levels of the members of a student group may improve the collaborative learning outcome in Wikis. Furthermore, since the performance of a student while working on a collaborative Wiki writing activity changes over time, a group formation process that balances the heterogeneity requires an adaptation mechanism that can utilize the change in students' performances to keep forming heterogeneous groups over time.
To test the effectiveness of ClassroomWiki in addressing the group formation and student assessment aspects, we have employed ClassroomWiki in a three-week-long collaborative writing experiment in a classroom with 145 students. The results of this deployment show that ClassroomWiki 1) was able to form student groups that yielded improved student performance and 2) provided a detailed and accurate view of student activities, which, in turn, allowed the course teacher to a) more accurately assess a student's contributions, and b) provide specific interventions when necessary, thereby improving student learning experience.
The remainder of the paper is organized as follows: In Section 2, we derive a set of design principles from a recently published research work that models the student learning in Wikis. Section 3 describes the ClassroomWiki architecture and the MHCF framework of group formation. Section 4 outlines our implementation while Section 5 describes our experiment setup and results. Finally, we conclude with our future work in Section 6.
2. Wiki as a Collaborative Learning Tool
In this section, we discuss a sociocognitive learning model (proposed in [ 13
]) that explains the student learning process in a collaborative Wiki-based system and discuss how that theory motivates us to incorporate a group formation algorithm and better tracking and modeling functionalities in ClassroomWiki.
2.1 A Model of Collaborative Knowledge Building with Wikis
Cress and Kimmerle [ 13
] model the collaborative knowledge building with Wikis as a two-component system composed of 1) a social system, i.e., the Wiki and 2) a cognitive system, i.e., the students' cognitive processes. Cress and Kimmerle discuss that through the structural coupling based on language
, the social system is able to effect changes into the participating learners' cognitive system. They model that coupling or influence with the process of externalization
. While working in the Wiki environment, learners contribute to topics or create artifacts that they have some knowledge on; which is the externalization process. On the other hand, the internalization process refers to the integration of the knowledge contained in the Wiki artifacts prepared by other learners. Thus, the externalization process of one learner contributes to the internalization process of another and this collaboration increases the knowledge of the learner who internalizes that knowledge. Furthermore, this internalization interacts with the knowledge a learner already has and produces emergent
knowledge—that is, knowledge that was neither the part of the externalized artifact nor was possessed by the learner.
So, according to Cress and Kimmerle, individual learning in the Wiki setting results from the interplay between the externalization and internalization processes of the learners and they describe this coevolution from the interplay of the two processes from the viewpoint of Piaget's model of equilibration [ 16
[, [ 17
], [ 18
]. This theory proposes that when the environment's knowledge and a person's prior knowledge do not fall in line, it causes a cognitive conflict which can be resolved in two ways, through: 1) assimilation, i.e., by adding
new congruent information to the prior knowledge or 2) accommodation, i.e., by modifying
the prior knowledge to better understand the environment's knowledge. So, while interacting with the Wiki, people learn as a result of the internalization and externalization processes, where this learning takes place due to accommodation and or assimilation occurring in both 1) the learner's cognitive processes and 2) the Wiki. When the learners are merely extending the Wiki by adding information, assimilation is taking place and accommodation occurs when the learners are extending the Wiki and reorganizing the existing information. To what extent these processes occur is determined by the learners' participation in a Wiki, which is also largely determined by their motivation. According to Cress and Kimmerle, this motivation is a function of 1) the incongruity between the learner's knowledge and the information in the Wiki and 2) the valence that Wiki topic has for the learner. The authors mention that a medium level of incongruity [ 19
] and high positive valence [ 20
] yields optimal learner motivation.
2.2 Motivation for Improved Group Formation and Individual Contribution Assessment in Wikis
After discussing the sociocognitive learning theories and a model that explains student learning in Wikis, we argue that 1) a heterogeneous group formation algorithm and 2) a detailed and accurate individual student contribution assessment mechanism would improve the collaboration and learning of the students in a Wiki.
First, a student group that consists of students with heterogeneous levels of knowledge would bring different perspectives
on the same assigned topic and would more likely generate cognitive conflicts (mentioned by Cress and Kimmerle [ 13
]) than a randomly formed or student-selected group. That increased cognitive conflict would then motivate the students to collaborate (i.e., change the Wiki content) so that the level of incongruity is reduced. That increased collaboration would then improve the participating students' collaborative learning outcome (e.g., the students' performance). Finally, the heterogeneity of the student groups should be balanced in such a way that the difference of the knowledge of the students is not so large
that it hinders their collaboration [ 21
]. So, a student group with students who have heterogeneous levels of knowledge/expertise may increase the cognitive conflicts among the students, and thus, their collaborative learning opportunities and we write the first guiding principle for group formation algorithm as follows:
Heterogeneity principle.The group formation algorithm should balance heterogeneity of learner expertise in a group in such a way that they are less likely to give rise to situations where the participating learners would be demotivated due to too high or too low incongruity between their expertise and the Wiki artifacts they are working on.
Note that the heterogeneity among the members of a student group would vary according to the environment— such as the collaborative writing assignment and the communication modes available—and the learner's characteristics. Furthermore, the expertise of the learners in a collaborative learning setting may evolve [ 22
] and their participations in their groups may change [ 23
] as they progress through their syllabus and coursework. Thus, the group formation algorithm has to model those changes and adapt its group formation technique accordingly. Thus, we may write our second principle as follows:
Adaptation principle.The group formation algorithm needs to track and model students' activities and performances to capture the changing cognitive states of the students so that it is able to form better groups over time.
Further, a detailed and accurate tracking of students' activities in the Wiki would also improve student collaboration and learning for the following reasons: First, as pointed out by the CSCL researchers [ 14
], free riding is one of the main problems in most CSCL environments. This occurs because many CSCL systems do not track and present a summary or model of the students' activities to the teacher, which would allow accurate assessment of student's effort. As a result, the free riding students receive credit without doing any work, which discourages
the hard working expert students from participating in the collaborative activities. Second, the lack of tracking yields lack of accountability for the students and may give rise to occasional irresponsibility, which reduces student participation in the Wiki environment [ 8
]. On the other hand, accurate assessment would raise a student's perception of valence since that student can then be held accountable for his or her contribution toward the final quality of his or her group's work. Third, the detailed information about student behavior collected by the tracking may allow the teacher to 1) provide specific and precise help to the students or groups who are not able to collaborate and 2) discover trends and patterns in students' collaborative behavior. Such trends and patterns are important since they have the potential of providing insights into the collaborative process of students leading to improvements in both the process and the environment. Finally, detailed tracking and modeling over time would allow the group formation algorithm to capture
and adapt to
the changing knowledge and collaborative behavior in the environment. So, our next guiding principle is the following:
Tracking and modeling principle.The Wiki should be able to track and present the students' contributions toward their group so that the scores they receive accurately represent the effort they have put in toward their groups' final outcome.
Although our discussion here motivates the need for a group formation method and better tracking and modeling of students in collaborative Wiki assignment, among the research approaches we have studied [ 1
], [ 2
], [ 3
], [ 4
], [ 5
], [ 6
], [ 7
], [ 8
], [ 9
], [ 10
], [ 11
], [ 12
], no one addressed the student group formation based on tracked student behavior in Wikis. However, Trentin [ 4
] addressed the evaluation of the individual student contribution aspect and none of them addressed the group formation aspect of collaborative Wiki writing. In [ 4
], Trentin combined student activity counts, such as the number of messages, with peer reviews to calculate the score of a student. However, that data were manually extracted from the Wiki traces. Such methods are error prone and difficult to implement in large classrooms and inconvenient for the teacher since it generally will require a large amount of work overhead if he or she wants to provide accurate and specific interventions.
ClassroomWiki is composed of four conceptual modules ( Fig. 1
) as follows:
1. Wiki (WIM),
2. communication (COM),
3. tracking and modeling (TAM), and
4. group formation (GFM).
First, the WIM allows the teacher to create and assign Wiki assignments to the students. For students, the WIM allows 1) revision and 2) versioning of their Wiki assignment text. Second, COM facilitates student and teacher communications through 1) assignment-specific topic-based forums used by the teacher and the student groups and 2) announcements and e-mails from the teacher to the individual students or student groups. Third, TAM tracks students' interactions with their group members and with the modules of ClassroomWiki to build a detailed student model. That model is then used to 1) better assess the students' individual contributions toward their groups' Wiki-related work leading to a) detection and prevention of free riding behavior and b) precise and specific interventions from the teacher to improve collaboration and 2) better group formation. Finally, the GFM allows the teacher to automatically form student groups randomly or by using the tracked student models and the MHCF framework.
Fig. 1. ClassroomWiki modules.
3.1 Wiki Module (WIM)
The WIM allows the teacher to create and assign a topic to the student groups in the course. Once assigned by the teacher, the student groups collaborate to create a Wiki on that topic which is evaluated by the teacher after the due date of the assignment. The WIM consists of the assignment and the versioning component.
The assignment component of WIM allows the Wiki teacher to create Wiki assignments for the participating students. The Wiki assignment specifies the topic, the requirements for the final submitted version (e.g., required sections, word limit, due date), and minimum size of the student groups. Once created by the teacher, the assignment component stores this specification which can then be accessed by the students (while they are collaborating) and by other modules (e.g., the group size is used by the group formation module).
The versioning component tracks and stores all changes (e.g., addition, deletion) made to the Wiki by all members of each student group. This tracking allows a student to view a color-coded (e.g., different colors for additions, deletions, and unchanged text) history of changes of the Wiki made by his or her group members.
3.2 Communication Module (COM)
ClassroomWiki's COM consists of two components: 1) a topic-based forum and 2) an announcement system. The topic-based forum in the COM facilitates the collaboration process of the students in two ways. First, while collaborating, the members of a student group can discuss their plan or approach of writing the Wiki, their revisions, and other Wiki-related questions and comments in the forum. Second, the forum allows the teacher to respond to questions posed by the members of a student group for their Wiki. The announcement system allows the teacher to notify the students about changes or other assignment-specific matters. Furthermore, the forum component supports the four forms of learning mentioned in Section 2.1 by allowing the students to discuss the Wiki assignments with their group members using a topic-based forum. This COM allows the students to discuss any ideas or concepts contained in the Wiki, and thus, assimilate and accommodate new knowledge from the forum while they are collaborating. Further, while the students exchange knowledge in the forum and resolve their cognitive conflicts through discussions in the forum, the forum itself transforms (i.e., external assimilation and accommodation in Section 2.1) due to those posted discussions. Note that the teacher is also able to participate in a student group's forum, where that participation impacts the group's knowledge on the Wiki topic and thereby enhances the assimilation and accommodation process of that group's members.
3.3 Tracking and Modeling Module (TAM)
The goal of the TAM in ClassroomWiki is to create and maintain a model of every participating student according to the tracking and modeling principle described in Section 2.2. This student model in ClassroomWiki is built using information regarding student activities that can be divided into the following five categories:
1. Active use—the actions of a student that push information onto his or her group's Wiki and change the content of that Wiki, e.g., the number of words: a) added, b) deleted, and c) rearranged.
2. Passive use—student activities in ClassroomWiki that pull information from his or her group's Wiki and do not result in a change in the contents of that Wiki. For example, the number of times a student views: a) the revision history of their group's Wiki, b) the topics posted by other group members, and c) the messages in his or her posted topics.
3. Interaction—a student's interactions with his or her group members while collaborating, e.g., the total number of topics created, the total number of replies posted, the size of his or her messages in words, and the average number of other group members who replied to a student's posted topic.
4. Survey response—a student's responses to the various surveys or questionnaires posted by the teacher. These surveys can be designed to capture a student's opinion about the effectiveness of his or her group, peers, or the ClassroomWiki itself. For example, a student may be asked to evaluate the contribution of another group member toward their group's Wiki.
5. Evaluation—the evaluation scores received by a student for all Wiki-related activities, e.g., a teacher's evaluation of a student for his or her contribution for the group Wiki.
is the set of all students in ClassroomWiki, Table 1
summarizes the information tracked by the Tracking and Modeling module for a student
in ClassroomWiki. The tracked information in Table 1
is used in ClassroomWiki to build student models that: 1) are used by the Group Formation module (GFM) to realize the MHCF group formation mechanism and 2) allow the teacher to assess the individual contribution levels of students facilitating specific and precise teacher interventions. The details of the use of this stored information are provided in Section 4.
Table 1. Information Stored by TAM for Student
Using the tracked information described in Table 1
, ClassroomWiki builds a model for each of the participating students. The student model
of a student
is defined as a two-tuple:
denotes the average performance of a student as an individual calculated from the tracked information on his or her contributions for the Wiki assignments completed at times
denotes the performance of a student as a group member calculated by combining the a) summary of that student's collaborative/interaction activities for the Wiki and b) relative individual contribution toward revising the Wiki (calculated by comparing a student's individual contributions with those of the other group members') for Wiki assignments completed at times
. The two components of the current student model in ClassroomWiki are calculated as the following:
in (2)-(8) are weights. Note that our design of (2)-(6) aims to achieve the following two goals: First, these equations capture the time-averaged performance (
) of a student with respect to his or her group (e.g., the relative values of the active use, passive use, interaction, survey response, and evaluation in Table 1
) while working in the ClassroomWiki environment. Second, the weights in this equation allow the teacher to customize
the model of a student to better capture his or her performance. For example, the teacher may want to adjust the weights to emphasize the importance of peer rating over the teacher evaluations and student interaction over peer rating by setting the weights as
. The student model
generated by the TAM is used for the following purposes:
To provide better assessment of individual student contribution—the five categories of student activity information stored in the student model help the teacher compare the effort or contribution of a student toward his or her group's Wiki against that student's group members' (in accordance to the Tracking and Modeling principle in Section 2.1). For example, a teacher may compare the total number of words that is added by a student with that of the average number of words added by his or her group members to estimate a student's contribution toward his or her group. This ability to compare the contributions can alleviate the free riding phenomenon since the students can be held accountable for not contributing to their group's Wiki. However, there could be scenarios where a student could try to game the ClassroomWiki system by, say, adding a large number of useless, trivial words. To counter this, the survey component comes into play. That is, the peer rating survey results are combined with the quantitative contribution assessment of the students such that a student who tries to game the system would receive a low peer rating from the group members when they observe the “ unnecessary word-adding” activity. This issue of improving the assessment of individual contributions, and thus, having more precise accountability, is further discussed in our future work.
The improved assessment of a student's individual contribution toward his or her group can also be used to prepare a detailed but summarized view of the members of each student group for the teacher so that he or she is able to provide specific and precise intervention if needed. For example, if there are free riding students in a student group, the information extracted from their models would allow the teacher to conveniently identify (even when classroom size is large), intervene, and motivate those students.
To improve group formation—the student models would allow the GFM to form student groups that contain a heterogeneous mix of students with varying levels of performances (as an individual (7) and as a group member (8)), i.e., implement the Heterogeneity principle discussed in Section 2.1. Furthermore, since a student's model is continuously updated, those models will capture the changes in the students' performances (as an individual (7) or as a group member (8)) while he or she progresses through the Wiki assignments. So, when the GFM forms student groups using the student models, the formed groups will reflect the changes in the students' behaviors, thereby implementing the Adaptation principle. For example, if during a Wiki assignment, a student improves his or her knowledge and contribution level toward his or her group, the student's model (i.e., in (7), (8)) would capture that change in terms of improved evaluation scores and increased contribution (forum, revision, etc.). The GFM would then use that changed model to assign the improved student to a different group with a more appropriate level of heterogeneity in future rounds.
3.4 Group Formation Module
ClassroomWiki's GFM allows the teacher to form student groups using two different group formation methods: 1) random and 2) MHCF. For random groups, the GFM forms student groups for a collaborative Wiki assignment by randomly choosing the specified number (i.e., according to the minimum group size) of students from the set of all participants in ClassroomWiki. For MHCF, the Group Formation module uses the MHCF framework (designed based on the principles described in [ 15
]) to form student groups. This framework assigns an intelligent agent to each of the participating students, where each agent maintains the model of its assigned student and utilizes that model to 1) probabilistically estimate the contribution of a student toward his or her group's Wiki, i.e., his or her performance as a group member, and 2) negotiate with other agents to form heterogeneous student groups. Note that the MHCF framework assumes a probabilistic environment where a student's average performance (as an individual (7) and as a group member (8)) can be estimated but not
accurately predicted. Based on this probabilistic view of the environment, an MHCF agent, on behalf of its user, negotiates with others to 1) collaborate to solve the current collaborative task well (i.e., improve the current task reward or score) and 2) improve his or her knowledge through collaboration to solve future tasks well (i.e., improve the future task reward or score) by forming heterogeneous student groups.
The MHCF framework's environment
is denoted by a 5-tuple
is the set of students,
is the set of agents where each agent
is assigned to a student
is the set of student groups,
is a set of tasks which the student groups collaborate to solve, and
is a 2-tuple
are two real-valued functions that estimate the probability of a student's current task and future task rewards when he or she joins a coalition. Here,
is defined as
is defined as
In (4) and (5),
is a set of the models (1) of the members of the potential student group
, where the group
is being formed to solve the task
. Note that, the functions
use the model of the members of a potential group to calculate the expected current task and future task rewards for a student to decide whether he or she joins that potential group to solve a task. The current task reward here represents the estimated reward a student expects to achieve for the current task for which the group is being formed. Furthermore, the future task reward here represents the expected reward for the future tasks by learning from the collaborations with his or her group members.
Group formation. Group formation in MHCF occurs in a set of negotiation rounds, where in each round, one agent is randomly selected to act as a proposer who negotiates with other agents in the framework to form a group for its assigned student. The negotiation of an agent is carried out in three steps: proposition, consideration, and notification. The steps are as follows:
Proposition: In the proposition step, the proposer agent chooses other agents ( is the minimum group size) and proposes a group that includes the students assigned to those chosen agents. The proposal from an agent to agent is , where is a set of models (not the ids) of the students in the proposed group are the expected current task and future task rewards (9), (10) for the task calculated from the perspective of agent .
Consideration: In the consideration stage, the proposed-to agent first compares its model stored by the proposer agent with its own model of student . If that model is not updated, in other words, if agent is unaware of the recent changes in the model of the student , the responding agent rejects the proposal and sends the updated model of to the proposer. Note that this notification from the responding agent allows a proposer to have updated view of the other potential members during the coalition formation round. This update procedure is important since each agent is assigned to a single student and may be unaware of the changes in the models of other students. If the proposer has the updated view of the responding agent's assigned student, the responding agent compares the expected current task and future task reward values of the proposed group to its current group. The responding agent leaves its current group to join the proposed group if the weighted sum of the current task and future task rewards is larger for the proposed group , i.e.,
Here, in (11), the and values are calculated by the proposer using the functions and (9), (10), respectively. So, in this negotiation process, an agent's decision regarding whether to join a group is determined by the value of these functions. Note that the number of times this entire negotiation process is run depends on the number of negotiation rounds, which is set as a multiple of the number of agents so that each agent is able to act as a proposer multiple times. Furthermore, to ensure that there is always an agreement among the agents, we set the current task and future task reward values (11) to zero if an agent cannot join a coalition (i.e., there is no agreement). Since (11) yields a nonzero value for any group, there will always be an agreement among the agents since it is better to be in any group than to be in no group.
Notification: If all of the chosen agents agree to join the proposed group, the proposer sends out a confirmation message to them notifying that they are now in the newly formed group. Otherwise, if any one of the responding agents disagrees, the proposer stops the negotiation process and waits for some other agent's proposal or its next turn to join a group.
Once the negotiation rounds end, the agents notify their assigned students about their respective newly formed groups and the details of the task they will collaborate to solve and the collaboration process begins.
MHCF group formation and the design principles.
The MHCF framework forms student groups based on the Adaptation and the Heterogeneity principles. First, with continuous tracking, the agents are aware of the current status of their respective students' models. So, when a student's model changes (say, a student becomes more attentive toward the coursework), the assistant agent would be able to adapt to that change accordingly and assign that student to a group that is expected to yield high current task and future task rewards. Second, note that the calculation of the expected current task and future task reward values is determined by the
functions, respectively. In our adoption of the MHCF framework into ClassroomWiki, the teacher can define this function according to the pedagogical strategies that he or she sees fit for the classroom. For example, if the teacher wants to increase the heterogeneity in the Wiki assignment groups, he or she may design the functions
in such way that if the models of the participating students indicate that their knowledge levels for the Wiki topic are heterogeneous, the current task and future task reward values are high. One such simple design can be implemented in the following manner: 1) let scg
be the array of average scores of the members of a group
, 2) specify
(10) functions as:
. Designed this way, the sum of current task and future task rewards (11) would be high for groups whose members have high average and standard deviation of scores.
We have used the open-source, Java-based Spring framework ( www.springsource.org) to implement ClassroomWiki as a web-based system. This implementation allows the teachers and students to participate in collaborative Wiki writing assignments from any computer that has an Internet connection and a web browser. In our current implementation, the ClassroomWiki modules (WIM, COM, TAM, and GFM) are programmed as plain Java objects that reside in the Spring framework. Here, the Spring framework acts as a container that 1) provides a repository (MySQL, dev.mysql. com) for the ClassroomWiki modules to store and retrieve information described in Table 1
and 2) stores the html webpages those act as the GUI for the modules. The Spring framework and the ClassroomWiki modules are hosted on a Java Glassfish Application Server (glassfish.dev.java.net), which serves the online user requests (e.g., access requests from teachers, students) by providing them the html pages generated by the ClassroomWiki modules in the Spring framework. Furthermore, we have used Repast (repast.sourceforge.net), an agent-based simulation framework, to realize the agents for the MHCF group formation mechanism in the group formation module (Section 3.4). Deployment diagram in Fig. 2
shows the implementation of ClassroomWiki modules. Furthermore, Fig. 3
shows the typical sequence of steps a student group and the teacher while they are interacting with ClassroomWiki to revise an assignment.
Fig. 2. ClassroomWiki's implementation using MySQL database, Repast, Spring framework, and Glassfish application server.
Fig. 3. Sequence diagram of group members (student1 and student2) and teacher's interaction with ClassroomWiki system.
Fig. 4. Resizable TinyMCE editor for students.
To implement the versioning functionality (Section 3.1), we have used an open-source Java library called DaisyDiff (code.google.com/p/daisydiff) to identify the added
, and reorganized
text by comparing the two versions
of a given html file. For example, for a given original html file, its modified version, and a CSS color code specification file (i.e., the colors of added, deleted, and unchanged words), DaisyDiff can determine the differences between them and generate a diff html file. This diff html file marks all changes (added, deleted, and unchanged words) according to the colors specified in the CSS file making it easy for the students to visualize
the changes made by their group members (see Fig. 5
). Furthermore, while generating that diff file, DaisyDiff allows the TAM to track the words that were added, deleted, or left unchanged by a revising student.
Fig. 5. Revision viewer in ClassroomWiki.
COM. The topic-based forum and the announcement system in the Communication module are also implemented as html webpages using Spring in our current implementation of ClassroomWiki. The webpage for the announcement system (Section 3.2) provides a form to the teacher, which allows him or her to write and submit announcements in html which is displayed to all students when they log in to the ClassroomWiki website. In the topic-based forum webpage, the forum messages are categorized according to their topics allowing the students to easily search, read, and respond to the topics/messages for each collaborative Wiki assignment.
The TAM is also implemented in Java to collect the student activity-related information (i.e., active use, passive use, survey response, evaluation, and interaction in Table 1
) and store this collected information in a MySQL database for other ClassroomWiki modules to use. TAM collects the student activity-related information by using Spring framework's request-driven Model-View-Control (MVC) architecture. Spring's MVC architecture is designed around a central servlet called the dispatcher servlet that acts as a gateway that passes the user requests to the ClassroomWiki modules, collects the result of the processing of the user request from the modules, and then generates the html pages that display the results to the requesting user.
To track all user interactions in the ClassroomWiki website, we have implemented the HandlerInterceptorAdapter
, a Java Class written in Spring, that intercepts all incoming user requests processed by the dispatcher servlet. Each request contains 1) the time and name of the requested webpage and 2) all variables associated with the requested webpage and the modules that process that user request. So, using the HandlerInterceptorAdapter, the TAM is able to collect all user activity-related information (
in Table 1
) in ClassroomWiki. This collected information is then used by the module to build, store, and update the student models (
in (7), (8)) in the MySQL database.
GFM. To implement the agents that realize the MHCF (Section 3.4) group formation method in the GFM, we have used the aforementioned Repast simulation. When instructed by the teacher, the GFM invokes the Repast environment creating one agent for each of the participating students. Upon initialization, each created Repast agent is provided the updated model of the student it represents. Once initialized, those Repast agents negotiate among themselves to put their assigned students to the groups that provide highest weighted sum of the current task and future task rewards (11).
As we have discussed in Section 3.4, the MHCF-agents' decision making relies on the design of the current task and future task reward functions, i.e.,
(9), (10). In our current implementation, we have designed these functions to promote the Heterogeneity Principle (Section 2.2). From the perspective of collaborative Wikis, the heterogeneity we would like to promote is the competence of a student as: 1) an individual contributing concepts, ideas, and Wiki-related content to his or her group Wiki and (2) a group member communicating, collaborating, and working with other members to complete the Wiki assignment. Thus, we use the student model
(1) built with the information collected by the TAM to estimate the performance of a student with the following:
denotes the performance of a student in ClassroomWiki,
are weights, and
are from (7) and (8), respectively. Using this performance value, we design the current task and future task reward value functions according to the Heterogeneity Principle (Section 2.2). For a group of students
Here, in (13), the current task reward is proportional to the average performance value of the members. That is based on the idea that when a group contains high-performing students, it is likely that they will be able to solve their current task, i.e., completing their writing assignment well. Then, in (14), the future task reward is inversely proportional to the standard deviation of the performances of the group members. A group that contains students with different levels of performances is more likely to generate cognitive conflicts (Section 2.1) that would motivate them to collaborate/contribute more to the Wiki, and thus, improve their performance levels through collaborative learning for future tasks. Thus, in our current implementation, the functions
provide a simple way for the agents to calculate the estimated current task reward
and future task reward
during group formation.
5. Experiment Setup and Results
We have deployed ClassroomWiki in an introductory history course (HIST202—America after 1877 Section 003) at the University of Nebraska, where ClassroomWiki was used to conduct a collaborative Wiki writing assignment. In the following, we describe the experiment setup. Then, we discuss how useful the students thought ClassroomWiki was as a collaborative learning tool based on survey results. Furthermore, we discuss ClassroomWiki's ability to track more detailed information (i.e., our implementation of Tracking and Modeling principle) than Blackboard's Wiki and that improved tracking's possible impact on the students' collaborative activities. Subsequently, we discuss how ClassroomWiki and the MHCF framework's group formation method (designed based on Heterogeneity principle) impacted the collaborative learning outcome of the students. Though not directly measured, we also observe indicators of the impact of the Adaptation principle. Finally, we present how ClassroomWiki's TAM's capability was used by the teacher for intervention and grading students, and could serve to inform redesigning of collaborative writing assignments.
5.1 Experiment Setup
In our experiment, the 145 participating students were divided into control (72 students) and treatment (73 students) sets by sorting the students in an array according to their average evaluation scores (
in Table 1
) in the class and then assigning the odd-numbered students to the control and the even-numbered students to the treatment set. This division mechanism was used to evenly distribute
the high-performing (and low-performing) students between the control and the treatment sets. Once the students were divided, the control set students were further divided into 14 groups randomly while the treatment set students were also divided into 14 groups using the MHCF framework using the GFM. Furthermore, while implementing our group formation algorithm, we have chosen weights
for (2)-(8) and (12)-(14).
For the initial ClassroomWiki assignments when we did not have a history of student attributes, to “bootstrap” the system, students' scores from three prior assignments were used: 1) a midterm exam, 2) a collaborative writing assignment, and 3) an individual essay writing assignment. These allowed us to estimate the student performances along three different ability dimensions, i.e., their ability to: 1) comprehend class material, 2) collaborate, and 3) analyze a given essay topic. Since, our agents tried to form student groups that had a high average and high standard deviation, it follows that such a group is likely to have members who are heterogeneous (cf., Heterogeneity principle in Section 2.2) in these three types of skills that are necessary to complete the ClassroomWiki's collaborative writing assignment.
Finally, we acknowledge that when MHCF framework is used, there is thus a bootstrap problem where, for the first round of group formation, MHCF does not have a history of student interactions and can only stratify the students based on their previous performances. We will further investigate the impact of this problem using a longer experiment in future (see future works in Section 6).
Once assigned to their groups, each student collaborated with his or her group members to prepare a collaborative Wiki writing assignment on the topic “ US as a super power
” for three weeks. In brief, student groups were first asked to choose a subtopic (e.g., Cuban Missile Crisis 1962) within the general topic by discussing with their group's members using the ClassroomWiki's Forum. Once the subtopics were chosen, the assignments were announced on ClassroomWiki. The student groups then collected and discussed evidences regarding their subtopics and collaborated to write their Wiki essay using ClassroomWiki's WIM. During this process, the teacher periodically monitored the summarized performances (e.g., number of topics or messages posted as described in Table 1
) of the student groups and provided group-specific forum messages and e-mails to help them collaborate better. At the end of the three weeks, the teacher reviewed each group's Wiki essay and scored each (0-100). Then the teacher converted a group's Wiki grade to the student members' individual grades by multiplying that group's grade with the relative contribution of that student. Here, the teacher used the following formula to calculate the student grade
for a student
is the member of group
's group grade,
are the student activity information tracked by the TAM in ClassroomWiki and are described in Table 1
in Section 3.3. Note that our experiment setup was double-blind
, i.e., neither the participating students nor the teacher who graded the student groups' Wikis and provided interventions knew which students belonged to the control or the treatment set. Furthermore, prior to this study, the students had completed collaborative writing assignments using the Wiki tool on the popular learning management system Blackboard. In our discussions below, we will also compare ClassroomWiki with Blackboard's Wiki.
5.2 Impact of ClassroomWiki
We have conducted a Wiki-rating survey [ 24
] among the students in the HIST 202 class to measure students' view of ClassroomWiki as a collaborative writing tool. The students' responses suggest that, on average (mean 22.28 and median 23 on a scale
), students were satisfied with the performance of ClassroomWiki as a collaborative writing tool. Furthermore, the students preferred ClassroomWiki slightly more (mean 3.4 on a [1,5] scale) than Blackboard's Wiki tool.
Overall student performance and collaboration.
Analyzing the “ All
” rows of Table 2
and Table 3
to investigate ClassroomWiki's impact on the collaborative learning outcome of the students, we compare their scores in the ClassroomWiki assignment (all students' evaluation scores in Table 2
) with other similar essay assignments ( Table 3
) and observe that the mean and median student scores in the three essay assignments do not show any clear trends or patterns. For example, there were no clear indications that the treatment students were better students to begin with. Furthermore, in the ClassroomWiki experiment, the students achieved lower mean and higher median scores than the other assignments except Blackboard's Wiki assignment. However, in Blackboard's Wiki assignment, students' scores were likely to be inflated since Blackboard does not
allow to the teacher to track individual student contributions ( noncontributing students
in Section 5.3). This comparison suggests that although some students have always achieved low scores in the tests (and thus, produced low averages for the class), while using ClassroomWiki, some
of those low-performing students have performed better
when they collaborated with their group members. The improvement of those low-performing students' performances then raised the median score of the students in the ClassroomWiki assignment. In our experiment: 1) ClassroomWiki's collaborative tools (e.g., versioning in WIM, assignment-specific forum in COM), 2) teacher's periodic reminders, and 3) enforced
accountability of each student's contribution due to our use of TAM could have motivated the students to collaborate with each other to improve the quality of their group's final essay. That improved participation then led to the improved median score.
Table 2. Student Evaluations in ClassroomWiki
Table 3. Student Evaluations in Other Tests
5.3 Impact of MHCF Group Formation
The individual and group student evaluations in both the treatment and control sets shown in Table 2
suggest that 1) the treatment set students achieved significantly higher individual scores (higher distribution mean with
) and 2) the treatment set student groups achieved significantly lower standard deviations (lower distribution mean with
) than the control set students. This suggests that the collaboration among the treatment set group members might have been better
than among those in the control set. This improvement in collaboration could be attributed to the MHCF framework's formation of heterogeneous student groups, i.e., group heterogeneity facilitated more cognitive conflicts and the members learned when they collaborated to resolve those conflicts (Section 2.2).
To investigate the collaboration process among the group members in both sets, we have looked at the correlations between the collaborative activities (revisions, forum message postings, etc., as described in Table 1
) of the students and their individual evaluation scores (
in (15)) in ClassroomWiki. Table 4
shows the analysis. In Table 4
, we see that the treatment set students who achieved higher scores in other classroom assignments (that occurred before the ClassroomWiki experiment) have also achieved higher scores in the ClassroomWiki assignment. However, the same pattern was not observed for the control set students. Furthermore, in the treatment set, the students who performed well in other previous classroom assignments posted relatively more topics and messages to their group's forum. However, in the control set, no such patterns were observed. These observed patterns suggest that the students who achieved high evaluation scores in other classroom activities helped their group members by posting more messages and topics and that those high-performing students were able to achieve relatively higher scores while helping their group members. This can also be explained by MHCF's formation of heterogeneous groups. In the heterogeneous student groups, when the low-performing students' contribution to the group Wiki generated cognitive conflicts (as discussed in the collaborative learning in Wikis in Section 2.1), it motivated the high-performing students to step in
and help their low-performing group members by posting topics and messages in the group's forum.
Table 4. Correlations between Student Activity and Individual Evaluation Score
Furthermore, Table 5
compares the collaborative activities of the students in the control and the treatment sets. We notice that the treatment set students had a higher count of collaborative activities (except the number of messages). This result combined with the treatment set students' 1) relatively better performance and 2) increased participation of the expert students suggests that the treatment set students' increased collaboration might have helped them to learn from their group members and achieve better scores.
Table 5. Collaborative Activities of Students
We have also compared the control and the treatment set's students' evaluations of their own groups and group members using a Team Rating survey [ 24
] and a Peer Rating survey [ 24
]. The results of these surveys suggest that the treatment set students rated their peers (21 versus 20) and groups (23 versus 22) only slightly better than the control set students, on average. One reason of this nonsignificant difference could have been that the high-performing students viewed the help they had to provide to their group members as extra work
and did not rate their groups high. Our analysis reveals that the correlation between the students' Wiki evaluation scores and the Team-Rating in the treatment set was higher
) than the control set students. We need to collect more data to further clarify this issue.
Noncontributing students.Table 6
shows the numbers of noncontributing students in HIST 202 class for other previous assignments. We see that, except the first assignment, the treatment set students had more noncontributing students, but in ClassroomWiki's assignment, the trend was significantly reversed
, i.e., more control set students failed to contribute. This again suggests that the treatment set's heterogeneous student groups were more motivated to collaborate, and thus, had a smaller number of noncontributing students.
Table 6. Number of Noncontributing Students in Other Tests
5.4 Tracking and Modeling in ClassroomWiki
Identifying and penalizing free riding.
Although it is difficult to accurately verify, there are positive indications that the student assessment done using (15) with the data collected by the TAM was able to capture
the level of individual contributions of the students. In the ClassroomWiki experiment, there were 13 students (10 in the control set and 3 in the treatment set) who did not revise their group's Wiki (i.e.,
in (15)) or post any forum topics or messages (i.e.,
in (15)). As a result, their individual student scores were 0. Furthermore, when compared to the Blackboard's Wiki assignment, we see that there were also 13 students who did not contribute in Blackboard's assignment. However, in Blackboard's Wiki, the teacher was able to catch noncontributing students only if
group of students failed to contribute. In ClassroomWiki, the data collected by TAM allowed the teacher to quickly identify each individual student's contribution
toward his or her group and penalize that student if his or her contribution fell below the instructor's threshold. Researchers [ 8
] suggest that the main problem regarding free riding is twofold: first, often collaborative learning tools do not allow the teacher to accurately capture student's contributions toward his or her group. Second, if the students perceive that their group members are not
held accountable for free riding, they feel aversion toward collaborative work. So, the teacher's ability to identify and penalize students who do not contribute toward their groups suggests ClassroomWiki's ability to address both these issues associated with free riding.
Note that our assessment of student contributions could have caused game playing among the students, where they post revisions to stay ahead of their group members in terms of contribution count/metric. Although it is difficult to be certain that no game playing occurred in the student groups, our experiment setup was designed to discourage
such game playing behavior. While presenting ClassroomWiki, we described to the students that the teacher would determine a threshold of student contribution based, on average, contribution of the students of the entire
classroom, then 1) students whose contributions fall substantially below
that threshold would be penalized and 2) contributing more than average would not yield them extra points. Moreover, students were only able to track their own group members' contributions in ClassroomWiki (and thus, would not know of other students' contributions in the class). So, in our experiment, student motivation
and the information
that they needed to game the system were both reduced. Finally, researchers [ 14
] have proposed alternate methods of assessing student contributions and free riding like self-reporting or group sign-off of contributions. Although, these assessment techniques could suffer from a different type of problem (e.g., all group members giving good ratings to one another because of previous acquaintanceship), we plan to compare these methods of student assessment with ours as future work (Section 6).
Improved assessment of student performance. We have also found the correlation between the students' Final Exam Score and the score they received for their ClassroomWiki assignment to be 0.75 and that between the Final Exam Score and the Blackboard's Wiki assignment to be 0.51. Treating the final exam in the class as the “ground truth” measuring how much the students have learned, this shows that ClassroomWiki's evaluations can more accurately measure—and to some extent predict—students' performances in the classroom than Blackboard's Wiki's. Finally, working with the instructor, ratings of the students' revisions of essays, their forum topics, and forum messages have been obtained. The correlation between the sum of the quality of contribution—based on the above ratings—and the students' scores that were assigned following the TAM's model in ClassromWiki is 0.88. This correlation indicates that TAM's model of student performances closely represented the quality of their contributions to their groups. Our analysis suggests that ClassroomWiki can effectively alleviate a common problem, i.e., the leveling effect, in collaborative learning tools where group members receive the same score even when they don't contribute equally.
Specific and precise teacher intervention.
To improve collaborations, the instructor of the HIST 202 course checked the summary of student activities (15) and sent e-mails
addressed to the whole classroom that praised the groups that contributed and encouraged the groups who needed to contribute more. Furthermore, the teacher monitored the group forums (
in Table 1
) to assess their progress and posted forum messages to provide specific
guidance to them.
Adaptation principle. One of the motivations behind our design of TAM was to help GFM to realize the Adaptation principle (Section 2.2), which consists of: 1) capturing the students' performances and then 2) utilizing the captured student performance to form high-performing student groups. If TAM's captured student model closely/truly represents the students' performances, over time, GFM would keep up with the change in students' performances and form better student groups. While we agree that to validate that ClassroomWiki realizes the principle would require a long-term experiment (see Section 6), a closer look at our analysis suggests that our implementation is able to realize this principle: 1) TAM is able to closely represent the students' performances and 2) GFM is able to utilize the tracked models to form student groups (Section 5.3), which improve student performance and collaboration.
We have presented ClassroomWiki, designed based on the educational research on modeling the collaborative learning process, to improve typical Wiki's functionalities in two aspects: 1) individual student contributions and 2) group formation. While typical Wikis track the changes made by the users, such tracking is from the perspective of the essay, and thus, student-centric statistics are not computed and presented readily, making assessment based on contributions difficult. Furthermore, typical Wikis do not provide functionalities to automatically form student groups for collaborative activities. Our ClassroomWiki provides a multiagent-based group formation mechanism that uses the tracked student information to form heterogeneous student groups to improve the collaborative learning outcomes of the students. We have reported on a three-week-long collaborative Wiki assignment in a university-level history course. Although not all results were statistically significant, our analysis suggests that ClassroomWiki may 1) improve the collaborative learning outcome of the students by its group formation framework, 2) help the teacher identify and penalize free riding students, and 3) facilitate specific and precise teacher interventions based on the tracked student activities.
Further investigations are necessary to better understand the impact of ClassroomWiki on the collaborative learning outcomes of the students, and our future work thus involves the following:
Improving the assessment of the qualitative aspect of student contributions in ClassroomWiki by estimating the quality of their edits and messages using natural language processing techniques (LingPipe tool—alias-i.com/lingpipe) such as:
- content-related phrase identification,
- sentence detection,
- stemming [ 25], and
- common words detection.
Implementing a self-reported and group-reported (i.e., signed off by the group) contribution assessment method. This is to better validate the impact of our current method of student assessment on the free riding of students.
Improving the MHCF group formation by incorporating a Bayesian Network to enable the agents to learn the current and future task reward functions that map the student models—and learner characteristics—to students' collaborative learning outcomes.
Obtaining more detailed results for our improved ClassroomWiki by running a more comprehensive, semester-long experiment with a large set of students for multiple collaborative writing assignments. In this experiment, we plan to:
- compare MHCF group formation with VALCAM [ 26]—another group formation method to provide a stronger comparative baseline,
- investigate the impact of MHCF on student performance when MHCF is able to utilize the student model built on a more detailed history of student activities, and
- collect more data to obtain results with higher statistical significance, and to further evaluate the impact of the three design principles.
This research is supported in part by US National Science Foundation grant CNS-0829647. The authors would also like to thank June Griffin and William Thomas for their feedback in developing and deploying ClassroomWiki and Leslie Working and Jason Heppler for their help in deploying ClassroomWiki.
N. Khandaker is with the Department of Computer Science and Engineering, University of Nebraska, 122F Avery Hall, Lincoln, NE 68588-0115. E-mail: email@example.com.
L.-K. Soh is with the Department of Computer Science and Engineering, University of Nebraska, 122E Avery Hall, Lincoln, NE 68588-0115. E-mail: firstname.lastname@example.org.
Manuscript received 29 July 2009; revised 3 Sept. 2009; accepted 3 Nov. 2009; published online 12 Nov. 2009.
For information on obtaining reprints of this article, please send e-mail to: email@example.com, and reference IEEECS Log Number TLT-2009-07-0129.
Digital Object Identifier no. 10.1109/TLT.2009.50.
received the BS degree (with honors) in physics from the University of Dhaka, Bangladesh, and the MS degree in computer science from the University of Nebraska Lincoln. He is currently working toward the doctorate degree in the Department of Computer Science and Engineering at the University of Nebraska Lincoln. His primary research interests include teamwork and coalition formation for human participants, multiagent coalition formation in uncertain environments, multiagent learning, computer-supported collaborative learning systems, and agent-based simulation. He is a member of the ACM and the AAAI and a student member of the IEEE.
received the BS (with highest distinction), MS, and PhD degrees with honors in electrical engineering from the University of Kansas. He is currently an associate professor in the Department of Computer Science and Engineering at the University of Nebraska. His primary research interests are in multiagent systems and intelligent agents, especially in coalition formation and multiagent learning. He has applied his research to computer-aided education, intelligent decision support, and distributed GIS. He is a member of the IEEE, the ACM, and the AAAI.