1. mostly summative,
2. designed and conducted by the teacher,
3. comprised of individual tasks (i.e., tests, quizzes, essays, etc.), a group task with each student receiving the same grade or a combination of group and individual tasks, and
4. nearly exclusively focused on cognitive outcomes (achievement).
2. an individual student typically has little influence on group formation and due to the coincidental presence of high or low achieving students or a free-rider, a group grade over- or underspecifies an individual students' competence, and equally active students may receive different grades only because of group formation outside of their control,
3. low- and medium-ability students generally profit more from group grades than their high-ability counterparts [ 28], and
4. unsatisfactory experiences with group grades often result in a reluctance for CL among students (and parents).
2.3.1 Challenge 1: Individual Level versus Group Level Since the CL approaches developed in the 1970s-1980s, the topic of whether the individual level or the group level should be assessed, that is, individual grades based on testscores versus group grades for group projects, is debated [ 39]. Obviously, this is connected to the learning metaphor adopted, but it would be an oversimplification to argue that proponents of the acquisition metaphor only stress individual-level assessment and that proponents of the participation, knowledge creation, and group cognition metaphor only stress group-level assessment.
Akkerman et al. [ 49] identified in their review of group cognition perspectives, three boundary studies that used concepts from both cognitive and socio-cultural approaches. Yet, all of them used the individual level as their starting point. Stahl's [ 48] view on group cognition takes the group level as the starting point, however, whereas Stahl views the individual as subsidiary to the group, Akkerman and colleagues call for more studies that aim to integrate both perspectives: “By either extending cognitive conceptualizations of group cognition to include social accounts of cognition or by extending socio-cultural conceptualizations of group cognition to include more precise accounts of individual cognition, we believe the studies on group cognition would be more complete” (p. 56).
Furthermore, Webb [ 67] concludes in discussing individual-level versus group-level assessments that the “students' competence may be misestimated in group assessment, but some groups may produce more misestimation than others” (p. 147) and provides a powerful argument for considering both the individual and group levels: “( ) scores on work submitted from group assessment should not be used to make inferences about the competence of individual students. Without data on group processes, scores from group assessment are better interpreted as what students can produce when working with others” (p. 149). Nevertheless, evaluation criteria and group functioning appear to affect group performance, which, in turn, may positively predict individual essay performance [ 26].
Finally, irrespective of whether the individual level or group level is the focus of assessment, it is important to consider that most educational groups are typically “ad hoc groups”: established in relation to a specific task, series of tasks or course and they only exist for the duration of that task, series of tasks or course. To some degree, the group product will be codified in an artifact (e.g., group report, dialogue, diagram, etc.), but the individual experience of that CL event will be transposed to future CL events. Hence, even if group-level interaction is considered as the engine for CL, the individual level cannot be dismissed.
2.3.2 Challenge 2: Convergence versus Similarity There is a vast array of terminology to refer to the anticipated outcome of group-level cognition. Commonly used nouns to refer to cognitive outcomes are knowledge, understanding, mental model, cognition, vision, representation, and meaning, which are commonly further specified by the following adjectives: convergent, same, shared, mutual, consensus, common, collective, equivalent, similar, and divergent [ 49], [ 68]. Since the inception of CSCL convergence has been a dominant measure and regarded as the most relevant outcome, that is, the more knowledge students have in common during or after a CL event, the more it can be assumed that CL resulted in cognitive benefits [ 68], [ 69], [ 70], [ 71]. Convergence operationalized as held in common after CL implies that students' individual knowledge must be the same.
However, it is much more likely that what individuals learn from CL is similar rather than the same. Consider the familiar example of a group vacation. When visiting the Acropolis in Athens, the entire group shares the event. Yet, when all photos by all group members are compared afterward, some photos by different group members will be exactly the same (copying or alerting each other), some of them will be similar (same object, but different perspective), other photos will be completely different (everybody aims for their unique scoop), and it is even possible that group members have no photos in common with one or more other group members. This everyday example clearly illustrates that a shared event need not result in the same individual experience of that event. In a similar vein, a CL event need not be experienced the same by each individual.
Fischer and Mandl [ 69] concluded that much less convergence (in terms of shared knowledge) was observed in a CSCL context in comparison to the potential for convergence given each participants' individual knowledge. Convergence operationalized as held in common/the same—irrespective of whether it is located in individual minds or in the interaction (discourse)—appears to be too strict as a criterion to determine whether a CL event results in learning. This led Weinberger et al. [ 68] to distinguish “convergence” from “equivalence,” and they defined the latter as “( ) learners becoming more similar to their learning partners with regard to the extent of their individual knowledge” (p. 417). However, “equivalent” is defined in [ 72] as “ equal to another in status, achievement or value [emphasis added],” whereas “similar” is defined as “having qualities in common.” Hence, similarity appears more adequate because it does not imply equality in the end state or result, for example, “achievement.” Moreover, divergent processes are also in operation [ 70]. Each individual student may develop an internal cognitive representation different from the one they achieved as a group [ 73], [ 74]. Miyake [ 75] concludes with respect to individual knowledge construction from CL that:
“(…) even during a highly collaborative comprehension activity, social sharing of the situation does not impede each participant from pursuing individualistic knowledge construction. Rather, the interactive process supported each to realize different perspectives to check and modify their understandings by making explicit the different perspectives, which are not within their individual repertoire” (p. 255).
In other words, knowledge that emerged during the CL event is internalized differently, given each participants prior perspective. Hence, individual students' knowledge may become more similar due to a CL event, but simultaneously their individual knowledge may differ to a large extent “because each participant works from a different starting schema, what is obvious and natural to one may not be so to the other” (p. 464) [ 74].
To conclude, convergence can best be construed as the pinnacle of CL. It is the most extreme instance of similarity, or in other words, the positive extreme of the “shared knowledge” dimension ranging from convergence (same; equal; shared), via similarity (analogue; parallel; partially shared), to divergence (different; disparate; unshared). Key to assessment of CL, then, becomes to determine when the observed degree of similarity can still be attributed to CL. One approach could be to determine the degree of transactivity, that is, the extent to which students refer and build on each others' contributions [ 68], [ 76] reflected in collaborative dialogue or individual products, or the extent to which students transform a shared artifact (e.g., a group report). In other words, it becomes essential to determine a situation dependent lower bound (threshold) for learning induced by a specific CL event.
2.3.3 Challenge 3: It's Not All about Cognition Reviewing the literature, it is apparent that cognitive outcomes are central to the assessment of learning in past and present (CS)CL studies [ 41], however, cognitive outcomes are not the only outcomes of CL. Slavin [ 16] already identified three major perspectives in cooperative learning research— the motivational, social (cohesion), and cognitive—and stated that they “( ) may be seen as complementary, not contradictory” (p. 52) and that there are many other outcomes like “( ) intergroup relations, self-esteem, acceptance of mainstreamed classmates, pro-social norms, and so on” (p.64). Social (cohesion) aspects, such as intergroup relations, are typically emphasized in the “Learning Together” approach [ 18] and the “Group Investigation” approach [ 20]. In the context of Group Investigation, there also appear to be positive effects in relation to aspects commonly associated with intrinsic motivation, such as interest, enjoyment, and (mutual) encouragement [ 17].
To a certain degree, the social dimension and associated outcomes are also considered in recent literature [ 27], [ 77], [ 78]. Kumpulainen and Mutanen [ 79], for example, distinguish three dimensions in their analysis of peer group interaction: 1) functional analysis (characterizes communicative strategies used by participants in social activity), 2)cognitive processing (examines ways in which students approach and process learning tasks in their social activity), and 3) social processing (focuses on the nature of social relationships that are developed in students' social activity). The social processing dimension describes peer group interaction in terms of collaborative, tutoring, argumentative, individualistic, dominative, conflict, and confusion. Tolmie et al. [ 80] recently studied the social effects of CL among 575primary schools students (aged 9-12) which revealed that
1. CL leads to a dual impact in terms of cognitive and social gains,
2. students' collaborative skills improve alongside understanding and optimal social relations need not be in place prior to collaboration,
3. social context (rural versus urban schools) did not affect cognitive or social gains; rather engagement in CL raises both cognitive and social gains counteracting prior social differences, and
4. convergence over time between transactive dialogue and collaborative skills (in terms of work relations) suggests that “( ) cognitive and social gains would appear to be interlinked, if distinguishable outcomes” (p. 188). In the context of CSCL, however, social interaction is still 1) often taken for granted, or 2) restricted to cognitive processes [ 81].
The motivational dimension and associated outcomes have also received increased attention in recent literature [ 82], [ 83], [ 84], [ 85], [ 86]. In contrast to an extrinsic operationalization of motivation during CL in terms of rewards (see Section2.1), present motivational perspectives—e.g., the “dual processing self-regulation model” [ 87], “self-determination theory” [ 88], and “person-object theory” [ 89]—share the premise that students have multiple goals with their subsequent motivations, actions, and affective responses. Likewise, students have multiple goals and motivations in the context of CL [ 84], [ 90]. Hijzen et al. [ 84], for example, found that mastery goals (“I want to learn new things”) and social responsibility goals (“I want help my peers”) prevail in effective CL groups. Furthermore, belongingness goals (e.g., “I want my peers to like me”) were more important than mastery goals in ineffective CL groups, whereas the opposite was observed for effective groups. Finally, students in effective CL groups were more aware of their goals compared to students in ineffective groups [ 84]. In CSCL research studies on motivational processes (including the role of emotions) are under-represented although motivation research is gaining interest in CSCL research [ 91], [ 92].
Consideration of outcomes other than cognition has direct implications for the design of CL assessment [ 39]: “If the emphasis is on using peer learning to improve subject-matter learning, it will lead to one kind of assessment design. If the emphasis is on promoting teamwork then design for assessment will need to be quite different” (p.419). In fact, akin to a probabilistic view on the design of CL [ 21], Strijbos et al. [ 93] argue that the design of “( ) assessment could be probabilistic as well, i.e., it should not focus on the specific learning goal X, but X' and X'' or even something unforeseen ‘U’ are equally probable collaboration outcomes that signal learning—intended or not” (p.247). This unforeseen outcome “U” could be cognitive, but it could equally likely be social or motivational.
1. learning from CL occurs through distributed emergent experiences,
2. what each individual learns is more likely to be similar than exactly the same (although the latter is not excluded),
3. it can affect different processes (cognitive, social, or motivational) for each individual, and
4. the distributed emergent experience is internalized differently by each individual given his/her prior experiences—even if the same process is affected!— for “( ) the experiences evoked in collaboration are highly variable and such variability is likely implicated in learning outcomes” (p. 162) [ 83].
1. Internal scripts. Conceptualizing CL as a GE implies that students' prior experiences will affect the collaborative process, for example, via internal collaboration scripts [ 114], which, in turn, may interact with the external scripts (e.g., instructions) given. Carmien et al. [ 115] ground this interplay of internal and external scripts in a distributed cognition perspective.
2. Time. The dynamic nature of the GE metaphor explicitly includes the notion of time, allowing for a wider scope in CL research, which has typically “( ) dwelt on situations that have no history” [ 83]. Presently, the notion of time is receiving more attention in (CS)CL in the sense of a) the sequence of actions during a CL event [ 116], [ 117] as well as b)longer term perspectives (e.g., day, week or month) [ 55], [ 56], [ 65], [ 118], [ 119].
3. Bridging. Sarmiento-Klapper [ 120] recently introduced the concept of “bridging.” Bridging activity aims at “( ) overcoming discontinuities emerging over the multiple episodes of interaction” (p. 91). In the context of Sarmiento-Klapper's work the bridging activities involve how students cope with changes in group constellation (one or several members are missing or substituted by other students not present during previous episodes) and how these changes affect their participation. Depending on the constellation students may position themselves differently [ 65], for example, the absence of a dominant group member may result in a previously quiet group member taking the lead. In view of the GE metaphor, the concept of bridging provides support for the role of the individual. The individual's interpretation of a distributed emergent experience constitutes what can be “bridged” between the CL events (or episodes). When a group performs a set of tasks in multiple constellations, the artifact(s) created (a dialogue, report) may still be available (short term), however, in the context of consecutive courses or over the span of a year (long term)—in which students collaborate in diverse and ad hoc groups—the artifacts are not easily available (a report is more likely retained as an accessible artifact than a dialogue) and unlikely to be accessed by students. Moreover, a report may contain the outcome of the CL event to some extent, but it will not convey how an individual experienced the input of fellow group members—other than what was internalized and available for recall. Hence, in long term settings participants are more likely to rely on internalized distributed emergent experiences. In sum, the concept of “bridging” helps to account for what Ellis and Goodyear [ 96] call “(…) the characteristics and capabilities that someone can ‘carry with them’ from one situation to another” (p. 7).
4. (Cross-age) peer tutoring. Whereas convergence has been useful (to some extent) for the analysis of CL outcomes, it is not applicable to peer tutoring (PT). In reflecting that CL outcomes can also be divergent (knowledge shared at the pretest is not necessarily shared at the posttest) Jeong and Chi [ 70] refer to prior work on tutoring in which it was difficult tofind traces of convergence, which could be due to the diverse roles. The limitation of the convergence measure is particularly prevalent in relation to cross-age PT, where the tutee is assumed to acquire the skills being taught by the tutor—and not the other way around. Learning for the tutee can still be conceptualized in terms of similarity or convergence, but the tutor is (in most cases) assumed to learn something completely different, for example, the skills on how to teach or explain particular subject matter to a younger student. Hence, cross-age PT implies that each student's learning from the interaction will not be convergent, maybe similar, but most likely divergent. Moreover, as “it is common in peer learning activities for students to have differentiated roles. Their assessable products may not be the same” (p. 422) [ 39].
5. Help seeking. The GE metaphor is also able to account for help-seeking practices, where the social relationship (i.e., trust in the help giver's competence) with the more able student, teacher, or parent might increase task persistence and subsequent cognitive gain for the help seeker [ 121]. In terms of learning as experience, the distributed emergent experience (help seeking/receiving) is internalized in terms of social skills by the more able partner (providing help) and by the help seeker in terms of cognitive skills (solving a problem).
6. Communities. Community perspectives on (CS)CL and Networked Learning (NL) typically consider other processes and outcomes of CL in addition to the cognitive benefits. Membership of a particular community of learners can, for example, have a significant impact on a participants' motivation to learn, and, in turn, this may result in the participant becoming an active contributor to the community and even helping others to learn [ 31], [ 55], [ 56], [ 65], [ 122]. Finally, although the GE metaphor describes a small group setting, it is precisely the fact that experiences of CL events are embedded in a larger local and cultural context that aligns the GE metaphor with community and NL approaches. Although socio-cultural perspectives are prevalent in the community-oriented studies, the terminology of “experiences” and “activities” of the GE metaphor, may help in reconciling the cognitive and socio-cultural perspectives.
1. there is no generic set of agreed-upon CL indicators that can be used for assessment of CL,
3. if available, the information collected by these tools (e.g., most systems collect some type of logfile data) is commonly not applied for teacher and student assessment of CL, and
4. actual teacher practices of CL assessment are sporadically investigated.
4.1.1 CL Mining
1. Mine access to system objects and student artifacts. This refers to a wide variety of standard collaboration objects in most software and online environments (such as a discussion forum, chat, and group space). In addition, the artifacts created by students during CL can be mined, such as
a. individual or group reports or assignments, wikipages, blogs etc.,
b. self-, peer, and teacher ratings of the artifacts,
c. peer and teacher feedbacks on artifacts, and
d. coconstructed whiteboard drawings, etc.
In short, mining of artifacts in its widest sense.
2. Mine student discourse and actions. Whether face-to-face, computer-supported or online, CL generates enormous amounts of process data—such as written text, discourse transcripts, video, audio, spatial movement (e.g., manipulation of 3D projections, interactive surfaces such as tabletops, whiteboards, touch objects, etc.)—that can be mined for analysis. Intelligent support could assist the selection of revelant indicators, as well as automated tagging once mined. Especially for dynamic tracking of transformations (trace data; see [ 129]). Examples of discourse mining tools are the Analytic Toolkit in KnowledgeForum [ 45], and the ForumManager developed by Jeong [ 139] which consists of an Excel file in which threaded Blackboard discussions (version 6.x and higher) can be imported and analyzed for descriptive information (e.g., number of posting by students, number and length of threads, etc.).
3. Mine system or instructional scripts/agents. Scripts consist of at least five components: learning objectives, type of activities, sequencing, role distribution and type of representation [ 140]. Each component can be used as an indicator for automated tagging and assessment (similar to conventional structured dialogue interfaces; [ 141], [ 142], [ 143]). Activities associated with a specific role, for example, peer assessor or peer assessee, can be associated with target behavior of interest. Finally, embedded agents (e.g., help prompts) can be mined for monitoring and assessment.
4.1.2 CL Analysis
1. Integrate multiple data sources. This is one of most challenging aspects for assessment of CL and for which intelligent support will be most welcome. Given the large (and ever expanding) variety of data that can be mined, the integration of multiple data sources is a rapidly growing necessity [ 144]. Dyke, Lund, and Giradot [ 145] developed “Trace Analysis Tool for Interaction Analysts” (Tatiana), which affords the analysis of (CS)CL multimodal process data (written discourse as well as video; including the synchronization of video and transcript) and also generates visualizations. In line with the GE metaphor, assessment of CL not only involves cognitive processes but also social and motivational processes. The latter two are typically more difficult to derive from discourse. In addition to objects, artifacts, discourse, scripts, and agents, self-report questionnaires can be used to assess students' perception of their current CL social or motivational state during collaboration (e.g., the Quality of Working in Groups (QWIGI) inventory [ 82]).
2. Analyze multiple levels simultaneously. The analysis of individual and group-level data is inevitable for assessment of CL. In addition, the relations between both levels should be analyzed. This includes basic analyses of group participation, for example, via Social Network Analysis (SNA) to identify patterns and community member roles [ 56], [ 146]. In relation to assessment of CL, the transformation of individual experiences through internalization of group experiences (which also includes social and motivational components in addition to cognitive components) is of particular interest. The application of multilevel modeling is particularly relevant to the analysis of CL processes and products [ 147], [ 148].
3. Analyze sequentiality and transformations over time. At a basic level generic logfiles, trace data, and digests can be used to determine the degree of interactivity at various stages during a CL event or entire course [ 149]. Jeong [ 139], for example, also developed a Data Analysis Tool as an extension to ForumManager for sequential analysis and identification and visualization of patterns. Alternative approaches to the analysis of sequentiality and transformation are, for example, user defined start-and-stop sequence indicators [ 117], latent growth curves [ 150], dynamic multilevel analysis [ 151], event-centered analysis [ 119]), and uptake contingency graphs [ 116]. Finally, recent developments in automated coding [ 152], [ 153] offer directions for natural language processing applications for CL monitoring and assessment.
4.1.3 CL Display
1. Awareness displays. Research on awareness originated in the area of Computer-Supported Cooperative Work (CSCW). For any type of collaboration, it is essential to know “who has done what and when” to coordinate group efforts. Over the past decade research on group awareness has rapidly expanded in CSCL. Hesse [ 154] distinguishes three broad types of awareness information: group awareness, situation awareness, and history awareness. Various types of group awareness information and visualizations are distinguished: social awareness (“who is around?” [ 77]), activity awareness (“who has done what?” [ 155], [ 156]), group performance and member contribution awareness (“where do we stand as a group?” and “how have individual members contributed to the group?” [ 126], [ 157], [ 158] [ 159]), and knowledge awareness (“who knows what?” [ 160]); as well as combinations of several types of awareness information [ 126], [ 161]. The information on the functioning of the group and individual members can be easily obtained through self- and peer assessments to assess cognitive (e.g., productivity) and social (e.g., reliability) aspects of CL, which can subsequently be displayed and applied to further individual and group reflections (see for example [ 126]). Although group awareness is the most common type of awareness in CSCL, situation awareness (e.g., representation of the CL task and the available scripts), and history awareness (e.g., prior CL experiences of one self and/or fellow group members) can be powerful tools for CL assessment by the teacher and the students.
2. Dynamic monitoring and assessment display. Teachers are typically confronted with the task to simultaneously monitor multiple groups, and sometimes these groups work in different or multiple environments. What teachers need, for example, is a dynamic display to monitor and assess progress of all groups or only a subset of all groups. To this end, RSS feeds could be applied. RSS feeds are popular in browsers and blogs, but these are not yet applied to dynamically allow a teacher or student to receive updates on specific parts of the CL environment (e.g., to only receive updates from additions to a specific forum, or only updates on new documents in a specific group space). The RSS feeds could signal the teacher of a potential collaboration breakdown, or help the teacher to monitor a group s/he suspects of being close to a breakdown. Subsequently, the teacher can activate a RSS feed tied to the discussion forum (or to the entire group space) in order to receive regular updates. Obviously, dynamic addition or removal of new feeds is required and will enable the teacher to monitor a forum in group A and the shared document space in group B. Once a group functions well (again) the feed can be removed. Ideally, a monitoring panel should allow for import from various platforms, e.g., Blackboard and Moodle. Alternatively, the teacher could activate “footprint” widgets and monitor student or group activity as they move through the online environment (trace data). Finally, whereas most generic systems provide information on actions performed, for example, which students or groups submitted their assignment, information on those students or groups who did not submit their individual contribution or group assignment is equally important—with respect to these students or groups, a teacher may want close(r) monitoring or even intervene. One endeavor in the direction of dynamic “mash up” tools is the Overherd project that specifically aims to alleviate the assessment and monitoring challenges associated with large lectures and multiple forums through automated analysis and multiple visualizations of user data [ 162].
3. User group adaptable displays. Obviously, different user groups have different requirements for displays. For example, different visualizations might be more appropriate for different educational levels (primary, secondary, and higher education). Intelligent support could help to extract information for CL displays depending on the specific user group [ 162]. As for the extent of student involvement in monitoring and assessment, the purpose—i.e., assessment at the end of CL (summative) versus monitoring during CL (formative)—has strong implications for the type of information that can be displayed.
• The author is with the Faculty of Social and Behavioural Sciences, Institute of Education and Child Studies, Centre for the Study of Learning and Instruction, Leiden University, PO Box 9555, 2300 RB Leiden, the Netherlands. E-mail: firstname.lastname@example.org.
Manuscript received 12 Apr. 2010; revised 27 Aug. 2010; accepted 31 Oct. 2010; published online 23 Nov. 2010.
For information on obtaining reprints of this article, please send e-mail to: email@example.com, and reference IEEECS Log Number TLTSI-2010-04-0059.
Digital Object Identifier no. 10.1109/TLT.2010.37.
Jan-Willem Strijbos received the MA degree from the Radboud University Nijmegen in 1999 and the PhD degree (with honours) from the Open University of the Netherlands in 2004. From 2005 to 2009, he was a postdoctoral researcher in the Institute for Child and Education Studies at the Leiden University, and he presently works there as an assistant professor. He is a member of the Consulting Board for Computers in Human Behavior and edited special issues on topics such as “CSCL methodology” ( Learning and Instruction, 2007), “peer assessment” ( Learning and Instruction, 2010), and “roles in CSCL” ( Computers in Human Behavior, 2010). He also edited the third volume in the Springer CSCL series, “ What We Know about CSCL” (2004). His current research interests are the design of (computer-supported) collaborative learning, peer assessment, peer feedback, and discourse analysis and methodology for (CS)CL analysis and assessment. He is a member of the EARLI and the ISLS.