An Ambient Awareness Tool for Supporting Supervised Collaborative Problem Solving
JULY-SEPTEMBER 2012 (Vol. 5, No. 3) pp. 264-274
1939-1382/12/$31.00 © 2012 IEEE

Published by the IEEE Computer Society
An Ambient Awareness Tool for Supporting Supervised Collaborative Problem Solving
Hamed S. Alavi , IEEE Student Member

Pierre Dillenbourg , IEEE Member
  Article Contents  
  Introduction  
  Relevant Research  
  Context: Recitation Section  
  Awareness Information  
  Awareness Tool: Lantern  
  User Study  
  Analysis  
  Discussion and Limitations  
  Conclusion  
  References  
Download Citation
   
Download Content
 
PDFs Require Adobe Acrobat
 

Abstract—We describe an ambient awareness tool, named Lantern, designed for supporting the learning process in recitation sections, (i.e., when students work in small teams on the exercise sets with the help of tutors). Each team is provided with an interactive lamp that displays their work status: the exercise they are working on, if they have called for help, since when, and on which exercise. Lantern, by providing this information, is meant to facilitate the interaction between tutors and teams, and to encourage collaboration among students. We report on a user study that examines the impact of Lantern on individual and group behavior in recitation sections. The results show how Lantern can improve the efficiency of tutor-teams interaction, increase the intrateam collaboration, and improve the structure of interteam communications. On the other hand, having a minimalist design, and being embedded in the classroom environment, it avoids diverting the focus of students from their main task and fades quickly in the periphery when not used.

Introduction
In university teaching, recitation sections are sessions in which students, in small teams, solve preassigned problems while one or more tutors provide support by giving explanation and guide. These sessions supplement lectures by providing 1) an informal learning atmosphere that allows for collaboration among students, and 2) an opportunity for students to benefit from a continuous interaction with the tutors. Group work, as it is well established in the CSCL research, can trigger learning processes such as explanation and argumentation through which students build up their knowledge. Second, tutoring is approved to be an effective educational strategy. Learners can gain higher performance in the tutoring compared to the traditional lecture-based classroom condition in terms of overall understanding and motivation [ 1 ], [ 2 ], [ 3 ]. Recitation sections, by providing both these learning opportunities together, have the potential to be extremely profitable.
Nevertheless, typical recitation sections are far from being optimal. Students tend to come to get the answer instead of elaborating the solution. They usually have a limited circle of friends with only whom they collaborate occasionally. Conversely, students complain that they have to wait long to receive help. In many cases, when choosing which team to assist, the tutor fails to notice or respond to the most pressing request.
In order to have a more accurate insight into the causes of deficiencies, we have conducted a field study of recitation sections in our university. The results, as are extensively discussed in our previous work [ 4 ] and recalled in Section 3.1, conclude that a major part of the shortcomings is originated from a lack of awareness information, both on the part of tutor and among students. It is not clear for the tutor who needs help, on which exercise and since when. Similarly, students have almost no clue how the others are progressing.
We propose and examine an ambient awareness tool, called Lantern ( Fig. 1 ), that gives information on the work status of students. We hypothesize that, by providing this information, our tool can facilitate the tutoring charge and trigger collaboration within and between teams. On the other hand, its ambient user interface avoids diverting the focus of student from their main task when not required. In this paper, we try to validate these hypotheses through an in-situ user study.




Fig. 1. Lantern is an interactive lamp that uses a simple visual grammar to display a piece of information.







The remainder of this paper is structured as follows. We first position our work within the relevant research domains in Section 2. Then, Section 3 gives a general picture of recitation sections, completed by the specific properties of the ones that we observed in our filed study. We explain the objectives and the design of our tool in Sections 4 and 5. The user study to assess its effectiveness and the results are described in Sections 6 and 7. We justify the main results of our experiment and discuss the limitations of the study in Section 8, and finally conclude in Section 9.
2. Relevant Research
Describing an educational technology designed to improve the efficiency of a tutored collaborative problem-solving activity, our work lies at the intersection of two domains: 1) CSCL research on tools for regulating teams' interactions, and 2) Help Seeking and Tutoring. In this section, we review these two research domains and position our work within each of them.
2.1 Computer-Supported Collaborative Learning
In CSCL, Jermann et al. provided a framework that categorizes collaborative learning supporting systems into three classes [ 5 ]: 1) mirroring systems, which display raw indicators to collaborators 2) metacognitive tools, which monitor the interactions, process the collected data and represents the state of interaction, and 3) coaching systems, which offer advice based on an interpretation of the learning indicators. We exemplify each category and compare our work against the others.
Chen [ 6 ] designed a tool, called Assistant, that monitors the collaboration, visualizes the processed data, and provides advice to the teacher in a distance learning context. It can also learn from teacher's feedback to improve its performance. Assistant is a coaching system and falls into the third category.
In the middle category (metacognitive tools), Avouris et al. [ 7 ] developed a collaboration environment called Synergo, for collocated and distance learning. Synergo monitors the activity, makes analysis and visualizes quantitative parameters like density of interaction, symmetry of partner's activity, etc. Moreover, Synergo provides teachers with useful information to manage the interactions occurring in the classroom.
Lantern fits in the first category as it mirrors the state of students without any preprocessing. Our work is also different than Chen's and Avouris' in terms of the level of interaction it considers. While Assistant and Synergo are mostly centered on interactions within one group, we also look at the interaction between groups and tutors as well as the interactions among groups.
2.2 Tutoring
Soliciting help, as an effort from the learner side, initiates a process of communication that involves both the learner and the tutor. The effectiveness of such process and the mutual role of learner and tutor has been a major topic within the research on "Tutoring." Here, we recall three well-established general findings of this research field:

    1. Students gain significantly higher performance in the tutoring compared to the traditional lecture-based classroom condition. This is measured mainly in terms of overall understanding, motivation, and task completion time [ 1 ], [ 2 ], [ 3 ].

    2. Tutors learn the subject that they tutor [ 8 ]. Cohen's analyzed 38 studies among which, in 33 cases, he found indications of tutors' learning [ 3 ].

    3. Tutors often do not have formal training in pedagogical skills; nevertheless, tutoring is effective [ 9 ], [ 10 ], [ 11 ]. They do not have particular expertise on when to give feedback, how to scaffold learning, when to give explanations, when to hold back explanations and allow the students constructs their knowledge through trial and error, although they are knowledgeable in the content domain. On the other hand, Cohen's metaanalysis shows that the impact of tutoring on learning is not significantly related to the amount of tutor's pedagogical skill or age differences between the tutor and the student [ 3 ].

In our work, we investigate the potential impact of using an ambient awareness tool on the efficiency of tutor-students interaction within the specific context of recitation sections.
3. Context: Recitation Section
As a university pedagogical practice, the recitation section is a complement to the lecture session. Students work on their assignments, individually or in small groups. Depending on the size of the class, a number of tutors are present to give help, hints, or in some cases public explanation. When they need help, students raise their hand and wait for the tutor. Recitation sections are meant to provide a semiformal learning atmosphere where students can practice the course material in collaboration with others while having a continuous interaction with the tutors.
In particular, the recitation sections we studied shared the following properties:

    • Attending the class was not mandatory for the students.

    • Each session lasted about 2 hours, while students were allowed to leave earlier.

    • Three to eight exercises were assigned for each session.

    • The exercises were usually theoretical, in that they only involved pen and paper.

    • One to three tutors ran each session.

    • Groups are formed freely by the students.

    • Twenty to thirty-five students attended each session, working individually or in small groups of at most 6.

For the rest of this paper, we refer to a student group as a team. A team could consist of only one student.
3.1 Field Study
We studied 11 recitation sections at our university. Three first-year Calculus courses given by three different lecturers and groups of tutors were observed during four consecutive weeks. Each course consisted of a series of weekly lectures as well as recitation sections. Observations were done silently, that is, we tried to retain the classes intact and to keep the dynamics of recitation sections as they take place normally. The results of this study is extensively reported in our previous work [ 4 ]. In the following, we recall some of the findings which make the basis for further discussions in this paper.
The interaction between teams and tutors appeared to be not efficient:

    • The teams do not raise their hand as soon as they need help, but wait for the moment when they can catch the attention of the tutor. They devote quite a lot of attention to monitoring the tutors' availability. In our observation, students spent 62 percent of their waiting time trying to catch the tutors' attention.

    • When a tutor is answering a question, she continuously monitors the room to check if there are other teams called for help, which also takes some attention.

    • The order of answering does not follow the order of help request in a fair way.

Note that, when referring to the efficiency of student-tutor interaction, we specifically focus on the time and effort that students need to spend for calling tutors and the tutors need to spend for managing the requests. Otherwise, once a tutor starts helping a student, the efficiency of their interaction depends on their communication skill, which is out of the scope of this paper.
Students form teams spontaneously and freely. They seem to do this mostly based on their former friendship connections. Team members seat in close proximity and go through the exercise set together, while each team has limited communication with some of its neighbors. In a session, there are 3-5 completely disconnected clusters of teams. Fig. 2 shows the model of interaction in recitation sections. In this figure, a link between two teams indicates that they communicated at least once during the session. Two teams from two clusters (shown as rectangle) never communicate with each other.




Fig. 2. Structure of interactions in recitation sections.







3.2 Summary
We speculate that, the inefficiency of learner-tutor interaction, and limited communication among teams, have one common root cause: lack of explicit awareness about student's status. For the tutors, when students raise their hand it is not apparent where the help is needed, how much time they have spent on it before calling for help, and whether there are other teams struggling with the same problem. Providing tutors with this information would support the help-management charge and improve their interaction with the teams. Similarly, adding to the knowledge of students about other teams might encourage them to take action when appropriate. For instance, one team could seek advice from another that seems to have completed an exercise the first team is stuck on.
The awareness tool that we propose in this paper build on this conjecture, which in turn will be validated through evaluating their effectiveness.
On the other hand, our tool must satisfy a set of design constraints. It should avoid

    1. violate the privacy of students,

    2. judge their performance,

    3. push students toward a situation of stressful competition, and

    4. distract students from their main task.

4. Awareness Information
Early on the design process, we came to believe that providing the following pieces of information strikes a reasonable balance between our design goals and constraints. For each team

    • The exercise on which the team is working,

    • The amount of time the team has spent on the current exercise since the beginning of the session,

    • If the team has asked for help,

    • Since when the team is waiting to receive help,

    • Since when the team is receiving help.

In order to display these data in the classroom setting, we developed an interactive lamp described in the following section.
5. Awareness Tool: Lantern
Lantern, shown in Fig. 3 , is a small and portable lamp which consists of five pairs of Light-Emitting Diodes (LEDs) installed in a column and covered by a blurring plastic cylinder. A microprocessor controls the LEDs.




Fig. 3. Lantern is controlled by turning and pressing and uses light to indicate its status. It records the user actions which can be downloaded through a USB port.







Each team is provided with a Lantern which makes use of a simple visual grammar to show the status of that team:

    Color: The color of the team's Lantern indicates the exercise they are currently working on.

    Intensity. The intensity of the light specifies the time that has been spent on the current exercise.

    Blinking. It indicates that the team is calling for help.

    Frequency of blinking. The faster the rate of blinking, the longer the team has been waiting.

Users interact with Lantern in two ways:

    Turn. By turning Lantern users choose an exercise.

    Press. In order to call for help, teams press on their Lantern.

However, the information that Lantern conveys is beyond merely the status of teams. It can inform a team if there are other teams who are struggling with the same problem as they are, if they call for help less or more often than the others and, in general, how they perform comparing to the other teams. The tutors can see who has called for help, since when and on what exercise. They also know if the calling team has tried enough before asking for help. If a majority of teams need help on the same exercise, the tutors might decide to give some public explanation on the board. The tutors can also detect the shy teams who never call for help even when not progressing. Moreover, Lanterns can give a quick overview to a visitor like the instructor of the course to see if some of the exercises are particularly time consuming, if there is need for more tutors, or simply to make sure that everything goes as expected ( Fig. 4 ).




Fig. 4. Lantern was used in several recitation sections.







We were aware of the fact that collaborative problem solving under supervision of tutors is an invaluable and rare learning opportunity, and thus any source of distraction could be extremely costly. Therefore, Lantern is designed in line with the idea of Ubiquitous Technologies and Ambient Interface which suggests giving information in an ambient way and embedding it in the user's surroundings. In the following, we briefly recall the notion of ubiquitous computing and review the works that investigate its application in education.
5.1 Ubiquitous Computing in Education
As envisioned by Weiser in 1988, the idea of Ubiquitous computing (Ubicomp) is to embed the computing power in the environment and everyday artifacts. The application of Ubicomp in education has been explored in two ways:

    1. The technology that supports learning is embedded in the classroom environment and artifacts.

    2. Taking advantage of the Ubicomp development, the educators design novel learning experiences outside of classroom while moving and provide learning material relevant to the social/physical environment.

We exemplify either of the categories: "Classroom 2000" [ 12 ] and "Subtle Stone" [ 13 ] for the first, and "Ambient Wood" [ 14 ] for the second one.
In 1995, Abowed launched the Classroom 2000 project. The initial goal was to empower classroom artifact such that they can save what has been discussed during a session as an indexed reference that students can later review. An electric board time stamps all the annotations by the teacher, and stores them along with the slide transition and the video record of the lesson. Immediately after the session, the captured lesson was made available through a software application.
Balaam et al. developed and studied an ambient tool called Subtle Stone with which students express their emotional experiences to their instructor during the lecture session at real time [ 13 ]. Subtle Stone is an interactive hand-held orb that changes color when squeezed and transmits the current color wirelessly to a tablet PC used by the teacher. The teacher, on her tablet, sees a physical map of the classroom overlaid by the students emotional data. This work builds on the theories that suggest students' emotion and reflecting on it can trigger and sustain academic motivation.
Ambient Wood is a set of tools, installed in woodland, that deliver digital information to the students when learning about ecology outdoor: The Probe is a hand-held device that allows for real-time measurement of light and moisture; The Periscope is a viewing tool showing prerecorded videos about the habitat, and The Ambient Horn replays sounds that represent plant or animal processes. All these tools were connected to an infrastructure that tracks the students in the woodland, registers the corresponding location to the collected data, and triggers location-based information delivery.
Lantern, similar to Classroom 2000, enriches the classroom environment with the computing power, and similar to Ambient Wood and Subtle Stone can move back and forth to the center and the periphery of the learners' attention.
6. User Study
In order to evaluate the effectiveness of Lantern, we conducted a user study. We were particularly interested to answer the following questions:

    Q1. Does Lantern make the tutor-team interaction more efficient?

    Q2. Does Lantern increase the intrateam collaboration?

    Q3. Does Lantern increase the interteam communication?

6.1 Setting
Third-year university students are observed during six regular and six Lantern-equipped recitation sections. Table 1 shows the basic parameters of the sections.
6.2 Data Collection
One observer (the first author of this paper) tried to note at real-time every interesting event, especially the collaboration among students which is mostly recognizable as verbal interaction and certain body postures. Considering the small size of the classrooms, it was not a problem to distinguish the topic of the conversations (relevant or irrelevant to the course material). More precisely, the observer with a one-minute precision registered whether each student is

    1. working individually,

    2. collaborating with a teammate,

    3. communicating with another team, or

    4. not engaged in the exercise set (e.g., Table 2 ).

In addition, from the data logged by Lantern, we know about the status of each team: when they start and finish an exercise, when they call for and receive help. Altogether, the following data have been collected from each session:

    1. When a student starts collaborating with another member of her team and when they finish it.

    2. When a student starts communicating with another team and when they finish the communication.

    3. When a team calls for help, when the tutor arrives to give help, and when the tutor leaves the team.

The collected data also can be represented by the status of each student, as visualized in Fig. 5 with one minute precision. In this figure, the student starts collaborating with a teammate at time $t_1$ . At time $t_2$ , her team call for help and starts waiting for the tutor who arrives at $t_4$ . Meanwhile, she stops the collaboration at $t_3$ . The tutor leaves the team at $t_5$ and she goes back to the problem solving mode.

Table 1. Observed Sessions in the Control and Lantern Conditions



Table 2. The Work Status of the Students Noted by the Observer







Fig. 5. Status of a student during displayed as a time diagram.







At the end of the experiment, the students answered a questionnaire. Among the questions, we analyze the responses to four questions as they are particularly relevant to the research question that we will try to address in this chapter:

    1. "Did using the interactive lamps have any effect on the exercise sessions? How?"

    2. "Did the lamps distract you? No—Only in the first session—Yes."

    3. "Is this correct: Before pressing on our lamp to call the teaching assistant, we discussed in our team if help is needed."

    4. "Is this correct: In typical exercise sessions (without lamps), before calling for help we discuss in our team if help is needed."

A series of informal interview with the tutors was also conducted.
7. Analysis
We start by defining some concepts which we will use in our quantitative analysis: a Demand $d_i$ identifies a help request from a team. The function $T_r (d_i)$ returns the time when the team raise hand to show the demand $d_i$ , and the function $T_a(d_i )$ returns the time when the tutor starts answering the demand $d_i$ . A set $D =\{d_1, \ldots, d_n\}$ includes all the demands that occur in a given recitation section, sorted in ascending order with respect to $T_a(d_i )$ . (That is, $d_{i+1}$ is the demand that gets answered right after $d_i$ .)
7.1 Learner-Tutor Interaction
According to our field study, teams do not raise hand as soon as they need help and thus hand raising is not an accurate sign for the beginning of the waiting period. In the following, we show 1) how significant this inaccuracy is, and 2) how we estimate the beginning of the waiting period.
Fig. 6 splits a recitation section into consecutive Busy and Free periods. In a Busy period, all the tutors are busy and in a Free period at least one tutor is available. A BF interval encompasses one Busy period and the Free period after it.




Fig. 6. A session encompasses the periods of time when all the tutors are busy or at least one tutor is available.







Fig. 7 shows the cumulative distribution of hand raisings within a single BF episode, from a set of observed recitation sections when Lantern was not used. One point at (50, 0.1), for example, illustrates that only 10 percent of the teams raise hand during the first half of the BF. As shown in this figure, the boundary between the Busy and Free periods, on average, is at 89.4 percent of the BF interval.




Fig. 7. The distribution of hand raisings during a BF interval, taken from a set of observed recitation sections in which Lantern was not used.







This curve is obtained by normalizing the length of all the BF iterations into the same unit of time. The fast growing slope of the curve at the end of the BF illustrates the fact that, in so many cases, the teams prefer to raise hand at the end of the BF: when one of the tutors just became free or looks to become free shortly. This fact reveals that teams self-regulate. They refrain from raising hand when there is low probability to receive help. This self-regulation implies that teams devote significant cognitive effort to monitoring the tutors' availability and therefore are not productive while waiting for them.
Let us suppose that, within a BF, the number of questions is uniformly distributed in time, i.e., for any team, the probability of facing difficulty at any time point within a certain BF is the same. We argue that this can be a valid assumption since the BF periods are fairly short (173 second in average). Based on this assumption, we compute the beginning of the waiting period for a demand $d_i$ as the middle of the BF into which it falls:


$$[T_a(d_{j+1})-T_a(d_j)]/2,$$


(1)



such that


$$T_a(d_j)<T_r(d_i)<T_a(d_{j+1})$$


(2)



and, consequently, the average waiting time as


$$\left(\sum^n_{i=1}{T_a(d_i)-[T_a(d_{j+1})-T_a(d_j)]}\bigg/2\right)\bigg/{n}.$$


(3)



Table 3 shows the time that each team, on average, had to wait during a session in the control and Lantern conditions.

Table 3. Average Waiting Time and While-Waiting Productivity for Each Observed Session in the Control and Lantern Conditions





7.1.1 While-Waiting Productivity (WWP) According to our observations, when teams have to wait for the tutor they decide between 1) immediately raise hand, keep it up and still do some problem solving or 2) chase the tutor with their eyes and wait until the right time when they can catch her attention ( Fig. 8 ).



Fig. 8. When students are waiting, they can raise hand and still work (the students in the first row), or monitor the tutor's state until she seems to becoming free and then raise hand (the student in the last row).







We define While-Waiting Productivity as the fraction of the waiting time that is not spent on chasing the tutors. We use this parameter as an indicator of the tutor-team interaction efficiency. In an ideal situation where all the teams always call for help immediately after they realize need help, based on our former assumption, the number of calls for help during a certain BF interval is uniformly distributed. We compute the WWP as the closeness of the distribution of calls for help to the ideal distribution. More precisely, for each demand, the difference between the time at which the actual hand raising happens and the time we compute with (1) as the beginning of waiting period gives the fraction of the waiting time that has been spent on chasing the tutor. The following formula gives the average WWP:



$$1{-}\bigg(\!\sum^n_{i{=1}} \left(\!\!T_r (d_i)\!-\!{[T_a (d_{j +1}){-}T_a (d_j)]\over {2}} \right) \!\bigg/{[T_a (d_{j{+1}}){-}T_a (d_j)]\over {2}} \bigg)\!\bigg/n.$$


(4)



We eliminate the questions which get answered immediately, as the productivity of a very short waiting period is negligible.

Table 3 compares while-waiting productivity in the Lantern and control conditions. The remarkable improvement from 41.3 to 91.1 percent validates our conjuncture that Lantern can reduce the required effort to interact with tutors (positive answer to question Q1).

Note that the performance of students during the waiting time is of special interest, since this is usually a high value period when students are challenged with and focused on exercises. Furthermore, as Table 3 shows, the waiting time can be considerably long, (depending on the number of demands and the number of tutors).



7.1.2 Revising Research Questions A while-waiting productivity of 91 percent means that students use 9 percent of their waiting time chasing the tutor. An interesting question is what do students do during the 91 percent? Specifically, do they collaborate and what is the influence of Lantern on that? We believed that, Lantern can encourage both intrateam collaboration and interteam communication when students are waiting for tutor. If members of a team need help while trying to solve a problem together, they can press on Lantern and continue the collaboration, whereas without Lantern they usually stop the collaboration and start chasing the tutor to get her attention. On the other hand, if a team waiting for help on an exercise realize that their neighbors have already solved that exercise may ask from that team, which triggers interteam communication. We thus, make our research questions Q2 and Q3 more precise as follows:

    Q2.1 Does Lantern increase intrateam collaboration while the team is waiting for tutor?

    Q2.2 Does Lantern increase intrateam collaboration while the team is NOT waiting for tutor?

    Q3.1 Does Lantern increase interteam communication while the teams are waiting for tutor?

    Q3.2 Does Lantern increase interteam communication while the teams are NOT waiting for tutor?

7.2 Intrateam Collaboration
We define four parameters for each session:

    $Col_{C}^{W}$ , $Col_{L}^{W}$ is the percentage of the waiting time each student spent on intrateam collaboration, averaged in the session, in the control and Lantern conditions, respectively.

    $Col_{C}^{P}$ , $Col_{C}^{P}$ is the percentage of the problem solving time each student spent on intrateam collaboration, averaged in the session, in the control and Lantern conditions, respectively.

Note that, in our analysis, we exclude the time intervals when the team is receiving help (like $t_4$ to $t_5$ in Fig. 5 ) because the interaction among students is highly influenced by the presence of the tutor while receiving help. We start by comparing $Col_{L}^{W}$ against $Col_{C}^{W}$ . For the case of collaboration that begins before the waiting period and lasts through it (for example, the first collaboration in Fig. 5 ), the second part ( $t_2$ to $t_3$ ) counts in the $Col_{}^{W}$ values. An unpaired t-test shows a significant improvement with Lantern ( $m_{lw}=42$ , $m_{cw}=11.17$ , $t[10]=10.97$ , $p<0.0001$ ): positive answer to question Q2.1.
One may explain this improvement as the direct effect higher while-waiting productivity. In order to test that, we modify the measures as follows:


$$\overline{Col}_{C}^{W}= {Col_{C}^{W}\over WWP_{C}} \quad{\rm and}\quad \overline{Col}_{L}^{W}= {Col_{L}^{W}\over WWP_{L}}.$$


In which, $WWP$ is the average while-waiting productivity of the session. An unpaired t-test shows that $\overline{Col}_{L}^{W}$ is still significantly higher than $\overline{Col}_{C}^{W}$ ( $m_{LW}=47.1$ , $m_{CW}=24.2$ , $t[10]=5.8$ , $p<0.001$ ).
We found no significant difference between $Col_L^P$ and $Col_C^P$ ( $m_{LP}=33.8$ , $m_{CP}=32.6$ , $t[10]=0.40$ , $p>0.1$ ), meaning that Lantern has no statistically significant effect on the intrateam collaboration when the team is not waiting: negative answer to question Q2.2. We were also interested to know if students collaborate more when they are waiting than when they are in problem solving mode, in either of the conditions. Comparing $\overline{Col}_C^W$ to $Col_C^P$ , and $\overline{Col}_L^W$ to $Col_L^P$ using a paired t-test,interestingly enough, we found out that in the control condition students collaborate less than usual when they are waiting ( $m_{CW}=24.2$ , $m_{CP}=32.6$ , $t[10]\;{=}$ $-2.62$ , $p<0.05$ ), whereas in the Lantern condition students collaborate more when waiting ( $m_{LW}=47.1$ , $m_{LP}=33.8$ , $t[10]=8.19$ , $p<0.001$ ). Fig. 9 summarizes the result of the above comparisons.




Fig. 9. Intrateam collaboration, across the condition (PC: $\overline{Col}_{C}^{P}$ , WC: $\overline{Col}_{C}^{W}$ , PL: $Col_{L}^{P}$ , WL: $Col_{L}^{W}$ ).







7.3 Interteam Communication
This section analyzes the effect of Lantern on interteam communication in terms of the duration of the interactions as well as their structure.


7.3.1 Duration To measure the duration of interteam communication, we define four parameters:

    $Comm_{C}^{W}$ , $Comm_{L}^{W}$ , for a given session, is the total time that two teams communicated while at least one of them was in Waiting Period, over the total waiting time of the session, in the Control and Lantern conditions, respectively.

    $Comm_{C}^{P}$ , $Comm_{L}^{P}$ , for a given session, is the total time that two teams communicated while both of them were in Problem Solving Period, over the total time that the teams were in Problem Solving Period, in the Control and Lantern conditions, respectively.

Note that, when a team calls for help, all its members switch from the Problem Solving Period to a new Waiting Period. Similarly, when the tutor leaves the team, all the members enter a new Problem Solving Period.

Fig. 10 compares these values across conditions. Table 4 shows the tests that we conduct to compare each pair of data sets and their results. In summary, Lantern did not significantly increase the duration of interteam communication, while the communicating teams are waiting or while they are doing problem solving, and therefore we negatively answer questions Q3.1 and Q3.2.





Fig. 10. Interteam communication over different time periods, across the conditions.







Table 4. Tests to Compare the Duration of Interteam Communication in the Control and Lantern Conditions





7.3.2 Rate of Occurrence To illustrate the different measures of possible interteam communication patterns, Fig. 11 shows a hypothetical example when 10 teams attended a session. In this figure, the number of links between two teams indicates the number of times that they communicated during the session. In this example, the total number of times that the teams communicated is 8.



Fig. 11. Structure of interteam communication.







We computed the total number of times when interteam communication took place in the observed sessions divided by the number of teams, and compared it across the conditions. Although the mean value in the Lantern condition is higher, unpaired t-test does not show a statistically significant increase ( $m_{c}=0.36$ , $m_{l}=0.48$ , $t[10]=1.64$ , $p=0.13$ ). Fig. 12 summarizes these values as a boxplot.





Fig. 12. Number of times that interteam communication happened over the number of team, across condition.









7.3.3 Structure of Interteam Communication In order to quantify the effect of Lantern on the structure of interteam communication, we define the following parameters for a session:

    Diversity of communication is the total number of pairs of teams who communicated at least once in the session. In Fig. 11 , the diversity of communication is 6. Note that when measuring the diversity of communication, several links between two teams is counted only as one.

    Number of communicating teams is the total number of teams who communicated at least once with another team. In Fig. 11 , there are eight communicating teams.

    Cluster is a subset of teams in which every team is directly or indirectly connected to every other team (corresponding to "clique" in social network theory). In Fig. 11 , there exist four clusters shown with dotted border lines.

Note that, even though these parameters are interrelated, increasing one does not necessarily yield an increase in the others. Each parameter is averaged over the number of teams in the sessions and analyzed using unpaired t-tests. The results show significant increase in all the parameters. The details of our analysis are given in Table 5 and Figs. 13 , 14 , 15 .

Table 5. Performed Tests on Interteam Communication







Fig. 13. Diversity of communication across condition.











Fig. 14. Number of communicating teams over total number of teams, across condition.











Fig. 15. Number of clusters over number of teams, across condition.







Table 6. The Users' Response when They Were Asked if Lantern Distracted Them



7.4 Qualitative Findings

    Competition among students. Only eight out of 55 students who filled the questionnaire answered "yes" to the question "Did you use Lantern to compare your team's performance against the others'?"

    Distraction. We asked the students and the tutors if Lantern distracted them. There were three choices: 1) No, 2) only in the first week, and 3) yes. Table 6 shows the answers from 55 students and five tutors.

    Synchronization. In the questionnaire, the students were asked if: 1) Before pressing on our lamp to call the teaching assistant, we discussed in our team if help is needed. 2) In typical exercise sessions (without lamps), before calling for help we discuss in our team if help is needed. Out of 55 students who filled the questionnaire, 31 selected only the first option, eight selected the both, nine selected only the second one, and seven did not answer. We also asked similar questions concerning the progress in the exercise set: 1) Is this correct: "I agreed with my team if we should turn the lamp and go to the next exercise. 2) Is this correct: "In typical exercise sessions (without lamps), before moving to the next exercise I agree with my team." Twenty-seven students selected only the first option, five selected the both, 10 selected only the second one, and 13 selected neither of the options.

    Similar questions. In the sessions with Lantern, there were eight cases where the tutors noticed that many teams were waiting to receive help on a certain exercise; they managed the situation by giving public explanation on the classroom board. Similarly, in many cases, two or three teams in one corner of the classroom needed aid on the same exercise; the tutor gave a semipublic explanation for the teams at that corner. The tutors adjusted the audience size to optimize their effort.

    Late/never help seeking. With Lantern, tutors can see if a team has spent much time on the current exercise and still hesitate to seek help from the tutors. In our interview with the tutors, when they were asked "how do you think Lantern can be helpful?" one replied: "With Lantern, the students' complaints is not the only way any more to see that they are in trouble." Then, he explained by refereing to the fact that there are always shy teams who never contact the tutors and that with Lantern he could recognize if they were progressing as expected or need help. The tutors have an internal standard that they use to monitor progress; Lantern supported it by giving a making the students' progress visible.

    Resource scheduling. In our interview with the tutors, when they were asked "how do you think Lantern can be helpful?" one replied: "On the lamp I could see immediately and even from far what the question was about (when students call). If another tutor knows that exercise better than me I would ask him to reach the team." He later mentioned: "students always argue that they have called us before others, with Lantern they have a proof." The information offered by Lantern could help at least one of the tutors to 1) pick the questions that he can confidently answer and 2) when there are more than one questions, answer them in a proper ordering by seeing who has called for help before the others.

8. Discussion and Limitations
In this section, we discuss the results of the study, design implications, and also the limitations of our work.
8.1 Lantern's Effects
Lantern improved while-waiting productivity, trivially because it can offload the charge of catching the tutors' attention. However, this approves that students adopted its functionality as replacement for hand raising.
Lantern increased intrateam collaboration while the team is waiting for the tutor. This can be explained by the combination of two facts: 1) according to the questionnaire, before pressing on Lantern the team members agree that they need help, which itself initiates a discussion, and 2) this discussion can last through the waiting time since no effort to catch the tutors' attention is needed. The quantity of the improvement, comparing to the total time that one student in average interacts with her teammates in the control condition is 22.1 percent.
Moreover, it has been shown that with Lantern students collaborate more with their teammates when waiting for the tutor than when they are doing problem solving. We conclude that, when students are waiting for the tutor, there is a high potential for collaboration, which is lost in chasing the tutor when Lantern is not used. As a consequence, the waiting time does not always need to be shortened: in some cases, using an awareness tool rather than adding new teaching assistants would maintain longer waiting times that become fertile grounds for collaboration to take place.
Lantern did not increase the duration of interteam communication. However, it changed the structure of interteam communication such that 1) each team communicated with a larger number of other teams (higher Diversity), 2) there were fewer teams who never communicated (more Communicating Teams), and 3) the knowledge of one team could spread over a larger part of the class (fewer Clusters). We explain this set of effects as the direct consequence of the main objective of Lantern: adding to the knowledge of students about other teams. The more students know about a specific team, the more they are likely to interact with that team.
8.2 Limitations
With Lantern, we tried to improve the process of Learning in recitation sections through facilitating the tutor-team interaction as well as encouraging students to collaborate with one another. The learning content, however, is not touched. Lantern cannot recognize whether a team that is currently working on the second exercise, has solved the first exercise correctly and completely. With no ability to judge on how the exercises are done or how helpful the interaction with the tutor is, Lantern can only show when a team start and finish working on an exercise, when call for and receive help, and how much time spend with the tutors.
Our user study was not a lab experiment. We were not able to control variables such as exercise difficulty, number of present tutors, number of students and teams, if the students were specially stressed because the assignments were graded or the short time to the final exam.
Data collection, consisting of noting all the students status every minute, is carried out by only one observer. We did not employ more human observers or videotape the sessions to keep the informal atmosphere of recitation sections as natural as possible. Damaging this atmosphere might reduce the students' tendency to freely converse with each other. On the other hand, by deciding to have only one observer, we traded consistency of collected data to the accuracy.
9. Conclusion
This paper introduces Lantern and analyzes its influence on tutor-teams interaction as well as the students collaboration in recitation sections. We show that, Lantern can 1) improve productivity of students while waiting for the tutors, 2) increases the intrateam collaboration taking place while the team is waiting for the tutor, and 3) have effect on the structure of interteam communication in such a way that, each team communicates with a larger number of other teams and that there are fewer teams who never communicate with others.
Furthermore, by providing information on the status of students, Lantern supports the classroom "Orchestration" concern as the tutors responsibility to manage the class needs and resources at real time.

    The authors are with the Swiss Federal Institute of Technology (EPFL), EPFL-CRAFT, Rolex Learning Center, Station 20, CH-1015 Lausanne, Switzerland. E-mail: {hamed.alavi, pierre.dillenbourg}@epfl.ch.

Manuscript received 3 Oct. 2011; revised 8 Feb. 2012; accepted 11 Mar. 2012; published online 13 Mar. 2012.

For information on obtaining reprints of this article, please send e-mail to: lt@computer.org, and reference IEEECS Log Number TLT-2011-10-0102.

Digital Object Identifier no. 10.1109/TLT.2012.7.

References





Hamed S. Alavi received two master's degrees in computer engineering and telecommunication systems from the University of Tehran and EPFL, respectively. He received the PhD degree in 2012 in the domain of ubiquitous learning technologies from EPFL. He is a postdoctoral researcher at Swiss Federal Institute of Technology in Lausanne (EPFL). His research interests include invisible/calm computing and its application to resolve concrete educational problems, especially the concern of classroom orchestration. He is a student member of the IEEE.





Pierre Dillenbourg graduated in educational science from the University of Mons, Belgium, and received the PhD degree in computer science from the University of Lancaster, United Kingdom, in the field of educational applications of artificial intelligence. He is a professor of learning technologies at the Swiss Federal Institute of Technology in Lausanne (EPFL). Previously, he was an elementary school teacher. He started to conduct research in learning technologies in 1984. He is the past president of the International Society for the Learning Sciences. His work covers various domains of computer-supported collaborative learning (CSCL), ranging from novel interfaces for face-to-face collaboration (interactive furniture, tangibles, paper computing) to more cognitive projects on dual eye tracking. He is a member of the IEEE.