The Collaborative Lecture Annotation System (CLAS): A New TOOL for Distributed Learning

Evan F. Risko
Tom Foulsham
Shane Dawson
Alan

Pages: pp. 4-13

Abstract—In the context of a lecture, the capacity to readily recognize and synthesize key concepts is crucial for comprehension and overall educational performance. In this paper, we introduce a tool, the Collaborative Lecture Annotation System (CLAS), which has been developed to make the extraction of important information a more collaborative and engaged process. The system relies on semantically constrained annotation, postannotation data amalgamation and transparent display of this amalgamated data. In addition to describing the CLAS, we report on a user experience study aimed at investigating students' perception of the utility of the tool.

Index Terms—Computer aided instruction, collaborative tools, distance learning, educational technology

Introduction

A major challenge facing all students relates to the capacity to quickly extract important concepts from lectures [ 1], [ 2]. On the other side of the lectern, instructors are frequently pressed to determine the best means through which to communicate and scaffold the course's central concepts. Here, we describe a video annotation tool that features a novel form of social navigation [ 3], [ 4], [ 5] inspired by recent research on event segmentation [ 6]. Specifically, the Collaborative Lecture Annotation System (CLAS) has been developed to 1) make the extraction of important information from lectures a collaborative and interactive process for students and 2) provide instructors with a suite of learning analytics to evaluate the efficacy with which they have communicated the central points in their lectures. In the following, we describe the CLAS architecture and an initial user experience study in a naturalistic educational context.

1.1 Video Lecture Annotation

The development of video lecture annotation tools is playing an increasingly important role in the evolving education landscape [ 4], [ 5], [ 7], [ 8], [ 9], [ 10], [ 11], [ 12], [ 13], [ 14]. This is driven in part by increasing class sizes, and calls for greater flexibility in course offering and timing. However, the development of video annotation platforms that retain and encourage collaboration is not without challenges [ 15], for instance attempting to design a tool that effectively caters to the diversity of learning and teaching approaches. Nonetheless, the extraordinary growth in online education [ 16] and the use of video lectures [ 17] demands a continued effort to innovate conceptually and technologically [ 18].

One of the most popular and well-researched platforms for collaborative video annotation in educational contexts is the Microsoft Research Annotation System (MRAS) developed by [ 7], [ 8]. Briefly, this platform presented the lecturer, the slides, and annotations anchored to time points in the lecture. Annotations consisted of comments and threaded responses to those comments. An example of an annotation might be a question about the presented material at a given point in the lecture (e.g., “what does she mean when she says X?”). These annotations could be viewed and responded to by users. In addition to public annotations, there were also private notes users could store. Thus, through reading, responding, and posting annotations the platform provides a video annotation tool that users can use to collaboratively and asynchronously interact with one another. A similar and more recently developed tool, virtPresenter (now part of the OpenCast Matterhorn Project), represents a similar tool [ 4], [ 5]. This tool captures, processes, and distributes educational audio and video. It also provides tools for students to engage with the material including in-video searching of text, bookmarking, and navigation. Included in the tool is a form of social navigation that allows users to traverse material by following traces caused by the actions of other users [ 3], [ 4], [ 5]. Specifically, users can review the accumulated viewing history of all users (i.e., what parts of the lecture other users have viewed) and also compare their own viewing history with that trace [ 5]. A similar concept, referred to as collaborative filtering, is used in recommender systems [ 19], [ 20].

The CLAS shares functionality with these (and other) lecture video annotation tools but features a form of annotation and social navigation that has not been developed in previous systems. In particular, the CLAS moves away from user defined annotations (e.g., a note taking tool; MRAS) and toward a system defined annotation strategy (i.e., the annotation has the same meaning for all users). This decision leads to important changes in how users interact with the system. For example, semantically constrained annotation alters the nature of the social navigation that can occur in the CLAS. Further details on the CLAS and how it differs from previous systems is provided below. It is important to note that the CLAS is not designed as an alternative to the MRAS, the virtPresenter/Matterhorn system or any related systems; rather the CLAS focuses on one particular mechanism for collaboration that is based on semantically constrained annotation, postannotation data amalgamation and transparent display of this amalgamated data. This mechanism represents the core concept underlying the CLAS and is independent of a specific implementation. Indeed, this approach to collaboration could be considered complementary to previous approaches in that it could presumably be integrated with a MRAS or virtPresenter/Matterhorn like annotation system that offers users a greater set of functionality (e.g., note taking) and alternative forms of social navigation. Furthermore, while the CLAS is implemented here as a video annotation tool, the underlying concept is not limited to such an implementation. For example, the same collaboration mechanism could be used in “live classes” with clicker technology [ 21].

The Collaborative Lecture Annotation System

The basic architecture of the CLAS is based on research in cognitive psychology and cognitive neuroscience on event segmentation [ 6], [ 22]. In these tasks, participants are presented with a dynamic stimulus and are asked to indicate when “meaningful” events occur. For the present purposes the critical observation is that there exists an impressive amount of agreement across different individuals on the starts/ends of these meaningful events [ 6]. Thus, by amalgamating individual event segmentations a consensus representation of the structure of the dynamic event can be produced. The idea motivating the CLAS is to extend event segmentation to an educational context and to use the resulting consensus representations as collaborative aids for students and instructors. As such, the CLAS represents an instance of computer supported collaborative learning (CLCS). This branch of educational psychology is concerned with how computers can be used to facilitate learning in groups [ 23]. The CLAS provides a medium through which students and instructors can engage in “meaning-making” around course content (i.e., the lecture).

The basic concept underlying the CLAS is straightforward. While watching a video captured lecture, each student indicates important points in the lecture by a simple button press (i.e., the spacebar in the current implementation; see Fig. 1 part A). Each button press represents a point-based semantic annotation [ 24] and indicates, “for this user something important happened at this point in the lecture.” A similar annotation feature is available in CoScribe [ 25], a print and digital document collaboration tool, and is referred to as semantic tagging or category tagging (see also collaborative tagging; [ 26]). These “importance markers” are akin to the points indicated in an event segmentation task [ 6], [ 22] wherein the event is a lecture and the segmentation is based on importance not meaningfulness. The use of a system defined point-based annotation as opposed to user-defined annotation [ 7], [ 8], [ 9] creates common ground between users. Specifically, student X's annotation has a similar meaning to student Y's annotation and so forth for each individual using the system. This common ground is fundamental to successful collaboration and is analogous to the importance of common ground in communication [ 27], [ 28]. Indicating a point simply via a button press also places less demands on working memory [ 29], [ 30] and is less disruptive to the flow of the lecture than note-based annotations. Though taking notes would also have its own benefits in an educational context.

Figure    Fig. 1. Levels of processing in the CLAS system.

Once finished, a student's point-based annotations are saved in a file on a server (see Fig. 1 part B). Thus, each student has his or her own personal set of annotations associated with each lecture. When the student revisits the lecture material (e.g., review of course material) they have the opportunity to upload their annotations. Once uploaded, the annotations are graphically presented in what we refer to as the individual graph (see Fig. 2 (Individual Graph)). This individual graph represents time in the lecture and the student's annotated points. Students can use their individual graph by clicking on points in the graph to navigate through the video lecture. Clicking in the graph repositions and plays the video (i.e., the lecture) to the point in time corresponding to the clicked region in the graph. This feature allows students to quickly access points in the lecture that they had previously indicated were important.

Figure    Fig. 2. Examples of individual and group graphs.

The collaborative nature of CLAS emerges when the software takes the individual point-based annotations and amalgamates them (see Fig. 1 part C) into what we refer to as the group graph (see Fig. 2 (Group Graph)). Thus, each student's personal set of annotations is amalgamated into a single graph. This group graph represents the consensus representation of the structure of the important events in the lecture. Thus, the group graph seeks to take advantage of the “wisdom of the crowd” [ 31], [ 32]. At present this amalgamation consists of recording the total number of individual annotations that occur in a particular time bin (currently 10 seconds in length). This count is divided by the total number of users contributing to the group graph thus resulting in a proportional measure of importance across the group as a function of time. In the current implementation, a time-based bar graph is used (similar to the individual graph) to represent importance ratings throughout the entire group. Higher bars represent the points that many students think are important and lower bars represent the points in the lecture that few students identified as important (see Fig. 2). The group graph represents a form of social navigation or collaborative filtering but differs from that used in previous applications [ 3], [ 4], [ 5]. For example, in the case of virtPresenter/Matterhorn the number of views of a given lecture segment across a group forms the basis of social navigation. However, number of views could be related to a number of different factors (e.g., importance, confusion, humor, etc.). The semantically constrained annotations in the CLAS render the past behavior (i.e., annotation history) of users transparent. The group graph's functionality mirrors the functionality of the individual graph. In the following sections, we highlight the potential uses of the CLAS for students, instructors, and the learning analytics community, we then report on a usability assessment of the tool. A usability study was chosen as the first evaluative step given that its educational impact will depend on this factor. Future research will assess the educational impact contingent on usability.

2.1 Students and the CLAS

In contrast to traditional note-taking practices, the point-based annotation process places minimal demands on working memory and reduces disruption to the lecture. For example, when compared to the cognitive demand and disruptive nature associated with writing, stopping the lecture or shifting attention from the lecture to a notepad [ 29], [ 30]. Despite the low cognitive load the point-based annotations provide a meaningful (and durable compared to memory, for example) external representation. In addition, research has demonstrated that memory can be enhanced for information encountered concurrently with a behaviorally relevant action [ 33]. This suggests the possibility that the act of hitting the button at particular moments in time during a lecture could help students encode that information. While the act of extracting important information through point-based annotation has benefits, it is important to note that note taking, in and of itself, has educational benefits as well [ 2], [ 29].

After annotating the lecture video there are a number of uses for the CLAS that could enhance the student learning process. First, individual graphs provide a reminder of the points the individual encoded as important during the lecture. The interface provides easy access to the annotated points to aid in the review of course material and the key identified concepts. Second, the group graph provides a representation of the key points in the lecture as determined by their classmates (and possibly individuals from previous classes). The information provided by the group graph is visualized in a simple and straightforward manner to facilitate personal interpretation. As noted above, the system induces common ground by constraining the annotation to importance. Thus, the group graph amalgamates individual responses to communicate what the broader class considers to be the important points within the lecture. This feature represents a form of control on information flow. Individual student note taking is largely idiosyncratic and collaborative tools that provide functionality to share individual notes (as many extant tools are designed to do; e.g., [ 8]) can quickly become overwhelming due to the vast number of notes. For example, reviewing a group graph formed by an average class of 80 students regarding perceived importance would be considerably more time efficient than reviewing the notes of those same 80 students.

The third student-based application for CLAS is aimed at supporting self-regulated learning. Self-regulated learning represents a suite of metacognitive skills including goal setting and monitoring, strategy use, and management of effort and motivation [ 34], [ 35]. Feedback represents a critical factor in supporting self-regulated learning as it provides a mechanism to monitor and compare progress (e.g., the appropriateness of a given cognitive strategy; [ 34]). Einstein et al. [ 1] demonstrated that the most successful students were also the best at identifying information of high importance in a lecture. Hence, the CLAS represents a tool that can provide useful feedback for both student and faculty in terms of assessing capabilities associated with this critical skill. More specifically, a student can compare their individual graph to the group graph as a benchmarking process. For example, through comparison with the group graph the student (or faculty) can discover that they have missed an important point (or that the group has). The relative alignment between the two graphs represents how correlated an individual's points are with the class as a whole (see Fig. 2). Large discrepancies between the individual graph and group graph could be indicative of students paying attention to the wrong types of information. This feedback can be used to modify strategies for identifying important information in lectures. In the current implementation, this comparison is visual (i.e., the student needs to compare their graph visually to the group graph). However, future versions could include features that compute the degree of correlation between the graphs and even graphically represent this region (i.e., highlight areas of greater or lesser overlap). Recent research has demonstrated that this type of collaboration, where individuals first offer an answer (i.e., I think these parts of the lecture are important) then compare their answer to others (i.e., the group graph), has a significant positive impact on student understanding of educational material [ 21].

Use of the group graph is of course dependent on there being sufficient annotations to make the group graph useful (i.e., the cold start problem). While this issue is limited to the first presentation of a lecture, it can be ameliorated by requiring students to view the lecture according to an instructor defined schedule and only releasing the group graph once a large enough sample size is collected. For example, if all of the students are required to view the lecture between Monday and Thursday (which the tool could also track), then the group graph can be released on Friday thus allowing students to review the material using the CLAS. Temporally staggering the initial annotation and release of the group graph would not be required (though the instructor could still use that option) beyond the first time a set of lectures were presented. The CLAS can combine annotations from multiple classes viewing the same content.

2.2 Instructors and the CLAS

On the other side of the lectern the instructor can also use the CLAS to evaluate the learning impact of presented lecture materials. The CLAS acts as means to record students' perception of importance in the context of lecture material. The instructor, as the individual setting out the educational goals, can use the group graph to assess the efficacy with which the important points are identified and communicated. For example, the instructor could create their own set of annotations for comparison to the class's annotations. Again this feedback is valuable in an educational context and is typically available only after an exam which is arguably too late for enacting any compensatory measures (e.g., class review of important points). The individual graph can play a similar role but at the level of the individual student rather than the class. For example, the instructor can survey an individual's annotations in order to generate potential explanations and remediation strategies for struggling students. Thus, the student's annotation record can be used by the instructor to help support the student's self-regulated learning. It is important to note that the control of the information flow with point-based annotations (i.e., restriction to perceived importance) is also critical for the instructor. While reviewing a group graph of perceived importance is relatively effortless, reviewing the notes of an entire class is cumbersome and ultimately unrealistic especially in the large class context (e.g., greater than 200 students).

2.3 Learning Analytics and CLAS

The concept of learning analytics represents an exciting new interdisciplinary field that is largely focused on optimizing learning outcomes via the collection and use of data available to educators. The CLAS represents a tool that offers a large amount of “throw off” data that can be analyzed in order to identify patterns and anomalies that can provide new insights into student learning. For example, individual student annotations (e.g., frequency), group annotations, and metrics of agreement between individual, group, and instructor annotations could all be used in conjunction with educationally relevant outcome measures (e.g., grades, retention) to track student, instructor, program, and university performance. The CLAS creates a digital representation of student comprehension of educational material, a representation that is transparent, analyzable, and archival and thus a valuable piece of data specific to an individual's learning process.

While the annotation activity itself represents a rich source of data, the CLAS also offers less direct, but possibly equally as rich, data in terms of users' clicking behavior. Specifically, by recording each click and its location user behavior at a finer grain can be tracked. For example, data might include the absolute and relative number of clicks on each of the individual and group graphs, the length of time between clicks, and the amount of time spent reviewing the video among many more. Once collected, these data can then be correlated with educationally relevant variables. For example, the click stream data can be correlated with performance in the course to determine patterns that yield both success (e.g., frequent clicks) and failure (e.g., less time reviewing lecture). Furthermore, these measures can be used as indicators in a predictive model of student outcomes and as a way to introduce and assess interventions. For example, following an intervention (e.g., extra help or tutoring) the click stream data can provide insight into whether or not the student changes his or her pattern of engagement with the class material. Together annotation and clicking behavior can be used to provide insight into student learning and therefore the development of predictive models relating to student support, academic performance, and learning dispositions [ 36], [ 37].

User-Based Evaluation

In the present investigation, we describe the first user experience study of the CLAS in a simulated educational context. The purpose is to assess how students use the CLAS and their perceptions of its utility. In the present investigation, we do not report an evaluation of the educational impact (e.g., learning gains) of the CLAS. Rather, the goal is conceptual development and tool development via the assessment of the user experience. Thus, the research focus is on whether students perceive the CLAS, and thus the concept underlying it, as useful.

Students took part in a two-session study. In the first session, students watched a full-length lecture video in the CLAS. Participants were instructed to treat the lecture as per their more familiar on-campus lecture experience. Participants were shown how to use the annotation functions in the CLAS and told that they would be tested in a second session on recall of the lecture material. In the second session, students were given 30 minutes to study the lecture using the CLAS that was followed by a test based on the lecture. Following the test, students filled out a detailed user experience survey in addition to a semi structured interview about their experience. Students were asked a number of questions aimed at addressing the general user experience. These questions were designed to answer a number of key research questions including 1) will students spontaneously use the individual and group graphs and if so how and how much 2) do students perceive the individual and groups graphs as useful, and 3) can they learn how to use the different types of graphs. While the use of the individual graphs will be familiar at least conceptually to students (i.e., it is similar to note taking), the concept of the group graph is novel to students and thus assessing how it is used and perceived will provide new insights into the potential use of different forms of annotation—an issue that extends beyond the CLAS specifically. In addition to these questions, we were also interested in obtaining user information specific to the current implementation (e.g., graphical user interface). This information can be used to further develop the CLAS specifically but can also be used to inform user preferences in the context of annotation software.

Discussion

We set out to answer a number of research questions in the user experience survey. We outlined three general questions in the introduction 1) will students spontaneously use the individual and group graphs and if so how, and how much; 2) do students perceive the individual and groups graphs as useful; and 3) can they learn how to use the different types of graphs. Results demonstrated that students would readily integrate the use of the individual and group graphs. Indeed, collectively the students used the group graph more than their own graph. Students also reported finding the individual and group graphs useful. Most importantly, they found the group graph useful as evidenced by their self-report and high use statistics. One of the most common responses to why the group graph was useful was that it often brought attention to important points the individual student might have missed. Consistent with this notion there was little overlap in the time points participants selected when using the individual and group graphs. Lastly, they also found the annotation tool easy to learn, indeed most did not think further training with the tool would have changed their use of it (i.e., they had mastered the tool in a short period of time). Taken together these results are promising and demonstrate the potential utility of the tool in a simulated educational context.

4.1 Improvements to the CLAS

The user experience study also revealed some notable avenues through which the CLAS can be improved. One of the most consistent comments was that the annotation lagged the important point in time. This is necessarily true given the lag between recognizing that a point is important and pressing the annotation button. For example, the process leading to an annotation will typically involve the student listening to the lecture, realizing that a given piece of information is important, and finally annotating that point via a button press. As a result, the annotation will mark a time after the initiation of the “important” utterance in the lecture and when a student clicks on that point in the individual or group graph it will take them to the “end” of the important point in the lecture (rather than the beginning). In future versions of the CLAS, a button that jumps the video back a given amount of time will be introduced. Pilot testing has identified the lag between the start of an important point and the time of the annotation to be about 2 seconds on average. Students can use this “jump back” option to temporally align their point-based annotation with the beginning of the utterance. This will provide for a more precise capture of the point-based annotation and specific lecture video time stamp.

A number of students mentioned wanting to be able to take notes. As described above the kind of point-based annotations used here have numerous advantages over full note annotations. That said, as we noted above in reference to other annotation systems (e.g., MRAS); the concepts underlying the CLAS could conceivably be integrated into an annotation system with additional functionality (i.e., both full note and point-based annotation options could be made available). For example, in a current iteration of the CLAS students can mark segments of the lecture and input temporally aligned text-based annotations. These time segments are then amalgamated (not the text) to form the group graph. This version of the CLAS is designed to take advantage of the strengths of both text-based annotation and point-based annotations. User testing of this implementation is currently ongoing.

Lastly, a number of students made comments about the user interface. While this was a prototype and little time was spent on aesthetics, a number of individuals noted the small size of the video and graphs. The size of the individual components derives, of course, from the need to represent each of these components on the same page. Future versions will attempt to address this issue, for example, by including a full screen button for the lecture video. It is important to note that the noted limitations of the current version of the CLAS, dealt largely with tangential issues (e.g., extra functionality) and were not directed at the core theoretical concept of the CLAS (i.e., a point-based collaborative annotation tool based on common ground and postannotation data amalgamation and display).

4.2 Future Directions and Limitations

There exist a number of future directions that can both address limitations with the present study and extend our understanding of the CLAS in educational contexts. For example, the present study was limited to a single study-test episode. The majority of courses will feature multiple lectures and extend over a longer time frame. Future assessment of the utility of the CLAS will need to involve longer running classes (e.g., multiple lectures) and multiple review episodes. In addition, a consideration of individual differences in motivation and learning style [ 40], [ 41] and a more detailed assessment of the various dimensions of the user experience [ 38], [ 39] could provide critical insights into how students will engage with the CLAS. One challenge that always threatens the validity of a user experience survey is demand effects. Specifically, users might explicitly or implicitly want to make the researchers happy and as a result provide positive feedback (e.g., [ 42]). While this was not true in general here (i.e., negative feedback was provided as well), future work can use more indirect measures (e.g., unscheduled adoption in an online class) as a means to assess user satisfaction.

The current user-based study focused on students (i.e., the primary user) and not instructors. Future work focusing on the instructor-based user experience could provide important insight into the tool. Besides the technological development discussed above, the potential efficacy of the CLAS system needs to be assessed. For example, systematic manipulation of the various features of the CLAS system (e.g., restricting students to the individual graph versus both the individual and group graphs) during review could highlight their relative contribution to test performance. Evaluations of usability often precede evaluation of educational impact given the latter will depend strongly on the former. That said, we have not demonstrated that the CLAS will improve educational outcomes. Future work will address this question directly.

Conclusion

The authors have described the development and a user experience test of a new tool for point-based collaborative lecture annotation. The results of the user study were positive and encouraging for future development of the CLAS.

References

• 1. G.O. Einstein, J. Morris, and S. Smith, “Note-Taking, Individual Differences, and Memory for Lecture Information,” J. Educational Psychology, vol. 77, pp. 522-532, 1985.
• 2. K.A. Kiewra, S.L. Benton, S. Kim, N. Risch, and M. Christensen, “Effects of Note-Taking Format and Study Technique on Recall and Relational Performance,” Contemporary Educational Psychology, vol. 20, pp. 172-187, 1995.
• 3. R. Farzan, and P. Brusilovsky, “AnnotatEd: A Social Navigation and Annotation Service for Web-Based Educational Resources,” New Rev. in Hypermedia and Multimedia, vol. 14, pp. 3-32, 2008.
• 4. M. Ketterl, J. Emden, and O. Vornberger, “Using Social Navigation for Multimedia Content Suggestion,” Proc. IEEE Fourth Int'l Conf. Semantic Computing, 2010.
• 5. R. Mertens, R. Farzan, and P. Brusilovsky, “Social Navigation in Web Lectures,” Proc. 17th ACM Conf. Hypertext and Hypermedia, U.K. Wiil, P.J. Nürnberg, and J. Rubart, eds., pp. 41-44, 2006.
• 6. N.K. Speer, K.M. Swallow, and J.M. Zacks, “Activation of Human Motion Processing Areas during Event Perception,” Cognitive Affective Behavioral Neuroscience, vol. 3, pp. 335-345, 2003.
• 7. D. Bargeron, A. Gupta, J. Grudin, and E. Sanocki, “Annotations for Streaming Video on Web: System Design and Usage Studies,” Proc. Eighth Int'l Conf. World Wide Web (WWW), 1999.
• 8. D. Bargeron, J. Grudin, A. Gupta, E. Sanocki, F. Li, and S. Leetiernan, “Asynchronous Collaboration around Multimedia Applied to On-Demand Education,” J. Management Information Systems, vol. 18, pp. 117-145, 2002.
• 9. A.J. Brush, D. Bargeron, J. Grudin, A. Borning, and A. Gupta, “Supporting Interaction Outside of Class: Anchored Discussion vs. Discussion Board,” Proc. Conf. Computer Support for Collaborative Learning (CSCL): Foundations for a CSCL Community, pp. 425-434, 2002.
• 10. B. Hosack, C. Miller, and D. Ernst, “VideoANT: Extending Video Beyond Content Delivery through Annotation,” Proc. World Conf. E-Learning in Corporate, Govt., Healthcare, and Higher Education, pp. 1654-1658, 2009.
• 11. S. LeeTiernan, and J. Grudin, “Fostering Engagement in Asynchronous Learning through Collaborative Multimedia Annotation,” Proc. INTERACT, pp. 472-479, 2001.
• 12. R. Mertens, H. Schneider, O. Müller, and O. Vornberger, “Hypermedia Navigation Concepts for Lecture Recordings,” Proc. World Conf. ELearning in Corporate, Govt., Healthcare, and Higher Education (E-Learn '04), 2004.
• 13. X. Mu, “Towards Effective Video Annotation: An Approach to Automatically Link Notes with Video Content,” Computers Education, vol. 55, pp. 1752-1763, 2010.
• 14. R. Schroeter, J. Hunter, and D. Kosovic, “Vannotea - A Collaborative Video Indexing, Annotation and Discussion System for Broadband Networks,” Proc. Knowledge Markup and Semantic Annotation Workshop (K-CAP '03), 2003.
• 15. J. Grudin, and D. Bargeron, “Multimedia Annotation: An Unsuccessful Tool Becomes a Successful Framework,” Communication and Collaboration Support Systems, K. Okada, T. Hoshi, and T. Inoue, eds., IOS, 2005.
• 16. F.A. Mayadas, J. Bourne, and P. Bacsich, “Online Education Today,” Science, vol. 323, pp. 85-89, 2009.
• 17. S. Chandra, “Experiences in Personal Lecture Video Capture,” IEEE Trans. Learning Technologies, vol. 4, no. 3, pp. 261-274, July-Sept. 2011.
• 18. C. Hermann, and T. Ottmann, “Electures-Wiki - Towards Engaging Students to Actively Work with Lecture Recordings,” IEEE Trans. Learning Technologies, vol. 4, no. 4, pp. 315-326, Oct.-Dec. 2011.
• 19. A. Clark, Natural-Born Cyborgs: Why Minds and Technologies Are Made to Merge. Oxford Univ., 2003.
• 20. D. Goldberg, D. Nichols, B.M. Oki, and D. Terry, “Using Collaborative Filtering to Weave an Information Tapestry,” Comm. ACM, vol. 35, no. 12, pp. 61-70, 1992.
• 21. M.K. Smith, W.B. Wood, W.K. Adams, C. Wieman, J.L. Knight, N. Guild, and T.T. Su, “Why Peer Discussion Improves Student Performance on In-Class Concept Questions,” Science, vol. 323, pp. 122-124, 2009.
• 22. D. Newtson, “Attribution and the Unit of Perception of Ongoingbehavior,” J. Personality Social Psychology, vol. 28, pp. 28-38, 1973.
• 23. G. Stahl, T. Koschmann, and D. Suthers, “Computer-Supported Collaborative Learning: A Historical Perspective,” Cambridge Handbook of the Learning Sciences, R.K. Sawyer, ed., Cambridge Univ., 2006.
• 24. M.J. Weal, D. Michaelides, K.R. Page, D.C. De Roure, E. Monger, and M. Gobbi, “Semantic Annotation of Ubiquitous Learning Environments,” IEEE Trans. Learning Technologies, vol. 5, no. 2, pp. 143-156, Apr.-June 2012.
• 25. J. Steimle, O. Brdiczka, and M. Muhlhauser, “CoScribe: Integrating Paper and Digital Documents for Collaborative Knowledge Work,” IEEE Trans. Learning Technologies, vol. 2, no. 3, pp. 174-188, July-Sept. 2009.
• 26. H. Sack, and J. Waitelonis, “Integrating Social Tagging and Document Annotation for Content-Based Search in Multimedia Data,” Proc. First Semantic Authoring and Annotation Workshop (SAAW '06), 2006.
• 27. H. Clark, Using Language. Cambridge Univ., 1996.
• 28. H. Clark, R. Schreuder, and S. Buttrick, “Common Ground at the Understanding of Demonstrative Reference,” J. Verbal Learning and Verbal Behavior, vol. 22, pp. 245-258, 1983.
• 29. K.A. Kiewra, S.L. Benton, S. Kim, N. Risch, and M. Christensen, “Effects of Note-Taking Format and Study Technique on Recall and Relational Performance,” Contemporary Educational Psychology, vol. 20, pp. 172-187, 1995.
• 30. A. Piolat, T. Olive, and R.T. Kellog, “Cognitive Effort during Note Taking,” Applied Cognitive Psychology, vol. 19, pp. 291-312, 2005.
• 31. M.C. Mozer, H. Pashler, and H. Homaei, “Optimal Predictions in Everyday Cognition: The Wisdom of Individuals or Crowds?” Cognitive Science, vol. 32, pp. 1133-1147, 2008.
• 32. J. Surowiecki, The Wisdom of Crowds: Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations. Random House, 2004.
• 33. J.Y. Lin, A. Pype, A.M.O. Scott, and G.M. Boynton, “Enhanced Memory for Scenes Presented at Behaviorally Relevant Points in Time,” PLoS Biology, vol. 8, p. e1000337, 2010.
• 34. D.L. Butler, and P.H. Winne, “Feedback and Self-Regulated Learning: A Theoretical Synthesis,” Rev. Educational Research, vol. 65, pp. 245-281, 1995.
• 35. P.R. Pintrich, and E.V. De Groot, “Motivational and Self-Regulated Learning Components of Classroom Academic Performance,” J. Educational Psychology, vol. 82, pp. 33-40, 1990.
• 36. L. Macfadyen, and S. Dawson, “Mining LMS Data to Develop an ‘Early Warning System’ for Educators: A Proof of Concept,” Computers & Education, vol. 54, pp. 588-599, 2010.
• 37. S. Dawson, “‘Seeing’ the Learning Community: An Exploration of the Development of a Resource for Monitoring Online Student Networking,” British J. Educational Technology, vol. 41, pp. 736-752, 2010.
• 38. F.D. Davis, “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Quarterly, vol. 13, pp. 319-339, 1989.
• 39. S. Petter, W. Delone, and E. McLean, “Measuring Information Systems Success: Models, Dimensions, Measures, and Interrelationships,” European J. Information Systems, vol. 17, pp. 236-263, 2008.
• 40. T.G. Duncan, and W.J. McKeachie, “The Making of the Motivated Strategies for Learning Questionnaire,” Educational Psychology, vol. 40, pp. 117-128, 2005.
• 41. P.R. Pintrich, D.A.F. Smith, T. Garcia, and W.J. McKeachie, “Reliability and Predictive Validity of the Motivated Strategies for Learning Questionnaire (MSLQ),” Educational Psychology Measure, vol. 53, pp. 801-813, 1993.
• 42. M.T. Orne, “On the Social Psychology of the Psychological Experiment: With Particular Reference to Demand Characteristics and Their Implications,” Am. Psychology, vol. 17, pp. 776-783, 1962.