# Context-Aware Recommender Systems for Learning: A Survey and Future Challenges

Katrien Verbert, IEEE
Nikos Manouselis, IEEE
Xavier Ochoa
Martin Wolpers
Hendrik Drachsler
Ivana Bosnic, IEEE
Erik Duval, IEEE

Pages: pp. 318-335

Abstract—Recommender systems have been researched extensively by the Technology Enhanced Learning (TEL) community during the last decade. By identifying suitable resources from a potentially overwhelming variety of choices, such systems offer a promising approach to facilitate both learning and teaching tasks. As learning is taking place in extremely diverse and rich environments, the incorporation of contextual information about the user in the recommendation process has attracted major interest. Such contextualization is researched as a paradigm for building intelligent systems that can better predict and anticipate the needs of users, and act more efficiently in response to their behavior. In this paper, we try to assess the degree to which current work in TEL recommender systems has achieved this, as well as outline areas in which further work is needed. First, we present a context framework that identifies relevant context dimensions for TEL applications. Then, we present an analysis of existing TEL recommender systems along these dimensions. Finally, based on our survey results, we outline topics on which further research is needed.

Index Terms—Adaptive and intelligent educational systems, personalized e-learning, system applications and experience

## Introduction

Recommender systems have been researched and deployed extensively over the last decade in various application areas, including e-commerce and e-health. Several recommendation algorithms, such as content-based filtering [ 81], collaborative filtering [ 49], knowledge-based filtering [ 17] and their hybridizations [ 18], are widely discussed in the literature and in several surveys of the state-of-the-art [ 2].

Also in the Technology Enhanced Learning (TEL) domain, the deployment of recommender systems has attracted increased interest during the past years. This interest is reflected in the growth of dedicated workshops and special issues on Social Information Retrieval [ 114], [ 36] and Recommender Systems [ 69] for TEL and is well justified. Whereas Google and other search engines are bound to have a higher recall as they index most of what is available on the Web, their precision for learning is low [ 34]. It is difficult to express a specific learning requirement through keywords. For example, search engines do a poor job when a learner needs content about “relativity theory,” oriented to high school level, with a duration of about 30 minutes. Finding relevant resources can be even more difficult when requirements are not always fully known by the learner, such as her level of competence or adequate technical format.

Recommender systems for learning try to address these challenges—i.e., they attempt to filter content for different learning settings. A recent survey of recommender systems in TEL has been elaborated by Manouselis et al. [ 70]. The authors presented an extensive overview of TEL recommender systems. In addition, evaluation perspectives on current research in this area and future challenges with respect to the evaluation of TEL recommenders were discussed.

The notion of context has started to attract significant attention in this research, as indicated by contributions to a recent special issue on Context-Aware Recommender Systems (CARS) [ 112]. Among others, advancements of network and mobile services and the growing tool and device landscape provide many new opportunities for the TEL domain to adjust itself to this landscape appropriately [ 102]. A new set of recommender systems for learning has been developed in recent years to demonstrate the potential of contextual recommendation.

From an operational perspective, context is often defined as an aggregate of various categories that describe the setting in which a recommender is deployed, such as the location, current activity, and available time of the learner. A first example of a context-aware recommender system for learning considers the location of the user and the noise level at this location as a basis to suggest learning resources [ 25]. If the learner is in a cafeteria, the noise level associated to this location might have an impact on her level of concentration and likelihood of interruption. Therefore, a contextual recommender would in such a context suggest learning activities to assess her knowledge on previously learned topics, e.g., through simple questionnaires that she can resume easily at a later point in time. A second example considers proximity between learners to support collaborative learning [ 65]. If a contextual recommender is able to detect people nearby who are working on similar learning activities, the system can suggest suitable peer learners to collaborate with. A third example also takes into account the device that the learner is currently using [ 112]. For example, if a learner is being recommended material to study the theory of relativity while commuting from work to school using a smartphone, short, audiovisual material that fits the screen of the smartphone should be more relevant than a long, text-only document. Accordingly, the recommender system should rank the short videos higher than the long documents. This reranking of the recommended resources is not possible if the system does not know about the context of the user.

In this perspective, new challenges emerge for capturing and understanding context and exploiting such information for creating intelligent recommendations adapted to current learner needs, without her being necessarily aware of the fact that such contextual variables (e.g., the noise level) are measured and taken into consideration. In this paper, we try to particularly investigate the way in which contextual information may be used by TEL recommender systems, as well as to assess the current maturity of work in this area. The research contribution of this paper is threefold.

1. First, we present a context framework for contextual recommendation in TEL. This framework identifies relevant context dimensions for TEL applications.
2. Second, the framework is used to drive an in-depth analysis of context-aware TEL recommenders.
3. Finally, based on an analysis of existing context-aware recommender systems, we outline directions of future research in this area.

In contrast to earlier surveys on recommender systems for learning [ 70], this paper focuses specifically on a particular set of recommender systems that incorporate contextual information in the recommendation process. As outlined by the context-aware recommendation community, much work is needed to advance this field [ 3]. We discuss the challenges that this community has identified in recent years, as well as how these challenges can be tackled for the TEL field.

The article is organized as follows: first, we present relevant background that situates this work within existing research on context-aware recommender systems. We also discuss particularities of the TEL application domain in this field. Then, we present a context framework for TEL that is used to drive the analysis of recommender systems. An analysis of existing context-aware recommender systems for learning along the dimensions of our context framework is presented in Section 4. Finally, we present future challenges for the development and evaluation of context-aware recommender systems for learning that are able to generate recommendations adapted to the current contextual needs of the user.

## Background

### 2.1 TEL and Recommendation

Technology enhanced learning aims to design, develop, and test sociotechnical innovations that will support and enhance learning practices of both individuals and organizations [ 70]. It is therefore a domain that generally covers technologies to support teaching and learning activities, including recommendation technologies that facilitate retrieval of relevant learning resources. In this section, we discuss particularities of the TEL domain for recommendation and existing work in this area.

#### 2.1.1 Particularities of TEL for Recommendation

Recommender systems are an extensively studied and well established field of research and application [ 2]. Major search engines like Google and electronic shops like Amazon have incorporated recommendation technology in their services in order to personalize their results. Unfortunately, the algorithms underlying regular recommender systems are not directly transferable to the area of TEL. The TEL area offers some specific characteristics that are not met by today's general purpose recommendation approaches [ 34].

The main difference is, of course, that each learner uses her own tools, methods, paths, collaborations, and processes. Consequently, guidance within the learning process must be personalized to an extreme extent. For example, rather than recommending resources that other users with similar interests have used, the recommendation must also respect the actual learning situation of the learner, including the learning history, environment, timing, and accessible resources.

Learning situations become even more complex due to the fact that pedagogical approaches differentiate between formal and informal learning processes. Both have different requirements for the learning environment and, as such, for the recommendation within the environment. Often, it is not possible to draw a clear line between formal and informal learning scenarios. For example, recommender systems need to deal with the tension of recommendations for activities liked by the learner and those required by the teacher [ 106]. Consequently, the need for massive amounts of data about the user and her activities within all of her learning environments is necessary to facilitate precise recommendations.

This leads to the problem of usage data availability. Many recommendation algorithms rely on massive amounts of usage data from numerous users to make precise recommendations. In TEL, this situation often does not occur. Instead, many learning activities take place with only a few learners participating. This situation occurs in both formal and informal learning settings, e.g., in courses [ 1] or in learning networks [ 35] that assemble and dissolve continuously.

At least some tailoring of the respective approach is necessary. In this paper, we outline how such tailoring is possible by taking the context of the learner into account in a much more specific way than applied in today's recommendation approaches. First, we present a brief overview of existing TEL recommender systems that rely on traditional recommendation algorithms. Then, we discuss context definitions and existing research on the incorporation of contextual information in the recommendation process.

#### 2.1.2 TEL Recommender Systems

Recommender systems in TEL are quite diverse. A recent survey of recommender systems in TEL has been presented by Manouselis et al. [ 70]. Most systems suggest learning resources [ 106] and/or people [ 43], [ 85] who can help with a learning activity. Course recommenders [ 42] typically provide advice to learners on courses to enroll in. Several social navigation systems [ 13], [ 15] rely on recommendation techniques to suggest resource sequences. While not traditionally considered as recommender systems, Intelligent Tutoring Systems (ITSs) [ 58] use information about the learner to suggest personalized hints while she is solving a problem.

Recommender systems usually rely on collaborative filtering, content-based filtering, knowledge-based filtering, or hybrid recommendation algorithms. A discussion of the advantages and drawbacks of the various techniques for TEL has been presented in [ 34]. These algorithms use information about users and resources to generate recommendations. Interestingly, most TEL recommender systems rely on profiles of learners or teachers that describe additional information as opposed to interests or preferences only. The knowledge level of the learner is often used to personalize recommendations, such as her knowledge of course concepts [ 21] or past academic grades [ 42]. Learning styles are also considered by some recommender systems in TEL [ 67], [ 92], often based on the Felder-Silverman [ 40] inventory.

Similarly, many systems rely on resource profiles that describe multiple attributes of resources. In addition to general characteristics like keywords, title, and author, many systems use educational metadata that describe for instance the difficulty level of a resource. This difficulty level is then used to recommend resources according to the knowledge level of the learner. In addition, the interactivity level, resource type (e.g., question, definition, or case study) and intended end-user role of the resource are often considered. These attributes are used to suggest different types of resources that are appropriate for the learning activity. Technical characteristics like the format of resources as well as annotation metadata that capture comments from users are also used. CoFIND [ 13] for instance relies on metadata that describe what users value in a resource. Examples defined by the authors are “detailed” or “simple to understand.”

Although multiple attributes are often considered, additional context dimensions (such as the current location, current activity or device) are not incorporated in most of these systems. In this paper, we investigate the way in which such additional contextual information may be used by TEL recommender systems.

### 2.2 Context in TEL

One of the most cited definitions of context is the definition of Dey et al. [ 29] that defines context as “any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves.” This definition is referenced extensively within various application domains, including by researchers in TEL [ 97], [ 38]. Dourish [ 32] suggests that context has a dual origin: 1) technical and 2) social science based. From a social perspective, Dourish argues that context is not something that describes a setting or situation, but rather a feature of interaction. Researchers in TEL have argued that this user-centered emphasis on factors affecting an activity is precisely what makes this notion of context meaningful for learning [ 121], [ 9]. From a technical perspective, there is a need to define context in a more specific way as an operational term [ 120]. To operationalize context, researchers have attempted to define context by enumerating categories. Early work of Schilit et al. [ 94], [ 29] divided context in three categories.

• Computing context, such as network connectivity, communication costs, and communication bandwidth, and nearby resources such as printers, displays, and workstations.
• User context, such as the profile of the user, location, people nearby, and social situation.
• Physical context, such as lighting, noise levels, traffic conditions, and temperature.

Chen and Kotz [ 22] added time as a fourth context category. Schmidt et al. [ 98] added a task category and define the following dimensions: the user, the social environment of the user, tasks, location, infrastructure, physical conditions, and time. Zimmermann et al. [ 129] list individuality, activity, location, time, and relations as fundamental context categories. Individuality is subdivided into four elements: natural entity, human entity, artificial entity, and group entity. This definition is one of the most comprehensive context definitions to date.

In TEL, such enumerations have also been proposed as an attempt to define the context of the learner or teacher as an operational term. Many enumerations are defined for mobile learning applications. Berri et al. [ 12] distinguish between technical and learner context elements. The first category deals with the technical aspects of mobile devices and their operational environment, including the capacity and bandwidth of the wireless network, and input (i.e., keypad) and output (i.e., small screen) constraints. The second category defines learner context elements, including aims and objectives of the learner, prerequisites, background, current level of understanding and subject domain. Beale and Lonsdale [ 9] argue that it is essential to capture interactions between the environment, the user, their tasks, and other users. Environment constitutes computing, time and physical context characteristics. Wang [ 115] defines six dimensions: identity, time, facility, activity, location, and community.

Derntl and Hummel [ 28] define world context that constitutes location and date and time, physical context (persons, books, journals, learning equipment), digital context (e-papers, e-collaborations, e-learning services), device context (hardware, software, connectivity), and learner information context (personal information such as name, expertise and interests and task specific information). The learner information dimension therefore covers both activity and user dimensions of other definitions. Li et al. [ 64] define five context dimensions: who (user), what (object), how (activities), where (location), and when (time). Many other definitions typically list three or four context dimensions. Further examples include definitions of Schmidt [ 97], Tankelevic iene and Damas evic ius [ 107], Kurti [ 60], and Hu and Moore [ 51].

Fig. 1 presents how these definitions relate to each other. The columns are derived from elements that researchers have tried to classify as context categories and the rows show the different context definitions. Whereas the terminology and scope of elements differ significantly, there is also a lot of overlap between existing context definitions. The first four definitions from Shilit et al. [ 94], Chen and Kotz [ 22], Schmidt et al. [ 98], and Zimmermann et al. [ 129] are generic definitions of context that have been used in many application domains. The other definitions have been proposed by researchers in TEL. Most of these definitions refer to similar context categories as defined by generic definitions: location, time, computing, user, activity, and social relations. Physical conditions, such as lighting and noise level, are defined less often in a TEL context. In contrast, resource context (including both physical and digital resources that are relevant to the user) is used more often within TEL applications.

Figure    Fig. 1. Context definitions.

Although the scope of context categories differs (i.e., some definitions specify only a few categories that cover several elements whereas other definitions attempt to define context categories more precisely), there are many similarities between the context definitions. The alignment in Fig. 1 presents how the existing context definitions relate to each. Such an alignment can be used to identify the meaning of context elements across different context-aware TEL applications and can be used to define mappings between contextual data representations. In Section 3, we present a context framework for learning that unifies the various definitions and that attempts to define context categories and data elements within these categories in a uniform and precise way.

### 2.3 Context-Aware Recommender Systems

Traditionally, collaborative, content-based, knowledge-based, and hybrid recommender systems deal with two types of entities, users and items. As discussed in Section 2.1.1, TEL applications have inherent additional complexities and may not fit well into the traditional 2D user/item approach based on ratings only. Of interest for TEL recommenders is the incorporation of additional information about learners and teachers and their context in the recommendation process. Such data can be used to adapt recommendations based on individual learner characteristics, such as learning goals and knowledge levels, and additional contextual information such as available time, location, people nearby, etc.

Pioneering work on context-aware recommender systems has been done by Adomavicius and Tuzhilin [ 3]. The authors researched approaches where the traditional user/item paradigm was extended to support additional dimensions capturing the context in which recommendations are made. Such contextual information can be obtained in a number of ways.

• Explicit context capturing relies on manual input from users. Registration modules are often used to capture information of users or rating modules are used to retrieve interests and preferences.
• Implicit methods capture contextual information automatically from the environment, for instance by obtaining the current location or device type.
• Contextual information can also be inferred by analyzing user interactions with tools and resources, for instance to estimate the current task of the user.

Different paradigms have been proposed to incorporate contextual information in the recommendation process. A first recommendation via context-driven querying and search approach uses contextual information to query or search a certain repository of resources (e.g., restaurants) and presents the best matching resources (e.g., nearby restaurants that are currently open) to the user. A second contextual preference elicitation and estimation approach is a more recent trend in context-aware recommender system research. This approach attempts to model and learn contextual user preferences. These recommender systems are built on knowledge of partial contextual user preferences and typically deal with data records of the form ${<}$ user, item, context, rating $>$ . Each record therefore captures how much a user liked a particular item in a specific context (e.g., weekend or weekday).

Three approaches have been identified to deal with such contextual preferences. In a contextual prefiltering approach, contextual information is used to filter the data set before applying a traditional recommendation algorithm. In a contextual postfiltering approach, recommendations are generated on the entire data set. The resulting set of recommendations is adjusted using the contextual information. Contextual modeling approaches use contextual information directly in the recommendation function as an explicit predictor of a rating for an item. Whereas contextual prefiltering and postfiltering approaches can use traditional recommendation algorithms, the contextual modeling approach uses multidimensional recommendation algorithms. Examples of heuristic-based and model-based approaches have been described in [ 3].

Several contextual recommender systems have been developed that use these paradigms in various application domains. Examples include context-aware recommender systems that suggest gas stations to a driver of a car [ 7], contextualized media delivery systems [ 124], [ 78] and intelligent tourist guides [ 3]. For example, COMPASS [ 110] is a recommender system that uses a context-driven querying and search approach to provide a tourist with information about nearby monuments, hotels and people. In an evaluation experiment, time and location were used to contextualize recommendations. Interestingly, the authors report that “last time visited” had a negative influence on the perceived usefulness of the system. These results illustrate that careful analysis of data that is taken into account is necessary when deploying contextualization algorithms.

The influence of various parameters on the recommendation process is therefore currently of major interest. This challenge has been identified by several authors. Yujie and Licai [ 125] outline that it is difficult to describe clearly and uniformly what types of contexts are truly needed in CARS because of the variety of application scenarios and user needs. Discovering valid context types and instances and then implementing them are therefore serious challenges that CARS should face and resolve. Adomavicius and Tuzhilin [ 3] identify the development of high-performing context-aware recommender systems and testing them on practical applications as an important challenge. They argue that most work on context-aware recommender systems has been conceptual, where a certain method has been developed, tested on some (often limited) data, and shown to perform well in comparison to certain benchmarks. Among others, they argue that there has been little work done on developing novel data structures and new system architectures for CARS that incorporate context sensors and various filters and converters in a modular fashion. A third important challenge is the evaluation and lack of publicly available data sets [ 3], [ 125]. In order to assess the impact of various contextual parameters, data sets are needed that contain contextual data. The interest in this area is reflected by the organization of several workshops and challenges related to, among others, contextual movie recommendation that have been organized in recent years [ 4].

Overall, the field of context-aware recommender systems is promising, but much work is needed to explore it comprehensively. In this paper, we explore the challenges outlined above for the development of CARS for learning. As a first step in this direction, we define a context framework for TEL that identifies relevant context categories for the analysis and development of CARS for learning. Then, we present an analysis of CARS that have been deployed in recent years within the TEL domain. In particular, we focus on context dimensions that are used to contextualize recommendations, context sensors that capture contextual information, contextual recommendation algorithms and evaluation methods and results.

## A Context Framework for TEL

The need to define and model context more precisely and in a consistent way has been identified by several researchers [ 9], [ 96]. A precise definition and model of context can facilitate the identification of what does and does not constitute context and can enable reuse and exchange of contextual data across applications.

To this extent, we introduce a simple classification of context information that is relevant to context-aware applications in TEL. This classification was constructed by the analysis and integration of existing context definitions. The categories and their context elements are detailed in this section. The classification is by no means exhaustive, but attempts to define context elements in a precise way. We also identify relevant standards and specifications that can be used to represent data elements within these categories in a uniform way. The classification is used in the remainder of this paper to drive the analysis of context-aware recommender systems for TEL.

### 3.1 Computing

Computing context has been researched extensively by the pervasive and mobile research community [ 61], including by researchers in the mobile learning area [ 99], [ 12]. Computing characteristics can be classified in three areas.

• Network includes static and dynamic properties of the network, such as maximum and available bandwidth.
• Hardware comprises input and output capabilities of devices, storage or CPU capabilities, etc.
• Software describes whether the delivery context supports certain APIs, document formats, operating systems, application-layer protocols, etc.

The acquisition and use of computing context is necessary to support intelligent interfaces that can for instance select suitable learning resources for the device that is used. The prevalent standards for describing computing characteristics are W3C Composite Capabilities/Preferences Profile (CC/PP) [ 55], User Agent Profile (UAprofile) [ 116], developed by OMA, and the Usage Environment Description (UED) standard which was standardized within MPEG-21 Digital Item Adaptation [ 113]. CC/PP defines only a basic structure of components and attributes without specifying a particular vocabulary of terms [ 109]. UAProfile adopts CC/PP and provides a concrete vocabulary mainly targeting mobile devices. UED defines both the structure and a comprehensive vocabulary. Device profiles can also be identified using the open source Wireless Universal Resource FiLe project (WURFL) [ 79]. WURFL is a community effort that collects a wide range of device descriptions.

### 3.2 Location

In addition to computing context, location context has dominated research on context-aware mobile computing to a large extent [ 98]. Location models have been proposed that capture human-readable and geometric information of objects, including persons and devices, and relationships between objects. These include: 1) proximity of objects within a space, 2) communicative ability, and 3) orientation, that can for instance indicate to which display device a user is looking [ 19].

Location contexts that are often referenced by learning applications include classroom, home, and outdoor [ 115] and several variations on these elements [ 127]. Some learning applications use more accurate resolution within these categories through a locating sensor, such as Global Positioning System (GPS) or Wi-Fi. Such quantitative models refer to coordinates with two or three dimensions [ 129]. Several standards and specifications have been elaborated to facilitate the exchange of location data. Endeavors in this area are among other driven by the Open GIS Consortium and ISO/TC21 [ 101], [ 6]. To our knowledge, their use in TEL applications is limited.

### 3.3 Time

Time context includes date and time information or, less specific, information about the week, month, or semester of the year. Time is often used in conjunction with other pieces of context, either as a timestamp or as time span, indicating an instant or period during which other contextual information is known or relevant [ 29].

### 3.4 Physical Conditions

Physical context describes the environmental conditions where the system or user is situated, and commonly includes measures for heat, light, and sound [ 83]. Whereas physical context has been researched extensively in home automation research, its use by TEL applications is still limited. In learning scenarios, lighting and noise are sometimes considered as important criteria [ 73].

### 3.5 Activity

Activity context reflects the tasks, objectives or actions of the user. Examples of models that have been elaborated in a TEL context are the Contextualized Attention Metadata (CAM) [ 122] and the UICO [ 84] models. Both models enable the capturing of actions of the user, which are comprised of events within an application, and session and time related data. Such data is then analyzed to infer information about the current task, objective or topic of interest of the user, for instance by using domain ontologies that describe the subject domain [ 10]. The models provide elements to capture additional contextual data, but do not define specific types of such data.

### 3.6 Resource

Resource context captures relevant characteristics of physical or virtual resources. The IEEE Learning Object Metadata (LOM) standard [ 62] is a standard for the description of learning resources. The elements are organized into several categories, including a general description of the resource, technical requirements and characteristics, educational characteristics, relations between resources, and annotation comments on the educational use of the resources. Other standards that are often used for the description of learning resources are the Dublin Core Metadata Element Set [ 118] and MPEG-7 [ 71].

### 3.7 User

Learner models have been researched extensively in the educational adaptive hypermedia and the educational user modeling research areas. In this section, we summarize briefly the main learner characteristics that have been proposed by Brusilovsky and Millán [ 16], Specht [ 103], and Nguyen and Do [ 74]. Relevant standards and specifications for partial representations of learner data are IMS LIP, IMS ePortfolio, IMS Enterprise, IEEE RCD, FOAF, and HR-XML. A detailed discussion of data elements available in these standards is beyond the scope of this paper. Interesting work in this area has been presented by Dolog and Nejdl [ 31].

#### 3.7.1 Basic Personal Information

Basic personal information typically includes identification information, name, contact information, affiliations, authentication information, information on accessibility, including language capabilities and disabilities, and other personal characteristics such as gender, age, profession, and educational level.

#### 3.7.2 Knowledge/Performance

The knowledge category represents prior knowledge levels of the learner [ 16]. Other researchers categorize this information under a performance nominator that stores information about measured performance of a learner through learning material [ 31].

#### 3.7.3 Interests

Learner interests capture interests or preferences of learners and are key characteristics to support personalization [ 74]. Values that are typically stored include search terms of the user, her tags, comments and resources she created, read, or rated.

#### 3.7.4 Learning Goals

The distinction is often made between short-term goals, where a learner intends to solve a certain problem, and long-term goals that are related to a course or plans for life-long learning. Goal hierarchies have been proposed that decompose higher level goals in subgoals [ 16].

#### 3.7.5 Learning and Cognitive Styles

Learners differ in their preferred way of learning presentation and cognitive processing. Examples for considering different cognitive styles are visual, textual, or auditory presentation of information. Different learning styles include the presentation of examples, presentation of theoretical knowledge, and practical exercises [ 26]. Among others, researchers in TEL often refer to the Felder-Silverman [ 40] and Honey and Mumford [ 50] inventory of learning styles that describes learning styles along several dimensions. An interesting analysis of learning style classifications has been presented in [ 53].

#### 3.7.6 Affects

The modeling and use of affective information is a popular research topic in various research areas. Researchers widely refer to Russell's [ 91] two-dimension “circumplex model of affect,” where emotions are seen as combinations of arousal and valence. The OCC [ 77] model is also referenced widely. This model specifies 22 emotion categories. In TEL, research on the influence of emotions on learning has also gained major interest in recent years. Some researchers have used Russell's model [ 100] as a basis to use emotional data to support learning. The work of the Affective Computing Group at MIT Media Lab [ 59] is also very prominent in this research area.

#### 3.7.7 Background

The background of the user is a common name for a set of features related to her previous experience outside the core domain of a specific system [ 16]. Elements typically modeled include experience of work in related areas, religion, and cultural characteristics.

### 3.8 Social Relations

Social relations describe social associations, connections, or affiliations between two or more persons [ 129]. For instance, social relations can contain information about friends, neutrals, enemies, neighbors, coworkers, and relatives. Other researchers identify community [ 56] as an important context dimension. In TEL applications, the distinction is sometimes made between experts, teachers and peer learners [ 52]. The FOAF specification [ 14] is often used to describe such user relations and roles.

### 3.9 Discussion

Fig. 2 summarizes the context categories and their elements that have been described in this section. The categorization is used in Section 4 to identify to which extent current context-aware TEL recommender systems are able to generate recommendations adapted to the current contextual needs of users.

Figure    Fig. 2. Context framework.

Existing work in recommender systems [ 11], [ 3] identifies a subtle difference between various parameters of contextual information. They refer to multidimensional spaces for representing contextual variables, and identify mainly three categories/dimensions within which variables may be grouped (i.e user, item, context). Nevertheless, the challenge is reaching towards an identification of variables, as well as their grouping in categories accepted by all researchers and developers. For instance, one could argue that physical conditions are part of location variables, or at least strongly linked to them. Or that learner models are actually user models that are not related to the context. To this end, reaching a generalization that will incorporate all single- or multidimensional representations in a single model can be considered unrealistic. Rather, one could see as a possible line of future work ways to map/link the various representations among them, so that contextual data may be exchanged among different systems. In Section 5 on future challenges, this is discussed under the prism of a Linked Data approach.

## Survey of Context-Aware TEL Recommenders

### 4.1 Context Dimensions

Fig. 3 summarizes the context elements that are used by the context-aware recommender systems we surveyed. Nine out of 22 systems consider the computing context of the user to suggest suitable learning resources according to hardware, software or network constraints. As illustrated in Fig. 4, Zhao et al. [ 127] rely on WURFL [ 79] descriptions of mobile devices and COLDEX [ 8] uses the CC/PP W3C recommendation [ 55]. The other systems rely on ad hoc representations of the computing context. Relying on existing standards and specifications offers interesting perspectives to extend, exchange and reuse computing related data for new and different application contexts.

Figure    Fig. 3. Context elements used.

Figure    Fig. 4. Overview of context-aware recommender systems.

The location context of the user is taken into account by ten systems. Lehsten et al. [ 63] detect location as a basis to infer whether the learner is attending a lecture. If she is not attending, a stream of the lecture or other relevant learning resources are recommended. Seven systems sense the location of the user and nearby objects to support situated or collaborative learning activities. For instance, Rogers et al. [ 89], and Zhou and Rechert [ 128] use location to suggest learning resources during outdoor learning activities. In MOBIlearn [ 65], location is detected for generating recommendations of both relevant learning resources and peers who are nearby in a museum. PERKAM [ 37] also uses location context to suggest learning resources and peer learners who are nearby. The system visualizes the distance to potential peer helpers.

Another set of context-aware recommender systems uses location context to support language learning. TANGO [ 76] uses nearby objects to suggest learning resources. For example, when the learner enters a meeting room, the system asks her where the remote control of the air conditioner is. If the learner scans the tag labeled on the remote control, the answer is correct. PALLAS [ 82] incorporates time and location dimensions to send automatic notifications to the learner when she is in the vicinity of a point of interest, such as a French art exhibition. In addition, related learning resources are recommended, such as resources related to a building.

Other systems track the location of the user as a basis to take into account physical conditions that are associated with this location—such as CALS [ 123] and TenseITS [ 25]. Noise level is for instance considered as an important criterion to estimate the possible level of concentration. Such physical conditions are used for learning resource recommendations. For instance, if a learner is on a bus where the likelihood of interruption is high, questions may be suggested to assess her knowledge on a previously learned topic that she can easily resume. Weather and lighting are also outlined as relevant physical conditions [ 128], but to our knowledge these elements have not yet been implemented in a prototype.

Ten systems consider the time context of the learner. Timestamp data is in most cases used in conjunction with location information to determine where a user is at a specific point in time [ 63]. Other systems work with interval data to contextualize recommendations according to the available study time of the learner [ 123], [ 25], [ 12], [ 97].

The activity context of the user is used by 15 systems. Many systems use the current task of the learner as a basis to suggest resources that are relevant for this task. 3A [ 38], APOSDLE [ 104], Berri et al. [ 12], C-LINK [ 95], Khribi et al. [ 54], and MOBIlearn [ 65] identify the topic of the activity to adapt recommendations to the current topic of interest. Nine systems capture and use actions of the user, such as a learner scanning an Radio-Frequency IDentification (RFID) tag or a Quick Response (QR) code as a basis to suggest relevant resources. For instance, Teng et al. [ 108] use this approach to bridge paper-based learning with online learning. More specifically, they recommend digital resources related to a text fragment or scaffolded questions the user is reading on paper.

Like traditional User x Item recommender systems for TEL that were described in Section 2.1.2, many systems rely on profiles of resources and users. In addition to general information like author, title, and keywords, both technical and educational characteristics of resources are often described. Also annotation metadata that capture comments of users are used. The IEEE LOM [ 62] standard is used by four systems to describe these elements. DEPTHS [ 52] relies on Dublin Core [ 118]. The learner profile captures in many cases the knowledge level of the learner in addition to her interests and preferences. Two systems consider the learning style of the learner.

Finally, social relations are used by nine recommender systems. DEPTHS [ 52] relies on the assumption that peer helpers who have already a close connection to the learner will be more likely to help a learner with an activity. These relations are represented using the FOAF [ 14] specification and used to suggest peer helpers. Chen and Chao [ 23] developed a system that connects traditional books with a web-based discussion forum. Learners receive messages from an online learning community based on their reading status, which they report through their mobile device. These messages are aimed to increase motivation and include links to additional learning resources. Within the Learning in Process (LIP) project [ 97], a context-aware recommender system was developed to suggest courses, learning resources and peer learners based on roles of users in a corporate environment. The approach is also deployed by APOSDLE [ 104] to support recommendation of learning resources and peers. The 3A recommender [ 38] targets context-aware recommendation in personal learning environments [ 41]. Context is measured and represented by different types of relations, including social relations and relations between resources. We elaborate in the next section how these contextual data are acquired.

### 4.2 Context Acquisition

As presented in Fig. 3, the majority of the recommender systems rely on a combination of explicit, implicit and/or inferred contextual data. We discuss briefly the context sensors that are used.

#### 4.2.1 Computing Context

Computing context is sensed implicitly by the surveyed systems. Information about the device is often transmitted by including the identifier of the device in the HTTP request [ 93]. This identifier is then used to retrieve relevant information about the device from a repository of device profiles. Alternatively, if no information about the device is available in the repository, information such as screen size is sometimes captured in the request header [ 12], [ 127].

#### 4.2.2 Location Context

Location context is also often sensed implicitly through GPS or Wi-Fi location sensors, or a combination of both [ 128]. MOBIlearn [ 65] uses an ultrasound positioning system. The approach is required in contexts such as museums where an accurate description of the location is required to identify the object closest to the user. Other systems that require precise location information often rely on an explicit method that requires a user to scan an RFID tag [ 76] or QR code when entering a room [ 108]. TenseITS [ 25] relies on an explicit approach where the user is asked to manually input the location type, such as home, university or transportation. Whereas such an approach may work to evaluate first prototypes, the acquisition can easily be automated for the location types that are listed by the authors.

#### 4.2.3 Time Context

Timestamp data is in all cases captured implicitly. Time interval data, such as available study time, is entered explicitly in work of Berri et al. [ 12], LIP [ 97], and TenseITS [ 25]. In CALS [ 123], a learning schedule is used that enables users to enter such data in a schedule that they can use to plan their learning activities. Whereas the approach also requires an effort from end-users, data acquisition is incorporated into their workflow.

#### 4.2.4 Activity Context

Many systems rely on explicit user interactions to capture information about the activity context. These interactions can be scanning a QR or RFID tag [ 108] or providing manual text input—such as the current line number during reading [ 23] or the current topic of interest [ 12]. The current task of the learner is in most cases entered manually by the teacher or the learner.

APOSDLE [ 104] and C-LINK [ 95] are examples of systems that infer the activity context from user interactions with tools and resources. For instance, APOSDLE uses a classification algorithm to detect the current task of the learner, based on a task model that is created by experts [ 84]. The algorithm uses among others the window title of the active application to infer information about the current task. In addition, a domain model is used to detect the current topic of resources. C-LINK [ 95] and Khribi et al. [ 54] extract terms from currently visited resources to infer the current topic of interest. The advantage of the approach is that it can be applied to other domains without the need for a domain taxonomy.

#### 4.2.5 Physical Conditions

Physical conditions are captured explicitly by the user or implicitly from the environment. Current prototypes only implement noise level indicators. TenseITS [ 25] relies on manual input to capture such information. CALS [ 123] uses a microphone to automate such acquisition.

#### 4.2.6 Resource Context

Descriptions of resources are often added manually by authors or experts. Such approaches are only suitable for systems that work on a closed corpus of resources. Other systems such as MOBIlearn [ 65] and C-LINK [ 95] use annotations of other users as a basis to capture relevant information about resources. C-LINK also automates metadata generation, e.g., by extracting keywords or by using automatic classification methods. Such an approach is also employed by APOSDLE [ 104], DEPTHS [ 52], and Khribi et al. [ 54]. Several metadata generation frameworks that integrate a broad set of algorithms to generate metadata have been elaborated in the TEL field [ 72], [ 20]. Such frameworks can potentially be used by some systems that currently rely on manual annotation.

#### 4.2.7 User Context

Information about the user is captured in different ways, depending on the type of information. CALS [ 123] and Santos et al. [ 93] capture information about learning styles. Both systems use a registration module that enables the user to select her best matching learning style based on the Felder-Silverman [ 40] inventory.

Interests of the user are captured explicitly through registration or rating modules [ 93], implicitly through interactions of the user with the system [ 128] or by combining both approaches. For instance, C-LINK [ 95] and COLDEX [ 8] use a rating module to capture explicit interest indicators and track which resources the user contributes or downloads as implicit interest indicators. PERKAM [ 37] uses a registration module to capture initial interest indicators. These indicators are refined when a learner adds resources to her folder. Such a combined approach is useful to avoid cold-start issues of recommendation algorithms. MOBIlearn [ 65] uses the time the learner spends on a location to estimate interest.

Knowledge levels are captured explicitly by some systems. For example, CALS [ 123] requests such information from the learner, whereas TANGO [ 76] relies on teachers to set knowledge levels indicators after examination. TANGO then refines knowledge levels during system use with an implicit approach. For example, when the learner provides correct answers, her knowledge level indicators are adapted. DEPTHS [ 52] also incorporates indicators based on the level of participation in activities. Many systems rely on an implicit approach only through self-assessment tools available in learning environments. COLDEX [ 8] infers the knowledge level of the learner based on the number of resources a learner has downloaded. There are obvious tradeoffs between the approaches: explicit examination by a teacher will provide a more accurate description of the knowledge level than an estimate based on number of downloads. Nevertheless, such estimates may be valuable in scenarios that unobtrusively try to obtain an indication of the knowledge level.

#### 4.2.8 Social Relation Context

Explicit approaches to capture social relations rely on a manual representation of the group structure [ 82] or the organization in corporate environments [ 95]. Implicit approaches use for instance enrollment data from learning management systems or social networks [ 23]. Other systems infer social relations by analyzing interactions between users [ 93], [ 52], [ 38] to obtain indications of the level of collaboration between different members of a group. DEPTHS [ 52] for instance uses such data to suggest peers who can help with a learning activity based on previous collaborations.

### 4.3 Contextual Recommendation Algorithms

As described in Section 2.3, different methods have been proposed to incorporate contextual information in the recommendation process [ 3], including recommendation via context-driven querying and search, contextual prefiltering, contextual postfiltering and contextual modeling.

Fourteen systems rely on a recommendation via context-driven querying and search method (see Fig. 4). These systems match contextual data to resource metadata in order to retrieve suitable resources. For instance, MOBIlearn [ 65] matches the location and current topic of interest to descriptions of learning resources and peers. TANGO [ 76] matches the knowledge level of the learner and her location to resource descriptions. Several systems rely on an ontology-based query method to filter suitable resources, including APOSDLE [ 104] and DEPTHS [ 52]. The main difference with traditional recommender systems, discussed in Section 2.1.2, is that additional context data are used to retrieve relevant resources from a repository.

Many other systems use a postfiltering algorithm to filter the results of a traditional recommendation algorithm based on contextual information. For instance, COLDEX [ 8] filters the results of a traditional recommendation algorithm based on computing context constraints. PL-CR $^2$ [ 68] and Berri et al. [ 12] filter the results based on both computing and time constraints. The postfiltering algorithm of CALS [ 123] also takes into account physical conditions. C-LINK [ 95], Zhao et al. [ 127] and the 3A recommender [ 38] use a contextual modeling technique that adapts a traditional recommendation algorithm to take into account contextual information. The 3A recommendation algorithm adapts a version of Google's PageRank algorithm to the particular modeling framework that considers relation context. C-LINK [ 95] and Zhao et al. [ 127] attempt to learn and model user preferences in contextualized user profiles. Such systems adapt traditional algorithms to incorporate multidimensional user preferences, for instance which resources a user liked when she was working on a particular activity. Whereas evaluation results are promising, potential drawbacks and additional challenges related to data sparsity have not been discussed.

More generally, there is a need to explore more comprehensively tradeoffs between the different approaches in order to develop a better understanding on which algorithms to use and how to combine them [ 3]. So far, only Santos et al. [ 93] compared the performance of different combinations of contextual recommendation algorithms. We discuss in Section 5.3 how such evaluations can be facilitated.

### 4.4 Recommendation Outputs

All surveyed recommender systems offer suggestions for suitable learning resources (see Fig. 4). In many cases, these resources are closed corpus resources that were made available and annotated to support the target learning activities, such as outdoor or language learning activities based on available objects. Nine systems enable recommendation of open corpus resources. These systems rely on automatic annotation [ 104], [ 54], [ 52] and/or annotations that are made by peers [ 95], [ 65], [ 38]. Seven systems provide suggestions for suitable peer learners or teachers in addition to learning resources. These people are suggested based on proximity [ 65] or social relations [ 52], [ 38], or a combination of both [ 37]. In addition, some systems include support for recommending messages to guide and motivate the learner. These messages are sometimes automatically triggered based on activity or location context [ 82] or provided manually by teachers or peers [ 23].

### 4.5 Evaluation

Whereas several promising context-aware recommender systems for TEL have been elaborated in recent years, most of these systems are still in prototype phase. Only a few systems have been used in more elaborate trial experiments—including LIP [ 97], COLDEX [ 8], PERKAM [ 37], MOBILearn [ 65], APOSDLE [ 95] and work of Rogers et al. [ 89] and Santos et al. [ 93].

#### 4.5.1 Learning Efficiency and Effectiveness

The effect on learning efficiency and/or learning effectiveness has been evaluated for five systems. Evaluation results of Rogers et al. [ 89] indicate that suggesting resources during outdoor learning activities helps children to reflect and has a positive effect on the learning outcome. The impact on learning effectiveness of PERKAM [ 37] was measured in an experiment that consisted of several phases, including a comparison of the knowledge learned with and without the system. Among others, results indicate that more learners were able to successfully complete the task within a fixed time interval when using the system. In a similar way, Zhao et al. [ 127] evaluated learning effectiveness in comparison with learners who use a text book only. Results indicate that users receive better test scores and that learning interest increased with more than 10 minutes. The used metric is somewhat unconventional, but at least some attempt is made to measure the effect on learning. Teng et al. [ 108] measured effectiveness in a comparative evaluation and conclude that there is no significant difference with students who did not use the recommender. Evaluation results of APOSDLE [ 8] indicate that the impact on learning effectiveness and efficiency is dependent on the domain in which the system is deployed. In highly specialized domains where knowledge is well documented, the approach turned out to have a bigger impact on learning than in domains with few available resources. Whereas conclusions are in some cases still preliminary, at least some efforts have been made to assess the impact of context-aware recommender systems on learning.

#### 4.5.2 Accuracy

Accuracy of recommendations has been evaluated for five systems. Khribi et al. [ 54] measured the accuracy of the approach with precision and recall measures. Results indicate that precision increases when more recently visited pages are considered to capture context and decreases when the number of recommended resources increases. Whereas the approach is promising, the number of visited pages considered is limited to four in the evaluation experiment and seems a bit arbitrary. The accuracy of PL-CR $^2$ [ 68] was evaluated in a simulation experiment. Results indicate that the method outperforms traditional content-based and collaborative filtering recommendation techniques with a significant increase of precision. Evaluation results of the 3A recommendation algorithm [ 38] indicate that the incorporation of relations outperforms standard collaborative filtering in terms of recall. In several experiments, Santos et al. [ 92], [ 93] gathered feedback on the accuracy of different types of recommendations. Among others, they conclude that recommendations based on learning styles are most appreciated, whereas collaboration recommendations were considered less relevant. Such evaluations are interesting, as they provide some insight into the relevancy of different contextual data in the recommendation process.

#### 4.5.3 Usefulness and Usability

The perceived usefulness and usability have been evaluated for more systems. These evaluations were conducted with a wide range of methods and data collection approaches such as questionnaires, interviews and log data. In some cases, these evaluations consisted of elaborate trial experiments where users were asked to use the system for an extended period of time—such as evaluations of APOSDLE [ 8], LIP [ 97], Teng et al. [ 108], and MOBIlearn [ 65]. In other cases, preliminary indicators were collected by asking users to perform certain tasks with the developed prototype [ 76]. Whereas the perceived usefulness was high in all cases, many usability issues have been identified. Such usability issues were identified for both context acquisition techniques as well as delivery of recommendations to end-users. Teng et al. [ 108] outline difficulties with scanning QR codes using cameras as a basis to capture activity context. Evaluation results of MOBIlearn [ 65] indicate that users were often confused because they did not understand why certain recommendations were made and why they changed. Similar issues were identified in usability evaluations of TANGO [ 76] and LIP [ 97], where usability issues were identified as critical for the use of the system. We elaborate in the next section on future research opportunities to address these issues.

## Future Challenges

Although several promising prototypes of context-aware recommenders for learning have been elaborated, important challenges remain in order to validate the developed prototypes and to assess their impact on learning. In this section, we briefly discuss the results of our survey and challenges involved in this endeavor, as well as how these challenges relate to what other researchers in this area have proposed.

### 5.1 Context Acquisition Challenges

Many of the surveyed systems rely on manual input from users. Whereas such manual input is useful to evaluate first prototypes, we believe that requesting such involvement from end-users may significantly hamper the uptake of context-aware recommendation for learning. In addition, several systems indicate that they use contextual data, but do not describe the methods that are used to capture these data (see Fig. 3).

An interesting future direction of research is the development of context sensors for learning that automate the acquisition of the context dimensions in a modular and generic fashion. Such modular approach has been proposed by several researchers, for context-aware systems in general [ 3], and TEL in specific [ 96]. Results of our survey indicate that the biggest challenge lies in capturing the user and activity context dimensions (actions, current topic of interest, tasks). Researchers in TEL have argued that factors affecting an activity is precisely what makes the notion of context meaningful for learning [ 121], [ 9]. However, in contrast to other dimensions such as computing, location and social relations that have been researched more extensively by the mobile learning and social network analysis communities, these dimensions still rely in many cases on explicit capturing methods. Whereas some promising examples of systems that automate activity context acquisition have been presented, the approaches either rely on task models that are created by experts [ 104] or have only been partially evaluated [ 54], [ 95].

To address this challenge, there is a need for interdisciplinary collaborative research from different research communities, including the learning analytics, educational data mining, adaptive hypermedia and user modeling communities. In recent years, much progress has been made in these areas to capture data from the learner in all her respective learning environments and to measure learning indicators related to low level activity data [ 90]. Learning context sensors for the various dimensions outlined above would enable researchers in the recommender systems for TEL field to reuse existing efforts in this area. As discussed in Section 2.1.2, many existing recommender systems for learning do not yet incorporate additional context dimensions. The availability of context sensors would be of major interest to incorporate contextual information in recommender systems for learning on a much larger scale than in today's prototypes. In addition to the exploration and use of existing techniques to extract contextual data, an interesting future line of research would be to investigate how contextual data can be collected implicitly by using a combination of open data sources and APIs. A simple example combines timestamp and location information with a weather service to suggest situated learning activities. Elaborating such scenarios for both capturing relevant contextual data as well as using such data for generating contextual recommendations constitutes another interesting area of research.

### 5.2 Context Representation Challenges

Current recommender systems rely on custom representations of contextual data. Some systems use standardized representations for describing resource and computing characteristics. The lack of a standard representation for contextual data prevents the sharing and reuse of data across systems [ 9], [ 96], [ 47].

To address this issue, several solutions have been proposed. Held et al. [ 47] indicate that contextual data should be structured, interchangeable, composable, uniform, extensible, and standardized. In addition, the authors present a novel format, Comprehensive Structured Context Profiles (CSCPs), that is based on RDF. The need for standardization has also been identified by Schmidt [ 96] and Richter and Pawlowski [ 88]. Schmidt [ 96] argues that one of the key success factors of technology enhanced learning in the last years has been the standardization activities. The author proposes to include context aspects in the standardization activities, such as SCORM or IEEE LOM. An interesting future line of research would be an integration of both research directions by elaborating a standardized representation of contextual data and mapping those data to existing standards and specifications, so that data compliant to these standards and specifications can be exchanged and reused. The development of a standard representation for both implicit and explicit user data, as well as relevant contextual data that can be associated to these data, will be taken up in a working group of the CEN Workshop on “Learning Technologies” (WS/LT). Initiatives such as LinkedEducation.org have a big potential to interconnect data compliant to existing standards and specifications. The interest in this research area is reflected in several research contributions to a recent workshop on linked learning [ 30].

### 5.3 Evaluation Challenges

In order to prove the successful application of contextual recommender systems in TEL, the evaluation of this technology needs to be further strengthened. Particularly interesting would be having more comprehensive evaluation studies that assess the impact of individual context elements on the recommendation process. Such evaluation has for instance been explored by Santos and Boticario [ 92]. For example, some context dimensions may have a negative influence on the recommendation process as outlined in [ 110].

In order to facilitate evaluation, a more structured and coordinated approach to identify the effects of the context dimensions and their combinations would benefit from generalizing the data according to the framework presented in this paper. Following this approach, existing research efforts can be aligned to enable comparative evaluation studies. The approach would also enable to identify possible gaps and to address them respectively.

In addition, we propose the creation of controlled experiments where the influence of different dimensions can be measured. These experiments could follow the same line as the TREC challenges organized in the Information Retrieval Community [ 46] and have already been conducted by the context-aware recommendation community. Examples include recommendation of suitable movies depending on time (e.g., the Christmas week and the week leading to the Oscars ceremony) and for a given mood [ 4]. For the creation of such experiments in the TEL field, data sets need to be collected from the use of TEL systems. Based on such data sets, correlations between different contextual variables may be discovered that could identify relevant contextual variables for TEL recommendation. Particularly relevant would be an exploratory investigation on the correlation between contextual variables and actions (i.e., ratings, downloads, reads, etc.) of users. Such an explorative analysis could identify potential differences in behavior based on contextual variables—such as day of the week, learning environment and available peers—and used to define meaningful recommendation challenges for the TEL field. Research opportunities to support this process are described in the next section.

### 5.4 Data Set Sharing Challenges

As drawing conclusions about the validity and generalizability of scientific experiments depends on the possibility of verification, repeatability, and comparisons of results [ 70], a collection of data sets is needed to compare the results of different recommendation algorithms and the influence of contextual data on the recommendation process [ 34]. In an increasing number of scientific disciplines, large data collections are emerging as important community resources [ 24]. A first step toward the identification and sharing of TEL data sets stems from the dataTEL theme team of the STELLAR Network of Excellence [ 33]—recently accepted as an EATEL Special Interest Group (SIG). A special dataTEL Cafe event took place during the RecSysTEL Workshop 2010 [ 69] to discuss the submitted data sets and to facilitate data set sharing in the TEL community. Details about the data sets and initial evaluation results of recommendation algorithms with these data sets have been reported in [ 111].

Although interesting data sets have been collected, so far only a few data sets are available that collect data of a large number of users. In addition, most data sets contain only information about learners or teachers and resources. In some data sets time context is captured. Additional context data is available in only one data set that was captured within the APOSDLE project [ 104]. However, the current sample is too small for experimental evaluations. The collection of large data collections that incorporate contextual information about learners is therefore still a major challenge that needs to be tackled.

To this extent, a framework is needed that enables 1) to share available data sets from recommender systems in TEL among researchers in this field and 2) to track the outcomes of context-aware recommendation algorithms on these data sets. Additional considerations related to the privacy of users will need to be researched to enable the sharing and reuse of data sets. We discuss this challenge in the next section.

The sharing of data sets is an open issue that is recognized by several other people. For example, following the National Science Foundation (NSF, USA), the JISC (United Kingdom) has opened a new research program (2011-2013) for the management of research data that “[...] is recognized as one of the most pressing challenges facing the higher education and research sectors. Research data generated by publicly funded research is seen as a public good and should be available for verification and reuse.” [ 86].

### 5.5 Privacy Challenges

Privacy and legal protection rights are a major challenge that needs to be tackled when capturing and using contextual data for recommendation. So far, researchers have often ignored privacy issues. However, if context-aware recommender technologies want to move beyond the current prototype phase, practical solutions regarding legal and privacy issues are needed.

The challenge needs to be tackled from two perspectives: 1) the privacy of the target users needs to be preserved in order to deploy current prototypes in real-life settings and 2) the sharing and exchange of data is a key requirement to enable comparative evaluation studies—as discussed in the previous section.

In principle, researchers do not want to harm the privacy rights of the target users and are willing to make scientific data available. However, they are inexperienced how to address both issues in a proper way. They are missing a condensed overview of the legal situation and practical solutions regarding data set sharing.

In order to overcome these challenges, guidelines need to be developed that document data protection laws like the European Directive on data protection 95/46/EC [ 39]. The main principles of this directive have been discussed in [ 66] and include 1) informed consent: users must be made aware of what data is being gathered and what it is being used for, 2) control: where consent has been given for the gathering and use of contextual data, users should be given information, access, and control over data, and 3) security: data has to be stored securely. Several frameworks to address these requirements are discussed in the literature [ 5], [ 44];

• In a privacy as confidentiality paradigm, privacy has been defined as “the right to be let alone” [ 117]. One way of achieving data confidentiality is to enable the use of information-based services, while either minimizing the collection of information, or securing the collected information from unauthorized access. Another way is to guarantee anonymity, e.g., by anonymizing the collected data.
• In a privacy as control paradigm, privacy is defined as the ability to control what happens with information. A reason for this notion is that revelation of data is necessary and beneficial under many circumstances and that control may help to prevent abuses of data. Identity Management Systems (IDMSs) [ 45] are an important class of tools that provide the ability to control information that is revealed.
• In a privacy as practice paradigm, the objectives are to make it possible to intervene in the flows of existing data by making transparent the way in which information is collected and used for decision making. We discuss in the next section how such an approach can be supported from a user interface point of view.

The documentation of these frameworks as well as research on the use of these frameworks in TEL settings is an interesting further line of research. Initial work in this area has been presented by Gürses and Vuorikari [ 44].

### 5.6 Interaction Challenges

Several challenges related to user interfaces were identified by researchers in this field. Usability evaluations of the LIP [ 97], MOBIlearn [ 65], and TANGO [ 76] systems indicate that the development of user interfaces is critical for context-aware recommender systems. Among others, people are often confused because they do not understand why certain recommendations are made and why these recommendations change.

As outlined in [ 48], it is important to explain the rationale behind recommendations to end-users. The complexity of recommendation algorithms often prevents users from comprehending recommended results and can lead to trust issues when recommendations fail. This complexity is often aggravated by contextual recommendation algorithms that use various types of contextual information in the recommendation process. In addition, recommendation results might change automatically when the context of the user changes. Such automatic updates can be confusing to the user.

To deal with this issue, it is important to provide explanations and justify decisions [ 75]. An important line of research in this area is the use of visualization techniques to provide users with insights in the recommendation process. As an example, social visualizations can help to explain recommendation results by explicitly exposing relationships among content and people [ 126], [ 57]. The PERKAM [ 37] system already uses visualization techniques to present the relationship and distance between peer learners. Such visualizations can be used to easily locate suggested peer learners.

Moreover, visualization techniques can increase understanding of the input and output of a recommender system. Such visualizations can enable the user to meaningfully revise input parameters and thus improve recommendations [ 105]. This objective is particularly important in contextualized recommendations that estimate relevant contextual elements based on user behavior. As the prediction of the current task or interest of the user is a challenging task, there is a need to develop mixed approaches that enable users to provide feedback and help steer this process. As such, the combination of visualization and recommendation techniques to empower users with actionable knowledge to become an active and responsible part-taker in the recommending process, instead of being the typical passive provider of just personal preferences and social connections, is a highly relevant research topic.

### 5.7 Towards Global Data Infrastructures

As it has been pointed out by D. Rehak in his Digital Content Manifesto [ 87] that is driving the US-based Learning Registry initiative, the information world is fragmented but still an abundance of learning information exists. For any learning activity, it can be suspected that somewhere relevant digital content can be found. The aim is to enable a learning layer over all this Web-accessible content, facilitating the deployment of contextualized services that will allow users to find and access relevant content more easily. In this sense, very large, cloud-based data infrastructures like the one that Learning Registry is setting up for the USA, are expected to provide a new perspective into the way that intelligent systems (in general) and recommender systems (in particular) will be developed for TEL. A similar example is the content infrastructure of the EU-oriented Open Discovery Space CIP PSP initiative ( http://www.opendiscoveryspace.eu) that also tries to create a very big data infrastructure collecting learning content and usage data around it. Such global learning data infrastructures can help in scaling up TEL recommender systems by allowing them to consume, process and use a rich variety of contextualized usage data streams, and thus enable novel forms of real time intelligence that can only become possible on extremely large data volumes. The existence of global data infrastructures is expected to really stretch the scalability, the robustness and reactivity of today's algorithms and systems, since there is going to be a need to meet a number of upcoming requirements. TEL, as well as other domains, will need to find ways to develop recommender systems that will be able to grow with the volume of data to be handled (being limited only by the availability of computing resources), to operate at the time scale of the processes they are designed to support (providing recommendations right at the time when requested or needed), and to be able to handle a large variety of data that will be often missing, corrupted or inconsistent (taking recommendations outside the lab simulations and in typical operating environments). We would expect this to become the next major research challenge in contextual recommender systems for TEL.

## Conclusion

In this paper, we have presented a survey of context-aware recommender systems that have been deployed in TEL settings. The research contributions of this paper are threefold. First, a context framework has been presented that identifies context dimensions for the analysis and development of context-aware recommender systems for TEL. This framework can be used to drive the discussion of contextualization of a variety of TEL applications. In addition, such a framework can facilitate comparative evaluation studies by aligning existing efforts in this area. Second, we have presented an in-depth analysis of context-aware recommender systems that have been deployed in educational settings. The analysis sheds light on the use of context in current prototypes, the techniques that are employed to incorporate these variables in the recommendation process and current practices to evaluate the potential impact of the approaches on the learning process.

Third, we have outlined future challenges for the development and validation of context-aware recommender systems for learning. Results of our survey indicate that there has been much advancement in the development of context-aware TEL recommenders in recent years. Many promising prototypes illustrate the potential and opportunities that these systems create. Nevertheless, important challenges related to the capturing and use of contextual data remain that need to be tackled in order to increase uptake and validate research efforts in realistic trial experiments. We hope that these ideas can help to further shape exciting and relevant research on context-aware TEL recommenders.

## Acknowledgments

The authors would like to thank Shlomo Berkovsky for his valuable comments on earlier versions of this manuscript. In addition, we would like to thank the anonymous reviewers for their suggestions that helped to improve this work to a great extent. Part of this work has been supported by the EU FP7 STELLAR Network of Excellence (grant agreement no. 231913). Katrien Verbert is a Postdoctoral Fellow of the Research Foundation—Flanders (FWO). The work of Nikos Manouselis has been funded with support by the EU project VOA3R - 250525 of the CIP PSP Programme ( http://voa3r.eu). The work of Martin Wolpers has received funding from the EC Seventh Framework Programme (FP7/2007-2013) under grant agreement no 231396 (ROLE). The contribution of Xavier Ochoa was supported by VLIR through the RIP Project ZEIN2010RIP09. The work of Hendrik Drachsler was supported by the Netherlands Laboratory for Lifelong Learning (NELLL) within the AlterEgo project.

## References

• 1. F. Abel, I.I. Bittencourt, E. Costa, N. Henze, D. Krause, and J. Vassileva, “Recommendations in Online Discussion Forums for E-Learning Systems,” IEEE Trans. Learning Technologies, vol. 3, no. 2, pp. 165-176, Apr.-June 2010.
• 2. G. Adomavicius, and A. Tuzhilin, “Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions,” IEEE Trans. Knowledge and Data Eng., vol. 17, no. 6, pp. 734-749, June 2005.
• 3. G. Adomavicius, and A. Tuzhilin, “Context-Aware Recommender Systems,” Recommender Systems Handbook: A Complete Guide for Research Scientists and Practitioners, L. Rokach, B. Shapira, P. Kantor, and F. Ricci, eds., pp. 217-250, Springer, 2011.
• 4. G. Adomavicius, A. Tuzhilin, and E.W. Luca, “Context-Awareness in Recommender Systems: Research Workshop and Movie Recommendation Challenge Categories and Subject Descriptors,” Human Factors, pp. 60558-60558, 2010.
• 5. R. Agrawal, and C. Johnson, “Securing Electronic Health Records without Impeding the Flow of Information,” Int'l J. Medical Informatics, vol. 76, nos. 5/6, pp. 471-479, 2006.
• 6. J. Albrecht, “Geospatial Information Standards. A Comparative Study of Approaches in the Standardisation of Geospatial Information,” Computers and Geosciences, vol. 25, no. 1, pp. 9-24, 1999.
• 7. R. Bader, E. Neufeld, W. Woerndl, and V. Prinz, “Context-Aware POI Recommendations in an Automotive Scenario Using Multi-Criteria Decision Making Methods,” Proc. Workshop Context-Awareness in Retrieval and Recommendation, pp. 23-30, 2011.
• 8. N. Baloian, P. Galdames, C. Collazos, and L. Guerrero, “A Model for a Collaborative Recommender System for Multimedia Learning Material,” Groupware: Design, Implementation, and Use, G.-J. de Vreede, L. Guerrero, and G. Marn Ravents, eds., vol. 3198, pp. 281-288, Springer, 2004.
• 9. R. Beale, and P. Lonsdale, “Mobile Context Aware Systems: The Intelligence to Support Tasks and Effectively Utilise Resources,” Mobile Human-Computer Interaction, S. Brewster and M. Dunlop, eds., vol. 3160, pp. 573-576, Springer, 2004.
• 10. R. Benlamri, and X. Zhang, “A Global Ontology Space for Mobile Learning,” Proc. IEEE Eighth Int'l Conf. Advanced Learning Technologies, pp. 49-53, July 2008.
• 11. S. Berkovsky, T. Kuflik, and F. Ricci, “Mediation of User Models for Enhanced Personalization in Recommender Systems,” User Modeling and User-Adapted Interaction, vol. 18, no. 3, pp. 245-286, 2008.
• 12. J. Berri, R. Benlamri, and Y. Atif, “Ontology-Based Framework for Context-Aware Mobile Learning,” Proc. Int'l Conf. Wireless Comm. and Mobile Computing, pp. 1307-1310, 2006.
• 13. C. Boyne, J. Dron, and R. Mitchell, “Evaluating Assessment Using N-Dimensional Filtering,” Proc. World Conf. E-Learning in Corporate, Govt., Healthcare, and Higher Education, pp. 268-274, 2002.
• 14. D. Brickley, and L. Miller, FOAF Vocabulary Specification 0.98, Namespace Document, http://xmlns.com/foaf/spec, 2010.
• 15. P. Brusilovsky, R. Farzan, and J. Ahn, “Comprehensive Personalized Information Access in an Educational Digital Library,” Proc. ACM/IEEE-CS Fifth Joint Conf. Digital Libraries, pp. 9-18, 2005.
• 16. P. Brusilovsky, and E. Millán, “User Models for Adaptive Hypermedia and Adaptive Educational Systems,” The Adaptive Web, P. Brusilovsky, A. Kobsa, and W. Nejdl, eds., vol. 4321, pp. 3-53, Springer, 2007.
• 17. R. Burke, “Knowledge-Based Recommender Systems,” Encyclopedia of Library and Information Systems, vol. 69, supplement 32, pp. 175-186, 2000.
• 18. R. Burke, “Hybrid Web Recommender Systems,” The Adaptive Web, P. Brusilovsky, A. Kobsa, and W. Nejdl, eds., vol. 4321, ch. 12, pp. 377-408, Springer, 2007.
• 19. M. Burnett, P. Prekop, and C. Rainsford, “Intimate Location Modeling for Context Aware Computing,” Proc. Workshop Location Modeling for Ubiquitous Computing, pp. 77-82, 2001.
• 20. M. Chatti, N. Muhammad, and M. Jarke, “ALOA: A Web Services Driven Framework for Automatic Learning Object Annotation,” Times of Convergence. Technologies Across Learning Contexts, P. Dillenbourg and M. Specht, eds., vol. 5192, pp. 86-91, Springer, 2008.
• 21. C.-M. Chen, H.-M. Lee, and Y.-H. Chen, “Personalized E-Learning System Using Item Response Theory,” Computers and Education, vol. 44, no. 3, pp. 237-255, Apr. 2005.
• 22. G. Chen, and D. Kotz, “A Survey of Context-Aware Mobile Computing Research,” technical report, 2000.
• 23. G.-D. Chen, and P.-Y. Chao, “Augmenting Traditional Books with Context-Aware Learning Supports from Online Learning Communities,” Educational Technology and Soc., vol. 11, no. 2, pp. 27-40, 2008.
• 24. A. Chervenak, “The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Datasets,” J. Network and Computer Applications, vol. 23, no. 3, pp. 187-200, 2000.
• 25. Y. Cui, and S. Bull, “Context and Learner Modelling for the Mobile Foreign Language Learner,” System, vol. 33, no. 2, pp. 353-367, 2005.
• 26. D. Dagger, O. Conlan, and V. Wade, “Fundamental Requirements of Personalised Elearning Development Environments,” Proc. World Conf. E-Learning in Corporate, Govt., Healthcare, and Higher Education, pp. 2746-2754, Oct. 2005.
• 27. P. De Bra, M. Pechenizkiy, K. van der Sluijs, and D. Smits, “GRAPPLE: Integrating Adaptive Learning into Learning Management Systems,” Proc. World Conf. Educational Multimedia, Hypermedia and Telecomm., pp. 5183-5188, June 2008.
• 28. M. Derntl, and K.A. Hummel, “Modeling Context-Aware E-Learning Scenarios,” Proc. IEEE Third Int'l Conf. Pervasive Computing and Comm. Workshops (PERCOMW '05), pp. 337-342, 2005.
• 29. A. Dey, G. Abowd, and D. Salber, “A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-Aware Applications,” HumAN-Computer Interaction, vol. 16, pp. 97-166, Dec. 2001.
• 30. “Navigating Learning Resources through Linked Data: A Preliminary Report on the Re-Design of Organic.Edunet,” Proc. First Int'l Workshop eLearning Approaches for Linked Data Age, S. Dietze, M. d'Aquin, D. Gasevic, and M.-A. Sicilia, eds., vol. 717, 2011.
• 31. P. Dolog, and W. Nejdl, “Challenges and Benefits of the Semantic Web for User Modelling,” Proc. Workshop Adaptive Hypermedia and Adaptive Web-Based Systems (AH '03), pp. 99-112, 2003.
• 32. P. Dourish, “What We Talk about When We Talk about Context,” Personal Ubiquitous Computing, vol. 8, pp. 19-30, Feb. 2004.
• 33. H. Drachsler, T. Bogers, R. Vuorikari, K. Verbert, E. Duval, N. Manouselis, G. Beham, S. Lindstaedt, H. Stern, M. Friedrich, and M. Wolpers, “Issues and Considerations Regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning,” Procedia Computer Science, vol. 1, no. 2, pp. 2849-2858, 2010.
• 34. H. Drachsler, H. Hummel, and R. Koper, “Personal Recommender Systems for Learners in Lifelong Learning Networks: The Requirements, Techniques and Model,” Int'l J. Learning Technology, vol. 3, no. 4, pp. 404-423, 2008.
• 35. H. Drachsler, H. Hummel, and R. Koper, “Identifying the Goal, User Model and Conditions of Recommender Systems for Formal and Informal Learning,” J. Digital Information, vol. 10, no. 2, pp. 4-24, 2009.
• 36. E. Duval, R. Vuorikari, N. Manouselis, and F. Van Assche, Proc. Second Workshop Social Information Retrieval for Technology-Enhanced Learning (SIRTEL '08), vol. 382, 2008.
• 37. M.M. El-Bishouty, H. Ogata, and Y. Yano, “PERKAM: Personalized Knowledge Awareness Map for Computer Supported Ubiquitous Learning,” Educational Technology and Soc., vol. 10, no. 3, pp. 122-134, 2007.
• 38. S. El Helou, C. Salzmann, and D. Gillet, “The 3a Personalized, Contextual and Relation-Based Recommender System,” J. Universal Computer Science, vol. 16, no. 16, pp. 2179-2195, Aug. 2010.
• 39. “Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the Protection of Individuals with Regard to the Processing of Personal Data and on the Free Movement of Such Data,” Official J. L 281 of 23.11.1995, pp. 31-39.
• 40. R.M. Felder, and L.K. Silverman, “Learning and Teaching Styles in Engineering Education,” J. Eng. Education, vol. 78, no. 7, pp. 674-681, 1988.
• 41. M. Friedrich, M. Wolpers, R. Shen, C. Ullrich, R. Klamma, D. Renzel, A. Richert, and B. von der Heiden, “Early Results of Experiments with Responsive Open Learning Environments,” J. Universal Computer Science, vol. 17, no. 3, pp. 451-471, 2011.
• 42. H. Garcia-Molina, “Flexible Recommendations in CourseRank,” On the Move to Meaningful Internet Systems (OTM '08), R. Meersman and Z. Tari, eds., vol. 5331, pp. 7-7, Springer, 2008.
• 43. J. Greer, G. McCalla, J. Cooke, J. Collins, V. Kumar, A. Bishop, and J. Vassileva, “The Intelligent Helpdesk: Supporting Peer-Help in a University Course,” Intelligent Tutoring Systems, B. Goettl, H. Halff, C. Redfield, and V. Shute, eds., vol. 1452, pp. 494-503, Springer, 1998.
• 44. S. Gürses, and R. Vuorikari, “Privacy Issues and Data Protection in Technology Enhanced Learning,” Proc. First Workshop Data Sets for Technology Enhanced Learning, http://homes.esat.kuleuven.be/~sguerses/talks.html, 2011.
• 45. M. Hansen, P. Berlich, J. Camenisch, S. Clauß, A. Pfitzmann, and M. Waidner, “Privacy-Enhancing Identity Management,” Information Security Technical Report, vol. 9, no. 1, pp. 35-44, 2004.
• 46. D. Harman, “What We Have Learned, and Not Learned, from TREC,” Proc. British Computer Soc.—Information Retrieval Specialist Group (BCS-IRSG) Conf., pp. 2-20, 2000.
• 47. A. Held, S. Buchholz, and A. Schill, “Modeling of Context Information for Pervasive Computing Applications,” Proc. Sixth World Multiconf. Systemics, Cybernetics and Informatics, p. 6, 2002.
• 48. J.L. Herlocker, J.A. Konstan, and J. Riedl, “Explaining Collaborative Filtering Recommendations,” Proc. ACM Conf. Computer Supported Cooperative Work (CSCW '00), pp. 241-250, 2000.
• 49. J.L. Herlocker, J.A. Konstan, L.G. Terveen, and J.T. Riedl, “Evaluating Collaborative Filtering Recommender Systems,” ACM Trans. Information Systems, vol. 22, no. 1, pp. 5-53, 2004.
• 50. P. Honey, and A. Mumford, Manual of Learning Styles. Peter Honey, 1982.
• 51. B. Hu, and P. Moore, “Context Modelling to Support Location Based Cooperative Mobile Learning,” Proc. Int'l Conf. Computer Supported Cooperative Work in Design, pp. 1372-1376, May 2006.
• 52. Z. Jeremic, J. Jovanovic, and D. Gasevic, Semantic Web Technologies for the Integration of Learning Tools and Context-Aware Educational Services, pp. 860-875. Springer, 2009.
• 53. C. Karagiannidis, and D. Sampson, “Accommodating Learning Styles in Adaptation Logics for Personalised Learning Systems,” Proc. World Conf. Educational Multimedia, Hypermedia and Telecomm., pp. 1715-1726, 2002.
• 54. M. Khribi, M. Jemni, and O. Nasraoui, “Automatic Recommendations for E-Learning Personalization Based on Web Usage Mining Techniques and Information Retrieval,” Proc. Int'l Conf. Advanced Learning Technologies, pp. 241-245, 2008.
• 55. C. Kiss, “Composite Capability/Preference Profiles (CC/PP): Structure and Vocabularies 2.0,” W3C working draft, Apr. 2007.
• 56. R. Klamma, M. Spaniol, and Y. Cao, “Community Aware Content Adaptation for Mobile Technology Enhanced Learning,” Proc. First European Conf. Technology Enhanced Learning: Innovative Approaches to Learning and Knowledge Sharing (ECTEL), vol. 4227, pp. 227-241, 2006.
• 57. J. Klerkx, and E. Duval, “Visualising Social Bookmarks,” J. Digital Information, vol. 10, no. 2, 2009.
• 58. K.R. Koedinger, and A.T. Corbett, “Cognitive Tutors: Technology Bringing Learning Sciences to the Classroom,” The Cambridge Handbook of the Learning Sciences, pp. 61-77, 2006.
• 59. B. Kort, R. Reilly, and R.W. Picard, “External Representation of Learning Process and Domain Knowledge: Affective State as a Determinate of Its Structure and Function,” Proc. Artificial Intelligence in Education Conf., pp. 64-69, 2001.
• 60. A. Kurti, “Context Modeling to Support the Design of Mobile Learning,” Proc. Fifth Int'l Conf. Soft Computing as Transdisciplinary Science and Technology (CSTST '08), pp. 536-541, 2008.
• 61. F.C.M. Lau, “A Context-Aware Decision Engine for Content Adaptation,” IEEE Pervasive Computing, vol. 1, no. 3, pp. 41-49, 2002.
• 62. IEEE P1484.12.2/D1, Learning Technology Standards Committee of the IEEE, Final Standard for Learning Technology - Learning Object Metadata, IEEE, July 2002.
• 63. P. Lehsten, R. Zender, U. Lucke, and D. Tavangarian, “A Service-Oriented Approach Towards Context-aware Mobile Learning Management Systems,” Proc. IEEE Eighth Int'l Conf. Pervasive Computing and Comm. Workshops (PERCOM), pp. 268-273, 2010.
• 64. L. Li, Y. Zheng, H. Ogata, and Y. Yano, “A Framework of Ubiquitous Learning Environment,” Proc. Fourth Int'l Conf. Computer and Information Technology (CIT '04), pp. 345-350, 2004.
• 65. P. Lonsdale, C. Baber, M. Sharples, W. Byrne, P. Brundell, and R. Beale, “Context awareness for MOBIlearn: Creating an Engaging Learning Experience in an Art Museum,” Proc. World Conf. Mobile and Contextual Learning (mLearn '04), pp. 115-118, 2004.
• 66. P. Lonsdale, C. Barber, M. Sharples, and T.N. Arvantis, “A Context-Awareness Architecture for Facilitating Mobile Learning,” Learning with Mobile Devices Research and Development, J. Attewell and C. Savil-Smith, eds., pp. 79-85, Learning and Skills Development Agency, 2004.
• 67. J. Lu, “A Personalized E-Learning Material Recommender System,” Proc. Int'l Conf. Information Technology for Application (ICITA), pp. 374-379, 2004.
• 68. J. Luo, F. Dong, J. Cao, and A. Song, “A Context-Aware Personalized Resource Recommendation for Pervasive Learning,” Cluster Computing, vol. 13, pp. 213-239, June 2010.
• 69. N. Manouselis, H. Drachsler, K. Verbert, and O.C. Santos, “RecSysTEL Preface 2010,” Procedia Computer Science, vol. 1, no. 2, pp. 2773-2774, 2010.
• 70. N. Manouselis, H. Drachsler, R. Vuorikari, H. Hummel, and R. Koper, “Recommender Systems in Technology Enhanced Learning,” Recommender Systems Handbook: A Complete Guide for Research Scientists and Practitioners, L. Rokach, B. Shapira, P. Kantor, and F. Ricci, eds., pp. 387-409, Springer, 2011.
• 71. J. Martinez, “MPEG-7 Overview,” version 10, Oct. 2005.
• 72. M. Meire, X. Ochoa, and E. Duval, “SAMGI: Automatic Metadata Generation v2.0,” Proc. World Conf. Educational Multimedia, Hypermedia and Telecomm. (ED-Media '07), pp. 1195-1204, 2007.
• 73. R. Nadolski, A. van den Berg, B. Berlanga, H. Drachsler, H. Hummel, R. Koper, and P. Sloep, “Simulating Light-Weight Personalised Recommender Systems in Learning Networks: A Case for Pedagogy-Oriented and Rating-Based Hybrid Recommendation Strategies,” J. Artificial Soc. and Social Simulation, vol. 12, no. 1, p. 4, 2009.
• 74. L. Nguyen, and P. Do, “Learner Model in Adaptive Learning,” World Academy of Science Eng. and Technology, vol. 45, pp. 395-400, 2008.
• 75. J. O'Donovan, B. Smyth, B. Gretarsson, S. Bostandjiev, and T. Höllerer, “PeerChooser: Visual Interactive Recommendation,” Proc. 26th Ann. SIGCHI Conf. Human Factors in Computing Systems, pp. 1085-1088, 2008.
• 76. H. Ogata, and Y. Yano, “Context-Aware Support for Computer-Supported Ubiquitous Learning,” Proc. IEEE Second Int'l Workshop Wireless and Mobile Technologies in Education, pp. 27-36, 2004.
• 77. A. Ortony, G.L. Clore, and A. Collins, The Cognitive Structure of Emotions. Cambridge Univ., July 1988.
• 78. H.-S. Park, J.-O. Yoo, and S.-B. Cho, “A Context-Aware Music Recommendation System Using Fuzzy Bayesian Networks with Utility Theory,” Fuzzy Systems and Knowledge Discovery, L. Wang, L. Jiao, G. Shi, X. Li, and J. Liu, eds., vol. 4223, pp. 970-979, Springer, 2006.
• 79. L. Passani, “WURFL - The Wireless Universal Resource File,” http://wurfl.sourceforge.net, 2009.
• 80. M.F. Paulsen, “Experiences with Learning Management Systems in 113 European Institutions,” Educational Technology and Soc., vol. 6, no. 4, pp. 134-148, 2003.
• 81. M.J. Pazzani, and D. Billsus, “Content-Based Recommendation Systems,” The Adaptive Web, P. Brusilovsky, A. Kobsa, and W. Nejdl, eds., vol. 4321, pp. 325-341, Springer-Verlag, 2007.
• 82. S.A. Petersen, and J.-K. Markiewicz, “PALLAS: Personalised Language Learning on Mobile Devices,” Proc. Fifth IEEE Int'l Conf. Wireless, Mobile, and Ubiquitous Technology in Education, pp. 52-59, 2008.
• 83. A.B. Randolph, and G.S. Hubona, “Organizational and Individual Acceptance of Assistive Interfaces and Technologies,” Human-Computer Interaction and Management Information Systems: Applications, D. Galletta and P. Zhang, eds., pp. 379-400, M.E. Sharpe, 2006.
• 84. A. Rath, D. Devaurs, and S. Lindstaedt, “UICO: An Ontology-Based User Interaction Context Model for Automatic Task Detection on the Computer Desktop,” Proc. First Workshop Context, Information and Ontologies (CIAO '09), p. 10, 2009.
• 85. M. Recker, and D. Wiley, “A Non-Authoritative Educational Metadata Ontology for Filtering and Recommending Learning Objects,” J. Interactive Learning Environments, vol. 9, no. 3, pp. 255-271, 2003.
• 86. C. Reffay, M.L. Betbeder, and T. Chanier, “Multimodal Learning and Teaching Corpora Exchange: Lessons Learned in 5 Years by the Mulce Project,” Int'l J. Technology Enhanced Learning, Special Issue on Datasets and Data Supported Learning in Technology-Enhanced Learning, vol. 4, nos. 1/2, 2012.
• 87. D.R. Rehak, “Digital Content Manifesto,” Proc. Invited Talk of the Fourth Int'l Workshop Search and Exchange of E-Le@rning Materials, http://www.learningregistry.org/news/digitalcontentmanifesto sem2010, Sept. 2010.
• 88. T. Richter, and J.M. Pawlowski, “The Need for Standardization of Context Metadata for E-Learning Environments,” Proc. Conf. Asia-Europe E-Learning Network (e-ASEM), p. 22, 2007.
• 89. Y. Rogers, S. Price, C. Randell, D. Stant, o. Fraser, M. Weal, and G. Fitzpatrick, “Ubi-Learning: Integrating Indoor and Outdoor Learning Experiences,” Comm. ACM, vol. 48, no. 1, pp. 55-59, 2005.
• 90. C. Romero, and S. Ventura, “Educational Data Mining: A Survey from 1995 to 2005,” Expert Systems with Applications, vol. 33, no. 1, pp. 135-146, 2007.
• 91. J.A. Russell, “A Circumplex Model of Affect,” J. Personality and Social Psychology, vol. 39, pp. 1161-1178, 1980.
• 92. O. Santos, and J. Boticario, “Users Experience with a Recommender System in an Open Source Standard-Based Learning Management System,” HCI and Usability for Education and Work, A. Holzinger, ed., vol. 5298, pp. 185-204, Springer, 2008.
• 93. O.C. Santos, J. Granado, E. Raffenne, and J.G. Boticario, “Offering Recommendations in OpenACS/dotLRN,” Proc. OpenACS/.LRN Conf., pp. 37-46, Nov. 2008.
• 94. B. Schilit, N. Adams, and R. Want, “Context-Aware Computing Applications,” Proc. First Workshop Mobile Computing Systems and Applications (WMCSA '94), pp. 85-90, 1994.
• 95. R. Schirru, S. Baumann, M. Memmel, and A. Dengel, “Extraction of Contextualized User Interest Profiles in Social Sharing Platforms,” J. Universal Computer Science, vol. 16, no. 16, pp. 2196-2213, 2010.
• 96. A. Schmidt, “Potentials and Challenges of Context-awareness for Learning Solutions,” Proc. LWA—LernenWissensentdeckung— Adaptivität, 13th Ann. Workshop SIG Adaptivity and User Modeling in Interactive Systems, pp. 63-68, 2005.
• 97. A. Schmidt, “Impact of Context-Awareness on the Architecture of E-Learning Solutions,” Architecture Solutions for E-Learning Systems, C. Pahl ed., ch. 16, pp. 306-319, IGI, Nov. 2007.
• 98. A. Schmidt, M. Beigl, and H.-W. Gellersen, “There is More to Context than Location,” Computers and Graphics, vol. 23, no. 6, pp. 893-901, 1999.
• 99. M. Sharples, “The Design of Personal Mobile Technologies for Lifelong Learning,” Computers and Education, vol. 34, nos. 3/4, pp. 177-193, 2000.
• 100. L. Shen, M. Wang, and R. Shen, “Affective E-Learning: Using Emotional Data to Improve Learning in Pervasive Learning Environment,” Educational Technology and Soc., vol. 12, no. 2, pp. 176-189, 2009.
• 101. A. Sorokine, “Spatial Data Exchange Formats and Standards: Comparative Study,” Proc. Bull. Geospatial Information and Technology (GISAssociation), vol. 4, no. 6, pp. 70-78, 1996.
• 102. A. Soylu, P. De Causmaecker, and P. Desmet, “Context and Adaptivity in Pervasive Computing Environments: Links with Software Engineering and Ontological Engineering,” J. Software, vol. 4, no. 9, pp. 992-1013, 2009.
• 103. M. Specht, “ACE - Adaptive Courseware Environment,” Proc. Int'l Conf. Adaptive Hypermedia and Adaptive Web-Based Systems (AH '00), pp. 380-383, 2000.
• 104. H. Stern, R. Kaiser, P. Hofmair, P. Kraker, S.N. Lindstaedt, and P. Scheir, “Content Recommendation in Aposdle Using the Associative Network,” J. Universal Computer Science, vol. 16, no. 16, pp. 2214-2231, 2010.
• 105. K. Swearingen, and R. Sinha, “Beyond Algorithms: An HCI Perspective on Recommender Systems,” Proc. ACM SIGIR Workshop Recommender Systems, p. 11, 2001.
• 106. T. Tang, and G. Mccalla, “Smart Recommendation for an Evolving E-Learning System,” Proc. Workshop Technologies for Electronic Documents for Supporting Learning, Int'l Conf. Artificial Intelligence in Education (AIED), p. 12, 2003.
• 107. L. Tankeleviciene, and R. Damasevicius, “Towards a Conceptual Model of Learning Context in E-Learning,” Proc. IEEE Int'l Conf. Advanced Learning Technologies, pp. 645-646, 2009.
• 108. D.C. Teng, N.-S. Chen, and C.-H. Lee, “Enhancing English Reading Comprehension by Integrating Direct Access to Digital Materials and Scaffolded Questionings in Paper Prints,” Proc. IEEE 11th Int'l Conf. Advanced Learning Technologies (ICALT), pp. 244-248, July 2011.
• 109. C. Timmerer, J. Jaborning, and H. Hellwagner, “Delivery Context Descriptions a Comparison and Mapping Model,” Proc. Ninth Workshop Multimedia Metadata, V. Charvillat R. Klamma, and R. Grigoras, eds., p. 18, http://ceur-ws.org, 2009.
• 110. M. Van Setten, S. Pokraev, and J. Koolwaaij, “Context-aware Recommendations in the Mobile Tourist Application COMPASS,” Adaptive Hypermedia and Adaptive Web Based Systems, pp. 235-244, Aug. 2004.
• 111. K. Verbert, H. Drachsler, N. Manouselis, M. Wolpers, R. Vuorikari, and E. Duval, “Dataset-Driven Research for Improving Recommender Systems for Learning,” Proc. First Int'l Conf. Learning Analytics and Knowledge (LAK '11), pp. 44-53, 2011.
• 112. K. Verbert, E. Duval, S.N. Lindstaedt, and D. Gillet, “Context-Aware Recommender Systems,” J. Universal Computer Science, vol. 16, no. 16, pp. 2175-2178, 2010.
• 113. A. Vetro, “MPEG-21 Digital Item Adaptation: Enabling Universal Multimedia Access,” IEEE Multimedia, vol. 11, no. 1, pp. 84-87, Jan.-Mar. 2004.
• 114. R. Vuorikari, N. Manouselis, and E. Duval, J. Digital Information, Special Issue on Social Information Retrieval for Technology Enhanced Learning, vol. 10, no. 2, 2009.
• 115. Y.-K. Wang, “Context Awareness and Adaptation in Mobile Learning,” Proc. IEEE Second Int'l Workshop Wireless and Mobile Technologies in Education (WMTE '04), pp. 154-158, 2004.
• 116. “User Agent Profile (UAProf),” WAP Forum, 2006.
• 117. S.D. Warren, and L.D. Brandeis, “The Right to Privacy,” Harward Law Rev., vol. 4, no. 5, pp. 193-220, Dec. 1890.
• 118. S. Weibel, J. Kunze, C. Lagoze, and M. Wolf, “Dublin Core Metadata for Resource Discovery,” IETF RFC 2413, 1998.
• 119. F. Wild, F. Mödritscher, and S. Sigurdarson, “Designing for Change: Mash-Up Personal Learning Environments,” eLearning Papers, vol. 9, pp. 1-15, July 2008.
• 120. T. Winograd, “Architectures for Context,” Human-Computer Interaction, vol. 16, no. 2, pp. 401-419, 2001.
• 121. N. Winters, and S. Price, “Mobile HCI and the Learning Context: An Exploration,” Proc. Context in Mobile HCI Workshop at Mobile (HC '05), p. 5, 2005.
• 122. M. Wolpers, J. Najjar, K. Verbert, and E. Duval, “Tracking Actual Usage: The Attention Metadata Approach,” Educational Technology and Soc., vol. 10, no. 3, pp. 106-121, 2007.
• 123. J. Yau, and M. Joy, “A Context-Aware and Adaptive Learning Schedule Framework for Supporting Learners' Daily Routines,” Proc. Second Int'l Conf. Systems, pp. 31-37, 2007.
• 124. Z. Yu, X. Zhou, D. Zhang, C.-Y. Chin, X. Wang, and J. Men, “Supporting Context-Aware Media Recommendations for Smart Phones,” IEEE Pervasive Computing, vol. 5, no. 3, pp. 68-75, July-Sept. 2006.
• 125. Z. Yujie, and W. Licai, “Some Challenges for Context-aware Recommender Systems,” Proc. Fifth Int'l Conf. Computer Science and Education (ICCSE), pp. 362-365, 2010.
• 126. S. Zhao, M.X. Zhou, Q. Yuan, X. Zhang, W. Zheng, and R. Fu, “Who Is Talking about What: Social Map-Based Recommendation for Content-Centric Social Websites,” Proc. Fourth ACM Conf. Recommender Systems (RecSys '10), pp. 143-150, 2010.
• 127. X. Zhao, F. Anma, T. Ninomiya, and T. Okamoto, “Personalized Adaptive Content System for Context-aware Mobile Learning,” Int'l J. Computer Science and Network Security, vol. 8, no. 8, pp. 153-161, 2008.
• 128. R. Zhou, and K. Rechert, “Personalization for Location-Based E-Learning,” Proc. Int'l Conf. Next Generation Mobile Applications, Services and Technologies (NGMAST '08), pp. 247-253, Sept. 2008.
• 129. A. Zimmermann, A. Lorenz, and R. Oppermann, “An Operational Definition of Context,” Proc. Sixth Int'l and Interdisciplinary Conf. Modeling and Using Context (CONTEXT '07), pp. 558-571, 2007.