The Community for Technology Leaders

An Experimental Assessment of the Use of Cognitive and Affective Factors in Adaptive Educational Hypermedia

Nikos Tsianos
Zacharias Lekkas
Panagiotis Germanakos
Costas Mourlas
George Samaras

Pages: pp. 249-258

Abstract—In order to assess the positive effect and validity of personalization on the basis of users' cognitive and emotional characteristics, this study presents three subsequent experiments. The first experiment explores the relationship of cognitive style and users' eye gaze behavior as to validate this specific psychological construct in the context of educational hypermedia. The second and third experiments present the effect of a set of human factors (cognitive style, visual working memory span, control/speed of processing, and anxiety) in an adaptive educational system. The eye tracking experiment demonstrated that eye gaze patterns are robustly related to cognitive style ( ${\rm n} = 21$ ), while matching the instructional style to users' characteristics was revealed to be statistically significant in optimizing users' performance ( ${\rm n} = 219$ ), with the exception of control/speed of processing. Based on this empirical assessment, this paper argues that individual differences at this intrinsic level are important and adaptation on these parameters through personalization technologies may have a positive effect on learning performance.

Index Terms—Adaptive hypermedia, computer-assisted instruction, psychology, human factors, personalization.


The notion of personalization and the development of adaptive hypermedia [ 1], [ 2] have indeed generated research in the area of e-learning, and corresponding educational systems have been developed, such as MOT and WHURLE [ 3], INSPIRE [ 4], the CS383 courseware [ 5], and Arthur [ 6], to name a few. A common characteristic of these applications is that learning style theories have been opted for as personalization parameters, even though researchers from the field of education express concerns regarding the use of these constructs [ 7], [ 8].

Nevertheless, this popularity of learning and cognitive style theories in user/learner profiling could perhaps be attributed to the fact that the proposed typologies are viable for implementation in hypermedia environments. On the contrary, educational and psychological theories that introduce terms such as attention, perception, memory, reading processes, language comprehension, thinking, and reasoning [ 9], are far more complex and profound in order to be mapped in a hypermedia setting.

Therefore, even though the entire spectrum of individual differences undoubtedly includes the aforementioned human factors, learning and cognitive style theories seem to still have a predominant role in the field of adaptive hypermedia research. The function of these typologies as " $\ldots$ an important interface at the border of personality and cognition" [ 10] is certainly of importance, but an approach that disregards the rest of the human factors involved in information processing would be inadequate, at least in search of a "significant difference."

In search of a model that combines the construct of cognitive style with other human information processing parameters, the authors have introduced a user/learner model that consists of three independent dimensions: 1) Cognitive Style, 2) Cognitive Processing Efficiency, and 3) Emotional Processing [ 11]. The first dimension is unitary, whereas Cognitive Processing Efficiency includes 1) Visual Working Memory Span (VWMS) [ 12] and 2) an aggregation of speed and control of information processing and visual attention [ 13]. The emotional aspect of the model focuses on different aspects of anxiety [ 14], [ 15], [ 16] and self-regulation. Fig. 1 illustrates this proposed user model that we refer to as "User Perceptual Preference Characteristics (UPPC)."


Figure    Fig. 1. User perceptual preference characteristics.

A corresponding adaptive hypermedia system has been built around this model [ 17], and there is a continuing process of evaluating our approach and reforming both the theoretical model and the system. This paper presents results that are gathered from experiments conducted throughout the assessment procedure, in order to clarify at some extent whether this combination of human factors is of importance in the area of educational adaptive hypermedia.

Theoretical Background

The rationale behind opting for the parameters that comprise our proposed user/learner profiling model has been thoroughly presented in previous publications [ 18]. In sum, besides satisfying the scientific criteria, a main prerequisite is the possibility to integrate the theories that are involved into a hypermedia system.

2.1 User Perceptual Preference Characteristics

2.1.1 Cognitive Style

First, cognitive rather than learning style has been used due to the fact that the latter is "a construct that, by definition, is not stable—it was grounded in process, and therefore, susceptible to rapid change" [ 19]. Moreover, we are interested in individual information processing parameters, whereas the social implications of other learning typologies are not examined at this stage of research.

Specifically, Riding and Cheema's Cognitive Style Analysis (CSA) has been opted for. The CSA is derived from a factor analytic approach on previous cognitive style theories, summarizing a number of different yet highly correlated constructs in two distinct independent dimensions [ 20]. This covers a wide spectrum of the former cognition-based style typologies, without going into unnecessary depth for the needs of hypermedia education. The dimensions are the wholist/analyst and the imager/verbalizer; the former is related to the structure and amount of learner control, while the latter to the type of resources that are presented in order to provide the necessary educational information.

2.1.2 Cognitive Processing Efficiency

Primarily, in search of a more coherent approach, the term of working memory [ 21] has also been introduced in our model as a personalization factor. A brief description of the working memory system is that it consists of the central executive that controls the two slave systems (visuospatial sketchpad and phonological loop) plus the episodic buffer that provides a temporary interface between the slave systems and the Long Term Memory [ 22]. Since Web environments are predominantly visual, we have focused on visual working memory [ 23].

Each individual has a specific and restricted memory span. Our system takes into account each users' VWMS, altering the amount of simultaneously presented information. The aim is to decrease the possibility of cognitive load in a hypermedia environment [ 24].

In parallel to VWMS, a number of other individuals' "cognitive processing efficiency" parameters are also measured. This term refers to "hardware" functions of the brain, based on Demetriou's architecture of the mind [ 25]. It is not a unitary concept, but an aggregation of learners' abilities: 1) control of processing (refers to the processes that identify and register goal-relevant information and block out dominant or appealing but actually irrelevant information), 2) speed of processing (refers to the maximum speed at which a given mental act may be efficiently executed), and 3) visual attention (based on the empirically validated assumption that when a person is performing a cognitive task, while watching a display, the location of his/her gaze corresponds to the symbol currently being processed in working memory and, moreover, that the eye naturally focuses on areas that are most likely to be informative).

2.1.3 Emotional Processing

An effort to take into account learners' emotional state has also been carried out. Our approach is entirely differentiated from affective computing [ 26], since we have focused exclusively on learners' levels of anxiety and their ability to control their emotions. At this level, we make use of the term "Emotional Processing," which includes 1) Emotional Arousal, which is the capacity of a human being to sense and experience specific emotional situations—with anxiety [ 14], [ 15], [ 16] as the main indication of emotional arousal and 2) Emotion Regulation, which is the way that an individual perceives and controls his emotions [ 27], [ 28], [ 29], [ 30].

2.2 Eye Tracking Differences in Cognitive Style

A different perspective on the issue of the importance of human factors, other than measuring the effect of personalization, could be the relationship of users' actual behavior in a hypermedia environment with theories that raise the issue of individual preferences and differences. The notion that there are individual differences in eye movement behavior in information processing has already been supported at a cultural level [ 31], at the level of gender differences [ 32], and even in relation to cognitive style (verbal-analytic versus spatial-holistic) [ 33].

To that direction, our intention was to cross-check the measurement of cognitive style with eye gaze patterns of users in a hypermedia environment, according to the data provided by an eye tracker. The motivation for this examination is that the construct of style has been originally developed for traditional educational settings, and therefore, it is ambiguous whether the preferences of learners in Web environments would be identified by such tools. Also, as mentioned above, style theories have been questioned by educationists, and it would be valuable to resolve at some extent the validity issue with an external measurement.

2.3 Implications and Research Questions

The greatest challenge is to extract from the above-mentioned theories the corresponding implications for an educational hypermedia environment. In order to establish a set of rules that would assist designers of educational applications to incorporate personalization based on human factors, we considered that an empirical evaluation of such rules, at an explorative level, would be necessary.

The eye tracking experiment did not require a personalization process, but served as a method of partial validation of the first dimension of the proposed user model. As it concerns cognitive style and VWMS, an elaboration of the personalization rules is rather explicit. On the contrary, for example, in order to experimentally assess the effect of personalization on individuals' cognitive processing speed, we imposed time limitations over the learning process; by manipulating time limits, we tried to explore whether there would be an effect on how learners perform (level of comprehension). Also, in the ambiguous field of emotions, the aesthetic enhancement of the system was expected to have a positive effect on highly anxious learners. Therefore, our research questions may be set forth as follows:

  1. Is the construct of cognitive style validated by its actual relationship with users' eye gaze patterns?
  2. Does matching online instructional style to users' cognitive style have a significant effect on their performance?
  3. Does providing the right amount of information according to users' VWMS promote effective information processing?
  4. Is the available amount of time in a learning setting related to users' cognitive processing efficiency, impacting on comprehension and performance?
  5. Is there any correlation between learners' performance and their levels of anxiety and emotional regulation? In that case, is the aesthetic enhancement of the environment supportive?

In order to elucidate the above-mentioned issues, we conducted three subsequent experiments in parallel with the development of the system, while the assessment methods were derived from the field of experimental psychology. Our efforts were also focused on "translating" our theoretical framework into personalization rules; it should be mentioned that the mapping of such a user profile on a hypermedia system is a complex procedure, due to the nonlinearity and the unforeseen interactions of human traits. However, this is the main challenge of our research work—the successful integration of theory into practice in a coherent way.


3.1 Eye Tracking Experiment

The methodology of the eye tracking experiment did not involve any personalization processes. It should be clarified that only the imager-verbalizer axis of the CSA was examined, because this dimension focuses on the preference for textual or visual information, which consequently can be mapped as learning objects in an adaptive system. In contrast, the analytic/wholistic dimension describes a rather intrinsic organization of information that is rather inappropriate, in our opinion, to measure with such an experimental method.

The experimental design was between participants. Each individual took the CSA test for the assessment of the imager/verbalizer axis of cognitive style and afterward participated in an online learning course about algorithms in computer science. The number of participants was 21 (12 females and nine males); they all were students from the University of Cyprus. Their mean age was 23, ranging from 20 to 26. It turned out that they were roughly equally distributed in groups according to their cognitive style: seven imagers, eight verbalizers, and six intermediates.

During the online course, an eye-tracker system recorded learners' eye fixations and tracking on the educational content. The procedure took place in a controlled environment, a computer lab, and each participant was alone during the experiment. It should be noted that the learning content consisted of a balanced (to the extent that would allow the delivery of the necessary information) number of visual and textual objects.

The dependent variables of our analysis were the calculated ratios of eye 1) fixations and 2) tracking. Fixation is defined as the focus on learning objects (eye gaze pause). In a sense, each visual or textual object may be considered as an Area of Interest (AOI), though it was not technically possible to measure the depth of focus on each object. Tracking refers to search patterns on the screen without focusing on specific areas.

The ratio in both cases is an image to text ratio on a scale of 1 to 10, with higher positioning on the scale implying a preference for images. For example, a ratio value of 10 (though realistically not possible) would mean that the user fixated exclusively on images (fixation ratio) or that he/she tracked only visual objects on the screen, avoiding anything else (tracking ratio). The positioning on these ratio scales displays a tendency that would ideally represent the users' style preference.

The ratios were calculated from the raw data exported by the system. These data included eye 1) scan paths on the x-y axes (tracking) and 2) number and duration of fixations and pupil dilations (fixations), all in relation to the visual and textual objects on screen (or areas). Also, fixations and tracking on the navigation menu of the learning environments were measured, though proved to be insignificant in this experiment.

The reason for converting fixations and tracking data into images to text ratios was that the CSA test is based on exactly such a ratio; thus, this conversion would allow the comparison of user behavior according to their style.

The duration (millisecond) of the experiment was measured, but only for a number of participants, due to an internal technical error. It should be noted that users were free to allocate as much time as they wanted to each Web page of the lesson.

The component that gathers and presents the eye tracking data was developed within and integrated in the AdaptiveWeb system. 1 A visual representation of users' tracking behavior was also available; though the differences between learners will be discussed in Section 4 of this paper, Fig. 2 illustrates how an imager differs in his eye tracking patterns from a verbalizer.

Graphic: Fig. 2. Differences in eye tracking of an imager (above) and a verbalizer (below) within the same educational Web page.

Figure    Fig. 2. Differences in eye tracking of an imager (above) and a verbalizer (below) within the same educational Web page.

3.2 Personalization Experiments

The experimental design of the two subsequent personalization experiments was a between participants memory test. In the first experiment, the effect of personalization on cognitive style was examined. The second experiment involved personalization on VWMS, speed of processing, and anxiety.

The procedure in both experiments was the same: users created their profiles through a series of psychometric tests, logged into the system, took the online course, and afterward participated in an online exam assessing their level of comprehension. The order of the psychometric tests was not predetermined, since users were free to choose which test to take by clicking on the corresponding links. As soon as they created their profiles by taking all tests, they were navigated to the online course, and upon the completion of the lesson, they clicked on a link that initiated a multiple choice exam on the subject they were taught.

The dependent variable was users' score at the memory test. The procedure was an in-class simultaneous learning activity, with students divided in groups of approximately 15 participants, and took place in computer labs of the Universities of Athens and Cyprus.

The total number of participants in these experiments was 219; all of them were students in the Universities of Athens and Cyprus, and their age varied from 17 to 22 with a mean age of 19. About 70 percent of the participants were females and 30 percent were males. The first experiment took place at the University of Cyprus, while the second was conducted at the University of Athens. The number of participants in each experiment was 138 and 81, respectively.

The academic subject was a computer science course on algorithms and flowcharts, which was chosen because the students of the departments, where the experiments took place, have absolutely no experience or previous knowledge on programming, due to the theoretical orientation of their curriculum. Participation in the experiments was voluntary, but most students were willing to take the course, as it provided additional help for them on a difficult academic subject.

In the first experiment, almost half of the participants, who had a style preference, received an online course that was personalized on their cognitive style, while the other half received a course that didn't coincide with their profiles (match/mismatch condition). It should be clarified that those who were classified as intermediates in both dimensions of the CSA were treated as a control group that received a balanced environment; in the second experiment, only users with high levels of anxiety were allocated to match/mismatch conditions, while those with normal/low levels of anxiety were treated as a control group. Also, in the case of VWMS, medium and high span users were not placed in a mismatch condition with additional content because that would not serve any purpose at this stage.

The allocation to the match/mismatch condition was quasirandom; each user that logged in was placed in the opposite from the previous user group, with the exception of intermediates and the control group. In the second experiment, a user could as well be in a matched condition regarding VWMS, but in a mismatched condition as it concerns speed of processing and anxiety. Thus, all combinations were possible and correspondingly considered in the statistical analysis.

3.3 Materials

Eye tracker: "Video Eyetracker Toolbox," manufactured by Cambridge Research Systems Ltd. It consists of a 50 Hz camera, an adjustable tower that stabilizes the head of the participant during the measurements, and the "Picolo Frame Grabber" PCI card. A calibration procedure for each participant was required in order to increase accuracy by minimizing errors and deviations. The manufacturer provided a library of Matlab instructions, the CRS Toolbox for Matlab, which was used for signal processing and data exporting.

Cognitive style: Riding's Cognitive Style Analysis, standardized in Greek and implemented in the .NET platform. This tool involves three phases: in the first phase, a series of sentences is presented to the participants; they are asked to respond whether the meaning of the sentence makes sense or not (true/false), by clicking on the corresponding computer key. Half of the 24 sentences require the individual to form a visual representation, while the other 12 involve abstract processing. The response time is measured and users are categorized accordingly (imagers/intermediates/verbalizers). The second and third phases involve the presentation of graphical schemes, examining individuals ability to integrate or distinguish parts of figures, classifying them as wholists, intermediates, or analysts.

Visual working memory span: Visuospatial working memory test [ 34], first developed on the E-prime platform (a software tool for developing psychometrical applications) and subsequently implemented in the .NET platform. A total of 21 figures are presented to the user, with increasing complexity as the test progresses. Each figure is presented for about 2 seconds before it disappears, and thereupon, the user has to identify the figure among five highly resembling ones. Each correct answer allows the user to continue to the next figure, until he fails to retain the visual information due to the increased complexity. Individuals are classified accordingly, with respect to their ability.

Cognitive processing efficiency: Speed and accuracy task-based tests that assess control of processing speed of processing and visual attention. Originally developed in the E-prime platform, these tests were also implemented on the .NET platform. The control and speed of processing tasks involve the presentation of words representing colors, in colored text. Participants have either to identify as fast as possible the meaning of the word regardless of the color of the text, or vice versa, by pressing designated keys. The visual attention test requires individuals to decide if a given stimulus is included within a complex of numerous resembling stimuli. The number of correct answers and the reaction time are measured in all three tests, classifying users according to their abilities.

Core (general) anxiety: Spielberger's State-Trait Anxiety Inventory (STAI)—10 items (Only the trait scale was used) [ 16].

Application-Specific Anxiety: Cassady's Cognitive Test Anxiety scale—27 items [ 15].

Current anxiety: Self-reported measure of state anxiety taken during the assessment phase of the experiment, in time slots of every 10 minutes—six time slots. An "anxiety bar" was presented on screen, and users were able to position themselves on a 1-5 scale. This measurement was mainly used in order to examine whether users are able to provide valid feedback regarding their emotional state, perhaps allowing the removal of some of the psychometric tests of the profiling procedure in the future.

Emotional regulation: This questionnaire was developed by us, and includes questions derived from emotional intelligence, self-regulation, and self-efficacy tests; Cronbach's α that indicates scale reliability reaches 0.718. The concept of using this questionnaire was also explorative.

Memory test: The memory test resembled the examination of the computer science academic course exam at the Department of Communication and Mass Media of the University of Athens. It included 14 multiple choice questions, with increasing difficulty. The first question was about very basic concepts of algorithms and components of flowcharts, while the following required learners to put into practice the newly acquired knowledge about algorithms, in order to complete missing parts of more complex flowcharts. As expected, the participants (as social studies students) did not perform very well, since the overall mean score was 57.29 percent.

3.4 Personalization Rules

A short description of the way that our system adapts to users' preferences is needed in order to provide the reader an insight to our research framework. The full system is available at

3.4.1 Cognitive Style

There are two dimensions of users' cognitive style that are mapped in the educational environment: the wholist/analyst scale affects the structure, the navigational patterns, and the amount of learner control, whereas the imager/verbalizer is related to the textual or graphical representation of information (where possible of course). Specifically, the differences in the analyst and wholist condition were the following:

Analyst environment: The navigation slide-in panel (located at the top of the screen) allowed users to navigate freely in the environment, with visible links for all the Web pages of the course. The links that were clicked were annotated, helping users to monitor their path. Additionally, a separate index of concepts was provided, enabling analysts to form an understanding of the course according to their own mode of information processing, increasing their level of control over the course. Also, links on terms within each Web page were provided, opening popup windows with definitions.

Wholist environment: The course was organized in sections, and the navigation slide-in panel hosted an overview of each section. Users were able to see descriptions of all Web pages within each section, as a form of external guidance and framing of the course. Their navigational path was essentially sequential, but they could visit previously seen pages, which were annotated on the navigational panel. Instead of providing links to definitions and an index of concepts, rollover text bubbles were included within the Web pages with corresponding explanations and definitions, maximizing the coherence of each section and minimizing the possibility of disorientation.

In sum, the difference mainly lies in the amount of external guidance and framing of the course, as suggested by the CSA theory.

In the case of imagers, diagrams and images were used as learning objects, instead of verbal descriptions. Where possible, instead of describing an algorithm process with text, a figure was provided. In the case of verbalizers, most of the learning material was textual, with the exception of flowcharts.

Finally, as it concerns the intermediate condition, an equal number of visual and textual learning objects were provided, while the navigational panel resembled the wholistic, albeit with the capability of free navigation in the Web pages of the course. Since according to theory intermediates are supposed to learn equally well in all conditions, the main idea was to provide a balanced environment with elements from all possible environments.

3.4.2 VWMS

Each user's visual working memory span is measured and classified. Users that have low levels of VWMS receive segmented content that unfolds gradually. The main idea is to alleviate the possibility of cognitive overload and is based on the notion that information processing is not sequential but parallel—therefore, the segmentation in clear-cut chunks may assist users' with low VWMS. Specifically, the Web pages were divided in logically coherent sections (from top to bottom), and users with low VWMS had to click on a link in order for the rest of the contents to be presented.

3.4.3 Cognitive Processing Efficiency

Since the term "efficiency" refers mainly to speed, in order to distinguish whether there is a relationship between users' ability and the time required to complete an online course, we set different time limits for each category of learners. Based on a previously conducted pilot study, users require approximately 35 minutes to complete the entire course. Therefore, users with low and medium speed were given 45 and 38 minutes, respectively, in the matched condition, and only 28 in the mismatched condition. Users with high speed were given 30 minutes in the matched condition and 25 minutes in the mismatched condition. A counter on top of the screen informed learners about the remaining time; we expected that framing the time allocated to the course would affect the amount of cognitive effort of the participants, and differences in performance would be revealed according to their ability.

3.4.4 Anxiety

In these first experiments, we were based on the results of the "core" and "application specific" anxiety questionnaires. The measurement of "current" anxiety and "emotion regulation" was used for exploratory reasons and for investigating the validity of such constructs—which is beyond the scope of this paper. In the case of high levels of anxiety (on behalf of the user), we provided aesthetical enhancement of the environment and further annotations; in a sense, the aesthetical aspect predominates over functionality (in terms of font size, colors, and annotations). This personalization rule was inspired by a design approach supporting that good aesthetics may assist in the alleviation of the effect of negative affect (such as anxiety) in cognitive processes [ 35].

Fig. 3 shows the same Web page of the online course, albeit differentiated (personalized) according to user preferences. The screenshot above is addressed to an analyst/verbalizer. Thus, the navigation menu on top allows free navigation and includes an index of concepts (see the popup menu that presents a definition), and on the main area of the page, the information is verbal.

Fig. 3. Two versions of the same Web page (analyst/verbalizer above, wholist/imager below) differentiated according to personalization techniques.

On the contrary, the screenshot below is provided to a wholist/imager, with high levels of anxiety. The navigation menu is sequential, with short overviews of each section, framing more coherently the entire course. In this page, information is conveyed through visual representations. Moreover, the popup window provides additional support, and though it is not visible in this screenshot, the above text is aesthetically enhanced. In the case of low VWMS, the page is segmented and presented in distinct phases, by clicking of the user.


4.1 Experiment I

The aim of the eye tracking experiment, as mentioned, was to investigate whether the imager/verbalizer axis of cognitive style is related to the eye gaze behavior of users in a hypermedia environment. This would provide additional validity to the concept of using cognitive style as a personalization parameter in adaptive systems: If users indeed behave according to their style preference, then content selection should accordingly be affected.

Since the variance of users' ratios of images to text fixations was homogeneous (Levene's ${\rm statistic}_{(2,17)} = 0.845{,}$${\rm p.} = 0.446$ ), one way analysis of variance was performed on the data. Indeed, there was a linear differentiation in users' fixations with respect to their cognitive style; imagers focused more on images, verbalizers on texts, and intermediates were placed in the middle. This difference is statistically significant: ${\rm F}_{(2,18)} = 6.074, {\rm p}. = 0.01$ . The actual differences in the calculated images to text ratio are shown in Fig. 4.

Graphic: Fig. 4. Calculated images to text ratios of eye fixations on a scale from 1 to 10 (textual to visual preference).

Figure    Fig. 4. Calculated images to text ratios of eye fixations on a scale from 1 to 10 (textual to visual preference).

Exactly the same applies with the calculated ratio of images to text tracking (imagers: 5.82, intermediates: 4.80, verbalizers: 4.27), albeit with even greater statistical effect and significance: ${\rm F}_{(2,18)} = 10.411, {\rm p}. = 0.001$ . Fixation and tracking on the menus of the Web Interface are more or less the same among categories with no differences observed.

As it concerns the time that users allocated to the entire course, which was unfortunately available only for 12 out of 21 participants due to an internal technical error, there was also an effect of cognitive style: imagers and intermediates devoted about the same amount of time, while verbalizers spent considerably less amount of time. Post hoc analysis of variance has shown that this difference on behalf of verbalizers is statistically significant compared to both imagers and verbalizers (see Table 1).

Table 1. Differences in Participants' Allocation of Time to the Online Course

The explanation of this finding is not as clear-cut as with the aforementioned results. It could be argued that the processing of visual stimuli and the interpretation of the meanings that are conveyed are a more time-consuming cognitive process; since verbalizers have a clear preference toward text, they allocate less time in the processing of text. However, according to Riding's theory, imagers also focus on textual resources, while the reverse is not observed; therefore, more time is consumed. With the case of intermediates on the other hand, it makes much sense that equal processing of all objects would require further allocation of time.

It should finally be mentioned that no gender differences were observed in any of the measurements, which was also the case with the personalization experiments.

Therefore, it is clearly indicated that the visual behavior of users in a Web environment, according to the eye-tracker measurements, depends on their cognitive style. These results also provide a form of validation for the effect of style in information processing within the context of hypermedia. This is of course a preliminary study conducted with a small number of participants, and it has to be replicated. Still, since the results are statistically robust, we believe that style could be considered as an important personalization factor in system design.

4.2 Experiment II

The first personalization experiment focused only on the construct of cognitive style as a personalization factor. Besides users' cognitive style, their VWMS was also included in their profile as a control variable. Participants had either a cognitive style preference or were classified as intermediates (no cognitive style preference). The latter was treated as a control group that has no need for a personalized environment, and received the intermediate balanced course. The remaining users were randomly allocated to a "matched" or "mismatched" group of learners. If cognitive style is of any importance, these two groups should have statistically significant different scores.

A $3 \times 3$ analysis of variance was performed (three groups of cognitive style and three groups of VWMS), since the variance of the dependent variable was homogeneous, in order not only to assess the effect of matching the environment to users' style, but also to control for the effect of VWMS. Indeed, learners that received matched environment ( ${\rm n} = 53$ ) outperformed mismatched learners ( ${\rm n} = 61): {\rm F}_{(2,137)} = 4.395, {\rm p} = 0.014$ . There was no main effect of VWMS, or interaction with cognitive style.

The group scores were 66.53 percent in the matched condition and 57.79 percent in the mismatched. Intermediates had a mean score of 58.58 percent. Post hoc analysis (see Table 2) has demonstrated that a difference actually exists only between matched and mismatched learners; intermediates ( ${\rm n} = 24$ ) do not seem to vary from the former groups, and they are more dispersed. Perhaps in the absence of a cognitive style preference, other factors may have a stronger effect on learners' performance in a hypermedia environment (such as those involved in the next experiment).

Table 2. Post Hoc Analysis of Learners' Scores in All Three Conditions of Experiment I

In sum, the argument that personalization on the basis of cognitive style may improve learners' information processing in a hypermedia environment can be supported; those who demonstrate cognitive style preference are indeed benefited. The mean difference of approximately 9 points should also be evaluated in relation to the small variation of participants' scores.

4.3 Experiment III

By controlling the cognitive style parameter (environment matched to this preference), users received either matched or mismatched environment in regard to each separate factor of our model (VWMS, cognitive processing efficiency, and level of anxiety). In order to distinguish the effects of matching/mismatching each factor, since the distribution of the sample was homogenous, a $2\times 2\times 3$ analysis of variance was performed; there were three groups of learners in the emotional categorization, since users with low levels of anxiety were treated as a control group. The composition of groups was the following:

  1. 19 mismatched low VWMS learners,
  2. 62 matched VWMS learners,
  3. 42 mismatched CPSE learners,
  4. 39 matched CPSE learners,
  5. 29 mismatched anxious learners,
  6. 22 matched anxious learners, and
  7. 30 participants in the emotional control group.

There was a significant main effect of matching the instructional style to users' VWMS ( ${\rm F} = _{(1,80)} = 4.501,{\rm p} \;{=}$$0.037$ ), and to their levels of anxiety ( ${\rm F} =_{(2,80)} = 3.128,{\rm p} \;{=}$$0.05$ ). Cognitive processing efficiency was not found to have a main effect on score or interaction with the other parameters. The differences in mean scores are demonstrated in Tables 3 and 4.

Table 3. Differences of Mean Scores in the Matched and Mismatched Conditions with Regard to Users' Levels of Anxiety

Table 4. Differences of Mean Scores in the Matched and Mismatched Conditions with Regard to Users' VWMS

Post hoc analysis of the differences between the three anxiety groups has demonstrated that the difference is statistically significant between matched and mismatched anxious users, with the control group scoring in between. The relatively moderate sample of the second experiment necessarily limits the level of analysis that can be applied. However, it is certainly encouraging the fact that there were found significant differences in learners' scores that can be attributed to the importance of taking into account factors such as those included in our approach; it seems that designing educational hypermedia with such factors left at chance may hamper the performance of users.

The finding that cognitive processing efficiency didn't affect users' performance may be explained by the fact that there were no real-time tasks involved in our online course; therefore, it would be difficult for this kind of individual differences to be revealed. It is also possible that a different approach to the personalization process or the experimental design could have provided different results.

Our methodology in this first endeavor to investigate the role of these human factors is of course not exhaustive. VWMS has been proved to be of importance as a parameter, and a certain effect of aesthetics has been demonstrated, but further empirical research is undoubtedly required.

Discussion and Conclusions

The results that are presented above may provide a good argument for incorporating human factors in educational adaptive hypermedia. More specifically, our research questions were answered as follows:

  1. Users' eye gaze patterns are indeed related to their cognitive style.
  2. Matching the instructional style to users' cognitive style promotes the learning performance.
  3. Segmenting the simultaneously presented information according to learners' VWMS benefits the information processing.
  4. Cognitive processing efficiency does not have an effect, nor is related to the amount of available time.
  5. The aesthetical enhancement of the environment is correlated to the increase of performance of anxious learners.

The eye-tracker study provided support for research on the construct of cognitive style in the context of adaptive hypermedia. The choice of the specific theory, its application in the hypermedia environment, and the measurement process were validated by an external measure, establishing a relationship with individuals' actual behavior in a Web environment.

As it concerns the personalization experiments, the findings are quite consistent with the psychological theories that are referred to in our framework and it seems that the difficult task of translating these theories into adaptation rules was at some extent successful. The differences in scores are not extreme, but an aggregation of these increases in performance may as well imply a far more efficient learning procedure. Our next step is the provision of educational environments that are fully adapted or nonpersonalized (baseline), and the comparison of these two conditions. Our expectations, as demonstrated by the above-mentioned findings, are that the differences will be far greater than marginal, also taking under consideration the results of the control groups that were used in some of the conditions of our experiments (see Section 4).

At this point, we should mention that there are several limitations in our study. First of all, the second personalization experiment was conducted with a numerically moderate sample. Though it is quite positive that it yielded statistically significant results, we are aware that these findings must be repeatedly confirmed. We have already designed and conducted a replication study with a larger sample and we are in the process of analyzing our data; the role of VWMS is found to be highly important.

Besides that, in terms of methodology, the unequal distribution of learners in some of the groups is also an issue of consideration, and a more elaborated sampling procedure is necessary in replicating and further validating these results. Moreover, additional control groups and conditions should be incorporated in the design of future experiments, controlling for the effect of the numerous parameters involved in the evaluation of our information processing model.

The sample issue also applies to the eye tracking experiment; the number of individuals is small in statistical terms, though this is common for studies that involve the use of tools such as the eye tracker.

Second, our experiments were conducted within a specific adaptive system, which may as well not be considered as representative of all possible hypermedia applications. The integration of our theories seems to be viable in this specific educational hypermedia system, but it should be nevertheless tested in other e-learning procedures. We have clarified that our interest is on individual information processing differences, and the interaction of these human factors with other parameters (predominantly socially oriented) should be examined.

In terms of future work, the latest addition to our model is the measurement of both subsystems of working memory, by adding the phonological loop span and the central executive function to our model. Our latest experimental approaches provide results that indicate the predominance of working memory span in information processing and our interest is currently focused on enhancing the corresponding personalization techniques. We are also working on providing physiological input to the system about the "current anxiety" levels of the user, by integrating biofeedback sensors in a computer mouse. As soon as signal processing problems are resolved, we expect that a real-time robust measure will be added in the user profiling procedure.

Nevertheless, the feedback that this study has provided us is encouraging, and in our opinion, there is quite some depth in personalization on individual differences. We certainly not consider our model as a rigorous construct, but as a framework that is driven by experimental research and methodology. The value of this approach for educational hypermedia designers is that the emphasis is placed upon the learner, exclusively on the level of a better understanding of the educational content. Since adaptive technologies offer the possibility of a highly personalized e-learning course, it would be rather obscure to not place users' intrinsic characteristics in the center of such an endeavor.


The project is cofunded by the Cyprus Research Foundation under the project EKPAIDEION ( $\Pi \Lambda$ HPO/0506/17).


About the Authors

Bio Graphic
Nikos Tsianos received the bachelor's degree in communication and media studies and the MSc degree in political communication and new technologies from the University of Athens, Greece, in 2003 and 2005, respectively. He is currently working toward the PhD degree as a research assistant at the Laboratory of New Technologies, Department of Communication and Mass Media. He participates in EU and national funded projects, and also collaborates on research projects with the Department of Computer Science, University of Cyprus. He has numerous publications on the field of adaptive educational hypermedia, including a Best Student Paper Award (Adaptive Hypermedia 2008 conference), and has coedited a book on human factors in e-learning. His main research interest is the effect of individual differences on information processing within the context of the Web and the corresponding personalization techniques. He has also conducted research in the field of locative media.
Bio Graphic
Zacharias Lekkas received the BA degree in philosophy and psychology from the University of Athens, and the PGDip degree in psychology and the MSc degree in occupational psychology from the University of Nottingham. He is currently working toward the PhD degree as a research associate at the Department of Communication and Media Studies at the University of Athens. He is interested in the role of emotions in Web-based educational systems, and has conducted empirical research on the effect of human factors such as anxiety, emotional moderation, emotional intelligence, self-efficacy, etc. Additionally, his research interests include the field of decision making support in adaptive hypermedia and the design of personalized training systems. His work has been published in conferences, journals, and edited books, and he is one of the awarded authors (Best Student Paper Award) at the Adaptive Hypermedia 2008 conference.
Bio Graphic
Panagiotis Germanakos received the BSc degree in computer science in 1998, the HND diploma of Technician Engineer degree in computer studies in 1996, the MSc degree in international marketing management from the Leeds University Business School in 1999, and the PhD degree from the University of Athens in 2008. Since 2008, he has been an assistant professor in the Department of Management and MIS at the University of Nicosia. He is also a research scientist in the Semantic and Cognitive Research Adaptivity Technologies (SCRAT) Research Group at the Department of Computer Science, University of Cyprus (since 2003), and on the Faculty of Communication and Media Studies, University of Athens (since 2004). He has several publications, including coedited books, chapters, articles in journals, and conference contributions, including a Best Student Paper Award (Adaptive Hypermedia 2008 conference).
Bio Graphic
Costas Mourlas has been an assistant professor in the Department of Communication and Media Studies at the National and Kapodistrian University of Athens, Greece, since 2002. His main research interest is in the design and development of intelligent environments that provide adaptive and personalized context to the users according to their preferences, cognitive characteristics, and emotional state. He has several publications including edited books, chapters, articles in journals, and conference contributions. Furthermore, he has coordinated and actively participated in numerous national and EU-funded projects.
Bio Graphic
George Samaras is a professor in the Department of Computer Science at the University of Cyprus. He has a number of patents relating to transaction processing technology and numerous book chapters, and technical conference and journal publications. He has served as a proposal evaluator at the national and international level and has been regularly invited by the European Commission to serve as an external project evaluator and auditor, ESPRIT and IST Programs (FP5, FP6, FP7). His research interests include e-learning and e-Services, Web information retrieval, mobile/wireless computing, context-based services, personalization systems, and database systems.
61 ms
(Ver 3.x)