Abstract—A key design goal of mobile learning is that its built-in experiences are enjoyable and proactive, empowering the learner with the knowledge and ability to self-manage. This implies that the benefits and critical success factors of mobile learning activities should not simply be based only on learning performance, but also on learners' experiences so that we can assess the added qualities of these activities over and above rote knowledge acquisition. To empirically demonstrate this premise, we report on an assessment of flow experience in three different learning spaces, where learners explore a built environment as part of a simulated security guard training program. Our results seem to show that the true cost-benefit ratio of mobile learning may not be evident in measures of learning performance or rote knowledge acquisition alone, but that mobile learning activities could provide a better learning experience by providing the conditions for optimal flow experience. Further, in the spatial navigation tasks given in our experimental context, it can be seen that simple learning performance measures can be highly dependent on the learner's spatial capability, but this was not the case for flow experience measures.
"The quality of the imagination is to flow and not to freeze."—Ralph Waldo Emerson
Although there are many definitions of mobile learning, there is a common thread through these definitions that we somehow involve electronic mobile devices in designing learning activities, e.g., handheld devices, laptop, or netbook computers. Indeed these devices are sometimes not so much mobile as merely portable, in the sense that they are often used within the static confines of a classroom (e.g., [ 1
]). One important issue that mobile learning researchers must, therefore, address is what benefits can we measure when we utilize such devices for learning, which may then enable us to create new and better types of learning activity?
The benefits that have been asserted for different contexts of mobile learning are many. Some studies regard the use of mobile devices as being primarily an issue of access. Small portable devices are cheaper and more easily used in the classroom than large desktop computer systems, so they bring the (assumed) benefits of e-learning to a wider constituency [ 2
]. Some more ambitious projects leverage the true mobility of such devices to explore outdoor natural or built environments, e.g., [ 3
], [ 4
]. In many cases, these utilize the devices' abilities to interact with external inputs such as Global Positioning System (GPS) data and bar codes. While these initiatives are usually touted as successful, there are some issues of cost and benefit that might need to be carefully assessed. A large expensive mobile learning project may indeed be successful, but perhaps the same money could have been equally well spent on alternative (and more sustainable) learning experiences. A number of projects have integrated mobile devices into more ambitious or immersive infrastructures, both in the classroom, e.g., [ 5
] and in the wider environment, e.g., [ 6
]. This raises the issue of the affordances of different learning tools; if we assume that mobile learning uses mobile devices, to what extent are the affordances of these mobile devices truly providing added value over alternatives? In some cases, the affordances of simple materials such as paper documents may be equally compelling [ 7
]. There is also a danger that we rely on the disruptive nature [ 8
] of such devices in measuring our success. One or two key features, such as GPS, may tilt the balance toward a mobile device in a particular context. However the affordances of (cheaper) alternatives such as maps and e-learning with budget laptops may, perhaps, achieve similar learning outcomes in situations where there is no compelling requirement for the unique functionality of handheld mobile devices.
The aim of the empirical studies described in this paper was thus to try to identify some particular outcomes from mobile learning that could be seen as addressing the contribution of mobile devices, when compared to both more traditional alternatives and immersive e-learning tools. For this experiment, it was important that we incorporated some notion of learning (or device) affordance (in this case, traditional pedagogy versus pedagogy based on a mobile device) and also mobility outside the classroom (in this case, e-learning pedagogy versus mobile pedagogy). The main focus of our analysis was to apply the concept of flow experience [ 9
] to the process of learning with mobile devices. Hence, in our experimental design, we considered three types of learning systems through which learners could briefly explore a physical (or virtual) workspace to be trained as security guards for a specific workplace as part of a simulated training program. With this empirical and exploratory study, we argue that mobile learning activities can achieve the desired conditions for learners' optimal flow experience. As a consequence, one can see the benefits and limitations of the mobile learning environment and the choices we must make when designing learning (or device) affordances within it. From this perspective, we will be able to see how mobile technology and its affordances can provide an effective learning experience and thereby influence a trajectory of mobile learning curriculum development.
2. Performance and Experience in Mobile Learning
Learning has been characterized in a number of ways. For example, traditional constructivism emphasizes that learning is intrinsically internal and personal, involving the generation of new understanding and knowledge and active changes in conceptual understanding. More recent views of learning, e.g., [ 10
] have asserted that intellectual development is largely at the mercy of self-control, by which learners may find their own way to make a learning situation personalized and sensitized to them. Such sociocognitive perspectives place emphasis on learning as an active, social process, in which the learner is a central participant in his or her own learning process, and collaborative interactions are viewed as a key determinant of the content of learning activities [ 11
Given the learning benefits of social participation, many mobile learning projects are oriented toward collaborative outdoor learning, e.g., [ 12
] rather than more formal classroom learning activities, e.g., [ 13
]. This is perhaps exaggerated by the fact that such projects have tried to develop new environments where mobile devices offer added value to the learners' educational experience. These environments complement traditional and more formal education by providing a new range of possibilities to students of the digital generation. This kind of blended learning
is, therefore, intended to educate outside the classroom not by replacing the face-to-face class, but by complementing it, and providing different ways of learning, emphasizing the importance of context
]. From this perspective, mobile learning seems to cater for certain curricular areas, such as archeology (refer to wetec.csumb.edu) or tourism [ 15
], more than others.
Although there have been many different learning activities undertaken within mobile learning projects, previous studies have mostly measured learning performance using quantitative metrics such as "time on task," "task completion rate," "user satisfaction," and so forth. These traditional learning performance measures might in some circumstances give a useful approximation of the benefits of mobile learning activities, but it may be problematic to rely on time data, or tests administered after learning activities, in order to draw inferences about learning outcomes. There may be many pragmatic reasons to apply these so-called " easy-to-use
" learning performance measures, but we should at the very least be aware that learning outcomes with mobile systems might be affected by unique factors. For example, Doolittle et al. [ 16
] empirically demonstrated that learning with a mobile-based multimedia learning environment is sensitive to individual differences in attention, pinpointing human working memory as an essential factor in dictating the success of mobile learning, which implies that mobile learners would not be able to perform highly task-focused (or onerous) learning activities. Hence, one of the theoretical foundations we assume in this paper is that learning performance measures might need to be interpreted together with individual cognitive differences.
Of course, any educational or learning activity concerns itself with improving learning outcomes or performance, and this should be also true of mobile learning. However, mobile learning needs to be measured on more than simply enhancing rote knowledge acquisition, since it is not easy to observe any significant instant benefits from mobile learning against institutional pedagogy (or distance learning). Instead, as many of its proponents claim, mobile learning is largely at the mercy of self-determination or self-management (i.e., self-control), which requires learners to find their own way to make the learning situation work for them. If this is the case, a logical research question, which is central to this paper, is how we can assess these new forms of learning activities.
A practical successful mobile learning example may suggest a possible answer. In Japan, the Nintendo DS console has been widely used as a mobile multimedia-based English learning tool, holding out the promise of fun and playful access to an otherwise frustrating learning activity beyond traditional institutional boundaries ( Cited in the Guardian Weekly, August 8, 2008). In this nation-wide project, the learning outcomes were simply measured by counting how many English words had been remembered by those using the Nintendo DS, as opposed to others who had not used it, and this was claimed as solid evidence of a preference for mobile learning. However, although the Japanese students were able to remember more English vocabulary with this game-based mobile learning system, this fact alone did not answer the question of what made this happen, in particular, how self-determination or self-management with the game console may be important factors.
A possible explanation for this would be that the best learning moments usually occur when a learner is stretched to the limit in a voluntary effort to accomplish something difficult, challenging and worthwhile [ 17
], consistently producing flow experience
] that sustains the learner's efforts to achieve something. In effect, to understand why some learning activities are more effective than others, and are able to sustain motivation, one should review the concept of flow experience as proposed by Csikszentmihalyi [ 9
]. This theory, which to a large extent provides the theoretical foundation of this paper, can help us to understand how learners might want to pursue whatever they are doing from internal motivation [ 18
It should be noted that this paper does not claim that learning performance measures or knowledge acquisition should be overlooked in mobile learning. Instead, we suggest that learning performance measurements of mobile learning (e.g., formative assessment or workload assessment) cannot alone justify the costs of a mobile learning system, when this cost might outweigh the advantages. However, we believe that mobile learning may exhibit less obvious qualities, less measurable over the very short term, that we may be able to evaluate from new analytical perspectives.
2.1. Measuring Learning Experiences
Assessing mobile learning from the perspective of user experience seems to have been a common practice in the mobile learning research community. For example, the Palm Education Pioneers (PEP) program [ 1
] tried to evaluate the effectiveness of handheld devices in real-world educational settings from the teacher's perspective. The participating teachers overwhelmingly stated that use of the handheld device resulted in more effective teaching activities, interestingly pinpointing that the mobility of the device was key to allowing more collaboration between teachers. By comparison, the NetCalc project [ 1
] showed the benefits of the handheld computer for the participating student groups, leading to motivational gains in the conceptually difficult domain of the mathematics subject being studied. Of course, tools such as classroom response systems [ 13
] could lead to increased student engagement, increased teacher awareness of student knowledge, and increased student understanding of content matter. Yet, while we are combining sophisticated mobile technologies for perceived benefits in user experiences, many mobile learning projects do not explicitly address exactly how we can present learners with appropriate learning experiences. By and large, empirical studies on positive attitudes or level of engagement have been undertaken, without addressing how we might systematically measure the learning experiences initiated by mobile learning technologies.
In this regard, Csikszentmihalyi's flow experience [ 9
] sheds light on a way to incorporate learning experiences with the known benefits of mobile learning. Though there are many different definitions of flow
, it is generally said that flow is a holistically controlled feeling where one acts with total involvement or engagement with a particular activity, with a narrowing of focus of attention
]. From a mobile learning perspective, it implies that, in order for learners to experience flow while engaged in a mobile learning activity, they must perceive a balance between their controls
and the challenges
of the activity, which should present them with playful interaction
, exploratory behavior
, and positive subjective experience
]. For instance, both the Savannah project [ 20
] and the Ambient Wood project [ 21
], [ 22
] allowed a high level of self-control over the learning content to construct a more pleasing learning experience. Given that self-control is intrinsic to mobile learning, the relative levels of challenge and skill may either facilitate or block the motivation to learn. That is, at a given moment, individuals are aware of a certain number of opportunities challenging them, while they assess how capable they are of coping with these challenges. If the challenges of an activity are beyond the individual's skill level, demanding more than the individual can handle, they may disengage from further learning. On the other hand, if the challenges are lower than the individual's skill level, boredom may be the result, also leading to disengagement. This interpretation may suggest why the Japanese students had such learning performance benefits from the Nintendo DS console. Similarly, to maximize the controls and challenges of a learning activity, playful interaction
and exploratory behavior
have been significantly drawn upon in some previous mobile learning systems, e.g., [ 17
], [ 20
], [ 21
], [ 22
], and these concepts are a core part of the optimal flow experience.
Yet, the role of flow experience in mobile learning activities is not easy to see, partly because behavioral measures may not be very helpful, but mostly because it is a subjective experience. There have been many attempts, using tools such as focus groups and questionnaires, which have aimed to capture the behavioral metrics of mobile learning, e.g., enjoyment or pleasurable use, and used them to measure aspects of flow experience. Most notably, Microsoft does extensive performance testing, mostly via focus groups, and measures enjoyment by interrupting the user's interaction every few minutes via a dialogue box asking them for their current level of engagement. However, collecting data with this elaborate mechanism in mobile learning would be, to say the least, challenging. In effect, although flow experience is considered to be a useful measure for evaluating mobile learning systems, very little empirical work has yet been undertaken. This, then, is the central focus of this study.
To apply flow experience in assessing learning outcomes, several Human-Computer Interaction (HCI) practitioners, e.g., [ 19
], [ 23
], [ 24
], [ 25
] have demonstrated that the optimal flow state in learning activities can be briefly characterized in four dimensions (rather than the nine dimensions originally proposed by Csikszentmihalyi [ 9
]). These four dimensions of flow incorporate the extent to which
1. the learner perceives a sense of control over the learning activity,
2. the learner perceives that his or her attention is focused on the learning activity,
3. the learner's curiosity is kept aroused during the learning activity, and
4. the learner finds the learning activity intrinsically interesting.
To explore these dimensions in more detail, we begin with control
; flow theory can be used to examine the process of achieving learning outcomes through control over one's learning activities. For a learning activity to encourage playful, exploratory behaviors, learners should experience a feeling of control over the whole learning activity, so they will be motivated to work on long learning tasks in the face of tempting distractions. One way that mobile learning activities may provide this feeling of control is by instantly adapting to feedback from the learner's activity in a way that is not possible with more static pedagogies, e.g., large group face-to-face classroom-based teaching. For example, the game-based learning activities with the Nintendo DS, described previously, presented the learning content by considering how memorizing English vocabulary could be controlled and shaped by a series of scaffolding activities. In this way, the learner's subjective state is able to control the information given by the game console. Second, as a consequence of the feeling of control over the learning activity when in the optimal flow state, the learner's focus of attention
is narrowed to a limited stimulus field (or content, in this case), filtering out irrelevant thoughts and perceptions. The person in the optimal flow experience becomes absorbed in the learning activity, and is more intensively aware of his or her own mental processes, thereby enhancing relevant mental activities such as remembering, thinking, feeling, and making decisions. The Savannah project [ 20
] reported that the students who participated experienced being highly "mesmerized" during their learning activities, and had their memories, thoughts, and feelings shaped by the mobile learning system. It is widely thought that attention is a sufficient tool for the task of improving the quality of learning experiences [ 23
], [ 24
], [ 25
]. However, learners are more motivated when the instructional design generates curiosity and interest about the content and learning context. The last two constructs in flow experience— cognitive curiosity
and intrinsic interests
—are considered more or less as the consequences of both control
and focus of attention
. They also allow self-determination or self-management of the learning activity. We should be aware, however, that while learning experiences that give cognitive curiosity can also give focused attention, curiosity does not necessarily grow as a consequence of attentive experiences. A further note is needed here. Csikszentmihalyi considered two types of curiosity; sensory curiosity and cognitive curiosity. This paper focuses on cognitive curiosity,
which engages learners in a specific topic in time-intensive learning activities. This type of curiosity is central to successful instructional design, by making learners aware that their knowledge structures are incomplete and inconsistent [ 26
]. Cognitive curiosity is dependent on determining what activity would be intrinsically interesting to learners, so that they may be strongly involved in the learning activity for its own pleasure and enjoyment, rather than for the utilitarian purposes that are more commonly emphasized in traditional instructional design. The Ambient Wood project [ 21
], [ 22
] and many game-based learning systems [ 17
] are examples that maximize cognitive curiosity and intrinsic interest in instructional design.
Prior research suggests that optimal flow experience in learning activities may lead to higher quality learning outcomes, encouraging learners to be more adaptable to changing environments or new learning content, and constructing creative solutions to problems with no known solutions. The research reported here focuses on flow experience and whether or not it would be a useful construct for characterizing and measuring the subjective mobile learning experience. Of course, this is not an exclusive outcome, as other learning outcomes may be measured using formative assessment by written tests or workload analysis. Balancing these factors is important, since flow experience in learning systems design usually results in longer task completion times, partly as a consequence of engendering more interest in the learning task itself.
2.2. Performance, Spatial Cognition, and Flow—Experimental Tasks
Among multiple factors underpinning the successful mobile learning application, mobility (positively or negatively) determines the usefulness of mobile learning, by which learners must physically tune in to their own social environment while they are learning. Inevitably, this mobility may have negative influences on learning performance, in that it asks the learner to attend to more than one information source when using the handheld device. For instance, while you are walking through a busy street, you need to pay attention to the whole street environment in order to avoid getting lost, run over, or hitting other pedestrians. Because of this mobility issue in any mobile learning design, navigation through a physical world is a task that consumes the majority of a person's attention resources. This spatial cognition issue, therefore, highlights one of the essential properties of the experimental task in the empirical study described in Section 3, in that the limits of our attention describe our limited ability to time-share the performance of two or more concurrent tasks in learning, and sometimes describe the limits in integrating multiple information sources (or learning content) on the mobile device.
To make a realistic and challenging experimental task that does lend itself to flow experience, we need to understand how this spatial cognition would work in mobile learning situations. According to Baddeley [ 27
], working memory consists of the phonological loop and visuospatial scratch pad, and though they are equally important in controlling attention, the latter seems to be more significant to mobile learning, in that it is posited to serve the maintenance of visual and spatial information over the short term. To empirically see the potential effect of visuospatial working memory, Stanney and Salvendy [ 28
] developed two different interfaces that were used to test the information search performance of high- and low-spatial individuals. The results showed that high-spatial individuals outperformed low-spatial individuals in mentally constructing a model of the organization and structure of embedded learning content. Further, Sjölinder [ 29
] suggested that individuals with low-spatial ability seem to be more directed to the semantic content rather than other learning information, and in a similar vein, Doolittle et al. suggested that animations with onscreen narrative text might result in difficulties for learners with low working memory [ 16
From a practical or applied perspective, understanding the implications of cognitive differences between people makes it necessary to consider the extent to which spatial ability and process are important for a particular learning task. We can also use the natural variation among participants to identify how those with more or less spatial ability complete challenging learning activities in different ways. Against this background, for the following empirical study of flow experience in mobile learning, we consider the variance in spatial cognition among participants. Note that there are many different tests that claim to measure individual differences in spatial ability, which vary in their identification of factors, e.g., [ 30
], [ 31
]. In this study, we employed the block rotation test [ 32
] (see Fig. 1
) as it has been widely used in the study of spatial cognition. In this test, the four options from the right-hand side of the figure are either altered in shape, or rotated by an angle, or both, compared with the example on the left. The task for each participant was to find two fully matched rotated shapes from the four options.
Fig. 1. Block Rotation Test. Participants were asked to find shapes from four options on the right that were identical to the shape on the left (Reprinted from Encarta Microsoft Corporation).
2.3. Pilot Study
A pilot study was carried out based on our understanding of a likely relationship between spatial ability and mobile learning activities. In particular, prior to the main experiment, we wanted to see if the individual difference in spatial ability should be regarded as an independent variable or not, and how it could be associated with the dependent variables. Also, our purpose was to identify whether this simulated navigation task would be sufficient to give flow experience to our participants.
A total of 16 students from Massey University voluntarily participated in this pilot study, having a similar tertiary educational background (Information Technology and Computer Science majors), in the age range 21-24, with an even gender distribution (eight males and eight females).
The first step was to classify the participants into two groups (high- and low-spatial ability) using the block rotation test. A total of eight participants were assigned to the high-spatial ability group and the other eight were classified as low-spatial ability individuals.
The pilot study was performed using the same task and mobile learning material as the main study (see Section 3). This learning session required the participants to create a mental map of the physical space within a building, locate five rooms within that map, and associate names with target rooms. Since differences in the paths taken by different participants might have an effect on memorization, the order of room visits was identical for all participants. Other potential variables [ 33
] that may interact with performance, such as the walls of a room, landmarks, or other external reference frames, were not fully controlled in this pilot study, which was motivated by a common navigation task in a real-world setting. (For further details of the experimental task, refer to Section 3.)
gives the mean learning performance by counting the number of correct answers regarding the five rooms. As one can see from this figure, there was a rather significant difference in our participants (
), which implies that the learner's spatial ability would have a significant effect on his or her learning performance.
Fig. 2. Learning performance (max: 100) between those who have high- or low-spatial ability.
Also, to assess if this navigation task was considered a pleasant experience, we collected the participants' verbal feedback when they had finished the experimental task. Of course, to fully understand this subjective experience, we might have employed more systematic questionnaires such as Webster et al.'s [ 34
]. However, in the sense that the main purpose of the pilot study was to figure out how our participants were thinking while doing this experiment, and what flow (or pleasure) they thought the whole mobile learning session engendered, this simple data met this purpose. Regardless of the level of spatial ability, almost all the participants (seven participants from each of the groups) felt that this learning activity was a pleasant experience, which we might presume to indicate that the flow experience provided by mobile learning is somewhat independent of cognitive capability.
Taken together, the outcomes of this pilot study first suggested that this navigation task could lend itself to sensitizing our participants to flow experience, and second that spatial ability could be an important independent variable for memorization. We presume, however, that a basic memorization task is not best served by flow experience, which indicates that separately measuring both flow experience and learning performance is necessary.
3. Method—Security Guard Training
The experiment presented here was suggested by our prior understanding of flow experience and the pilot study described in the previous section. To empirically investigate the research question described in Section 1, a simulated learning program, such as might be used for training security guards, was developed. This learning program was intended to allow trainee security guards to explore a physical space (or virtual space in the case of game-based learning) to become competent in securing specific premises. The pilot study showed that this navigation task in the building was challenging, so it could lend itself to flow experience. Three types of instructional design were developed to see the differences in both learning performance and flow experience; traditional map-based instruction (see Fig. 3
), a game-based system (see Fig. 4
), and a mobile learning system (see Fig. 5
). The first (map-based) condition was regarded as a control condition, and the game-based system was built upon the assumption that game-based learning activities would present a highly engaging and immersive learning experience [ 17
], [ 35
]. However, this game-based system did not provide the same physical mobility as the real world, enabling us to consider the differences between virtual reality and reality when assessing flow experience in mobile learning.
Fig. 3. Traditional instruction design—the first experimental condition (as a control group) was presented on a paper-based map and the other information relevant to security training services was provided as an additional document.
Fig. 4. Game-based instruction design—the second experimental condition was presented in a virtual 3D space. The other information relevant to security training services was provided with dialog boxes on the display.
Fig. 5. Mobile-based instruction design—the last experimental condition replaced the traditional instruction design with a Personal Digital Assistant (PDA). The other information relevant to security training services was presented with animated content on the PDA.
Note that the three instructional methods did not provide exactly the same training environment to participants. In particular, the game-based instruction method did not provide the same real-world environment cues that were available in both the traditional map and mobile systems. This game-based learning did not allow the participants to physically visit the rooms or interact with the real building environment, so potential environmental cues that might be memorable, such as the names of the nontarget rooms and perhaps a drawing or a crack in the wall, a sign, even the true shape of a space, were not available. This would inevitably affect performance in creating a mental map of the space, locating targets within that map and memorizing the names of target rooms. Hence, it seems likely that the virtual game-based learning is significantly more difficult to use than the other two methods, and that this difficulty might have an effect on flow experience. On the other hand, the game-based learning tool emphasized more attentive, immersive, and playful learning activities. Another important issue was the relative affordances of the paper-based and mobile systems. It should be noted that the mobile system included both way finding support and interactive features not present on the paper map.
To minimize the effects of extraneous conditions as much as possible, all the possible landmarks in the building environment were concealed for both the map-based and mobile learning. Further, for the virtual game, we provided a 3D map of the premises as realistically as possible that allowed the participants to see all the landmarks covered and perceive the depth or the relative volume of each room via the space view option.
Other information relevant to security training services, such as security key codes for each room, rules, and professional practices, was delivered via different media, i.e., written documents in the traditional instruction design, dialogue boxes in the game-based system and animated multimedia content in the mobile learning system. These subtle differences that might influence the learning activities in this experiment are carefully considered in the following analyses, in the sense that this experimental task abstracts out the important features of the experience that trainee security guards would have in reality.
Despite its limitations, this simulated patrolling task within the building was frequently observed by the university's professional security guards, who showed great interest.
A total of 53 subjects from Massey University voluntarily participated in the study, none of whom had physically explored the premises before, and around half of them (25 out of 53) were females. All of them had a similar tertiary educational background (Information Technology and Computer Science majors), in the age range 19-26. The map-based learning system was a control condition, and only 9 out of 53 participants were assigned to this treatment. There were differences in the size of each group (see Table 1
) due to the random way participants were assigned to conditions, and to increase the power of the two experimental conditions of interest, the number of participants in the control condition (i.e., the paper-based instruction) was intentionally minimized.
Table 1. Learning Performance (Max: 100)
3.2. Experimental Design
The experimental design was a between-subjects factorial. The three types of system given were the main between-subjects independent variable. Note that one important feature in this experiment is that the participants with the mobile-based or paper-based system are not sitting attentively at a desk. Thanks to this mobility feature, navigation through the physical world is a task that might use the majority of a person's attention resources; therefore, the participant's spatial capability was considered as another independent variable in the corresponding analyses.
The dependent variables used the ratings on 12 questions regarding the flow experience (see Section 3.4). With regard to learning performance measures, two dependent variables were considered: 1) the number of correct answers about the five rooms to be located (plus their security key codes and other norms regarding security rules) and 2) the NASA-TLX scale [ 36
] that measures disturbance to primary task performance and is useful for an estimate of the attentional demands of a task, which can negatively impact learning performance [ 37
As depicted in Fig. 3
, the traditional paper-based security training system contained a map of the five rooms to be visited, their security key codes and their names on a sheet of paper. The game-based learning system (see Fig. 4
) was created using Active Worlds (activeworlds.com) with the same information, but allowing the participants to navigate the experimental space virtually. A PDA with narrative (textual) information developed using Microsoft PowerPoint was used for mobile learning as shown in Fig. 5
The participants were first provided with the instructions regarding the experiment. These gave information about the experiment, the purpose of the study, and the data protection policy. They were then randomly assigned into one of the three experimental conditions.
To control a possible covariate effect of the independent variable in the analysis (i.e., spatial capability), all the participants were first asked to perform a block rotation test [ 32
]. They were given 10 such questions to complete in 6 minutes. Later, to classify our participants, the base value of 60 (out of 100) of the mental rotation test score was used, as indicated in many related spatial capability studies. A total of 27 participants were assigned to the high-spatial ability group and the others were classified as low-spatial ability individuals in the following analyses.
After each participant completed the mental rotation test, he or she was asked to use either the paper-based, game-based, or mobile learning system for around half an hour. As they completed the learning session (the premises contains 23 rooms on the second floor, and the participants were guided by the system given to visit five of them, plus remembering other information related to security services), workload assessment measures from the NASA-TLX scale were applied using software developed by the Naval Research Laboratory (nrl.navy.mil). All of them were then asked to write down the locations of the five rooms, security key codes, other norms regarding security rules, and their names on the map. Then, the 12 questions relating to flow experience were rated on a five-point Likert scale. These questions were based on Webster et al.'s categories [ 34
], which have proven useful in many previous studies on flow theory:
- When using the mobile device (paper/game), I felt in control over everything.
- I felt that I had no control over my learning process with the mobile device (paper/game).
- The mobile device (paper/game) allowed me to control the whole learning process.
• Attention focus
- When using the mobile device (paper/game), I thought about other things.
- When using the mobile device (paper/game), I was aware of distractions.
- When using the mobile device (paper/game), I was totally absorbed in what I was doing.
- Using the mobile device (paper/game) excited my curiosity.
- Interacting with the mobile device (paper/game) made me curious.
- Using the mobile device (paper/game) aroused my imagination.
• Intrinsic interests
- Using the mobile device (paper/game) bored me.
- Using the mobile device (paper/game) was intrinsically interesting.
- The mobile device (paper/game) was fun for me to use.
As depicted in Table 1
, our participants with the game-based learning system performed the experimental task poorly, when compared with the other two experimental systems, which might indicate limitations to this type of instructional design in training programs where physical exploration is essential. Note that the virtual game-based learning did not allow the participants (20 out of 53) to physically visit the rooms, so this performance measure was not of great indicative value in this learning activity. This will be further discussed later.
In contrast, a comparison between the paper-based and mobile systems seems to indicate that a nonsophisticated learning material (paper-based learning in this case) would lead to similar learning performance outcomes as mobile learning, despite differences in learning and device affordance (though the difference in results was not statistically significant). This result would be in line with our previous understanding of the marginal performance benefits of mobile learning. Note that both systems asked the participants to physically visit the five rooms, which positioned mobility at the center of the learning affordances of the two systems, against the virtual game-based learning where there was no mobility.
A one-way analysis of variance was conducted on the learning performance, revealing that there was significant effect of the system given (
). A Tukey test (at
) was performed to further examine the effect of the systems, indicating that the game-based learning program led to poorer performance than the other two, but that these were not significantly different from each other.
These results may be influenced by individual differences in spatial capability [ 28
], [ 29
], which may have the effect of the covariant independent variable, and additional analyses were thus carried out. Tables 2
give the mean learning performances by the participant's spatial capability. For high-spatial ability individuals, there seemed to be no significant difference, but the low-spatial ability subjects seemed not to see the benefits of the mobile system. This is very much in line with Sjölinder's study [ 29
Table 2. Learning Performance by High-Spatial Individuals (Max: 100)
Table 3. Learning Performance by Low-Spatial Individual (Max: 100)
As clearly depicted in both Tables 2
, spatial ability is indeed having an effect on the learning performance (i.e., covariate). A one-way between analysis of variance was conducted for the statistical control of spatial ability, respectively. A Tukey test (at
confirmed that the mobile system led to significantly poorer performance for those of low-spatial ability, which was not the case for those who have high-spatial ability. This implies that the learner's spatial ability, when associated with mobile learning, can influence learning performance.
The data can be taken to suggest that, at the very least, care is needed when designing mobile learning systems and that individual differences in learners' spatial capability need to be considered to assure the learning benefit. Those who do not have high-spatial capability may not gain the same performance benefit from a given system as those who do. Also, mobility should be to the fore of this type of instruction design, a factor not present in the virtual game-based learning.
As another performance measure, the NASA-TLX subjective workload assessment tool was applied, which derives an overall workload score based on a weighted average of ratings on six subscales [ 36
]: mental demand
, physical demand
, temporal demand
, and frustration level
. This measure is thought to be useful for an estimate of the attentional demands of a task, considering that demands are relevant to poor learning performance. Table 4
summarizes the data collected, together with the mean scores for the six workload items, respectively.
Table 4. Mean Ratings of Workloads (Max:100)
Overall, the game-based learning system exhibited the highest (nonphysical) workload demands. A Tukey test showed that the game-based system was notably distinct from the other two experimental conditions. In each workload item, a one-way between-subjects analysis of covariance (spatial ability as covariate independent variable) was applied, confirming that, regarding mental demand, those who do not have high-spatial capability may not gain the same performance benefit from a given mobile system as those who do, in line with the results above. Then, one-way ANOVA analyses on each workload item indicated that there are significant differences in terms of physical demands and effort. Both of these factors were further analyzed by Tukey tests, revealing that the game-based system was significantly better than the other two conditions in the "physical demand" subscale, but that was not the case for "effort."
Taking these results together with the learning performance depicted in Table 1
, they suggest that the game-based learning system was of less value in this type of exploration-based instructional design and, further, that individual differences in spatial cognition should be considered as an important factor when assessing the performance benefits of mobile learning. However, note that these analyses of learning performance do not lend themselves to generalization of other types of learning situations. Indeed, this experimental task intrinsically benefits from active participation and interaction with the real world, but the virtual game-based learning just hinders this social process, instead being directed toward playful and pleasant interaction.
Contrary to the two performance measures above, the ratings of flow experience revealed a rather different pattern, which may suggest the distinctive nature of a mobile learning experience, and also the benefits of playful game-based learning. "Learning control" asks whether a learning activity encourages exploratory behaviors, with the expectation that learners who experienced feelings of control over learning activities would give a higher rating. "Attention focus" is another sufficient condition for achieving flow experience, being narrowed to a limited stimulus, so higher ratings on these questions mean more engagement in learning activities. "Cognitive curiosity" and the desire to attain competence with the learning application may motivate learners to develop more skills where the challenges are matched by the user's current skill set, so higher ratings on these questions imply willingness to use the system again. Finally, Webster et al. [ 34
] termed "intrinsic interests" as subjective experiences during interactions that are characterized by perceptions of pleasure and involvement. Higher ratings on these questions mean the learners are so intensively involved in the learning activity that nothing else seems to matter.
gives the mean ratings for the three experimental settings across the four contributors to the flow experience, where it can be seen that there is an effect of the system. In most cases, except learning control, mobile learning gives the highest ratings, and the virtual game-based learning follows, results which differ from the learning performance measures above (note that the game-based learning was the worst in learning performance, but not for flow experience). In particular, both cognitive curiosity and intrinsic interests seem to be critical motivators of the learning experience among our participants, which would motivate them to make the required effort to learn to use the sophisticated mobile-based or game-based instructional design. For each factor, a one-way between-subjects analysis of variance was applied, followed by a Tukey test (at
). Note that before applying the one-way ANOVA, a test for linearity confirmed no explicit linear relationship between the dependent variable (ratings) and the covariate (spatial ability) for each system (paper, mobile, and game); therefore, we proceeded with an analysis of variance, respectively. Indeed, it is striking that flow experience seems to be independent of individual spatial cognition, though more detailed experiments would be needed to confirm this. A possible explanation for this result would be the "novelty effect
" of learning and the device affordances of both the mobile and the virtual game-based learning. Two of the contributors in the flow experience, i.e., cognitive curiosity and intrinsic interests, may be primarily driven by using the technology itself, which is new, and not the content. On the other hand, noting that the learning performance measures described above were about the content rather than the technology, individuals' spatial cognition does not seem to be critical in having a high level of flow experience arising from the technology affordances. Further accounts of these findings are considered in the discussion.
Table 5. Mean Ratings of the Flow Experience
Taken together, the benefit of mobile learning can be seen by the optimal flow experience aroused by "cognitive curiosity" and "intrinsic interests," compared to the other learning systems.
5. Conclusionsand Discussion
This paper has empirically investigated the flow experience effect as one potential analytical tool to explain the benefits of mobile learning. We have contextualized these comparative benefits within an experiential workplace learning landscape that underscores the significance of the flow experience associated with mobile learning.
In our examination of flow experience, comparisons of the three configurations allowed us to assess the potential value of the mobile learning experiences available in this experimental context. The evaluation of learning performance suggested that a nonsophisticated technology (i.e., paper-based) could also be useful to such educational activities, which implies no significant benefits from mobile learning technologies. However, the evaluation of flow experience explicitly confirmed the significant advantages of the mobile learning and game-based learning systems over the traditional pedagogy. Neither of these twofold possibilities had been demonstrated empirically before. Fig. 6
illustrates our findings that the benefits of mobile learning should be perceived as the intersection of both learning performance and experience in these experimental contexts.
Fig. 6. Mobile learning can have both performance and experience benefits in this exploratory workplace learning (i.e., security guard training), where situationally aroused.
The empirical data presented in the previous section make a case for the relevance of flow experience to mobile learning and provide some insights as to why it might be more effective than previous learning performance measures (i.e., formative assessment or workload assessment). Flow experience [ 9
] was devised by analyzing learners' subjective experiences, or an exploratory experience where learning outcomes are not yet firmly applied, as was the case with this mobile instructional design. In the case of instructional design using traditional pedagogies that are well established—such as face-to-face classroom learning or e-learning—such experiences may not easily be achieved. Arguably, at least in workplace learning, mobile learning, where situationally aroused and when sufficiently motivated, seems to encourage people to have more cognitive curiosity and intrinsic interest than might be secured by other instructional designs [ 38
5.1. Knowledge Acquisition and Situated Learning
These interpretations can be seen from two perspectives. The first is that of convenient knowledge acquisition. Security guards with either the handheld device or the paper-based learning material could acquire knowledge through constructing and internalizing their own cognitive structures through the given learning activities [ 39
], [ 40
]. However, given that self-control aroused by "cognitive curiosity" and "interests" was more intrinsic to mobile learning rather than alternatives, the relative levels of challenge and skill seem to highly facilitate the motivation to learn with the handheld device. That is, at a given moment, the security guards with mobile learning are able to be aware of a certain number of opportunities challenging them, and can assess how capable they are of coping with these challenges. On the other hand, if the challenges are lower than the individual's skill level, which is more likely with the limited affordances of paper-based learning, boredom may be the result, also leading to disengagement. Here, to maximize the controls and challenges of a learning activity, playful interaction
and exploratory behavior
has been significantly drawn upon in previous mobile learning systems as the core part of the optimal flow experience
, e.g., [ 20
], [ 21
], [ 22
The second perspective is that of building up knowledge by active participation, in which knowledge is directly obtained by participation in practices located in specific environments. In this study, it can be seen that, via mobile learning or paper-based learning, knowledge is directly gained through participating in a social process, and learning is displayed by becoming more central in these processes [ 11
]. Perhaps this accounts for why individuals' spatial cognition in our experimental setting was highly associated with their learning performance. By comparison, the game-based learning shows the unique way in which knowledge is represented, organized, processed, and manipulated only in individuals' minds, under the traditional constructivism perspective [ 41
], centered on what a single individual was capable of learning in a setting isolated from everyday activities. Hence, this situated learning process exemplified in this paper can refocus on the situated and distributed nature of cognition applied to thinking, learning, and doing in workplace settings [ 11
]. In effect, learning with mobile technologies should be now seen as situated within both physical and psychosocial contexts and distributed between a person and the tools he or she is using [ 42
5.2. Limitations of This Study
No single measure can assess all learning outcomes. Traditional learning performance measures may evaluate instructional design factors such as task completion time. In contrast, flow experience may work at an abstract and subjective level of learning outcomes. However, we do not have any direct evidence from real-world mobile learning applications that our accounts are practically applicable. Perhaps this is the major limitation of this empirical study, that the participants were third party learners, not security guards themselves.
Another question is how the flow experience will scale up to handle real instructional designs. We have as yet to gain experience of the flow experience in real instructional design projects and so quite how this will work out is an empirical question. However, flow experience is presented here as a key perspective that captures self-initiated curiosity and interests to be worthy of attention in designing new learning activities. Practitioners, as well as researchers, should thus embrace the notion of learning experiences, for a better understanding of the important values that mobile learning can provide.
5.3. Further Research
The results of this study raise several questions that could be pursued in future work. The limitations of our study suggest further research to explore how learners would attune themselves differently to different mobile learning experiences. The experimental task in this empirical study was a spatial navigation task that required spatial cognition in performance. Hence, without considering many other learning tasks that would not fall into this category, we cannot clearly assert the relative effectiveness of these different forms of learning systems, with respect to the type of content to be learned and the type of learning outcomes used to assess the benefits of mobile learning. As well as considering other learning tasks, our observations also point out two important areas of further research that could be advanced. First, we might explore other individual differences, such as attention level [ 43
], age, education, or professional training level and so forth; second, we might pursue a longitudinal study to see how flow experience might make a difference in students' learning performance using longer term measures of mobile learning outcomes.
This work was partly supported by the Sunmoon University Research Grant of 2009. Thanks are also due to the anonymous reviewers for their constructive comments on the original draft. Hokyoung Ryu is the corresponding author, and the authors are in alphabetical order.
• J. Park is with the Division of Information and Computer Science, Sunmoon University, Asan, Chungchungnam-Do 336-708, South Korea. E-mail: firstname.lastname@example.org .
• D. Parsons and H. Ryu are with the Institute of Information and Mathematical Sciences, Massey University, Auckland, New Zealand, NSMC 102904.
E-mail: email@example.com, firstname.lastname@example.org.
Manuscript received 28 May 2009; revised 3 Aug. 2009; accepted 22 Dec. 2009; published online 5 Jan. 2010.
For information on obtaining reprints of this article, please send e-mail to: email@example.com, and reference IEEECS Log Number TLT-2009-05-0094.
Digital Object Identifier 10.1109/TLT.2010.1.
1. Please note that in general Tukey tests are more reliable for large sample sizes (at least more than 5 or so in each cell); however, to make this analysis consistent with the following analysis, the simple solution of using Tukey tests was adopted.
received the master's degree in 1987 and the PhD degree in 1990 in information engineering from Osaka University, Japan. He is currently a full professor in the Division of Information and Computer Science at the Sunmoon University, Korea. His current research interests include internet ethics and e-learning. He is a vice president of The Ethics Council on Internet and also a vice president of the Korea Information Processing Society.
received the master's degree in electronics and computer science from the University of Southampton, United Kingdom, in 1996 and the PhD degree in information technology from Nottingham Trent University, United Kingdom, in 1999. He has worked both in the university sector and in commercial software development. His is currently a senior lecturer in information technology at Massey University, Auckland, New Zealand. He has published a number of books on computer programming, Web development, and mobile learning. He is an editor-in-chief of the International Journal of Mobile and Blended Learning
and a member of the International Association for Mobile Learning.
received the master's degree in industrial engineering from the Korea Advanced Institute of Science and Technology in 1999 and the PhD degree in psychology from the University of York, United Kingdom, in 2003. He is currently a senior lecturer in information technology at Massey University, Auckland, New Zealand. His current research interests include human-computer interaction and mobile learning, publishing more than 70 peer-reviewed articles in conferences, journals, books, and book chapters on these research themes. He is a member of the British Computer Society's Human-Computer Interaction Group, and also an associate editor of both the International Journal of Mobile and Blended Learning
and the International Journal of Mobile Learning and Organisation