The Community for Technology Leaders

Empirical Evidence of Priming, Transfer, Reinforcement, and Learning in the Real and Virtual Trillium Trails

Maria C.R. Harrington, IEEE

Pages: pp. 175-186

Abstract—Over the past 20 years, there has been a debate on the effectiveness of virtual reality used for learning with young children, producing many ideas but little empirical proof. This empirical study compared learning activity in situ of a real environment (Real) and a desktop virtual reality (Virtual) environment, built with video game technology, for discovery-based learning. The experiences were in the form of two field trips featuring statistically identical wildflower reserves. While the results support that the Real is superior for learning activity, they also show that the Virtual is useful for priming and reinforcing in-curriculum material, or for situations when the real environment is inaccessible. Offering the Virtual first primes for learning activity in the Real; if used second, it reinforces the Real experience, as supporting evidence shows significant transfer effects. Thus, the Virtual may serve educational goals, if used appropriately, and can come close to the Real. As informal learning environments, such as field trips and video games, are accepted as motivational, an attitudinal survey was conducted postexperiences to capture motivational factors at play, to aid in comparison and contrast, and to provide context to the empirical results on learning activity in situ; however, more work is needed.

Index Terms—Child-computer-environment interaction, child-computer interface, discovery-based learning, educational simulation, evaluation/methodology, serious games, human factors in software design, human-computer interaction, human information processing, informal learning, intrinsic learning, salient events, simulation, modeling, visualization, software psychology, virtual reality, user-centered design, user interfaces.


In May 2007, amid the springtime bloom of wild trillium at a long-term NSF deer exclosure study site, a group of elementary school students from a suburban Pittsburgh school district went on a field trip—with a twist. In this study of the Virtual Trillium Trail (VTT), students took both real and virtual field trips to the protected site, with the real trip functioning as the control.

A driving question behind the VTT was: Could a desktop virtual environment (Virtual) draw on elements of simulations, games, and educational software to enhance perceptual, cognitive, and emotional experiences?

The best way to approach this design problem was to frame it within the context of a proven, successful real-world model: the field trip. The real-environment field trip (Real) is such a baseline model that, at its best, embodies discovery-based learning experiences beyond the confines of the classroom.

In comparing a real field trip to a virtual one through empirical evaluation of in situ discovery-based learning activity and postexperience attitudes, the study held to the assumption that Real is preferable to Virtual; this common sense “given” is too often ignored in studies centered on the effectiveness of virtual environments, perhaps due to the theoretical idealism of designers. In contrast, the study underscored this primary assumption, thereby allowing for empirical, statistical measurements that provide instructional designers with tangible ways to implement highly realistic virtual reality (VR) applications and reality-based simulations (see Fig. 1), as an extension of the classroom, as educational simulations.


Figure    Fig. 1. An image of the Virtual Trillium Trail, an educational simulation, a simulated ecology for discovery-based, intrinsic learning.

The central software design focus was on the child and the child's needs. Personalization is used to advantage in intelligent tutoring systems [ 1], where extreme personalization is the goal, and where each student can have a personally unique and meaningful experience. As is advocated in user-centered design (UCD) [ 2] and human-computer interaction (HCI) [ 3], the child, as the main user, drives the software requirements and assessment (see Fig. 2).


Figure    Fig. 2. An illustration defining intersection of the research interests of a simulated ecological environment for education (SEEE).

While formal statistical hypotheses were stated to structure the work of the study, the study itself was driven by the desire to explore, generate insights, and frame future research in this area. The results inform design and development of virtual environments for independent and intrinsic learning. The empirical analyses reported here are unique and have not been reported elsewhere; however, they are part of a larger body of work, and the ethnographic and nonparametric results have been reported and published elsewhere [ 4]. This submission to the IEEE TLT is an empirical analysis using parametric statistics to show differences in learning activity between Real and Virtual discovery-based learning activity, transfer effects, and attitudes, where the paper published in [ 4] is an ethnographic report and a nonparametric analysis of the experiences. The two papers complement and support each other, while allowing detailed analysis from either a qualitative or a quantitative approach.

1.1 Framing the Design Problem in Broader Context

Climate change is the defining problem of the age. Learning about ecology requires tools that inform, inspire, and enlighten, as well as tools that increase perception, awareness, and constructive action. Highly realistic virtual reality applications and reality-based simulations are unique tools for learning, as students may transcend time, space, distance, and scale to develop an accurate and complete understanding of complex relationships. However, not much is known about the design and construction of such tools for stimulating perception, learning, and creating; hence, the need for research that goes beyond Presence [ 5] or the “feeling of being there,” to build tools that aid learning, problem solving, decision support, and creativity. Future research is required to understand how these tools might act as conduits to increase the probability of enhanced awareness, knowledge acquisition, and constructive creativity, with respect to our ecosystems and survival.

Real-time, interactive virtual reality allows one to experience different time periods or scenarios. Simulations are data-based, often abstract, and usually, not real-time interactive. Educational software allows for information search, exploration, annotation, and augmentation but is usually not executed in a form intended to enhance perceptual, intellectual, and emotional experiences. Could virtual reality merge with simulations, games, and educational software to provide a new combination of features? Capturing the feel of the game Myst [ 6] but retaining scientific information? Could such new tools deepen children's understanding of plants in their backyards as well as develop appreciation for ecologies in distant lands? The challenge was to design and build such virtual experiences to be truthful, effective, and motivating.

Modeling an informal, discovery-based science and ecology field trip provided the ideal opportunity to meet this challenge based on real-world best practices. The Trillium Trail field trip, offered by the Audubon Society of Western Pennsylvania and led by expert educators and naturalist guides [ 7], is currently part of school enrichment programs for science and ecology; first established in the 1960s [ 8], its teaching methods and curriculum are a superset of the Pennsylvania State educational standards. These outdoor classrooms offered an ideal situation to simulate and model in the software.

Additionally, the plant population distributions at the Real field trip location have been extensively studied by Susan Kalisz, PhD, who has been conducting biological field research at Trillium Trail since 1994 [ 9]. These deer exclosure data sets—free from overgrazing, man-made roads, power lines, or sewers—provided the required virgin population distributions on which to build and statistically extrapolate the virtual plant populations in the desktop VR visualization. Thus, the physical content of the VTT consists of unique set of statistical, scientific visualization of biological plot study data, Real field trip user activities, and Audubon Society educational content. Interestingly, this offers a reality to the students that is about to slip away, as we are on a cusp in time between past and the future ecologies, and only within the fenced areas of the deer exclosures is the forest protected.

Theoretical Background

Past reports on educational virtual reality were proof-of-concept, technical investigations, or educational case studies [ 10], [ 11], [ 12], [ 13], [ 14], but none offered a multidimensional synthesis of the child-computer-environment interaction or stated a comprehensive and complete design problem. Seminal research in virtual reality for education showed evidence of knowledge gains [ 15], [ 16]. Pioneering research in immersive virtual reality and ubiquitous environments explored the problem and documented emotional reactions such as enjoyment and sense of play [ 17], [ 18], [ 19].

Ecology simulations were attempted [ 20], [ 21], [ 22], [ 23] but left the educational impact undetermined. Related research in biology education using high-fidelity simulations showed convincing results [ 24], and thus, indicated that high-fidelity simulations were needed for such learning systems. Features from high-fidelity military [ 25] and medical [ 26] training applications, used for procedural knowledge transfer and shortened learning curves, were evaluated. Analyses of those applications' critical features were important in the design of the first prototype of the Virtual Trillium Trail and in the investigation of the ideal child-computer-environment interface. Horizontal functionality of user interfaces for virtual reality applications was developed and investigated with success, especially in the use of annotation and augmentation [ 27], which is required for effective information and knowledge inquiry and acquisition. Research on navigation in virtual reality shows that route and wayfinding transfer is effective [ 28], [ 29] as an aid in exploration. Data visualization systems, geographical information systems, and landscape architecture systems enhance decision support [ 30] and serve to increase the perception and understanding of complex interconnected relationships.

Orthogonal research suggests design criteria and desirable features, such as emotional and motivational affects that relate to play [ 31] and a sense of “flow” [ 32]. The sustained and independent play elicited by noneducational games [ 33], [ 34] may offer design clues for educational software. Thus, discovering and understanding the elements in existing virtual reality software that create such intrinsic motivation is of investigative concern.

Nonetheless, while VR technology for education and learning is maturing [ 36], [ 37] not much is known about how the virtual world interfaces with the user in learning: The interactions, interferences, and convolutions of “form and function” are indeed complex and dynamic. The Virtual Trillium Trail design challenge was to create a baseline study required for an empirical comparison and contrast of causal factors found in the interactions of child, environment, and user interface in the Real and the Virtual environments.

The overriding goal was to define the ideal virtual environment for independent and intrinsic learning. The problem space is represented by the intersection between learning, real and virtual environments and the user interface. The proposed conceptual model represents the intersection of the child's mental model, knowledge ( $\Delta$ Knowledge), the user interface ( $\Delta$ UI), and the environment ( $\Delta$ VE) as a dynamic interdependent network (see Fig. 2).

The Virtual Trillium Trail System

3.1 The System

The Virtual Trillium Trail application was developed between 2005 and 2008. The first 2007 prototype ran on a Dell XPS Gen 2 laptop and required a high-end NVDIA GeForce Go 6800 Ultra graphics card. The virtual field trip reported in this paper occurred in a PC-lab classroom at the University of Pittsburgh (see Fig. 3). Each PC had the application installed; the application ran as an Unreal map in a stand-alone instance of the Unreal Tournament application [ 35].

Graphic: Fig. 3. A photograph of students using the Virtual Trillium Trail in the PC lab at the University of Pittsburgh, May 2007, with the naturalist guide pointing to a virtual fact card on the projected screen.

Figure    Fig. 3. A photograph of students using the Virtual Trillium Trail in the PC lab at the University of Pittsburgh, May 2007, with the naturalist guide pointing to a virtual fact card on the projected screen.

Using common desktop PCs was advantageous because students were familiar with the keyboard and mouse. The keyboard arrows allowed them to move forward, backward, and to either side. The mouse could be used to pan and rotate the view. The control keys were used to switch modes: Flying (Ctrl-F), walking (Ctrl-W), jumping (Ctrl-J), and running or swimming occurred automatically based on environmental context. The controls were known, guessed, or learned quickly. The one-ear headset was easily used for sound. Accessing the textual fact cards consisted of moving the cursor close to the cards, positioning in front, and then reading.

While problematic for real classroom deployment, video game technology represented a low-cost, high-fidelity platform for rapid development and use in research in 2005-2007 [ 38]. The PC lab had 12 Dell desktop computers, but all lacked the high-end graphics cards desired. While typical of most equipment found in most schools in 2007, this lack negatively impacted the frame-refresh rates, causing them to slow to between 3 and 10 frames per second. Future performance should improve frame rates with faster equipment and/or with compiled, more efficient runtime code.

The VTT was a prototype system that simulated approximately one square mile of terrain and biological plot study and transect data [ 9]. This was simply done by importing GIS data into ESRI [ 39] to generate accurate data-derived DEM terrains. As such, some features were not in the DEM database, so features such as ravines, waterfalls, caves, or rock cliffs were manually added. While the resulting model is not an exact reproduction of the terrain and ecology, it is a very close statistical population density clone at a moment of time, and one based on the biological plot study data and terrain data. The 10-meter samples of real plant data sets were used to create virtual plant data sets. Furthermore, clusters of plants that were not in the original biological plot studies but were observed in the real environment were manually added to improve the realism and accuracy.

Footpaths and the fact cards used in the real field trip were simulated and added to the virtual field trip. The footpaths gave the students a natural affordance for navigation. The Virtual fact cards simulated the Real fact cards, and were composed of a photograph, a schematic drawing, and text including scientific, biological, and ecological facts.

The virtual plants were 3D computer graphic models created in Maya [ 40], textured with photographs taken on-site at the Real location. Over 1,500 locally gathered photographs were taken in 2006 for this purpose. In this prototype, only 36 species of the 102 indigenous plants were modeled. As can be seen in Fig. 4, the level of realism is very high for a 2007 system.

Graphic: Fig. 4. Photographs paired with 3D computer graphic models of the flowers used in the May 2007 Virtual Trillium Trail prototype.

Figure    Fig. 4. Photographs paired with 3D computer graphic models of the flowers used in the May 2007 Virtual Trillium Trail prototype.

3.2 Research Importance of Scientific Visualization

This research is different from previous projects in several important ways. A critical point in this research is that the Virtual Trillium Trail is based on a real place so as to minimize any programmer's or designer's unintentional introduction of misconceptions. “What you see is what it is” (WYS-I-WII) was the paradigm followed. Basing elements on reality was a critical design factor for success, as the goal was to create an authentic simulation of reality for intrinsic learning.

When is it acceptable for the software designer or the educator to deviate into fantasy?

In the past, most systems were “What you see is not what it is.” Regarding prior systems, there were three main concerns with the choice of technology for learning goals, as the medium can influence the meaning:

  1. the image quality was often low;
  2. the environment was dependent on a designer's interpretation;
  3. the navigation, controlled by the designer, restricted freedom of movement.

For example, cartoon-like images used either for style or lower costs resulted in images that lacked detail. Other “virtual field trips” deployed in desktop VR environments, such as Quest Atlantis [ 41] or River City [ 42], used a type of multiuser virtual environment (MUVE) [ 12] based on a low-fidelity platform by Active Worlds [ 43]; they were constructed to reflect a fictitious environment, and therefore, are theoretically capable of introducing unintentional misconceptions. While such variants may be needed and desirable for some educational objectives, it is important to be clear about how a tool can influence learning, and to use it with intention, or guard against either intentional or ignorant misuse.

The Virtual Trillium Trail differs critically from prior virtual environments for education systems which allow for complete freedom of movement and object selection in that it is:

  1. a data-based simulation of terrain and plant population species;
  2. composed of graphics that are high-fidelity, photorealistic approximations of the real location, ecology, and plants;
  3. based on a real informal educational curriculum;
  4. based on real informal learning activity and interaction.

3.3 Implementation

The real field trip was to the real Trillium Trail, a wildflower reserve located outside of Pittsburgh, Pennsylvania, and the virtual field trip utilized the Virtual Trillium Trail, a high-fidelity virtual environment simulation of the real-world location installed in a PC-lab classroom in the School of Information Sciences at the University of Pittsburgh. Each experience lasted 1.5 hours, and the study with this sample was conducted over the first two weekends in May 2007.

The field trip activities were carefully controlled to be the same, except that the Real was in the real outdoor location and the Virtual occurred through the VTT software run in a computer lab at the University of Pittsburgh. In the Real, the students followed the naturalist guide on the trail and were presented with information, or the guide responded to the spontaneous inquiries of the students. In the Virtual, each student sat at a PC station, used an earphone on one ear, and independently selected when to listen to the naturalist guide and when to explore independently. At times, students would find something of interest and share with the guide and the class, while at other times, they would share information with the closest student in the room. Frequently, the students did not listen to verbal narration but instead were off exploring the software on their own. The students positioned the mouse to select a direction, tapped on the arrow keys to navigate, and used the space bar to select an object's fact card for more information on an onscreen object. At other times, the students navigated to an audio Sprite and listened to the concept, story, or lesson. The Sprites made it easy for the younger children to listen to and hear the entire concept, whereas the fact cards allowed students to read and flip through, read several times, or leave once some basic information was reviewed. When students found objects of interest, they would mark them on the paper maps distributed at the beginning of the session.

The obvious difference was that the Real was fully multimodal in that a student could smell, taste, and touch plants as well as perceive the temperature and feel the wind in the context of the environment. The students had to stay on the footpath, as it was a nature reserve that restricted human activity, and thus, the wayfinding and navigation were restricted to, essentially, a linear route. Additionally, there were surprises, conveniently provided by nature: a doe and her fawn, a mother turkey hen on her nest, a salamander in the stream, the cry of a red-tailed hawk, or a woodpecker pounding. The Virtual only had plants, so there was no opportunity for interaction with insects, amphibians, reptiles, or mammals.

However, the Virtual did allow for the students to fly, travel off-trail, and freely explore the entire space independently. The simulation allowed for the student and the teacher to interact in dynamic, real-time, synchronous, and asynchronous ways. It also allowed the teacher to present alternative views, such as a fly-through in the forest canopy, not possible in the Real. A more detailed account of the activity is published in Harrington [ 4].

Overview of the Study

4.1 Population and Participants

An overview of the study is presented here, but for a detailed ethnographic report, see Harrington [ 4]. There were 12 volunteer students from a local suburban public elementary school. They were all from a high socioeconomic class with identical high ratings in Enjoyment of Nature and Computer Knowledge on their preexperience user profile survey. This was an ideal user profile for software design activities, since a system that cannot perform adequately under ideal conditions with an ideal user profile would probably not perform well in challenging situations that presented confounding variables. Due to the complexities of working with real children and given the homogeneous student profile (10 of 12 from the same class, curriculum, teacher, preexperience profile statistically identical), the preexperience knowledge was assumed to be practically the same. All students volunteered in compliance with the US federal regulations that protect human subjects under research.

4.2 Materials

Materials included a preexperience user profile demographic survey, a 2D map for recording personally salient and meaningful information in situ, a field guide reference book [ 44], a posttest following either one real or one virtual experience [ 4], a postexperience interview and survey [ 4], and a follow-up microworld study whose results are not reported here.

In the lab, the students were able to use the software in a typical classroom that was equipped with a standard overhead projector; desks with PCs for each student; and with keyboard, mouse, and a one-earphone headset to facilitate both classroom conversation and private listening to the sounds made by the software. The curriculum was the same in all conditions. One group experienced the Real first and then, a week later, the Virtual, while the other group experienced the Virtual first and then, a week later, the Real, so that a meaningful comparison could be made. The educational material embedded in both the Real and the Virtual field trips was based on the fourth grade Natural Communities curriculum provided by the Audubon Society of Western Pennsylvania, located at Beechwood Farms [ 7].

The 2D map (see Fig. 5) was a critical tool used to record learning activity in situ as free-will, student-initiated annotations. The students were instructed to record anything of interest on their map. The maps were numbered with each student's unique ID, and all annotations were recorded after each field trip—Real and Virtual—so that they could be classified by experience type.

Graphic: Fig. 5. An example of the 2D map students used to annotate personally important and meaningful finds.

Figure    Fig. 5. An example of the 2D map students used to annotate personally important and meaningful finds.

4.3 Methods and Procedure

The Virtual Trillium Trail was created as a high-fidelity software simulation with the highest degree of accuracy. Every attempt was made to control confounding variables and to make a comparison feasible. However, there are differences. These differences represent the innate differences between the two environments. The two environments had the same curriculum, the same naturalist guide teaching, and the same maps, books, and fact cards.

All students received a preexperience user profile survey before experiencing their first field trip, Real or Virtual. An immediate posttest was administered, which showed no difference in scores [ 4]. A week later, they experienced their second field trip under the opposite condition, Real or Virtual. Each field trip lasted for 1.5 hours. The same expert naturalist guide conducted all the field trips. All parents were welcome to be present at all times.

All students had a 2D map and a wildflower guide book during each field trip. The students were encouraged to use their books for reference and to ask questions at will. In addition to the guide telling stories and pointing to items of interest along the way, students could find fact cards placed next to flowers and plants. They were instructed to mark anything of interest on their map, and in this way, an explicit count of objects and events of personal significance in both environments was captured for all students. This count is the main empirical measurement reported in this paper. It is a proxy for personally meaningful and salient information gathered in the in situ learning activity.

The data comparison methods consisted of measuring, in both the Real and Virtual environments, the student's in situ annotation activity by group, by environment, by order, and by type. As the annotation proxies resulted in a reliable object count, it is our claim that such a method is superior to the automatic data logging so often used, because it reduces the noise and gives the researcher an explicit student-created record of events and objects that the student recorded as perceived, observed and meaningful.

The postexperience survey consisted of 14 questions, representing different affective dimensions rated for each experience, Real and Virtual. So, there are two result sets for each question per student. A 5-point Likert Scale was used: 1 = Not at all, 2 = Somewhat, 3 = Average, 4 = Mostly, and 5 = A great deal. The results were applied to comparing and contrasting the real and virtual experiences.

Research Design

The investigative research design used a one-way, within-subject ANOVA, with repeated measures in counterbalanced order. In this design, the factor is Environment, with two levels, Real and Virtual. This design was required for a meaningful postexperience attitudinal survey that allowed direct comparison and contrast of Real and Virtual. Each subject acted as his/her own control [ 45], which allowed for a powerful and elegant statistical comparison of Real and Virtual. Each student's in situ map annotations resulted in a count per cell, and thus, produced the data required for the repeated measurement. Automatic data-logs were not used as they proved too noisy to be useful. This design was required for a meaningful postexperience attitudinal survey that allowed direct comparison and contrast of Real and Virtual. Additionally, there were two groups, thus allowing for a one-way, between-subjects ANOVA test, with Group 1 experiencing the Real-Virtual order and Group 2 experiencing the Virtual-Real order. Finally, t-tests showed significant transfer effects.


There are several different hypotheses required for complete and accurate comparison. First, using a one-way, within-subject ANOVA, with repeated measures in counterbalanced order, three hypotheses become viable at a level of significance of $\alpha = 0.05$ . One hypothesis is between the environments, Real and Virtual, and the other is between the Orders, First and Second. A subsequent set of hypotheses, using t-tests, investigates transfer of learning activity and the impact of priming, at a level of significance of $\alpha = 0.01$ . Finally, and to give context to the research results, an empirical comparison using a one-way, within-subject ANOVA is employed to generate attitudinal rankings on the learning environments, at an $\alpha = 0.05$ level of significance.

6.1 Hypotheses for Environment and Order

Total in situ map annotations recorded in the Real environment should be higher than in the Virtual, as there will be more nature-driven events and signals from the forest, and thus, more total learning. Total learning in this study context is defined as a holistic, contextual, complex causal chaining of multidimensional factors, and the activation of complete knowledge ontologies, even though such knowledge may be exogenous to simple in-curriculum tests:

H1: Hypotheses for Environment Impact on Total Activity:

$$\eqalign{&H1_{0}{:}\; \mu \;Total\; Activity_{(Real)} = \mu \;Total\; Activity_{(Virtual),}\cr &H1_{a}{:}\; \mu \;Total\; Activity_{(Real)} > \mu \;Total\; Activity_{(Virtual).}}$$

Slicing out the subset of data, which represents in-curriculum, Plant-Only learning activity is expected to be equal in the Real and Virtual environments, as the curriculum and environments are statistically identical:

H2: Hypotheses for Environment Impact on Plant-Only Activity:

$$\eqalign{&H2_{0}{:}\; \mu \;Plant\hbox{-}Only\; Activity_{(Real)}\cr &\qquad\quad = \mu \;Plant\hbox{-}Only\; Activity_{(Virtual),}\cr &H2_{a}{:}\; \mu \;Plant\hbox{-}Only\; Activity_{(Real)} \cr &\qquad\quad\ne \mu \;Plant\hbox{-}Only\; Activity_{(Virtual).}}$$

The Second experience will show more learning activity than the First for Total and for Plant-Only in situ map annotations. Expected are order effects, as repetition increases activity, wayfinding, spatial awareness, and ability; thus, the Second experience should outperform the First:

H3: Hypotheses for Order Impact on Activity:

$$\eqalign{&H3_{0}{:}\; \mu \;First\; Experience = \mu\; Second \;Experience,\cr & H3_{a}{:}\;\mu \;First\; Experience < \mu\; Second \;Experience.}$$

6.2 Hypotheses for Transfer Effects

Transfer effects should be observed in both directions, from Real to Virtual, and from Virtual to Real:

H4: Virtual Primes for and Transfers to Real:

$$\eqalign{& H4_{0}{:}\; Total \;Activity{:}\; \mu \;Real_{(First)} = \mu \;Real_{(After\; Virtual),}\cr &H4_{a}{:}\; Total\; Activity{:}\; \mu \;Real_{(First)} < \mu\; Real_{(After \;Virtual).}}$$

H5: Real Primes for and Transfers to Virtual:

$$\eqalign{&H5_{0}{:}\; Total\; Activity{:}\; \mu \;Virtual_{(First)} = \mu \;Virtual_{(After\; Real),}\cr &H5_{a}{:}\; Total\; Activity{:}\; \mu \;Virtual_{(First)} < \mu \;Virtual_{(After \;Real).}}$$

Transfer effects for Plant-Only in situ map annotations should be stronger than the Total in situ map annotations statistical tests, as the content and visualization fidelity for plant life are statistically identical in both modalities:

H6: Virtual Primes for and Transfers to Real:

$$\eqalign{&H6_{0}{:}\; Plant\hbox{-}Only\; Activity{:}\; \mu \;Real_{(First)} \cr & \qquad\quad= \mu\; Real_{(After\; Virtual),}\cr &H6_{a}{:}\; Plant\hbox{-}Only\; Activity{:}\; \mu\; Real_{(First)}\cr &\qquad\quad < \mu \;Real_{(After\; Virtual).}}$$

H7: Real Primes for and Transfers to Virtual:

$$\eqalign{&H7_{0}{:}\; Plant\hbox{-}Only\; Activity{:}\; \mu\; Virtual_{(First)} \cr &\qquad\quad= \mu \;Virtual_{(After\; Real),}\cr &H7_{a}{:}\; Plant\hbox{-}Only\; Activity{:}\; \mu \;Virtual_{(First)} \cr &\qquad\quad< \mu\; Virtual_{(After\; Real).}}$$

6.3 Hypotheses for Attitudes

Comparisons of attitudinal ranks will show relative perceived strengths and weaknesses of Environments. All attitudes resulting from the Real and Virtual, respectively, will give supporting evidence to be equal to, greater than, or less than, dependent on overall multisignal parity, not including signal convulsion:

$$\eqalign{&H8_{0}{:}\; \mu \;Attitudes_{(Real)} = \mu\; Attitudes_{(Virtual),}\cr &H8_{a1}{:}\; \mu\; Attitudes_{(Real)} > \mu \;Attitudes_{(Virtual),}\cr &H8_{a2}{:}\; \mu\; Attitudes_{(Real)} < \mu\; Attitudes_{(Virtual).}}$$

Detailed Empirical Results

7.1 Overview of Results

Empirical learning activity, as measured by the map annotations, is higher in the Total data set for the Real environment (see Table 1). However, learning activity for the data subset of Plant-Only map annotations is identical. Notably, order effects show the addition of a Second experience, Real or Virtual, to result in more learning activity than resulted from the First experience alone (see Fig. 6). Transfers occur in both directions, with the stronger effect shown in the Plant-Only data subset. Overall, students preferred the Real to the Virtual as the more compelling learning environment.

Table 1. Data of the Total and Plant-Only Map Annotation Counts by Environments

Graphic: Fig. 6. Bar charts showing environment and order effects on in situ learning activity.

Figure    Fig. 6. Bar charts showing environment and order effects on in situ learning activity.

7.2 Results for Environment and Order

$$H1_{a}{:}\; \mu \;Total\; Activity_{(Real)} > \mu \;Total\; Activity_{(Virtual).}$$

According to the above accepted hypothesis, ${\rm H1}_{\rm a}$ , the Real Environment results in significantly higher Total learning activity than the Virtual. A one-way, within-subject ANOVA, with repeated measures in counterbalanced order, shows a significant and strong effect, ${\rm F}(1,11) = 4.68$ , ${\rm p} = 0.05$ , thus giving supporting evidence that the Real ( ${\rm M} = 4.5$ , ${\rm SD} = 2.71$ ) resulted in more Total learning activity than did the Virtual ( ${\rm M} = 2.83$ , ${\rm SD} = 3.43$ ). However, this relationship did not hold for the subset Plant-Only data set, as shown below in $H2_{0}$ :

$$\eqalign{&H2_{0}{:}\; \mu \;Plant\hbox{-}Only\; Activity_{(Real)}\cr &\qquad\quad= \mu \;Plant\hbox{-}Only\; Activity_{(Virtual}).}$$

The Real and Virtual environments are equal when the in-curriculum, Plant-Only learning activity data subset is analyzed separately. For the data subset of Plant-Only map learning activity, a one-way, within-subject ANOVA with repeated measurements in counterbalanced order shows no effect ( ${\rm F}(1,11) = 0.00$ , ${\rm p} = 0.95$ ). Thus, for Plant-Only data subset learning activity, the Real ( ${\rm M} = 2.75$ , ${\rm SD} = 1.96$ ) and the Virtual ( ${\rm M} = 2.83$ , ${\rm SD} = 3.43$ ) are identical.

In terms of Order, the Second experience resulted in higher learning activity than the First, independent of Environment, as shown in ${\rm H3}_{\rm a}$ :

$$H3_{a}{:}\; \mu \;First\; Experience < \mu\; Second \;Experience.$$

A one-way, within-subject ANOVA with repeated measures gives supporting evidence that the Second experience, independent of the environment, results in higher map annotation learning activities for both the Total map annotations, ${\rm F}(1, 11) = 16.23$ , ${\rm p} = 0.002$ , and the Plant-Only map annotations, ${\rm F}(1, 11) = 49.00$ , ${\rm p} < 0.000$ . The Second Total map annotation mean ( ${\rm M} = 5.75$ , ${\rm SD} = 2.18$ ) was greater than that of the First ( ${\rm M} = 1.58$ , ${\rm SD} = 2.53$ ), and the Second Plant-Only map annotation mean ( ${\rm M} = 4.82$ , ${\rm SD} = 2.12$ ) was also greater than that of the First ( ${\rm M} = 0.75$ , ${\rm SD} = 1.42$ ).

7.3 Results for Transfer Effects

If there is no transfer of skill, in situ learning activity should be the same. If there is a significant difference, then there is evidence of transfer. The t-test for two independent groups (one-tail, posthoc analysis) has been carried out to help explain the results.

For the Total map annotations, there was a significant difference, ${\rm t} \;\!\!=\;\!\! -2.29$ , ${\rm df} \;\!\!= \;\!\!10$ , ${\rm p} \;\!\!= \;\!\!0.023$ (one-tailed, $a \;\!\!= \;\!\!0.01$ ), between the conditions, Real (First) ( ${\rm M} = 2.8$ , ${\rm SD} = 2.6$ ) and Real (After Virtual) ( ${\rm M} = 5.8$ , ${\rm SD} = 1.9$ ), in the Independent Samples t-test. This result gives supporting evidence of the value of priming for the Real first with the Virtual, as the mean value significantly increased from 2.8 to 5.8 counts:

$$H4_{a}{:}\; Total\; Activity{:}\; \mu \;Real_{(First)} < \mu\; Real_{(After\; Virtual).}$$

If transfer effects behave like order effects, could not the opposite relationship be true? Could we see transfer from Real to Virtual? In the past, such an analysis was impossible for life-critical training, such as pilot flight training, medical student operation simulation, or combat military training. But, for educational and learning applications, the question is valid and viable. The results for Total map learning activity show the Virtual (After Real) ( ${\rm M} = 3.8$ , ${\rm SD} = 2.4$ ) to be significantly higher, ${\rm t} = -3.88$ , ${\rm df} = 10$ , ${\rm p} = 0.00$ (one-tailed, $\alpha = 0.01$ ), than the Virtual (First) ( ${\rm M} = 0.0$ , ${\rm SD} = 0.0$ ) in the Independent Samples t-test. This result gives supporting evidence of the value of reinforcement of the Real with the Virtual, as the mean value significantly increased from 0.0 to 3.8 counts. Note that no learning activity occurred in the Virtual when used First and independently:

$$H5_{\rm a}{:}\; Total\; Activity{:}\; \mu\; Virtual_{(First)} < \mu\; Virtual_{(After\; Real).}$$

For the Plant-Only map annotations, the Real (After Virtual) ( ${\rm M} = 4.0$ , ${\rm SD} = 1.26$ ) resulted in significantly higher activity ( ${\rm t} = - 2.83$ , ${\rm df} = 10$ , ${\rm p} = 0.00$ (one-tailed, $\alpha = 0.001$ )) than the Real (First) ( ${\rm M} = 1.5$ , ${\rm SD} = 1.8$ ) in the Independent Samples t-test. This result gives supporting evidence of the value of priming for the Real with the Virtual, as the mean value significantly increased from 1.5 to 4.0 counts:

$$H6_{a}{:}\; Plant\hbox{-}Only\; Activity{:}\; \mu\; Real_{(First)} < \mu \;Real_{(After\; Virtual).}$$

There was a significant difference between the Plant-Only annotations for the Virtual (First) and the Virtual (After Real) ( ${\rm t} = - 5.38$ , ${\rm df} = 10$ , ${\rm p} = 0.000$ (one-tailed, alpha $= 0.01$ )). There were more annotations in the Virtual (After Real) ( ${\rm M} = 5.67$ , ${\rm SD} = 2.58$ ) than in the Virtual (First) ( ${\rm M} = 0.00$ , ${\rm SD} = 0.0$ ) in the Independent Samples t-test. This result gives supporting evidence of the value of reinforcement of the Real with the Virtual, as the mean value significantly increased from 0.0 to 5.67 counts:

$$\eqalign{&H7_{a}{:}\; Plant\hbox{-}Only\; Activity{:}\; \mu \;Virtual_{(First)} \cr &\qquad\quad< \mu \;Virtual_{(After\; Real).}}$$

7.4 Results for Attitudes on Environments

Student subjective attitudes were gathered using a postexperience survey [ 4]. The comparative ranks as a natural scale—on usability, attitudes, emotional reactions, esthetic assessments, and subjective reflections for both the Real and Virtual environments—give requisite framing and context to the results. Each environment was experienced in opposite counterbalanced order and directly compared using a one-way, within-subject ANOVA (see Table 2). The survey did not capture Plant-Only data, but allowed the student to compare and contrast the total learning experience.

Table 2. Real and Virtual Attitudinal, Emotional, and Affective Ranks (One-Way ANOVA, Within-Subject, Repeated Measurements)

7.4.1 Results on Attitudes

Attitudes ranked as equal represent an exciting accomplishment in interface design and virtual environment design for intrinsic learning environments, as the Virtual matches the Real. Attitudes ranked the same were: Awe and Wonder, Sense of Calm, Assessment of Beauty, Disinterest, Sense of Excitement, Level of Curiosity, and Desire to Share. Furthermore, the attitude of Desire to Share was statistically identical, ${\rm F}(1,11) = 0.00$ , ${\rm p} = 1.0$ , thus suggesting that programs like the VTT may prove effective for teaching collaboration and team-building:

$$H8_{0}{:}\; \mu\; Attitudes_{(Real)} = \mu\; Attitudes_{(Virtual).}$$

7.4.2 Real Ranked Higher than Virtual

Presence ranked higher, ${\rm F}(1,11) = 11.89$ , ${\rm p} = 0.00$ , for the Real ( ${\rm M} = 4.5$ , ${\rm SD} = 0.34$ ) when compared to the Virtual ( ${\rm M} = 2.83$ , ${\rm SD} = 0.83$ ). Inquiry ranked higher, ${\rm F}(1,11) \;{=}$$6.22$ , ${\rm p} = 0.03$ , in the Real ( ${\rm M} = 4.67$ , ${\rm SD} = 0.34$ ) when compared to the Virtual ( ${\rm M} = 3.67$ , ${\rm SD} = 0.03$ ). Learning ranked higher, ${\rm F}(1,11) = 12.79$ , ${\rm p} = 0.04$ , for the Real ( ${\rm M} = 4.83, {\rm SD} = 0.17$ ) when compared to the Virtual ( ${\rm M} = 2.17$ , ${\rm SD} = 0.04$ ):

$$H8_{a1}{:}\; \mu\; Attitudes_{(Real)} > \mu\; Attitudes_{(Virtual).}$$

7.4.3 Virtual Results on Attitudes Trending Higher than the Real

The Virtual trended higher in Level of Frustration and suggested a higher trend for Desire to Create, a need to Reexperience, and the ranking of Exploration. The students ranked Level of Frustration, ${\rm F}(1,11) = 3.36$ , ${\rm p} = 0.10$ , higher for the Virtual ( ${\rm M} = 2.17$ , ${\rm SD} = 0.48$ ) when compared to the Real ( ${\rm M} = 1.83$ , ${\rm SD} = 0.48$ ). The Desire to Create trended higher for the Virtual ( ${\rm M} = 4.33$ , ${\rm SD} = 0.33$ ) when compared to the Real ( ${\rm M} = 3.33$ , ${\rm SD} = 0.92$ ). A need to reexperience ranked higher for the Virtual ( ${\rm M} = 3.67$ , ${\rm SD} = 0.56$ ) when compared to the Real ( ${\rm M} = 2.67$ , ${\rm SD} = 0.61$ ). As expected, Exploration ranked higher for the Virtual ( ${\rm M} = 4.83$ , ${\rm SD} = 0.17$ ) when compared to the Real ( ${\rm M} = 3.33$ , ${\rm SD} = 0.67$ ):

$$H8_{a2}{:}\; \mu\; Attitudes_{(Real)} < \mu \;Attitudes_{(Virtual).}$$

Discussion of Results, Correlation Probe, and Future Work

The obvious limitation is due to the sample type. It was a volunteer sample, as is required by the federal law, and as such, the findings cannot generalize to students who do not like nature or who do not like computers. One must also view these results through the lens of a homogeneous sample of 12 subjects. However, in situ activity of map annotations is high ( ${\rm n} = 85$ ) and postexperience attitudes reported are sufficient ( ${\rm n} = 28$ ). Using a one-way within-subject ANOVA, with repeated measurements in a counterbalanced, design is a sound statistical method, as is the use of the t-tests. Setting levels of significance, with $\alpha$ set at 0.05 or 0.01, and reporting actual p-values are acceptable methods for identifying the findings most relevant to instructional design with virtual environments.

There is supporting evidence that the Real environment is the significantly superior learning environment when compared to the Virtual for Total map annotation, and there is evidence that the attitudes rate the Real experiences as higher overall. The ethnographic observation of the salamander find recorded in the Real environment was reported as a Salient Event [ 4]. Empirical recorded map annotations give evidence of the importance of such Salient Events, as 100 percent of the Second Group in the Real environment saw and recorded the sighting of a salamander.

No animals were available or seen in the Virtual. Such out-of-curriculum Salient Events may be important for long-term episodic memory of the entire curriculum in that they provide an anchor for the entire knowledge ontology, and thus, are of future investigative concern. Such Salient Events, if properly seized by the teacher, are important triggers for Teachable Moments [ 46]. This evidence provokes important user interface design and learning sciences questions on the interaction of the environment, the software, and learning.

There are significant order effects for both the Total map annotations and Plant-Only map annotation learning activity. The Second experience, independent of environment, results in higher learning activity for both in- and out-of-curriculum material. Therefore, the Virtual can prime for the Real, especially for desired activity within curriculum. In the learning activity of a field trip, the desired activity was to increase perception, observation, inquiry, and recording of information. Repetition, Real or Virtual, increased the desired learning activity.

Furthermore, significant transfer of learning activity occurs when both experiences are combined, independent of order. However, the Real to Virtual transfer effect is stronger, perhaps because of the obvious richness inherent in the Real environment. These results together suggest a new approach, one not considered in life-critical training simulators: that of offering the Virtual after the Real to allow for reflection, collaboration, and reinforcement. Thus, the results suggest using the Virtual First to prime and transfer learning of in-curriculum material for the Real, then following the Real with the Virtual to reflect, collaborate, and reinforce material learned. Given the stronger impact for in-curriculum material of Plant-Only data, the data show that the closer the simulation resembles the Real, the more powerful the transfer effect. In other words, as the simulation approaches reality, we can expect the transfer effect to become the order effect.

The postexperience attitudinal survey produced comparative ranks classified as equal or higher in the Real, or as attitudes trending higher in the Virtual. The attitudes that are equal represent an exciting accomplishment in interface design, learning sciences, and virtual environments. Their correspondence means that the software matched the students' perception of and reaction to reality insofar as the Real was the standard. Attitudes ranked as the same were: Awe and Wonder, Sense of Calm, Assessment of Beauty, Disinterest, Sense of Excitement, Level of Curiosity, and Desire to Share. Therefore, if such emotional reactions are pedagogically desirable, then a high-fidelity virtual environment is as effective as the Real. An important finding for research in social, collaborative virtual worlds [ 11], [ 12], [ 47], [ 48] was that the Virtual is statistically identical to the Real for the emotional reaction of Desire to Share.

Presence is cited as an important attribute of virtual environments [ 5]. Research and application of multimodal, immersive environments could yield higher learning activity in the Virtual and could also increase subjective rankings by the students. Here, Presence ranked higher in the Real, as the Real environment represents the highest degree of Presence possible, with many signals and redundancy gains. However, the Virtual was not completely without Presence; it just ranked significantly lower. This result opens important new questions for future research on the importance of Presence in virtual environments for learning. In addition to multiple signals replicated, the degree of Immersion [ 49] can be a factor that impacts Presence. The Virtual Trillium Trail system was a desktop, not an immersive, implementation [ 50]. Future research directions will move toward testing each factor for impact on Presence and Immersion.

Inquiry also ranked higher in the Real, supported by the higher in situ activity of the map annotation counts. Recorded in the field notes were observations of children using their imagination as they intently explored and discovered in the Virtual, with many pretending to “fly over the forest leaf canopy as if a red-tailed hawk,” “run through the woods like a deer,” or “swim in the stream like a fish.” The rate of active inquiry observed in the PC lab appeared to increase in the Virtual, but as the students had both hands on the computer, they may not have recorded items of interest on their paper maps.

Learning ranked higher in the Real, as more activity occurs therein. There were more Total map annotations in the Real, there were Salient Events, and consequently, there were higher subjective rankings of Presence and Inquiry. While the learning activity was equal for Plant-Only annotations, which represent in-curriculum material, from the students' perspective, they saw, inquired, and learned more in the Real.

To give depth to the discussion, and despite the doubt cast by the high number of tests, correlations were nonetheless used to probe for future research ideas. While Presence ranked higher in the Real, it is not correlated with Learning in the Real (Spearman Rank Order Coefficient, ${\rm r} = -0.05$ , ${\rm p} = 0.88$ ). However, while Presence ranked lower in the Virtual, it is significantly correlated to Learning in the Virtual (Spearman Rank Order Coefficient, ${\rm r} = 0.79$ , ${\rm p} = 0.00$ ). These statistics suggest that different factors apply to the Real and the Virtual. Presence may not be important for Learning in the Real, but may be important in the Virtual. The students interpreted Presence as the “feeling of being there,” and many students perceived the question as silly or irrelevant with respect to the Real environment, as they were there.

Inquiry was ranked higher in the Real and was significantly correlated to Learning in the Real (Spearman Rank Order Coefficient, ${\rm r} = 0.69$ , ${\rm p} = 0.01$ ), as compared to Inquiry in the Virtual, which ranked lower and did not have a significant correlation (Spearman Rank Order Coefficient, ${\rm r} = 0.30$ , ${\rm p} = 0.35$ ). The students interpreted Inquiry to mean the ability to ask questions. Some field-note observations were that, in the Real, each child took turns asking questions of the guide. This behavior contrasted with that observed in the PC lab, where not only was each child immersed in his/her own world, but each had to compete for the guide's attention. They would all ask questions at the same time, and about different objects relevant to their unique experience of the moment, creating competition for the guide's attention.

Significant correlation between Beauty and Learning in the Virtual (Spearman Rank Order Coefficient, ${\rm r} = 0.76$ , ${\rm p} = 0.00$ ) was found, but this was not found in the Real (Spearman Rank Order Coefficient, ${\rm r} = 0.00$ , ${\rm p} = 1.0$ ), even though Beauty ranked identically in both Environments. The other interesting significant correlation is between Presence and Beauty in the Virtual (Spearman Rank Order Coefficient, ${\rm r} = 0.64$ , ${\rm p} = 0.03$ ), a correlation not found in the Real (Spearman Rank Order Coefficient, ${\rm r} = 0.49$ , ${\rm p} = 0.11$ ). These results suggest complex interaction between emotion, perception, and learning in Real and Virtual spaces, interactions that are worthy of future analysis and research.

The level of Frustration trends higher ( ${\rm p} = 0.10$ ) (see Table 2) for the Virtual; future work may prove that faster machines impact this finding. The Desire to Create shows promise ( ${\rm p} = 0.14$ ) (see Table 2) in the Virtual and most likely resulted from the anticipated follow-up Microworld study in which students created their own Virtual Trillium Trail with all of the system's plant and terrain assets [ 51]. It also points toward new research directions into motivators of intrinsic learning with anticipated follow-up creative activity. The desire to reexperience shows promise ( ${\rm p} = 0.15$ ) (see Table 2) in the Virtual, thus, suggesting a use of virtual environments when voluntary repetition is desired, and especially where the leveraging of the power of the transfer effect is needed. Moreover, the level of Exploration shows promise ( ${\rm p} = 0.18$ ) (see Table 2) for the Virtual and could be one of the most promising advantages of virtual learning environments over the Real, as one can explore freely in the Virtual, without harm to the real, delicate, dangerous, or inaccessible ecosystem.


Significantly higher Total learning activity, combined with higher attitudinal ranks for Learning in the Real, gives supporting evidence to the claim that more Total learning activity occurs in the Real. However, Plant-Only learning activity is identical in Real and Virtual, which gives supporting evidence to the claim that the Virtual may be used for in-curriculum material. Thus, the Real is the superior learning environment overall, but the Virtual shows value for carefully targeted learning objectives of in-curriculum material, especially when the real environment is not available.

This result does not imply that schools should substitute “expensive” real-world field trips with “inexpensive” virtual reality field trips, but it does suggest the value of virtual environments for learning, if used properly. The empirical data show order and transfer effects. Therefore, for schools to maximize the learning impact of real-world field trips, they should prime the students before, and reinforce afterward, with follow-up virtual reality field trips. If only one experience is possible, use the Real. If the Real is impossible, carefully use the Virtual.

Great scientific exploration and discovery can only logically come from real and true environmental awareness, curiosity, a sense of wonder, and the ability to inquire when we recognize that we do not know. We must remember to “know that we do not know,” and therefore, cannot program artifacts, such as virtual environments, that are as authentic as the real world. These new tools require sensitive understanding to maximize their potential while respecting their limitations. We can design and build virtual environments for ecology education to support particular goals of factual, conceptual, and contextual awareness; strategically use features of free navigation and information annotation; and respect the interaction of form and function. At the same time, we should not use these powerful emotional and intellectual tools to mislead, frame, sell, or advance a biased or destructive agenda. In final conclusion, virtual environments present opportunities for research of the child-computer-environment interface. However, much future research is required to understand the complex interactions of learning in virtual worlds and to responsibly harness this powerful new technology as it approaches the truth.


The author wishes to thank Dr. Peter Brusilovsky, major PhD dissertation advisor, for his kind assistance and critical guidance; Dr. Susan Kalisz, professor of biology at the University of Pittsburgh; and Jessica Dunn, graduate student researcher in Pitt's Biology Department, for their Trillium Trail data sets. She would also like to thank Ms. Gabi Hughes, coordinator of the Environmental Education Program, Audubon Society of Western Pennsylvania, for her participation and advice. Most importantly recognized are all of the teachers, parents, and students for volunteering.


About the Authors

Bio Graphic
Maria C.R. Harrington received the BS degree in economics with a minor in art from Carnegie Mellon University and the PhD degree in information science from the School of Information Sciences at the University of Pittsburgh in 2008. She is now an assistant professor in the Department of Computing Science at Slippery Rock University, Pennsylvania, and the CEO of Virtual Field Trips, LLC. She has been an adjunct professor and visiting lecturer of human-computer interaction, new media art and design, and computer science. She served on the ACM SIGGRAPH Education Program Committee. She is a member of the IEEE.
58 ms
(Ver 3.x)