Multi-User Virtual Environments for Learning: Experience and Technology Design
OCTOBER-DECEMBER 2012 (Vol. 5, No. 4) pp. 349-365
1939-1382/12/$31.00 © 2012 IEEE

Published by the IEEE Computer Society
Multi-User Virtual Environments for Learning: Experience and Technology Design

Abstract—Multi-User Virtual Environments (MUVEs) are often used to support learning in formal and informal educational contexts. A technology-based educational experience consists of several elements: content, syllabus, roles, sequence of activities, assignments, assessment procedures, etc. that must be aligned with the affordances of the technologies to be used. The design process, therefore, has to follow a dual track: the design of the educational experience as a whole and the design of the MUVE. Each design process has some degree of independence, while, at the same time, the two design processes are also deeply intertwined. The paper proposes a novel approach to design (both for the educational experience and the MUVE): a "biological lifecycle" design, where evolution (for survival and fitness) is crucial, while anticipating all the requirements (creating an engineering blueprint) is very challenging. This paper is based upon a number of large-scale case studies, involving nearly 9,000 high-school students from 18 countries in Europe, Israel, and the United States. Substantial educational benefits were achieved by these learning experiences, at the center of which were MUVEs. It cannot be claimed that MUVEs were the only factors for generating these benefits, but for sure they were exceptionally important components.

Introduction and Context
This paper is about Multi-User Virtual Environments (MUVEs) for education. The main argument is that MUVEs, if properly used within the context of a well thought-out educational experience, can provide a substantial leverage to a learning process. To this end, the design of the experience and the design of the MUVE must be strongly intertwined. The requirements of the educational experience influence the MUVE's design while in their turns the features of the MUVE influence how the experience can be conducted. The design team, therefore, has to be very multidisciplinary, requiring education experts and MUVE experts to work together on an equal level basis: no side dictates the requirements to the other and a satisfactory synthesis must be reached. There is an additional complexity: being the educational experience innovative (as it often happens using technologies), even the experts cannot fully design it upfront. A complex design process is therefore involved, more similar to what may be called a "biological" design (or evolution) than to a traditional engineering design.
The paper draws on the authors' experience with four different MUVEs that were extensively deployed in real school environments between 2002 and 2008. Almost 9,000 students from Europe, the United States, and Israel were involved, as part of their formal education: i.e., at school and as a curricular activity, not as experimentation in a laboratory. The experiences were designed and deployed by HOC-LAB of Politecnico di Milano, with the technological support of GSA Lab of University of Salento (in Lecce, Southern Italy). A remarkable educational impact, as recognized by teachers and students, has been achieved with all of them [ 1 ].
The ancestor MUVE was "Virtual Leonardo," developed in 1999 in cooperation with the Museum of Science and Technology Leonardo da Vinci of Milan (Italy). The core idea, relatively innovative in the pre-Second Life era, was to build a 3D museum where virtual visitors could look at the exhibits and chat with other visitors. It was a scientific success (awarded "Best online exhibition" at the Museums and the Web Conference that year) but it was also a real-life failure: it did not provide the feeling and the emotion of being in a museum at all. Moreover, the activities were shallow and people's interest would wane after a short time. On the positive side, the socialization appeal was strong: people loved meeting and interacting with each other in the virtual world. It was clear that something could be built upon this.
The first educational experience based on a MUVE was Shrine Educational Experience (see http://www.seequmran .net, 2002-2004), developed in cooperation with the Israel Museum of Jerusalem. 1,500 students (from junior and high school) from Europe and Israel were involved. The subject was the Dead Sea Scrolls (which the museum is in charge of preserving) and related historical, sociological, and religious issues. The MUVE graphics reminded the museum's architecture ( Fig. 3 ), but what was going on was an educational experience, not an ordinary museum visit.
The second (and the largest) educational experience was Learning@Europe (L@E, http://www.learningateurope.net, 2005-2008), made possible by a grant of the international Accenture Foundation. More than 6,000 high school students from 18 European countries and three classes from the West Point Academy, United States, were involved, for a complex experience dealing with European identities and European history.
Stori@Lombardia (S@L, http://www.storialombardia.it, 2004-2006) was developed in cooperation with the regional government of Lombardy (Italy). The subject was medieval history and more than one thousand Italian students were involved. The format and the technology developed for the previous experiences were repurposed for a completely different cultural context.
Learning@SocialSport (L@SS, http://www.learningat socialsport.it, 2007-2009) was developed in cooperation with the Italian Accenture Foundation, Benetton (the fashion group) and the Italian Olympic Committee. The experience dealt with the ethical and social issues of sport, and it was aimed at young Italian athletes from sport clubs. Almost 500 participants (teens) were involved.
Table 1 provides a synoptic view of the four experiences above described.

Table 1. Participants in the Four Educational Experiences



In the remaining of the paper the focus will be on Learning@Europe, being it the most complex and comprehensive of the above experiences.
2. State of the Art
Multi-user 3D environments have become popular (also due to the success of Second Life) in a number of fields, especially eEntertainment and eMarketing. They can also be developed for Cultural Heritage in a broad sense, as with Kenderdine's Ancient Olympia [ 2 ], "Home of the Gods" Johnson's Monticello, the "Theban Mapping Project" [ 3 ], and Wolf's Quest [ 4 ]. While most of the cultural heritage 3D virtual worlds can be categorized as "informal education" (i.e., a situation where the visitor does not necessarily have a precise learning goal, but she may be expected to learn something), several studies have been reported about the use of MUVEs in "formal education," i.e., situations where a group of pupils in a class, possibly under the guidance of a teacher, have precise learning goals to fulfill [ 5 ]. Among the best-known examples are: Barab's Quest Atlantis, a persistent virtual world where children engage in curriculum-related quests to save an imaginary land from environmental disaster [ 6 ], [ 7 ]; Dede's River City [ 8 ], where teams of middle-school students investigate the social, health, and environmental causes of an epidemic in a virtual town; Bers' Zora [ 9 ], a virtual environment used by children with psychological, mental, or physical problems, who can find a way to express themselves by manipulating virtual objects and characters; AppEdTech [ 10 ], a graphical MUVE designed to support graduate students working over distance; AquaMOOSE 3D [ 11 ], a MUVE designed for the construction and investigation of parametric equations; MOOSE Crossing [ 12 ], a text-based MUVE designed for children aged 9-13; Revolution [ 13 ], a multiplayer role-playing game where students experience history and the American Revolution by participating in a virtual community set in Williamsburg; Whyville [ 14 ], a graphical MUVE designed for children between middle childhood and adolescence to communicate with old friends, to learn math, science, and history, and to build online identities; Critical Life [ 15 ], a MUVE that allows student nurses to practice their clinical skills using the Second Life platform; and Virtual Singapura (VS) [ 16 ], an intelligent agent-augmented multi-user virtual environment modeled on early 19th century Singapore, and a variety of artificial intelligence entities called intelligent software agents that act as the learning companions for the learners.
Looking at recent research overviews and surveys [ 17 ], [ 18 ], [ 19 ] on MUVE studies [ 20 ] and their findings [ 21 ], [ 22 ], a number of research questions have been addressed, for example: do games and virtual worlds work for all learners/subjects? How do we assess learning when it's happening in games and virtual worlds? How does the kind of learning that happens in games and virtual worlds map onto curriculum standards? Authors have a broad understanding of how MUVEs can be designed to support the situated and distributed nature of learning and thinking [ 5 ], recognizing the Distributed Cognition—which states that "knowledge and cognition is distributed across objects, individuals, artifacts, and tools in the environment" [ 23 ]—as a contributing theory for 3D virtual worlds in education. MUVEs also draw on Experiential theory, which states that learners gain a deeper understanding of a complex subject by cycling through "concrete experience, reflective observation, abstract conceptualization, and active experimentation" [ 24 ], and Constructivist theory, whose components include scaffolding, experimentation, and collaboration [ 15 ]. As MUVEs are designed to give students problems with multiple paths to the solution, performance-based assessments, such as proposals or final reports, seem to better assess the pedagogical benefits [ 25 ]. While teachers must be aware that extra time is needed to incorporate MUVEs into curricular activities, researchers recognize general benefits for students since 3D virtual worlds can assist with improving self-efficacy [ 26 ], and can provide environments that immerse the student in a specific role and task, encouraging and engaging him/her to become an explorer and experimenter [ 15 ]. Many published studies report on the impact evaluation of MUVEs in formal education. For example, Kennedy-Clark et al. [ 27 ] focus on analyzing the impact of structure in inquiry-learning activities in Virtual Singapura, showing that "adopting a low structure initial activity in inquiry-learning can result in better learning outcomes than using an initial high-structure activity." Researchers in [ 28 ] present a review of the emerging use of MUVEs to support interactive scientific inquiry practices revealing that "MUVE-based curricula can successfully support real-world inquiry practices based on authentic interactivity with simulated worlds and tools." Researchers in [ 29 ], [ 30 ] conducted interesting case studies collecting data and evidence about motivational aspects of the use of MUVEs in managed learning, i.e., as software tools and digital content specifically intended to support learning activities. Authors recognize that "maintaining student engagement is a major concern in higher education, especially when concepts become more sophisticated and coursework becomes more complex." Other works report on in-field observation and evaluation studies on collaborative and virtual interactive learning environments, both from a teacher's point of view, stressing his/her habit changes [ 31 ], and from a student's perspective [ 32 ], analyzing the "positive quantitative findings (the students' advancement in the knowledge and problem-solving generic skills concerned) of the study, with a combination of quantitative and qualitative methods of inquiry."
Looking at MUVEs from a more technical viewpoint, in an international survey on virtual world environments conducted in February 2010 interviewing educators, the New Media Consortium (a community of hundreds of universities, colleges, museums, research centers; http://www.nmc. org) classified the predominant software platforms. Second Life, as was to be expected, was the most commonly used (75.8 percent). The other surveyed platforms are: Open Sim (1.1 percent), Teleplace (0.7 percent), ActiveWorlds (0.4 percent), Blue Mars (0.4 percent). At the question "Where would you place Virtual Worlds on Gartner's Hype Curve [ 33 ] today?" participants answered as follows: Slope of Enlightenment (35 percent)—although the press may have stopped covering the technology, some businesses continue to understand the benefits and practical applications; Trough of Disillusionment (31 percent)—the technology enters the "trough of disillusionment" because it fails to meet expectations and quickly becomes unfashionable; Between Peak and Trough (17 percent); Peak of Inflated Expectations (7 percent)—a frenzy of publicity typically generates overenthusiasm and unrealistic expectations; Between Trigger and Peak (6 percent); Plateau of Productivity (4 percent)—the benefits of the technology become widely demonstrated and accepted. These data indicate that people are starting to see effective uses for virtual worlds, albeit also at a much more realistic level than early visions imagined. At the same time, at the question "What is the number one barrier to broader adoption of virtual worlds by your institution?" participants answered as follows: Learning Curve (33 percent), the lack of understandable or effective Pedagogical Models (17 percent), Negative Perceptions (10 percent) due to the negative depiction of virtual worlds in the press, Cost (6 percent). Fig. 1 shows the Gartner's Hype Curve for emerging technologies (August 2010).




Fig. 1. Gartner's Hype Cycle for emerging technologies (Source: Gartner, August 2010).







Researchers [ 34 ] have proposed specific software environments providing 3D virtual worlds to support collaborative eLearning, such as CVE-VM [ 35 ], DeskTOP [ 36 ], DigitalEE and DigitalEE II [ 37 ], Viras [ 38 ], and NICE [ 39 ], defining specialized software architectures for Web-based Educational Virtual Environments (EVEs) and Web3D EVEs [ 34 ].
Considering the number of research proposals that have been made relating to MUVEs in recent years (see, for example, the National Science Foundation (NSF) website in the US or the Community Research and Development Information Service in Europe), the quantity of such platforms and projects are foreseen to increase. Although instances of 3D virtual worlds for education are available and that specific technologies have been proposed both by industry and academia, as the NMC survey observes, pedagogical models, and organizational formats are strongly needed to introduce MUVEs into the context of formal education. This paper aims at reporting knowledge about how pedagogical requirements influence the design and the development of MUVEs for education. The approach is empirical, proposing credible evidence from international teachers on the job who created innovation in schools, where 3D collaborative technology played an important role.
3. Designing Educational Experiences and MUVEs
This section aims at clarifying how the educational experiences introduced in Section 1 were designed. Given the complexity of the artifacts, it is impossible to describe all the details; the interested reader is thus referred to [ 1 ], [ 40 ]. Here the overall flavor and some relevant examples will be provided.
Two design processes took place in parallel, continuously influencing each other: the design of the overall educational experience and the design of the MUVE. Both processes went on according to an approach that in [ 41 ] was baptized "biological life cycle": a nonlinear strategy, described in the next paragraph, quite different with respect to more common engineering design strategies.
3.1 Biological Life Cycle
The "try-monitor-modify" approach that was taken to design the educational experiences and the MUVEs is more similar to the "design" of an animal, say for example an armadillo (an apparently weird result), than to the design of a well engineered artifact, say for example a bridge (an apparently rational result). The analogy is the following: the school environment is like an ecosystem, where a new living creature (i.e., an educational experience based on MUVEs) is plugged in. The creature must survive (i.e., be accepted by teachers and students) and be functional (i.e., generate the expected outcome). Rather than deciding upfront all the details in a well-engineered blueprint, the designer in a biological life cycle creates a first version that may work, puts it in the environment and monitors what goes on: whether the creature thrives, how it affects and is affected by the environment. If something does not work properly or room for a better living is perceived, the creature must be modified immediately, making it to "evolve." After several rounds of "evolution," the creature is well developed: it may look weird and not fully rational (like an armadillo), but it works.
It is possible to identify several specific features that make a biological life cycle different from other life cycles more common in software engineering [ 42 ], where the closest well known approach is probably the so called "agile design" [ 43 ]. These are the most relevant features of a biological life cycle [ 41 ]:

    1. Trial and error. It is impossible to identify in advance all the possible dangers for the creature in the environment; if something becomes really dangerous, a quick solution must be devised and the creature modified. The ecosystem, in addition, evolves on its own and must be continuously monitored.

    2. By chance evolution. An unexpected event/characteristic in the environment may modify the creature.

    3. Arbitrariness of the solutions. It is the armadillo better (or optimized) with respect to other animals living in the same environment? It is hard to tell, and somehow it is an irrelevant question. What matters is surviving and getting the job done.

    4. Redundancy. Biological creatures sometimes have too many features. Is the design of an armadillo streamlined? Or, taking another example, do human beings really need five fingers on each hand? Why not four or six? Other animals have taken different solutions.

    5. Persistence of irrational features. Biological creatures often exhibit features they do not need anymore, which were probably necessary in previous stages of the evolution. It is more economical to leave them there, rather than streamlining everything.

    6. Irreducible wholeness. Survival and fitness for the job is ensured by the whole, not by individual parts. Clearly some parts are more important than others, still the whole has its own meaning which goes beyond the mere sum of the components.

    7. Apparent chaos. An armadillo may look chaotic, but a good natural science scholar can provide an explanation for each of its features (even those not needed any more).

Given the above cycle, we had a blueprint describing all the features that did not look like a well-organized engineering design; there was, instead, a complex map, tracking how each feature was related to requirements or to an "evolutionary step." Traceability of the design decisions became a crucial aspect [ 44 ], over the attempts of deciding all the details upfront.
Three quick examples may help to better understand the biological life cycle approach.
First example: the creature thrives better. Initially there were two "players" (i.e., two users of the MUVE) for each participating class, each one with his/her own avatar; later a third player, without avatar, was added. As better explained below in this section, there were two reasons for this: improving the organization in the class (since an additional group of students, surrounding the player, could be formed) and providing the opportunity to explore issues more in depth (since additional players had the role of answering complex open-ended questions).
Second example: evolution by genetic mutation. At one point it was detected that, for the mistake of a programmer, avatars could visualize scenes "through the eyes" of another avatar. This unusual feature was much appreciated by the users, so it was decided not to fix the bug. The rules of some games were instead modified, in order to better exploit this functionality that became an "advanced feature" of the design.
Third example: evolution forced by a conscious choice. The chat analysis showed that indexical words like "here," "there," "left," and "right" were difficult to interpret in the virtual 3D environment. The problem was exploited in a new game where a "blind" user had to move through a maze full of obstacles, under the guidance (via chat) of a remote user who could see the obstacles. "Find your way" became one of the most successful games in the MUVE.
3.2 Educational Experience Design
The two keywords for the design were: "entertainment" (there had to be fun) and "education" (there had to be substantial educational benefits for the participants). On the basis of previous experiences of the authors and following the suggestion of a group of experts, two lists of requirements were drawn for each of the two keywords [ 1 ], [ 45 ].
An example of entertainment requirement is: "promote collaboration." Collaboration with remote users (teammates of a remote class) was psychologically very important in order to build an "international virtual experience."
An example of education requirement is: "foster additional research activities." Assignments were quite open. They ranged from the collection of material evidences of the students' own country's history, such as monuments, street names, statues, to surveys: students interviewed people in the streets on their perception of national identity. More difficult assignments implied researching over a specific aspect of one's own history (e.g., the formation of national borders), then comparing results to those of the foreign team partners: the discovery of analogies among different European countries was one of the most culturally intense moments of the experience.
The educational experiences were designed as blended-learning experiences, stretching over a period of seven weeks. Four classes (from four different countries) were organized into two teams (two classes against two other classes). This arrangement forced remote collaboration among the classes of the same team and friendly competition among classes of different teams. Fig. 2 shows the design of the Learning@Europe experience: four MUVE sessions are intermingled with several traditional and less traditional learning activities.




Fig. 2. Learning@Europe: overall design of the experience.







Traditional activities included studying and discussing the background material, performing research, and assignments in the class and cooperating with the remote class of the same team: it was thanks to these activities that most of the learning occurred. Still, they were all in view of the 3D sessions, which in the designers' intention as well as in the users' perception were the real core of the experience. The analogy is with training for an Olympic race: the race may last for a few seconds, while the training takes years; the training, however, makes sense only in view of the Olympic race. The main "trainers" were the teachers, who assigned roles to the students, made sure that the background materials were studied and the assignments completed, and facilitated the experience overall.
Each session in the MUVE had a specific theme:

    1. Introduction.

    2. History of the participating countries.

    3. The general pattern of nation-building in Europe.

    4. Assignments' discussion.

Two problems related to content almost immediately arose: what content to provide and where to provide it. It was important to provide basic notions about history, and even more important to provide deep understanding of complex issues, such as national identities in Europe, in view of the development of different attitudes (tolerance towards different identities, for example). In addition, a fair competition required that all the participants would start from the same basis and this could only be guaranteed by providing all of them with the same background content. Cost limitation prevented the engagement of professional writers, therefore the solution was to interview leading experts; interviews were then synthesized into readable documents. For Learning@Europe, more than 20 experts (European sociologists, political science experts, and historians) were interviewed, offering a multifaceted view of historical phenomena across Europe; more than 50 documents were derived from these interviews. In addition, chronologies and maps were prepared, for each European country. Another example of biological life cycle: it turned out that these documents could be difficult for younger students (14-15 years of age), therefore a simpler and shorter version of each was created. All discussions, questions, quizzes, and assignments in the experience were based on these documents.
A conscious design decision was that the MUVE could not be the place for content delivery, if not for minimal parts. First of all, it was important to avoid getting several users downloading material at the same time, in order not to stress the connections, which were quite bad at that time. In addition, and most important, each session lasted one hour only, with four classes (80 students) connected: it would have been a waste to have them spending a lot of time reading content instead of interacting with each other. It was a lesson learned in the first versions of the MUVEs, in which long texts were put over interactive boards in the virtual environment. Avatars would stand in front of them, apparently frozen, for several minutes: users were reading the texts. The overall impression was that nothing was going on and the MUVE was dead. It was thus decided to use MUVEs for what they are very good at: socialization, lively discussion, meeting with other people, etc. MUVEs were thus designed as meeting places (for multicultural exchanges), playgrounds (for games of different kinds), and competition-grounds (for quizzes and discussions).
Collaboration, within the class and among remote classes, was a high-level requirement responsible for many a design solutions. As a consequence, many activities required the synchronized action of two players of different classes. For quizzes, negative scores for incorrect answers were introduced, in order to encourage careful consultation via chat before throwing in an answer "just to try and see." Assignments required as final step to compare the findings between the two remote classes; after answering the question "how did religion play a role in the formation of your nation state?" for example, two remote classes (say from Poland and France) had to compare their answers and draw some conclusions from the comparison. Since social spaces were not popular at that time, three forums were introduced: one for discussing with experts, one for global discussion, and one for private conversations with the team mates.
3.3 MUVE's Design
The design of the MUVE consisted of several components: the "architecture" of the worlds, the avatars' powers and looks, the interactive elements (boards and panels displaying content items), the chats, etc.
The overall architecture was designed having in mind some general requirements: flexibility (content had to be customized rather quickly), accessibility (all schools had to be allowed to participate, not only the top ones with state-of-the-art technical equipment) and cost-effectiveness. The aesthetic quality and the faithfulness of the reproduction of real life buildings were not compelling requirements; previous experiences and the outcomes of initial surveys showed that the intrinsic graphic quality or precision did not affect the users' reaction nor their appreciation of the experience [ 46 ].
Despite the fact that the core of the background content was not delivered through the MUVE, there were still several content items to be made available in there: small texts and pictures about the participating classes, data e short presentations, quizzes and questions, etc. These content items had to change frequently, according to the participating classes and the countries involved; in the peak periods, the environment had to be reconfigured by educational staff (i.e., not technical people) up to three times per day. Therefore the 3D world was treated almost like a theatre stage, where backgrounds and objects on the scenes could be easily changed according to the needs.
Additional requirements influenced more fine-grained solutions. The labyrinth of the treasure hunt, for example, was redesigned several times in order to improve orientation and to make the game more challenging.
Design of the avatars' powers was a particularly difficult task. In some circumstances users had to be very "powerful" (i.e., be allowed to fly, to look through other avatars' eyes, to activate hotspots placed anywhere in the environment …) in order to play challenging games while in others, for example during a discussion, they had to focus on the chat. The design decision was to have "super-users" (i.e., staff people guiding the MUVE sessions) who were in charge of controlling what avatars (as a group or individually) could and could not do (e.g., walking, chatting, flying, using different visualizations, etc.). More specifically this choice was dictated by three higher-levels requirements: 1) social control in a 3D environment is more difficult than in real life and there must be a quick and effective way to react to misbehavior (e.g., forbidding an avatar from chatting); 2) games must be engaging, and special powers add to the engagement; 3) the participating students have to be kept together as a group, to keep the pace of the educational collaborative experience (tutors could, for example, forcedly gather all the users in one place).
As regards the avatars' number, only two avatars for each class were allowed in the environment. The original design choice was based upon a technical observation: too many avatars moving would require sending too many "refreshing messages" to participating computers, which often had poor graphics and connection. After the initial trials, good educational reasons also surfaced up: the environment was crowded enough with 10 avatars (two for each one of the four classes and two for the online tutors) and more avatars would have added confusion. Moreover, it was observed that groups of students spontaneously surrounded the students in charge of moving the avatar ( Fig. 3 ), supporting them, suggesting the right answers, discussing what to do next. Group work and team building were effective.




Fig. 3. Students in action during a MUVE session.







Therefore, in spite of the fact that later on the refreshing algorithm was improved, the number of avatars was not increased. Another kind of evolution took place instead, dictated by two factors: first, for classes with more than 20 students two players were too few to keep everybody busy; second, and more important, quizzes, and questions in the MUVE were too short to assess a deep understanding of the cultural issues raised by the background material. But it was considered unwise to keep avatars idle for 5-6 minutes, while pondering how to answer difficult questions. The design choice, in the end, was to add a third player connected but without an avatar. The 4 new players (one for each class) "met" in a special chat room: they had to answer difficult open questions, writing a short text of 10-12 lines in cooperation with the remote team mates. The best performing students, again supported by a team of other students, were selected in each class for this task. At the end of their job these players would join the global chat in the 3D environment, together with the avatar players. A two-fold benefit was obtained: better organization for each class and fostering deep understanding of the issues.
As explained above, sessions within the MUVEs were meant as the core of the overall experience: their design was reworked many times over the years and defined to the tiniest details. The sessions' main role was to provide the motivation and engagement that in turn would fuel the performing of all the educational (offline) activities. The cultural and educational goal of the experience was also present through quizzes, discussion, and content-based games. In order to accomplish all this and also following the authors' experience with other 3D environments, it was decided to keep users busy all the time long. A very strict schedule was organized, where every 10-12 minutes something new had to start. Fig. 4 illustrates as an example the various steps of session 2.




Fig. 4. Example of online session's storyboard: session 2, Learning@Europe.







At first, students presented their homework (their team's presentation), previously uploaded in the environment (1-2). A quiz, on historical facts and characters, was asked and a discussion followed (3.1). A third player answered open questions in a parallel chat, in collaboration with the remote team mates (3.2). The answers provoked a discussion in the global chat, becoming the occasion for additional learning. A Treasure Hunt took place in a labyrinth: students of the same team had to collaborate to select various objects (4) on the basis of cultural clues. A final overall discussion (not shown in the figure) closed the session.
In the next section, the technical details for the construction of the MUVEs will be presented.
4. The Supporting Technology
The current version (WebTalkCube) of the platform is the result of a number of refinements over previous versions. A few, overall, requirements were the starting point.

    • Simplified deployment. As the target was European public schools, the setting of the client had to be simple and without any specific technical installation. In fact, schools with very basic equipment and connection were able to participate.

    • Efficient repurposing, i.e., being able to quickly and inexpensively adapt the environment for different educational experiences (from Learning@Europe to Storia@Lombardia). This was never achieved as configuration in the strict sense: some level of reprogramming was always needed, but the amount of manpower involved was kept at a minimum (ranging from a few days to a couple of weeks).

    • Rapid configuration, i.e., being able to quickly set up the environment for different runs of the same experience. Nontechnical people had to reconfigure everything, without calling in programmers.

    • Collaboration control. Collaboration features had to be finely controlled even while a session was in progress since, like in a real class, different educational activities (each with its own rules) would follow one another.

A number of technical concerns that shaped the software architecture were also considered ( Fig. 5 ):

    • Client-side: the choice of deploying the client side in the form of a browser plug-in versus a stand-alone application.

    • Server-side: the need to keep a coherent shared state of the virtual world (e.g., the positions of objects and avatars).

    • Middleware and network: the responsiveness of the system. The sense shared by the participants of being together was influenced by the middleware architecture, thus a suitable Internet connection and a careful setting of the proxy/firewall were needed to reduce refresh delays, avoiding unwanted interruptions of the experience flow.

    • Collaboration: the granularity of the control on interactions among users-objects-world (who can do what and in which way, how the objects react to actions, how the actions modify the virtual space, and the shared state).

    • Tools: the richness of a toolset that provides back office and authoring funcob.

Fig. 5 provides a component view of the WebTalk software architecture. Two subenvironments separate the main responsibilities: staff use an offline environment for authoring experiences and analyzing interactions; students and tutors access an online environment during sessions.




Fig. 5. WebTalk software architecture.







Each client has three main components: the XML parser determines which objects must be loaded by the 3D renderer on the scene and how they can interact on the basis of the XML description. The application server provides content, graphic objects, and XML files via an http protocol. During session runs, clients update their own local object properties. The collaboration server propagates those properties to the corresponding shared objects, keeping a coherent shared state. The authoring environment provides a session configurator to customize XML files, an experience configurator to define the schedule of each collaborative session, and some analysis tools.
Two main points derived from the requirements: 1) great variability of microdesign needs, 2) a large number of setting variants. Traditional 3D environments mostly fail to support these needs, as their authoring systems (when pro-vided) are unsuitable for nontechnical users. To overcome this, the environment is based on a declarative (XML-encoding) description of the virtual world, including

    • Static properties: describing how the world appears (in terms of geometries, textures, object positions), i.e., encoding 3D graphic scenes using an XML syntax as well as the VRML or X3D.

    • Dynamic properties: representing how actions in the world can evolve through explicit rules called "cooperation metaphors," i.e., interaction patterns among users and between users and the environment. These rules encompass various aspects, like the way users can gather in groups to talk or navigate the virtual space, or how the state of the graphic objects is visualized. For example, a 3D virtual museum may encompass the "Guided Tour of the Museum" cooperation metaphor defined as follows: "Leadership in groups is tokenized, so any participant can acquire it and become a "tour guide." Other participants can follow the one which acquired leadership, and pull inside their point of view what the guide is looking at (in simple words, seeing from the eyes of the guide), using the "external avatar" visualization mode. This allows making very interesting virtual guided tours inside the virtual museum.

5. Educational Requirements and Design Solutions
As explained in Section 3, in a biological life cycle process, reaction to the environment is crucial. It is necessary to proceed through incremental adjustments, dealing with problems as they emerge, making changes and testing the effect, adding new features and removing when possible the useless ones.
In this section, a sample of how the various requirements (organized according to the educational benefits) influenced the design decisions, both for the MUVE and for the educational experience, is presented. A rough estimate is that 50 percent of the design was decided at the beginning and 50 percent was defined according to how "the creature evolved." All the examples discussed here are from Learning@Europe [ 46 ].
5.1. Cognitive Benefits
The overall educational goal of Learning@Europe was to bring students "to know more about each other's country's history" by enabling them "to reflect on the similarities of the processes of nation building and to consider Europe as a common frame of reference sharing values beyond the nation state" (L@E scientific committee official statement; http://www.learningateurope.net). This ambitious goal, making European students feel "European," was translated into a number of specific benefits: 1) knowledge acquisition of the European countries' histories; 2) deep understanding of the historical processes that occurred across Europe at the time of the nations' formation; 3) the capacity of critically re-elaborating the knowledge acquired. Table 2 shows the most relevant design solutions.

Table 2. Educational Requirements and Design Solutions for Cognitive Benefits (L@E)



5.2. Social Benefits
MUVEs are intrinsically cooperative, otherwise, it would not make sense to be many in one place. Therefore, a MUVEs-based educational experience must promote cooperation in all possible ways. In addition, many pedagogical theories put an accent on cooperation [ 48 ], [ 49 ], [ 50 ] as a key factor for triggering learning, especially if there is an external incentive or compensation, as in this case: winning the competition [ 51 ], [ 52 ]. Cooperation was essential all along the various moments of the blended experience, i.e., during the online sessions and between one session and the next, and also among remote peers as well as within the class, and thus various strategies were used to nurture cooperation among the learners. Table 3 shows how the cooperation requirements were fulfilled.

Table 3. Educational Requirements and Design Solutions for Social Benefits (L@E)



5.3 Media Literacy Benefits
A possible definition for media literacy is the ability to "effectively create, use and communicate information" in a digital world [ 53 ]. Table 5 shows how "media literacy" benefits were taken into account in the experience design.
5.4 Motivational Benefits
Motivation is a key factor in education: it can be defined as the willingness to make an effort in view of a meaningful goal. Engagement, instead, is more related to how attractive an activity is in itself. If both engagement and motivation are there, the educational experience is likely to be very profitable [ 54 ]. Table 4 shows how the design took into account the challenge of keeping the participants engaged and motivated throughout the experience.

Table 4. Educational Requirements and Design Solutions for Motivational Benefits (L@E)



5.5 "Attitude" Benefits
Learning@Europe had the goal of paving the ground for a "European identity" by putting forward the idea that identity is not given by birth, but it can be acquired through an historical-cultural process. Table 6 shows the main design solutions for this goal.

Table 5. Educational Requirements and Design Solutions for Media Literacy Benefits (L@E)



Table 6. Educational Requirements and Design Solutions for "Attitude" Benefits (L@E)



6. Evaluation
This section introduces the evaluation of the educational experience. The evaluation of the MUVE per se is not dealt with, since from the initial surveys it was clear that users were not able to comment specific aspects of the MUVE: their statements were all supportive and quite generic, with no hint that could be used to inform the redesign process in any way. Much more insightful were the comments and data about the educational impact. Still, MUVEs were the cornerstone of the educational experience, so assessing the educational impact can be considered an indirect way to assess the quality of the MUVE's design.
Assessing the impact of ICT in education is not an easy task, as Erstad's acknowledge [ 55 , p. 21]: "The most important point I have learned by studying the impact of ICT on Norwegian education in the last 10 years is the complexity and multilevel aspects of such innovations." An additional complexity was given by the fact that for all the experiences described in this paper the goal was not to develop just factual knowledge (which could be easily tested) but to foster deep understanding of complex issues and to favor a change of attitude. Reeves and Hedberg [ 56 ] recommend triangulation whereby multiple measures of a variable are considered to converge on a more accurate estimate of the value of that variable. Evaluation of a phenomenon as complex as ICT in education is more a matter of building a case for impact with multiple sources of evidence than it is in finding one absolute objective measure of impact [ 56 ], [ 57 ]. A large variety of data collection methods was adopted:

    Surveys to teachers: teachers were administered five online questionnaires; one before the start of the experience and one after each step of the experience.

    Surveys to students: students were administered two online questionnaires: one before and one after the experience.

    Session reports: after each online session, the two tutors (guide and helper) had to jointly fill in an observation-evaluation report.

    Forum reports: tutors filled in a weekly report about how the cultural discussion and the cooperation were going.

    Direct observations: observers went to monitor what was happening in the classroom. The method was rarely used (four to five classes per year) due to its high cost and for the fear of interfering with the learning process.

    Post-analysis of online sessions: online sessions were recorded and analyzed.

    Assignments evaluation: the tutors evaluated all the assignments done by the students.

    Focus groups with teachers at the end of the year (involving 20 teachers on average).

Specific evaluations have been discussed elsewhere (see [ 1 ], [ 58 ], [ 59 ]). For this paper, in order to simplify the discussion, the focus is on a specific year of Learning@Europe. In the school year 2006-07, Learning@Europe involved a total of 80 classes of students from 54 high schools in 16 European countries, with 1,615 students and 116 teachers. Eighty one MUVE sessions were performed (one had to be repeated due to a serious network failure that affected an entire Italian region). 1,824 surveys were collected from ca. 60 percent of teachers and ca. 35-50 percent of students, plus 81 surveys from the online tutors, 294 weekly and final forum reports, and 80 assignments' evaluations. Each survey was five to six pages long, and it was based on almost 30 questions, some of which were multiple-answer questions and others open ended. All sessions were recorded (with Camtasia). A focus group was held at the end of the school year with 20 teachers. The interested reader can access the teachers' and students' surveys in the website ( http://www.learningateurope.net, "download" area). Some of the most relevant findings are presented in the next sections.
6.1 Cognitive Benefits
Cognitive benefits were rated quite high, not only in terms of knowledge acquisition but also of critical thinking: students learned how to critically analyze historical phenomena and to apply general concepts to their local context ( Figs. 6 and 7 ).




Fig. 6. Learning@Europe 2006-07; teachers' opinion about cognitive benefits (61 respondents).











Fig. 7. Learning@Europe 2006-07; students' self-perception (498 respondents) about cognitive benefits.







The evaluation of the assignments was carried on by experts who scored their quality. The following tasks were assigned: 1) interviewing peers about their perception of national identity, 2) collecting and presenting material evidence of the students' national identity (e.g., flag, national anthem, monuments, etc.) and c) reflecting on a given historical theme, such as "the Enlightenment" or "the influence of religion in the process of nation-formation," and comparing reflections with the international team partners. The evaluation was based on a four-point scale: poor, acceptable, good and very good. Sixty nine percent of the delivered assignments were scored above acceptability (i.e., as acceptable, good, or very good).
6.2 Social Benefits
Forums' discussions were carefully monitored by the tutors and a weekly report and a final report were made for each. In year 2006-07, 63 forums were active. In 83 percent of them, tutors report that students were curious to know each other, and eager to talk about their cultures, their countries and their history. In 95 percent of forums students were friendly to each other, on a personal basis. In 76 percent of the forums students collaborated on homework assignments; the team manager (a student) was active or very active in 81 percent of team forums, encouraging peers to contribute to discussions ( Fig. 8 ).




Fig. 8. Learning@Europe 2006-07; teachers' opinion (61 respondents) about group work.







A teacher reported (focus group): "Each one's skills were resources for the class. They understood that, by playing their role well, the whole team would benefit. I saw none of the usual jealously for those who controlled mouse and keyboard: they stood together, united to win."
6.3 Motivational Benefits
According to teachers, in 76.2 percent of classes at least 70 percent of the students were actively involved in the activities for the session, and in 73.9 percent of the classes more than two thirds of the students participated actively during the sessions. The interaction with foreign peers and the competition were identified as the key motivational elements ( Fig. 9 ).




Fig. 9. Learning@Europe 2006-07; teachers' opinions about motivations (61 respondents).







Students seemed to prefer the interaction with foreign students above all, in the MUVE and also in the forums. They also appreciated the group-work with their class-mates (average score: 3.72 and 3.68 out of 5). The highest average rating (3.81) is given to the overall L@E experience, rather than to any of its specific activities. Surprisingly, the games score only slightly higher than the questions in the 3D world, and their ratings are a little lower than the discussions in the parallel 2D chat without 3D graphics, where competition is purely cultural ( Figs. 10 and 11 ).




Fig. 10. Learning@Europe 2006-07; students' opinion about what was engaging (498 respondents).











Fig. 11. Learning@Europe 2006-07; students' opinion about motivation (498 respondents).







Most of the participating students experienced a high level of engagement. During the focus group at the end of the year, a French teacher reported: "They were so enthusiastic and excited that it was difficult for us to calm everybody down; it is so different from a usual lesson that I'm not sure they have realized they have communicated in English; you know, French students are so reticent to speak English, and in this situation you could feel that everything was going well." Another teacher said: "It may sound strange but …after this experience, some of my students got more motivated and proficient in other school subjects."
6.4 Media Literacy Benefits
Participation to the learning experience implied using a number of technology tools and supports, ranging from the MUVEs themselves to forums, email, internet (searching for content), HTML programming, etc. Thus it came as no surprise that media literacy ( Fig. 12 ) was one of the main benefits.




Fig. 12. Learning@Europe 2006-07; teachers' opinion about media literacy (61 respondents).







6.5 Attitude Benefits
A change of attitudes only occurs when a deep impact is achieved and motivation is raised (see similar outcomes in [ 39 ]). As Fig. 13 shows, more than 93 percent of teachers rated good or higher the improvements in the students' attitudes toward other cultures, and 85.2 percent reported a good or higher increase in students' motivation at school in general. Also, almost 70 percent of teachers reported a general increase of proficiency.




Fig. 13. Learning@Europe 2006-07; teachers' opinions about "change of attitude" (61 respondents).







Students were also aware of the fact that their attitudes were affected by the learning experience (see Fig. 14 ). "Now I understand that Europe is ME," a student said. Many students realized that history can be an interesting subject. Others changed their minds about the usefulness of English and of computers as learning tools.




Fig. 14. Learning@Europe 2006-07; students' opinions (498 respondents) about change of attitude.







A teacher reported (focus group): "Before starting Learning@Europe, my students felt just French. Now they feel French AND European." Students commented: "Thanks to L@E I respect more other cultures. I became more tolerant and respectful towards people from other countries"; "now I think that Europe is not only an idea, but something concrete." The forums also gave evidence of significant changes of attitude ( Fig. 15 ).




Fig. 15. L@E 2006-07. A contribution to the public forum from a Latvian student.







The learning experience was a complex one, with many factors concurring on it. The overall pattern that emerges from the evaluation is that the benefits were real and substantial, even in the difficult context of formal education in several different countries, each with a different educational system.
7. Conclusions
The conclusions that can be drawn from the above is that MUVEs can effectively empower complex educational experiences in formal education. There remains three nontrivial questions: 1) What was the specific role of the MUVE in the learning process? 2) Which features of the MUVE were the necessary ingredients for the successful educational outcomes? 3) What is the overall lesson learned about design?
As regards the first question: the MUVEs played a crucial role in all the educational experiences discussed in this paper but at the same time it must be acknowledged that they could not have "done the job" alone. But this applies to any tool used in education: "technology for education" does not mean that "technology makes education" nor that "technology covers all the aspects of education." The MUVEs were felt as the cornerstone of the experience, which was called by everyone a "3D-world experience" even if the time spent in the MUVE was a small percentage of the whole experience. Students were engaged by the MUVE sessions: they typically "asked for more" at the end of each of them and they very reluctantly logged out.
As regards the second question: MUVEs were conceived to support playful social interaction as a stepping stone to facilitate learning. This meant: to support activities rather than content delivery; to keep actions fast-paced to avoid users getting bored; to surprise users with unusual features; to alternate serious discussions with games; to have superusers (the staff tutors) who kept everything working and on schedule; to display the users' generated content (pictures, presentations …) in the 3D world, which was fascinating for the students at that time; to alternate activity games in which moving and exploring the environment was relevant with discussions in which paying attention to the chat was relevant.
As regards the third question, the answer is more complex. First of all, the parallel design processes of the experience and of the MUVE, with strong interrelation between the two, must be acknowledged from the beginning. The experience design generates requirements for the MUVE and the MUVE suggests possibilities or requirements for the experience. This implies a multidisciplinary design team working together: the standard setting, with pedagogy experts defining requirements and developers implementing them, does not work well. Second, the need for an evolutionary "biological life cycle" has to be acknowledged. The initial design must be put in the field, closely monitored and continuously modified. Evolution never stops, and the pace of the evolution must be fast, in order to keep track with problems and opportunities created by the environment (i.e., by teachers, students, educational goals, technical affordances, etc.). The third lesson is to keep well in mind the educational requirements. In an educational experience customer satisfaction is important, but double checking what is actually being learned is far more important. In the pedagogy community, many critics of technology-based solutions emphasize the fact that the engagement that technology often provides is not a benefit on its own: it must be leveraged to obtain educational benefits.
Better understanding the complex relationship between the design of technology-based educational experiences and the educational outcomes is the next research step. In addition, a specific concern is "inclusion" of best and least performing students [ 60 ], [ 61 ]. Technology in fact, as also shown by the experiences discussed in this paper, seems very good at involving even those students that remain at the borders in ordinary educational activities. This research will take into consideration a broad range of technology-based educational formats [ 62 ].

Acknowledgments

The authors thank all the people that contributed to the success of their programs: technical people, educators, teachers, and students. They also thank their experts (more than 20) who were interviewed in order to create specific content items, and the pedagogy experts (Thomas Reeves and Michael Orey of the University of Georgia, Athens, and Jennifer Preece of the University of Maryland, College Park), who helped to interpret the evaluation data. They are also grateful to their partner institutions (the Museum of Science and Technology in Milan, the Israel Museum in Jerusalem) and various sponsors (Accenture International Foundation, Accenture Italian Foundation, and the Regional Government of Lombardy). The work is partially funded by the Italian Ministry of Education, University and Research (MIUR) under the FIRB RBNE07RCPX L4A (Learning for All) research project.

    N. Di Blas and P. Paolini are with the Department of Electronics and Information, Politecnico di Milano, Piazza Leonardo da Vinci 32, Milano, Lombardy 20133, Italy. E-mail: {nicoletta.diblas, paolo.paolini}@polimi.it.

    A. Bucciero is with the Graphics and Software Architectures Lab (GSA-Lab), Department of Engineering for Innovation, University of Salento, Via per Monteroni, I-73100 Lecce, Italy.

    E-mail: alberto.bucciero@unisalento.it.

    L. Mainetti is with the Graphics and Software Architectures Lab (GSA Lab), Department of Innovation Engineering, University of Salento, Via Monteroni, I-73100 Lecce, Italy. E-mail: luca.mainetti@unisalento.it.

Manuscript received 10 Oct. 2011; revised 1 Apr. 2012; accepted 6 July 2012; published online 12 July 2012.

For information on obtaining reprints of this article, please send e-mail to: lt@computer.org, and reference IEEECS Log Number TLT-2011-10-0104.

Digital Object Identifier no. 10.1109/TLT.2012.16.

References





Nicoletta Di Blas is an assistant professor with the Department of Electronics and Information, Politecnico di Milano. She teaches communication theory at the Politecnico di Milano and communication for cultural heritage at the University of Lugano. Her interests focus on multimedia multichannel applications, serious games design, and impact evaluation.





Paolo Paolini is a full professor at the Politecnico di Milano. He is a scientific coordinator of HOC-LAB ( hoc.elet.polimi.it), where he has managed several projects related to e-learning (3D collaborative worlds, digital storytelling, interaction for children) and e-culture since 1998.





Alberto Bucciero received the PhD degree from the University of Lecce in 2006. He is a tutor in software engineering and computer graphics courses and a lecturer in data management at the University of Salento. His main research interests include enhanced-learning technologies, 3D virtual environments, and service-oriented architectures.





Luca Mainetti is an associate professor of software engineering and computer graphics at the University of Salento. His research interests include web design methodologies, notations and tools, and collaborative computer graphics. He is a scientific coordinator of the GSA Lab— Graphics and Software Architectures Lab ( www.gsalab.unisalento.it). He is the Rector's delegate at the ICT.