The Community for Technology Leaders

A Mobile Mixed-Reality Environment for Children's Storytelling Using a Handheld Projector and a Robot

Masanori Sugimoto, IEEE

Pages: pp. 249-260

Abstract—This paper describes a system called GENTORO that uses a robot and a handheld projector for supporting children's storytelling activities. GENTORO differs from many existing systems in that children can make a robot play their own story in a physical space augmented by mixed-reality technologies. Pilot studies have been conducted to clarify the design requirements of GENTORO from both technological and practical viewpoints. A user study indicates that GENTORO's ability to enable manipulation of a robot using a handheld projector in a physical space can enhance children's embodied participation in, and their level of engagement with, their storytelling activities, and can support children in designing and expressing creative and original stories.

Index Terms—Storytelling, children, handheld projector, robot, mobile mixed-reality, artificial, augmented, and virtual realities, collaborative learning.


Children create their stories at various times in their daily life. For example, they will often improvise a story by using puppets while playing. Making children create and tell stories is one of the teaching methods used in primary education [ 36]. There has been much research into supporting such story creation and story expression activity via the use of computational media. In accordance with this research, the term “storytelling” in this paper is used to refer both to children's “story creation” and to “story expression.”

In Japanese elementary school classes, children are asked to become individual characters in a given story, and read aloud their words by imagining their feelings and understanding scenes of the story [ 18]. Such storytelling activities are effective for developing the children's capabilities, such as linguistic skills, logical thinking, communication, and imagination [ 36]. However, we believe that if children can create their own story and express it visually and aurally in a physical space, as if they were producing a film, it will be more effective for enhancing children's creativity and imagination. Therefore, we have proposed a system called GENTORO ( Fig. 1) that uses a robot and a handheld projector [ 17], [ 27]. In GENTORO, children collaborate to create an original story by themselves and to express it dynamically in a physical space by manipulating a robot via handheld projectors. The features and effects of GENTORO can be summarized as follows:

  • In GENTORO, children can express their story in an immersive environment where physical and virtual spaces are integrated.
  • Because children can use a robot that behaves in a physical space as a character of their story, they can express the story as if they were producing a film or a Tokusatsu (special effects movie) [ 30].
  • The mobility of handheld projectors and robots enhances the children's embodied participation in their storytelling activities.

In this paper, the design, development, and evaluation of GENTORO are discussed. GENTORO's novel use of a handheld projector and a robot in a physical space makes it necessary to investigate how GENTORO supports children's storytelling activities, and the positive and negative effects GENTORO has on these activities. Moreover, the design requirements for making GENTORO acceptable to children as a tool for supporting their storytelling must be made clear.


Figure    Fig. 1. GENTORO supports children's storytelling in a physical space via the child's use of a handheld projector and a robot.

In the design and development of GENTORO, therefore, the authors aimed for minimal functionality and performance, to make GENTORO's behavior as stable as possible while retaining its features. To clarify the design requirements, pilot studies with schoolchildren were conducted by using the CoGAME system [ 15], developed by the authors, which had been confirmed to work stably in previous experiments. Development of the current version of GENTORO was based on requirements derived from the pilot studies, and was evaluated via user studies. Based on questionnaire surveys and video analyses, those aspects of GENTORO that enhance children's embodied participation, their collaboration and coordination, the level of their motivation and engagement, and the support for their creative storytelling became clear.

The organization of the paper is as follows: the next section briefly introduces work related to GENTORO. The following two sections discuss issues related to the design, development, and implementation of GENTORO via pilot studies. Then the GENTORO user study is described, with the lessons learned from its results and future work being discussed.

Related Work

Storytelling support systems for children can be divided into two categories [ 11]:

  • Desktop-based storytelling support systems: a typical feature of systems in this category is that children create a story and make story characters play in the virtual world. Some systems enable children to interact with characters shown on a fixed monitor, using conventional input devices such as mice and keyboards or novel input devices.
  • Physical-space-based storytelling support systems: a typical feature of systems in this category is that children play the roles of characters in their story in a physical space. They can interact with other children or artifacts in an immersive environment enhanced by mixed-reality technologies.

Fate2 [ 13], KidPad [ 14], and JabberStamp [ 21] are examples of desktop-based storytelling support systems. Fate2 supports children in collaborative story building in web-based 2D and 3D virtual spaces. KidPad uses local tools so that children can draw story scenes simultaneously and easily. JabberStamp enables children to synchronize their drawings and voices by using computationally enhanced stamps and trumpets as input devices. Howland et al. [ 16] propose a card-based interface for scripting plot events.

Many systems have been proposed to support children's storytelling activities by enhancing their interactions in a physical space. StoryRoom [ 1] supports children as authors rather than participants in their story in a room-sized storytelling environment. In [ 20], a physical programming tool that enables children in StoryRoom to create their own interactive environments in an intuitive manner is proposed. The Magic Carpet system [ 4] augments KidPad [ 14] with tangible interfaces to support children's storytelling in a classroom setting. KidsRoom [ 6] creates an immersive storytelling environment by tracking multiple children's positions and movements. In PETS [ 10], children use a pet robot that displays emotions and behaviors based on their story. StoryMat [ 24] is an interactive play mat that records and recalls children's storytelling activities. Similarly, Dolltalk [ 32] uses a computationally augmented doll and enables children to record their story and hear it back in different voices. In SAGE [ 31], a conversational stuffed toy is used for children's story authoring. Magic Story Cube [ 39] uses a head-mounted display (HMD) and a foldable cube for children's interactive storytelling.

The major difference between these predecessors and GENTORO is, as described in the previous section, that GENTORO enables children to conduct their storytelling using a robot and handheld projectors as if they were making a movie in a mixed-reality environment, which enhances their collaboration, coordination, and embodied participation.

The GENTORO system is related to several research topics in addition to children's storytelling, such as interaction techniques using a handheld projector and techniques for enhancing human-robot interaction in mixed-reality environments. Cao et al. propose interaction techniques using handheld projectors and passive pens for displaying information that is hidden in an environment [ 7]. Zoom-and-Pick [ 12] enables the user of a handheld projector to conduct accurate zooming and pointing actions in hand-jittering and image-distorted situations. RFIG Lamps [ 23] enables localization of the object position using a cluster of handheld projectors with adaptive projection techniques and photosensor tags.

Collett and MacDonald developed an augmented reality (AR) system for displaying visual information from robot sensors in robot programming [ 8]. Virtual Humanoid [ 25] is an AR system that synchronizes a humanoid robot and a virtual avatar using an HMD. Young et al. developed a system that captures an image of a simple robot (iRobot, Roomba) via a camera and shows the robot and expressions on its cartoon-like face on a user's tablet PC to promote human-robot interactions [ 38]. Augmented Coliseum [ 28] is a game environment based on display-based computing (DBC) that uses a display device (a projector or an LCD display) as a tool for estimating the position of a robot or for sending commands to move it.

GENTORO has inherited from our previous system called CoGAME [ 15]. Several new functions have been implemented for supporting children's storytelling, such as a story simulation module, a scene change function using a Wii controller, and a notification function that visualizes the recognition status of a robot. GENTORO's details are described in the following sections.

Design of GENTORO

3.1 Overview

GENTORO aims to support children's story creation and story expression. Story creation in GENTORO includes the design of a script and a visual and auditory rendering of the script. Story expression in GENTORO is a performance of the script, rendered via a handheld projector and a robot. In this paper, the children's storytelling activities are divided into three processes, as shown in Fig. 2. These are the story design process for writing a story script, the story rendering process for rendering the script, and the story expression process for expressing the story in a physical space.

Graphic: Fig. 2. Children's storytelling activities using GENTORO.

Figure    Fig. 2. Children's storytelling activities using GENTORO.

In the story design process, children individually and collaboratively brainstorm and discuss a theme for their story, its plot, and its characters. A character in their story is to be played by a physical robot. Based on the story script, decisions are made about the words of the characters, the narration, and rough sketches of individual scenes to be visualized graphically by a handheld projector. In the story rendering process, children draw detailed sketches and specify the behavior of the robot, including how it should move within a scene and what should happen if the robot approaches or collides with a virtual object in the scene. The children then conduct simulations on their PC to confirm the rendering of their story, to improve the scene drawings, and to develop the robot movements to make their story more impressive. Finally, in the story expression process, the children actually manipulate the robot and express their story via a handheld projector that projects graphical images as story scenes.

3.2 The Handheld Projector

In GENTORO, children are asked to use a handheld projector to control the movement of a robot in their storytelling. We know there are systems or toys (e.g., Candy Train) that enable children to control a robot in different ways. The reasons for using a projector can be summarized as follows:

  • In these systems or toys, children arrange special physical papers or tiles where scenes are drawn so that a robot can follow a designated path on them. One advantage of using a projector is that animated and dynamic scenes can be easily expressed, which is difficult when using physical papers or tiles. Moreover, papers and tiles may get dirty or broken, which is avoided when projected scenes are used.
  • Using a projector is a common method for implementing AR techniques. These techniques are used for collaborative work and learning applications in an immersive environment and have been confirmed as useful for raising the level of participants' motivation (e.g., [ 26], [ 37]). In this study, by including a mobile projector for robot control, it is expected to further enhance the level of children's commitment to and embodied participation in their storytelling activities.

There are several problems when a projector is used, such as a dark environment being called for, current projectors being neither lightweight nor small, and a projected scene often becoming blurred (without autofocus). However, a tiny and lightweight laser projector that can solve these problems came on the market in 2010 (e.g., [ 19]). Therefore, although we used an available handheld projector at the time of the system development and have tried to clarify problems related to the projector via user studies, one of our challenges is to explore the possibilities of new interaction techniques and applications that use a mobile projector and a robot for supporting children's play and learning.

3.3 Issues to Be Investigated

GENTORO does not explicitly support children in the story design process, because the goal of this research project is to clarify the effects of using a robot and a handheld projector. The remainder of this paper, therefore, focuses on the design of GENTORO for supporting children's story rendering and expression. As the users of GENTORO are elementary school children, it may not be appropriate to implement all the functions under consideration. If the functions implemented are too complicated or difficult for children to use, or these functions are sometimes unstable, the children will be distracted from their storytelling activities. Therefore, design decisions must be made about which functions to implement, what kinds of problems might occur, and how they might be solved from both a technological and a practical viewpoint. The following design issues were investigated via pilot studies:

  • Issues related to story rendering processes.
  • Issues related to story expression processes.

The CoGAME system ( Fig. 3) used in this pilot study is a system for an entertainment application that enables multiple players to cooperatively manipulate a physical robot without any sensors and guide it to its destination using a handheld projector. In CoGAME, a robot is recognized by a camera carefully attached to a projector so that their optical axes become parallel and as close as possible ( Fig. 4). With this alignment, it is possible to estimate accurately the position of the robot in the projected scene via an image captured by the camera. The robot developed by the authors has a PIC microprocessor, a Bluetooth communication module, and two wheels, each of which is attached to a small servomotor. The robot is controlled by the mobile PC connected to the projector, to make the robot follow a path drawn on the projected image. For cooperative navigation, players take turns. That is, transfer of robot control from one player to the other occurs by placing projected images of the two players so that the paths of the projected images connect, and the robot can move from one projected image to the other.

Graphic: Fig. 3. Three children manipulating two robots simultaneously in the revised version of CoGAME.

Figure    Fig. 3. Three children manipulating two robots simultaneously in the revised version of CoGAME.

Graphic: Fig. 4. How to recognize and manipulate a robot.

Figure    Fig. 4. How to recognize and manipulate a robot.

Several limitations of CoGAME were found when used to support children's storytelling. The major limitations to be overcome were that: 1) it could control only a single robot, 2) its control device (i.e., a handheld projector) was not designed to be used by child players, and 3) CoGAME lacked functions to support story rendering processes such as scene drawing and story simulations.

Following discussions between the authors and schoolteachers collaborating in this research project, it was decided to retain the feature of CoGAME that enables players to manipulate a robot via handheld projectors. This decision was made because: 1) using mobile equipment is expected to enhance the children's embodied participation in their story expression tasks in a mobile mixed-reality environment, 2) children can feel that they are active creators or producers of a film or a Tokusatsu that makes physical entities play in the physical world, and 3) the cooperative nature of a robot control task may raise the level of children's collaboration not only in the story expression processes but also in the story rendering processes.

In this section, two pilot studies for identifying the design requirements for GENTORO using CoGAME are described. The studies were videotaped and postexperiment questionnaire surveys and interviews with children and schoolteachers were conducted.

3.4 Pilot Study 1

3.4.1 Overview

Pilot Study 1 was conducted to make explicit the requirements for supporting children's story expression processes. The following main issues were investigated:

  • Problems or difficulties when children use a handheld projector for controlling a robot, and solutions.
  • The merits and demerits of multirobot control.

Before the pilot study, several functions of the CoGAME system were extended. These include multirobot control and the visual representation of interactions between the robots. For example, when two robots approached each other, the robots started to rotate, or displayed expressions such as “ouch” or “hello, where are you going?” in a balloon on the projected image.

The pilot study was carried out during a single day in November 2007 at a public elementary school near the authors' laboratory (Kashiwa Chiba prefecture, Japan). Twenty-four schoolchildren (fourth to sixth grade, aged 9-12) voluntarily participated during after-school hours. The children were divided into eight groups of three and asked to control two robots (a turtle and a rabbit) using a handheld projector, as shown in Fig. 3. The mobile equipment used by the children comprised a small lightweight LED projector (Mitsubishi LVP-PK, weight 0.5 kg) of reasonable resolution (800 × 600) and brightness, a lightweight USB camera (Logicool Qcam for Notebook Pro, weight 0.04 kg), and a mobile PC with sufficient computational power (Sony VAIO VGN-UX90PS, weight 0.52 kg). All the children in each group held the projector in their hands and carried the PC in a shoulder bag, as shown in Fig. 3. In the pilot study, individual children in each group first manipulated a single robot separately using their own projectors, but occasionally one of them manipulated two robots simultaneously using his/her own projector. The study lasted about 150 minutes, with each group using the system for 15 to 20 minutes.

3.4.2 Results

The major findings of the study obtained via questionnaires and video analyses were as follows:

  1. Twenty-two children (92 percent) could easily understand how to manipulate the robot and could control its movement as they intended.
  2. Via the questionnaires, 18 children (75 percent) were found to have a strong interest in the interactions between two robots. Four groups (50 percent) repeatedly improvised expressions for the robots when they approached each other (e.g., “hey, nice to meet you again”) and started their rotating movement (e.g., “oh, you are too strong, help me $\ldots$ ”).
  3. Two children (8 percent) in two groups could not control the robot stably because they had difficulty in holding the handheld projector.
  4. The system sometimes failed to recognize robots via the camera, which made robot control by the children impossible. In this case, the children could not understand what had happened and why they could not control the robots.
  5. Transfer of robot control from one projector to another did not always work smoothly, and this often irritated the children.
  6. When small children (in many cases, the younger children) used the system, they had to control the robot via a small projected image because of the short distance between the screen (floor) and the projector. This was a problem, because text or visual objects shown in the projected image were too small to be recognized.
  7. The display of a robot's words in a balloon was legible if there were only a few letters (less than 10 Japanese letters). Otherwise, the small size of the letters made it difficult to read the words. In addition, it was almost impossible for the children to keep a constant distance between their projector and the floor while manipulating the robot, which meant that with no autofocus function in the projector the projected image often became blurred.

Findings 1 and 2 represented positive results for GENTORO. Especially, Finding 2 indicated that the physical interactions between multiple robots could inspire children storytelling. However, the remaining findings were negative results. For Finding 3, it was necessary for the projector to be designed so that children could hold it easily. As the projector used in the pilot study is not as light and small as a cellular phone, which enables even children to hold it one-handed, the design of the hardware to enable the projector to be held easily with both hands must be considered.

With Finding 4, it became clear that improvements in robot recognition and the information display in cases of recognition failure were critical. The system used in the pilot study recognized the position and orientation of each robot by identifying different blinking patterns of three infrared LEDs mounted on its surface. If the camera attached to a projector is located directly above the robot, it can capture all LED signals and the robot is completely recognized. However, in a multirobot situation, with the robots moving away from each other, some robots frequently went beyond the field of view of the camera. In that case, to control the robots, the children had to step away from the robots and tilt the camera to enable all the robots to be within its field of view, which could lead to robot recognition failure. This means that, in that implementation of the system, simultaneous multirobot control was not always stable, and visualizing interactions between multiple robots, which engaged the children (Finding 2), did not always work properly.

There are several technical approaches to this problem, such as using a highly functional camera with a wide-angle lens instead of the small lightweight USB monocular camera, or using visual markers, similar to ARToolKitPlus markers [ 33], which show higher recognition rates than infrared LEDs. However, the authors did not take either approach, because, in the former approach, a highly functional camera is expensive (more than $\$1{,}000$ (US) at the time of this study) and attaching it to a projector would make the projector heavier. In the latter approach, the markers would be too obtrusive [ 34] and might distract children from their storytelling activities. Our primary goals are to clarify the effects of using a handheld projector and a robot for children's storytelling and to investigate the design of a storytelling support tool that is acceptable to children. Therefore, the design decision was made that a lightweight camera should be used for identifying a single robot, with the issue of simultaneous multirobot identification being postponed.

There were several reasons for Finding 5, including the robot recognition problem described above and hand jitter that made the recognition of the robot unstable when it was at the edge of the projected image. Different control transfer methods should be investigated.

For Finding 6, it is possible to make a projected image larger by using a wide-angle lens or mirrors to increase the distance to the floor from the projector. However, using a wide-angle lens requires nonlinear real-time image calibrations that involve considerable computational cost. Adding mirrors leads not only to difficult optical problems (e.g., optical axis adjustment) but also to a heavier projector. Video analyses of the pilot study indicated that tall children (in many cases sixth graders) could hold the projector sufficiently high to show a large projected image. Therefore, it was decided to restrict the target users to sixth graders in the absence of improved hardware components for the projector.

Finding 7 indicated that the visual representation of text for the words of characters or the narration via a projected image was not ideal, and other methods such as auditory representation should therefore be considered. To provide an autofocus function for sharpening the projected image, one solution is to measure the distance to the floor via external sensors such as ultrasonic sensors. However, a preliminary experiment proved that the measurement was not always accurate, and that it was difficult to adjust the focus automatically and precisely. Adding sensors also meant that the projector became heavier. Therefore, it was decided not to add a sensor for distance measurement and to omit an autofocus function.

3.5 Pilot Study 2

3.5.1 Overview

For Pilot Study 2, software modules to support story rendering processes were implemented. These modules included a scene drawing module, a robot path setting module, and story simulation modules. The children used a tablet PC (HP tc4200, $1{,}024 \times 768$ resolution) for scene drawing tasks. They then set the path for robot movement in each scene. The direction of movement for a robot was specified by a pen stroke on the PC. When the children had completed the scene drawing and the robot path setting, they then conducted story simulations to check the appearance of the rendering of their story on the tablet PC, as shown in Fig. 5. During the simulations, each scene automatically changed to the next scene at a specified time. The duration of each scene was decided by the time needed for children to speak the words of the characters and the narration of the scene. The time taken for each scene was also used to control the speed of the robot for their story expression in the physical space.

Fig. 5. The user interface of the story rendering modules.

Pilot Study 2 was carried out during a single day in March 2008 at the same elementary school as Pilot Study 1. Seven children (sixth graders aged 12, four boys, and three girls) who had participated in Pilot Study 1 volunteered. The children were divided into two groups of three and four. For this study, the children first spent about one hour on their story design, discussing the theme of their story and its script. The children then spent about one hour using the software modules for their story rendering. Finally, they spent 30 minutes expressing their story three to four times. The children spoke the words of the characters, narrated their story, and manipulated a robot by handheld projectors.

3.5.2 Results

From Pilot Study 2, the following issues emerged via video analyses:

  • The children were not confused by the use of the scene drawing module. However, they did not favor the module for the drawing task. As the resolution of the tablet PC was low compared with pen and paper, and the children were trying to draw finely decorated objects, they were not satisfied with their drawings and tried to draw scenes repeatedly.
  • The children could easily set a path for the robot in each scene by using the robot path-setting module.
  • The story simulation module was effective, with the children trying to check several things such as the movement of the robot and the tempo of their story. The children specified the duration of each scene and used the simulation module for the rehearsal of their story expression tasks.
  • In the story expression tasks, each scene changed to the next scene automatically after the specified duration, and the active projector (the projector showing the projected scene for controlling the robot) switched from one child's projector to another accordingly. However, the automatic scene transfer did not always occur as intended. The timing of the scene change was not sufficiently synchronized to the children's speech because children often spoke the characters' words and the narration faster or slower than expected.

The observations of the study and postexperimental inquiries to children and schoolteachers elicited the following facts:

  • The children put physical objects (their belongings such as erasers and pencils) on the floor and then expressed their story. These objects were used either as landmarks to lead the robot or as meaningful scene objects mentioned in their story.
  • The children spent longer than expected on their story design and the rendering processes. In particular, they had difficulty in deciding on their story theme because, individually, they had different ideas and had to spend a long time making the ideas converge to an agreeable theme. Children tend to lose their concentration and interest if much time is needed to complete a task. Therefore, it is better to ask children to select their story theme from some alternatives, rather than ask them to invent it by themselves. Similarly, the number of scenes drawn by the children should be limited.
  • During the expression of their story, two children in the group worked on the manipulation of the robot using the handheld projector, and the others spoke the characters' words and the narration. It was found that speaking a character's words and the narration while confirming the moves of the robot and considering the timing of scene changes was too difficult for one child alone. During the story expression tasks, each child concentrated on his/her own task and did not pay full attention to the others' tasks. Therefore, to make the story expression successful, the number of children in a group should be increased, and each child's task load should be reduced.

System Configuration of GENTORO

4.1 Design Requirements

Based on the lessons learned from the pilot studies described in the previous section, the design requirements for GENTORO were specified as follows:

  • Story rendering function.
  • Story expression function.

Other practical issues related to children's storytelling are as follows:

  • Because of the size and weight of currently available handheld projectors, the target users in elementary school should be sixth graders.
  • The number of children in a group should be five or more. For a group of five children, two children should work on the manipulation of the robot via the handheld projector, one child on speaking the characters' words, one child on the narration, and one child as the “director” who gives directions to the other children to enable coordinated and synchronized behavior. This child manipulates a scene control device, as described below.

4.2 System Configuration

Fig. 6 shows the system configuration for the current version of GENTORO. The children's scene drawing on paper is scanned and stored in the image database in Scalable Vector Graphics format. The robot path setting module then loads each scene image and asks the children to set a path with a stylus pen on the tablet PC. The simulation module plays the story by changing scenes and moving the robot using the specified times for individual scenes.

Graphic: Fig. 6. System configuration of GENTORO.

Figure    Fig. 6. System configuration of GENTORO.

The story expression function inherited many functions from the CoGAME system, but there are two important enhancements for supporting story expression processes. The first enhancement is to use a Wii controller device (from Nintendo) to control scene changes, as shown in Fig. 7. The child who works as the director can use the device as a real filmmaker would use a clapperboard. For example, the director can control a story expression task such as “start,” “scene change,” or “end” by swinging the device. The buttons on the controller can also be used for the same control function.

Graphic: Fig. 7. Scene control via the Wii controller.

Figure    Fig. 7. Scene control via the Wii controller.

The second functional enhancement is a visual indicator to show the robot recognition status. There are two types of indicator signal: “suspension” and “warning.” If the recognition of a robot by the camera on a projector fails, the “suspension” status is shown via the projector (see Fig. 8). A “warning” indicator lets the children know that the recognition level is becoming low. The recognition level is calculated by using the position of the robot's infrared LEDs captured via the camera and a small lightweight three-axis accelerometer sensor (weight 0.006 kg) attached to the projector ( Fig. 4). When the camera is directly above the robot, the LEDs mounted on the robot are located near the center of the image captured by the camera. Then the $Z$ -axis value of the accelerometer sensor will be almost the same value as the acceleration of gravity and the $X$ -axis and $Y$ -axis values will be approximately zero. By evaluating the data captured by the camera and the accelerometer sensor, the “warning” indicator can be shown in a visually unobtrusive manner when necessary. To enable children to conduct manipulations of the robot easily, a grip that is held by both hands is attached, as shown in Fig. 9.

Graphic: Fig. 8. A visual indicator signaling “suspension.”

Figure    Fig. 8. A visual indicator signaling “suspension.”

Graphic: Fig. 9. The grip for a handheld projector.

Figure    Fig. 9. The grip for a handheld projector.

The hardware components of the current version of GENTORO comprise a tablet PC and a scanner for supporting the story rendering processes, handheld projectors, mobile PCs, USB cameras (the same models as described in Pilot Study 1), accelerometer sensors (NEC-TOKIN MDP-A3U9S), and a server PC for supporting story expression processes. The accelerometer sensor is used not only for the “warning” indicator, but also for making a distorted projected image rectangular, as discussed in [ 15]. The server PC is used for managing the data related to story simulations (scene image, robot path, etc.), the robot (battery level, speed, etc.), and the mobile PCs (the active mobile PC that controls the robot, robot recognition level, etc.). WiFi communication is used to send control commands from a mobile PC to the robot via the server. Data transmission between the server and the robot, and between the server and the Wii controller, uses Bluetooth communication.

User Study

5.1 Overview

The user study of GENTORO took place over two weekend days in June 2008 at an elementary school in Kobe, Hyogo prefecture. A schoolteacher explained GENTORO to the children in his classroom and invited them to participate in the study. Twenty-five out of 40 sixth graders (aged 11 to 12 years, 13 boys and 12 girls) volunteered. The children were divided into five groups of five by randomly allocating two or three boys to each group. Before the user study, children were asked to create an “ itinerant turtle” story by using GENTORO. Each group was then asked to select one of seven story themes. Four themes were selected (“dream” by two groups, and “environmental problem,” “friendship,” and “future” by the other three groups). Individual children decided their roles in their group. Each child then manipulated the robot via the handheld projector, which took 10 minutes per group.

Three groups participated in the user study on the first day and two groups participated on the second day. Story expression tasks were executed on a field in their classroom (a $3.6 {\;\rm m} \times 3.6 {\;\rm m}$ white panel). More than 30 physical objects (stuffed toys, LEGO blocks, miniature cars, etc.) were made available to enhance the children's storytelling. Each group was asked to create their story twice. In the first and second trial, 60 minutes were allotted for the story design and story rendering processes, and 30 minutes for the story expression process. In the second trial, children were not allowed to change the theme of their story, but were encouraged to revise the story based on the experiences and findings of the first trial. In this user study, children were instructed to use four scenes and design the story by following the style of a four-panel comic. With Japanese children in composition classes being taught to write essays based on a four-phase structure comprising “introduction,” “development,” “turn,” and “conclusion,” it was not difficult for the children to design their story by following these instructions.

The children discussed and decided on a story script based on their theme and roughly sketched each scene. During the discussions, they tried including several physical objects in their story. In the story rendering processes, the children helped each other and drew scenes collaboratively using magic markers. After the experimenters scanned the scenes drawn on paper, children could then use the robot path setting module and the story simulation module without difficulty. They conducted their story expression tasks three times using the handheld projector, the robot, and the Wii device. The story expressed by the children was videotaped, edited, and uploaded to a web server in their classroom by the experimenters. On the next day of the user study, the children watched the videos of their own and the other groups' stories, and reflected on their storytelling activities in free discussions.

5.2 Results

The results of the user study using GENTORO were discussed with the children via postexperimental inquiries and video analyses. The inquiries included usability and free-answer questions about GENTORO, and lasted 30 minutes. During the user study, two experimenters and two video cameras (fixed and mobile) were assigned to each group. One experimenter did the videotaping and another took notes to identify the observed time of children's noteworthy utterances (e.g., about the design of robot moves or projected scenes, or about the selection and usage of physical objects), which facilitated the video analysis tasks.

Story expression using GENTORO. GENTORO provided children with a mobile and immersive story expression environment by using a physical robot and a handheld projector. Table 1 summarizes the children's responses to the usability questions, which were generally positive. Further inquiries were conducted with children who had responded negatively, and several drawbacks in the current version of GENTORO were found. The major drawback mentioned by the children was with the brightness and sharpness of projected scenes, which could seem dark and blurred. No serious problems occurred with the control of the robot via the handheld projector. However, some children stated that they could not move the robot during the story expression tasks as they intended. This problem was traced to the changed rotation speed of individual robot motors as the robot battery ran down, causing the robot to move in the wrong direction sometimes. Using physical objects in story expression was generally accepted by the children.

Table 1. Summary of Inquiry Results

Feelings about film and Tokusatsu making. Children repeatedly discussed the paths of robots and arrangements of physical objects during story rendering and story expression processes. A child working as a director often gave suggestions to the other children, aiming to make their story expression more successful. In the discussion on the next day of the weekend user study, many children explained their ideas about the moves of the robot, projected scenes, and the usage and arrangement of physical objects (e.g., “I would not use so many trees, but bring in more animal objects to make the scene more impressive”). This indicated that children using GENTORO had viewpoints similar to those of film directors or Tokusatsu creators.

Effects of mobility. GENTORO is different from many existing systems, and does not make children act out the characters of their story in the physical space. However, GENTORO enhanced the children's embodied participation and commitment via their story expression tasks. These were enabled by the mobility of the projector and the robot. The children who controlled the robot had to walk around in the play field and the other children, working as directors or speaking the narration or the words of characters, also had to follow the moving scenes and the robot. The mobile nature of GENTORO made children engage strongly in the story expression processes and act in a coordinated manner.

5.3 Enhancing Children's Creative Storytelling

5.3.1 Episode Analysis

To investigate how GENTORO affected the children's stories, one group's discourse during the story design processes in the first and second trials was analyzed. The story theme of the group was “friendship.” In the first trial, they devised the idea of using a pit to represent a character's comeback from his life crisis. In the second trial, they were inspired by the movement of the robot in their story expression process and devised a different idea, namely dropping physical objects for the robot (the left picture in Fig. 1). In the following transcripts, phrases and sentences in square brackets have been added by the authors to clarify the meaning of the discourse.

From this discourse, it can be seen that, via the story expression tasks in the physical space, the children tried to make the scenes of their story more dynamic by moving not only the robot but also other physical objects. The resulting story indicates that the children could use their imaginations and express ideas in a creative manner.

5.3.2 Evaluations Using Creative Product Semantic Scale (CPSS)

Overview. To clarify the effects of GENTORO in supporting children's creative storytelling quantitatively, videos of their stories captured by experimenters were evaluated using the CPSS method [ 5]. CPSS asks nonexpert evaluators to evaluate products in three dimensions, namely “Novelty,” “Resolution,” and “Elaboration and Synthesis.” The original CPSS includes 55 subscale items, each of which is represented by an adjective, and asks evaluators to rate each item on a 7-point Likert scale. As this is too time-consuming and burdensome for evaluators, a simplified version of CPSS [ 35] that uses 15 subscale items out of 55 is often employed. A more simplified CPSS method using six subscale items to evaluate children's creativity is discussed in [ 29]. In this study, by following the method proposed by White and Smith [ 35], 15 subscale items from the 55 items were selected to reduce the workload of evaluators, while retaining the meanings of the three dimensions proposed in the original CPSS as much as possible. The selected subscale items were “novel,” “unusual,” “unique,” “original,” and “fresh” from the “Novelty” dimension, “logical,” “makes sense,” “relevant,” “appropriate,” and “adequate” from the “Resolution” dimension, and “skillful,” “well-made,” “well-crafted,” “meticulous,” and “careful” from the “Elaboration and Synthesis” dimension.

In this evaluation, two stories created by each group of children in the first and second trials were compared. Children changed their story in the second trial after expressing their story in the first trial. Therefore, the purpose of this evaluation was to clarify how experiences of physical and embodied interaction (collaborative story expressions using a robot, artifacts, and handheld projectors in a mobile setting) affected the children's story.

As for the evaluators, they are requested to be familiar with the corresponding domain [ 3]. Therefore, six graduate students (male, aged 25-30) who have been studying educational technologies for at least two years and involved with educational practices through interactions with children in elementary schools were recruited.

Each of them was first asked to watch a one-minute example story on a laptop PC and instructed about their task, which was to rate the stories based on the 15 subscale items. Then the evaluators received 10 video clips comprising five pairs of stories from the first and the second trials by each group of children. The duration of each video was less than three minutes. To prevent order effects, the experimenters selected a pair of stories and instructed each evaluator to rate one of the two stories in random order. When they had rated one pair of stories, they wrote brief comments to explain their ratings. The evaluation lasted about 70 minutes.

5.3.3 Result

In order to confirm the internal reliability and consistency of the evaluators, Cronbach's alpha coefficient [ 2] was calculated for each subscale item. One guideline threshold for reliability and consistency is 0.7, and all the values shown in Table 2 are greater than this threshold. Therefore, the evaluators' internal reliability and consistency can be considered satisfactory.

Table 2. Cronbach's Alpha Coefficient for Each Subscale Item

Fig. 10 shows the rating results by the evaluators in terms of the 15 subscale items. As shown in this figure, the average scores of stories in the second trial are all higher than those in the first trial. Concerning the within-subject tests for each subscale item, 13 out of the 15 items were of significance, namely the story in the second trial being more “novel” $({\rm F}(1,4) = 11.98,{\rm p} < .01)$ , “unusual” $({\rm F}(1,4) = 6.67,{\rm p} < .05)$ , “unique” $({\rm F}(1,4) = 4.67,{\rm p} < .05)$ , “original” $({\rm F}(1,4) \;{=}$$8.17,{\rm p} < .01)$ , “logical” $({\rm F}(1,4) = 6.48,{\rm p} < .05)$ , “makes sense” $({\rm F}(1,4) = 11.41,{\rm p} < .01)$ , “appropriate” $({\rm F}(1,4) \;{=}$$5.95,{\rm p} < .05)$ , “adequate” $({\rm F}(1,4) = 8.01,{\rm p} < .01)$ , “skillful” $({\rm F}(1,4) = 6.26,{\rm p} < .05)$ , “well-made” $({\rm F}(1,4) = 13.75{,}$${\rm p} < .01)$ , “well-crafted” $({\rm F}(1,4) = 10.29,{\rm p} < .01)$ , “meticulous” $({\rm F}(1,4) = 5.28,{\rm p} < .05)$ and “careful” $({\rm F}(1,4) = 11.43{,}$${\rm p} < .01)$ . The evaluators explained the reasons why they rated a story as “novel,” “unusual,” “unique,” and “original,” such as the inclusion of an unexpected plot, usage of physical objects, or rendition (e.g., singing a song). Also the evaluators commented that stories in the second trial were more “logical” and “makes sense,” because they were more understandable than those in the first trial. One of the evaluators mentioned that stories in the second trial were not boring because they were more “meticulous.” The evaluators rating a story as more “skillful” explained that children could smoothly manipulate a robot using the handheld projector and change scenes, which might be because children were more familiar with GENTORO in the second trial.

Fig. 10. Average scores of CPSS subscale items.


In this section, the lessons learned and the findings about the design, development, and evaluation of GENTORO are discussed.

6.1 Effects of the Robot and the Handheld Projector

The novel feature of GENTORO is that it uses a robot and a handheld projector, which enables children to express their story successfully in a physical space. Several functions investigated via the pilot studies have not been implemented in the version of GENTORO that was used in the user study. However, the children could express their story in a creative manner by integrating a robot, mobile projected images, and physical objects. Therefore, GENTORO supports children's storytelling activities differently from existing systems, and shows different effects.

The robot used in GENTORO, which is very similar to an inexpensive radio-controlled toy car, does not have smart functions and sensors. However, postexperimental questionnaires and interviews proved that children had considerable interest in the robot, because it behaved like a living thing and always followed a path on a moving projected image. This feature of the robot, which cannot be achieved by a radio-controlled robot manipulated by a traditional remote controller, seemed to motivate children effectively in conducting their tasks. In addition, because many Japanese children have been exposed to the Tokusatsu culture by TV programs, they could easily understand storytelling tasks using GENTORO without confusion. An interesting comment from children in the postexperimental discussions was that they wanted to include not only scenes from the children's (the third person) viewpoint, but also from the robot's (the first person or story character) viewpoint, to make their story more like a film or a Tokusatsu.

6.2 Creativity Support

The effects of GENTORO in supporting children's creative storytelling were evaluated via episode analysis and the CPSS method. Moreover, schoolteachers commented in the postexperimental interviews and inquiries that children could devise an original story and express it in a creative way. The teachers reported that the constraint given to the children (to create and express a story in four scenes) enhanced their deeper discussion and thinking. In fact, the children included recent news items and knowledge from their science classes to make the story more realistic. However, the current version of GENTORO is just a tool for rendering and expressing a story, and does not support story design processes. Therefore, one of the challenges of the GENTORO project is to investigate techniques for supporting the design of children's stories.

6.3 Collaboration and Coordination

In using GENTORO, children experienced different types of group activity. The children worked in a collaborative manner in their story design and rendering processes. On the other hand, the children worked in a coordinated manner in their story expression processes. Individual children had different tasks but were asked to synchronize with each other. The children completed these group activities successfully because they shared the same interest (a robot augmented by a projector) and explicit goals (to design and express their own original story), and were well motivated to commit to them, even though the nature of the activities varied.

6.4 Learning Support in School Education

Motivation is a critical factor in learning [ 9]. Because GENTORO can raise the level of children's motivation for their tasks, applying it to a learning support system should be useful. GENTORO could be used in several aspects of school education, such as language classes and drawing classes. The authors are now planning to use GENTORO in English classes in junior high schools for students to design and express their own English stories. There is strong demand for the support of English learning by nonnative English speakers such as Japanese students.

6.5 Technical Improvement

There are many technical tasks left undone, such as object recognition using multiple robots. In the current version of GENTORO, the robot is like a dress-up doll: it becomes a turtle by wearing a green-colored decorated cloth. By using recent projection techniques, it may become possible to represent any character on a 3D object, as discussed in [ 22]. For scene drawing, GENTORO enables children to create a static background image. Some children asked to draw dynamic scenes, using animation to represent a blowing wind or rippling water, or using sound effects to increase the level of reality. To enable children to create visually and aurally enriched stories, improvements and extensions to GENTORO will be investigated.

6.6 Mobile and Fixed Settings

The mobile nature of GENTORO contributed to the children's involvement in and engagement with their storytelling activities. However, some technical problems, such as unstable object recognition and the small projected image, may be alleviated by fixing the position of the camera or the projector. For example, they could be attached to the ceiling. Comparative studies of “mobile” and “fixed” settings should be conducted to clarify their merits and drawbacks in children's storytelling.

6.7 Toward Design Guidelines

GENTORO is a novel storytelling support system that has many research issues to be investigated and tested from technical and practical viewpoints. Therefore, it is currently difficult to specify exact design guidelines. However, some findings may point to design guidelines:

  • Because of the size and performance of current hardware devices, the target children must be of a certain height. For Japanese children, this means sixth graders or older. However, this situation may change with the recent development of small-scale projector technologies, as discussed in Section 3.2.
  • To avoid losing children's concentration by using too much time, it is better to limit the number of story scenes. (Four-scene stories were designed in this study).
  • To enhance embodied participation, individual children should be assigned tasks with moderate difficulty levels, which promote their collaboration and coordination.
  • The appearance of a robot may affect the children's interest and engagement level. In this study, the “cute” appearance of the turtle robot attracted interest, particularly among the girls. However, cultural differences should be considered in discussing this issue.

More explicit design guidelines will be investigated in the future work.


This paper describes a system called GENTORO, which uses a robot and a handheld projector to support children's storytelling activities. GENTORO differs from many existing systems in that it can make a robot act out a story created by children in a physical space. As GENTORO is a novel system, two pilot studies were conducted to identify its design requirements. A user study of GENTORO indicated that the features of GENTORO enhanced children's embodied participation in and engagement with their tasks, and supported the design and expression of their creative and original stories. As the primary goals of this research project were to clarify the effects of the novel features of GENTORO and to explore its possibilities, several technical issues requiring investigation have not been addressed at this stage. Future work on the project will improve and extend the functions of GENTORO, and evaluate its ability to support children's creative storytelling via additional user studies.


About the Authors

Bio Graphic
Masanori Sugimoto received the BEng and MEng degrees from the Department of Aeronautics and Astronautics and the DrEng degree from the Interdisciplinary Course on Advanced Science and Technology, University of Tokyo, Japan, in 1990, 1992, and 1995, respectively. Currently, he is an associate professor in the Department of Electrical Engineering and Information Systems, Graduate School of Engineering, University of Tokyo. His research interests include human computer interaction, mobile and ubiquitous computing, human robot interaction, acoustic imaging, entertainment computing, and mixed reality. He is a member of the IEEE.
63 ms
(Ver 3.x)