The Community for Technology Leaders

Reality Environments

Joseph A. Paradiso, Massachusetts Institute of Technology Media Lab
James A. Landay, University of Washington

Pages: pp. 14-15

The genesis of a ubiquitously networked sensor/actuator infrastructure, leveraged by the increasingly low cost of microelectronics, sensors, and wireless technologies, is endemic to pervasive computing. As the independent application-siloed sensor/actuator networks now being deployed begin to converge through common standards, the world is becoming covered by a seamless electronic "nervous system" that extends across places, things, and people. Concurrently, although 3D virtual environments have been around for decades, their deployment has recently exploded with the advent of massively shared online virtual worlds. Dedicated immersive games such as World of Warcraft and general purpose worlds like Second Life each now boast more than 10 million subscribers.

We call the ubiquitous mixed reality environment that comes from the fusion of these two technologies cross-reality. Sensor networks can tunnel dense real-world information into virtual worlds, where this data is interpreted and displayed to dispersed users. Interaction of virtual participants can incarnate into the physical world through a plentitude of diverse displays and actuators. We can envision a user's interface into this environment as an extension of human perception and interaction, augmenting our five senses well beyond the canonical "here and now" and redefining the meaning of presence.

Although augmented reality applications are no stranger to Pervasive Computing readers, we distinguish cross-reality in that the conduits to and from virtuality will be everywhere, not restricted to head-worn or other wearable/mobile devices. Similarly, cross-reality's virtual digs aren't just filled with inanimate data—they're also social environments, populated with human-driven avatars and other representations of residents and sensor data. The manifestation of virtual phenomena in the real world will become a function of available and appropriate devices and conduits. Such scalable virtuality should seamlessly range from ambient displays to mobile devices to full immersion.

Speculative fiction has progressively honed stimulating visions that presage cross-reality frameworks. Consider, for example, Neil Stephenson's Snow Crash, Verner Vinge's Rainbow's End, and Charles Stross' Halting State. Real work, however, is now emerging from research labs and innovative companies are beginning to explore this space. This special issue includes several articles that provide a glimpse into some of these initiatives.

The first article, "Using Sensor Inputs to Affect Virtual and Real Environments," by Beth Coleman from MIT's Comparative Media Studies program, serves as an introduction to these concepts by surveying three projects that leverage the convergence of online virtual worlds with sensor networks, data sources, and group interaction. Companies are beginning to implement cross-reality ideas, and mixed-reality artists have long been prodding frontiers that hint at cross-reality. Beth's article reflects this, as two of the projects she covers are commercial endeavors and the other is an online interactive art installation.

The second article, "Metaphor and Manifestation—Cross-Reality with Ubiquitous Sensor/Actuator Networks," by Joshua Lifton, Mathew Laibowitz, Nan-Wei Gong, and Joseph A. Paradiso, outlines several projects from the MIT Media Lab's Responsive Environments Group aimed at interfacing people with ubiquitous sensor networks. These range from Second Life environments that let users browse and interact with distributed sensor/actuator nodes and ubiquitous media portals to mobile environments for in-situ browsing, interaction, and context-scripting of sensor networks. The authors also describe a badge-based approach for managing dynamic privacy in physical environments that host dense sensing and media capture.

The third article, "Event Processing Support for Cross-Reality Environments," by Nihal Dindar, Çağri Balkesen, Katinka Kromwijk, and Nesime Tatbul from ETH Zurich, presents a general-purpose complex event-processing framework called DejaVu that detects real-world phenomena from sensor data and manifests it in virtual environments. They demonstrate DejaVu through SmartRFLib, a representation of a library in Second Life that's driven by RFID data generated by a real-world library as people enter and exit with tagged books and artifacts.

The final article, "Through-Walls Collaboration," by Bruce Thomas and Wayne Piekarski, describes several projects from the Wearable Computing Lab at the University of South Australia that explore what they call through-walls collaboration, leveraging the convergence of fixed and mobile virtual environments to enable dynamic collaboration in augmented reality applications. As mobile augmented reality begins to fuse with sensor networks and shared online virtual worlds, it subscribes to the notion of cross-reality that this special issue explores.

This issue also includes a Spotlight contribution from Daniel Horn, Ewen Cheslack-Postava, Tahir Azim, Michael J. Freedman, and Philip Levis that overviews a collaboration between Stanford and Princeton aimed at defining a general purpose framework for online virtual worlds called Meru. Meru is designed for massive scalability, taking inspiration from perception in the physical world, where, for example, allocated bandwidth between virtual constructs scales with their proximity. Meru is posited to evolve into a standard that could federate disparate virtual spaces while seamlessly accommodating information flowing to and from the real world (which the authors term "space zero").

As they present a 3D environment well suited to human perception, we can interpret shared immersive virtual worlds as something of a "front panel" to pervasive computing, where users interact with each other and information from the real world as it's extrapolated, processed, and represented before being projected back down into reality through appropriate incarnations. Cross-reality is by nature at an exciting crossroads, where many currently breaking trends (for example, shared online virtual worlds, pervasive sensor/actuator networks, social computing) are converging—an overlap that promises a potential that the articles in this issue only begin to hint at.

About the Authors

Bio Graphic
Joseph A. Paradiso is an associate professor of Media Arts and Sciences at the Massachusetts Institute of Technology Media Laboratory, where he directs the Responsive Environments Group, which explores how sensor networks augment and mediate human experience, interaction, and perception. In addition, he codirects the Things That Think Consortium, a group of industrial partners and Media Lab researchers who explore the extreme fringe of embedded computation, communication, and sensing. Paradiso has a PhD in physics from MIT's Lab for Nuclear Science. He's a senior member of the IEEE and the American Institute of Aeronautics and Astronautics, and a member of the American Physical Society, the ACM, and Sigma Xi. Contact him at
Bio Graphic
James A. Landay is an associate professor in computer science and engineering at the University of Washington. His research interests include automated usability evaluation, demonstrational interfaces, ubiquitous computing, user interface design tools, and Web design. Landay has a BS in electrical engineering and computer science from the University of California, Berkeley, and an MS and PhD in computer science from Carnegie Mellon University. Contact him at
64 ms
(Ver 3.x)