This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Real-Time 3D Human Capture System for Mixed-Reality Art and Entertainment
November/December 2005 (vol. 11 no. 6)
pp. 706-721
ASCII Text
x 
 
Ta Huynh Duy Nguyen, Tran Cong Thien Qui, Ke Xu, Adrian David Cheok, Sze Lee Teo, ZhiYing Zhou, Asitha Mallawaarachchi, Shang Ping Lee, Wei Liu, Hui Siang Teo, Le Nam Thang, Yu Li, Hirokazu Kato, "Real-Time 3D Human Capture System for Mixed-Reality Art and Entertainment," IEEE Transactions on Visualization and Computer Graphics, vol. 11, no. 6, pp. 706-721, November/December, 2005.
 
 
BibTex
x
 
@article{ 10.1109/TVCG.2005.105,
author = {Ta Huynh Duy Nguyen and Tran Cong Thien Qui and Ke Xu and Adrian David Cheok and Sze Lee Teo and ZhiYing Zhou and Asitha Mallawaarachchi and Shang Ping Lee and Wei Liu and Hui Siang Teo and Le Nam Thang and Yu Li and Hirokazu Kato},
title = {Real-Time 3D Human Capture System for Mixed-Reality Art and Entertainment},
journal ={IEEE Transactions on Visualization and Computer Graphics},
volume = {11},
number = {6},
issn = {1077-2626},
year = {2005},
pages = {706-721},
doi = {http://doi.ieeecomputersociety.org/10.1109/TVCG.2005.105},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Visualization and Computer Graphics
TI - Real-Time 3D Human Capture System for Mixed-Reality Art and Entertainment
IS - 6
SN - 1077-2626
SP706
EP721
EPD - 706-721
A1 - Ta Huynh Duy Nguyen,
A1 - Tran Cong Thien Qui,
A1 - Ke Xu,
A1 - Adrian David Cheok,
A1 - Sze Lee Teo,
A1 - ZhiYing Zhou,
A1 - Asitha Mallawaarachchi,
A1 - Shang Ping Lee,
A1 - Wei Liu,
A1 - Hui Siang Teo,
A1 - Le Nam Thang,
A1 - Yu Li,
A1 - Hirokazu Kato,
PY - 2005
KW - Index Terms- 3D viewpoint
KW - mixed reality
KW - tangible interaction
KW - art
KW - entertainment.
VL - 11
JA - IEEE Transactions on Visualization and Computer Graphics
ER -
 
A real-time system for capturing humans in 3D and placing them into a mixed reality environment is presented in this paper. The subject is captured by nine cameras surrounding her. Looking through a head-mounted-display with a camera in front pointing at a marker, the user can see the 3D image of this subject overlaid onto a mixed reality scene. The 3D images of the subject viewed from this viewpoint are constructed using a robust and fast shape-from-silhouette algorithm. The paper also presents several techniques to produce good quality and speed up the whole system. The frame rate of our system is around 25 fps using only standard Intel processor-based personal computers. Besides a remote live 3D conferencing and collaborating system, we also describe an application of the system in art and entertainment, named Magic Land, which is a mixed reality environment where captured avatars of human and 3D computer generated virtual animations can form an interactive story and play with each other. This system demonstrates many technologies in human computer interaction: mixed reality, tangible interaction, and 3D communication. The result of the user study not only emphasizes the benefits, but also addresses some issues of these technologies.

[1] ARToolKit, http://www.hitl.washington.eduartoolkit/, 2005.
[2] CAVE Quake II, http://brighton.ncsa.uiuc.edu/prajlichcave Quake /, 2005.
[3] MXRToolkit, http://sourceforge.net/projectsmxrtoolkit /, 2005.
[4] OpenCV, http://sourceforge.net/projectsopencvlibrary /, 2005.
[5] Point Grey Research Inc., http:/www.ptgrey.com, 2005.
[6] RGB “Bayer” Color and MicroLenses, http://www.siliconimaging.comRGB Bayer.htm , 2005.
[7] A. Amory, K. Naicker, J. Vincent, and C. Adams, “The Use of Computer Games as an Educational Tool: Identification of Appropriate Game Types and Elements,” pp. 311-321, British J. Educational Technology, 1999.
[8] R. Azuma, “A Survey of Augmented Reality,” Presence, 1997.
[9] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre, “Recent Advances in Augmented Reality,” IEEE Computer Graphics and Applications, Nov./Dec. 2001.
[10] S. Bjork, J. Falk, R. Hansson, and P. Ljungstrand, “Pirates!— Using the Physical World as a Game Board,” Proc. Interact 2001, IFIP TC. 13 Conf. Human-Computer Interaction, 2001.
[11] A.D. Cheok, K.H. Goh, W. Liu, F. Farzbiz, S.W. Fong, S.Z. Teo, Y. Li, and X. Yang, “Human Pacman: A Mobile Wide-Area Entertainment System Based on Physical, Social, and Ubiquitous Computing,” Personal and Ubiquitous Computing, vol. 8, no. 2, pp. 71-81, May 2004.
[12] A.D. Cheok, X. Yang, Z. Zhou, M. Billinghurst, and H. Kato, “Touch-Space: Mixed Reality Game Space Based on Ubiquitous, Tangible, and Social Computing,” J. Personal and Ubiquitous Computing, pp. 430-442, 2002.
[13] J.M. Hasenfratz, M. Lapierre, and F. Sillion, “A Real-Time System for Full Body Interaction,” Virtual Environments, pp. 147-156, 2004.
[14] T. Horprasert et al., “A Statistical Approach for Robust Background Subtraction and Shadow Detection,” Proc. IEEE Int'l Conf. Computer Vision (ICCV '99) Frame Rate Workshop, 1999.
[15] H. Kato, K. Tachibana, M. Tanabe, T. Nakajima, and Y. Fukuda, “Magiccup: A Tangible Interface for Virtual Objects Manipulation in Table-Top Augmented Reality,” Proc. Augmented Reality Toolkit Workshop (ART03), pp. 85-86, 2003.
[16] K.N. Kutulakos and S.M. Seitz, “A Theory of Shape by Space Carving,” Int'l J. Computer Vision, vol. 38, no. 3, 2000.
[17] S.P. Lee, F. Farbiz, and A.D. Cheok, “Touchy Internet: A Cybernetic System for Human-Pet Interaction through the Internet,” SIGGRAPH 2003, Sketches and Application, 2003.
[18] T.W. Malone, “Toward a Theory of Intrinsically Motivating Instruction,” vol. 5, pp. 333-369, 1981.
[19] R.L. Mandryk and K.M. Inkpen, “Supporting Free Play in Ubiquitous Computer Games,” Proc. Workshop Designing Ubiquitous Computer Games (UbiComp), 2001.
[20] G. Markus, W. Stephan, N. Martin, L. Edouard, S. Christian, K. Andreas, K.M. Esther, S. Tomas, V.G. Luc, L. Silke, S. Kai, V.M. Andrew, and S. Oliver, “blue-c: A Spatially Immersive Display and 3D Video Portal for Telepresence,” Proc. ACM SIGGRAPH, 2003.
[21] W. Matusik, C. Buehler, and L. McMillan, “Polyhedral Visual Hulls for Real-Time Rendering,” Proc. 12th Eurographics Workshop Rendering Techniques, 2001.
[22] R. Mester, T. Aach, and L. Dmbgen, “Illumination-Invariant Change Detection Using Statistical Colinearity Criterion,” Proc. DAGM Symp. Pattern Recognition, pp. 170-177, 2001.
[23] T. Oshima, K. Satoh, H. Yamamoto, and H. Tamura, “Ar2 Hockey System: A Collaboration Mixed Reality System,” vol. 3, no. 2, pp. 55-60, Trans. VRSJ, 1998.
[24] S.J.D. Prince, A.D. Cheok, F. Farbiz, T. Williamson, N. Johnson, M. Billinghurst, and H. Kato, “Live 3-Dimensional Content for Augmented Reality,” IEEE Trans. Multimedia, submitted.
[25] S.J.D. Prince, A.D. Cheok, F. Farbiz, T. Williamson, N. Johnson, M. Billinghurst, and H. Kato, “3D Live: Real Time Captured Content for Mixed Reality,” Proc. Int'l Symp. Mixed and Augmented Reality (ISMAR), 2002.
[26] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, “Rtp: A Transport Protocol for Real-Time Applications,” Internet Eng. Task Force, Audio-Video Transport Working Group, 1996.
[27] S. Seitz and C. Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 1997.
[28] M. Seki, H. Fujiwara, and K. Sumi, “A Robust Background Subtraction Method for Changing Background,” Proc. Fifth IEEE Int'l Workshop Applications of Computer Vision, 2000.
[29] G. Slabaugh, W.B. Culbertson, T. Malzbender, and R. Schafer, “A Survey of Volumetric Scene Reconstruction Methods from Photographs,” Proc. Int'l Workshop Volume Graphics, 2001.
[30] G. Slabaugh, R. Schafer, and M. Hans, “Image-Based Photo Hulls,” Proc. First Int'l Symp. 3D Data Processing, Visualization, and Transmission, 2002.
[31] G.G. Slabaugh, W.B. Culbertson, T. Malzbender, M.R. Stevens, and R.W. Schafer, “Methods for Volumetric Reconstruction of Visual Scenes,” Int'l J. Computer Vision, 2004.
[32] R. Szeliski, “Video Mosaics for Virtual Environments,” IEEE Computer Graphics and Applications, Mar. 1996.
[33] H. Tamura, H. Yamamoto, and A. Katayama, “Mixed Reality: Future Dreams Seen at the Border between Real and Virtual Worlds,” IEEE Computer Graphics and Applications, vol. 21, no. 6, pp. 64-70, 2001.

Index Terms:
Index Terms- 3D viewpoint, mixed reality, tangible interaction, art, entertainment.
Citation:
Ta Huynh Duy Nguyen, Tran Cong Thien Qui, Ke Xu, Adrian David Cheok, Sze Lee Teo, ZhiYing Zhou, Asitha Mallawaarachchi, Shang Ping Lee, Wei Liu, Hui Siang Teo, Le Nam Thang, Yu Li, Hirokazu Kato, "Real-Time 3D Human Capture System for Mixed-Reality Art and Entertainment," IEEE Transactions on Visualization and Computer Graphics, vol. 11, no. 6, pp. 706-721, Nov.-Dec. 2005, doi:10.1109/TVCG.2005.105
Usage of this product signifies your acceptance of the Terms of Use.