This Article 
 Bibliographic References 
 Add to: 
Interactive 3D Model Acquisition and Tracking of Building Block Structures
April 2012 (vol. 18 no. 4)
pp. 651-659
We present a prototype system for interactive construction and modification of 3D physical models using building blocks.Our system uses a depth sensing camera and a novel algorithm for acquiring and tracking the physical models. The algorithm,Lattice-First, is based on the fact that building block structures can be arranged in a 3D point lattice where the smallest block unit is a basis in which to derive all the pieces of the model. The algorithm also makes it possible for users to interact naturally with the physical model as it is acquired, using their bare hands to add and remove pieces. We present the details of our algorithm, along with examples of the models we can acquire using the interactive system. We also show the results of an experiment where participants modify a block structure in the absence of visual feedback. Finally, we discuss two proof-of-concept applications: a collaborative guided assembly system where one user is interactively guided to build a structure based on another user's design, and a game where the player must build a structure that matches an on-screen silhouette.

[1] D. Anderson, J. Frankel, J. Marks, A. Agarwala, P. Beardsley, J. Hodgins, D. Leigh, K. Ryall, E. Sullivan, and J. Yedidia, Tangible interaction+ graphical interpretation: a new approach to 3D modeling. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 393-402, 2000.
[2] P. Baudisch, T. Becker, and F. Rudeck, Lumino: tangible blocks for tabletop computers based on glass fiber bundles. In Proceedings of the 28th international conference on Human factors in computing systems, CHI '10, pages 1165-1174, New York, NY, USA, 2010. ACM.
[3] P. J. Besl and N. D. McKay, A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell., 14 pp. 239-256, February 1992.
[4] M. Bray, E. Koller-Meier, and L. J. V. Gool, Smart particle filtering for high-dimensional tracking. Computer Vision and Image Understanding, 106(1) pp. 116-129, 2007.
[5] D. J. Campbell, J. D. Miller, S. J. Bannon, and L. M. Obermaier, An Exploration of the Nanoworld with LEGO Bricks. Journal of Chemical Education, pages 602-606, 2011.
[6] R. J. Campbell and P. J. Flynn, A survey of free-form object representation and recognition techniques. Comput. Vis. Image Underst., 81(2) pp. 166-210, February 2001.
[7] G. K. Cheung, S. Baker, and T. Kanade, Visual hull alignment and refinement across time: A 3d reconstruction algorithm combining shape-from-silhouette with stereo. Computer Vision and Pattern Recognition, 2 p. 375, 2003.
[8] B. Curless and M. Levoy, A volumetric method for building complex models from range images. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques SIGGRAPH 96, pages(Annual Conference Series) pp. 303-312, 1996.
[9] P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin, and M. Sagar, Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, SIGGRAPH '00, pages 145-156, New York, NY, USA, 2000. ACM.
[10] S. Feiner, B. Macintyre, and D. Seligmann, Knowledge-based augmented reality. Commun. ACM, 36 pp. 53-62, July 1993.
[11] S. Fuchs and G. Hirzinger, Extrinsic and depth calibration of tof-cameras. In Computer Vision and Pattern Recognition, pages 1-6. IEEE, 2008.
[12] S. R. Harding and M. F. D'Eon, Using a LEGO-based communications simulation to introduce medical students to patient-centered interviewing. Teaching and Learning in Medicine, 13(2) pp. 130-135, 2001.
[13] B. K. P. Horn and J. G. Harris, Rigid body motion from range image sequences. CVGIP: Image Underst. , 53 pp. 1-13, January 1991.
[14] H. Ichida, Y. Itoh, Y. Kitamura, and F. Kishino, Interactive retrieval of 3d shape models using physical objects. In Proceedings of the 12th annual ACM international conference on Multimedia, MULTIMEDIA '04, pages 692-699, New York, NY, USA, 2004. ACM.
[15] S. Izadi, A. Davison, A. Fitzgibbon, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, and D. Freeman, KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11, pages 559-568, New York, New York, USA, Oct. 2011. ACM Press.
[16] R. Jota and H. Benko, Constructing virtual 3d models with physical building blocks. In CHI 2011 Extended Abstracts, pages 2173-2178. ACM, May 2011.
[17] S. B. Kang and K. Ikeuchi, The complex egi: A new representation for 3-d pose determination. IEEE Trans. Pattern Anal. Mach. Intell., 15 pp. 707-721, July 1993.
[18] K. N. Kutulakos and S. M. Seitz, A theory of shape by space carving. Int. J. Comput. Vision, 38 pp. 199-218, July 2000.
[19] J. Lee, Y. Kakehi, and T. Naemura, Bloxels: glowing blocks as volumetric pixels. In ACM SIGGRAPH 2009 Emerging Technologies, SIGGRAPH '09, pages 5:1-5:1, New York, NY, USA, 2009. ACM.
[20] D. B. LeGoff, Use of LEGO as a therapeutic medium for improving social competence. Technical Report 5, Bancroft Neurosciences Institute, Haddonfield, NJ 08034, USA. 2004.
[21] M. Levoy, K. Pulli, B. Curless, S. Rusinkiewicz, D. Koller, L. Pereira, M. Ginzton, S. Anderson, J. Davis, J. Ginsberg, J. Shade, and D. Fulk, The digital michelangelo project: 3d scanning of large statues. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, SIGGRAPH '00, pages 131-144, New York, NY, USA, 2000. ACM.
[22] X.-Y. Li, C.-H. Shen, S.-S. Huang, T. Ju, and S.-M. Hu, Popup: automatic paper architectures from 3d models. In ACM SIGGRAPH 2010 papers, SIGGRAPH '10, pages 111:1-111:9, New York, NY, USA, 2010. ACM.
[23] M. R. Marner and B. H. Thomas, Augmented foam sculpting for capturing 3d models. IEEE Symposium on 3D User Interfaces, pages 63-70, April 2010.
[24] Y. Mori and T. Igarashi, Plushie: an interactive design system for plush toys. ACM Trans. Graph., 26:45 pp. 1-458 July 2007.
[25] R. A. Newcombe, S. J. Lovegrove, and A. J. Davison, Dtam: Dense tracking and mapping in real-time. IEEE International Conference on Computer Vision, 1, 2011.
[26] R. A. Newcombe, D. Molyneaux, D. Kim, A. J. Davison, J. Shotton, S. Hodges, and A. Fitzgibbon, KinectFusion: Real-Time Dense Surface Mapping and Tracking. ISMAR, pages 127-136, 2011.
[27] H. Newton-Dunn, H. Nakano, and J. Gibson, Block jam: a tangible interface for interactive music. In Proceedings of the 2003 conf. on New interfaces for musical expression, NIME '03, pages 170-177, Singapore, Singapore, 2003. National University of Singapore.
[28] R. Raskar, P. Beardsley, P. Dietz, and J. van Baar, Photosensing wireless tags for geometric procedures. Commun. ACM, 48 pp. 46-51, September 2005.
[29] R. Raskar, G. Welch, and W. chao Chen, Table-top spatially-augmented reality: Bringing physical models to life with projected imagery. In In: Proceedings of the 2nd IEEE and ACM international workshop on augmented reality (IWAR99), pages 64-73. IEEE, 1999.
[30] S. Rosenthal, S. K. Kane, J. O. Wobbrock, and D. Avrahami, Augmenting on-screen instructions with micro-projected guides: when it works, and when it fails. In Proceedings of the 12th ACM international conference on Ubiquitous computing, Ubicomp '10, pages 203-212, New York, NY, USA, 2010. ACM.
[31] F. Rottensteiner and C. Briese, A new method for building extraction in urban areas from high-resolution lidar data. International Archives of Photogrammetry Remote Sensing and Spatial Information Sciences, 34(3/A) pp. 295-301, 2001.
[32] S. Rusinkiewicz, O. Hall-Holt, , and M. Levoy, Real-time 3d model acquisition. ACM Trans. Graph., 21 pp. 438-446, July 2002.
[33] S. Rusinkiewicz and M. Levoy, Efficient variants of the icp algorithm. In 3D Digital Imaging and Modeling, pages 145-152. IEEE, 2001.
[34] M. Schlattmann, F. Kahlesz, R. Sarlette, and R. Klein, Markerless 4 gestures 6 dofreal-time visual tracking of the human hand with automatic initialization. Comput. Graph. Forum, 26(3) pp. 467-476, 2007.
[35] R. Schnabel, R. Wahl, and R. Klein, Efficient ransac for point-cloud shape detection. Computer Graphics Forum, 26(2) pp. 214-226, June 2007.
[36] E. Sharlin, Y. Itoh, B. Watson, Y. Kitamura, S. Sutphen, and L. Liu, Cognitive cubes: a tangible user interface for cognitive assessment. In Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, CHI'02, pages 347-354, New York, NY, USA, 2002. ACM.
[37] J. Slaney and S. Thiebaux, Blocks world revisited. Artif. In tell., 125 pp. 119-153, January 2001.
[38] A. Tang, C. Owen, F. Biocca, and W. Mou, Comparative effectiveness of augmented reality in object assembly. In Proceedings of the SIGCHI conference on Human factors in computing systems, CHI '03, pages 73-80, New York, NY, USA, 2003. ACM.
[39] M. A. Templin and M. K. Fetters, A Working Model of Protein Synthesis Using LEGO. The American Biology Teacher, 64(9) pp. 673-678, 2002.
[40] R. Y. Wang and J. Popović, Real-time hand-tracking with a color glove. ACM Trans. Graph., 28 pp. 63:1-63:8 July 2009.
[41] T. Weise, T. Wismer, B. Leibe, and L. J. V. Gool, Online loop closure for real-time interactive 3d scanning. Computer Vision and Image Understanding, 115(5) pp. 635-648, 2011.
[42] G. Welch and E. Foxlin, Motion Tracking: No Silver Bullet, but a Respectable Arsenal. IEEE Computer Graphics and Applications, 22(6) pp. 24-38, 2002.
[43] M. P. Weller, E. Y.-L. Do, and M. D. Gross, Posey: instrumenting a poseable hub and strut construction toy. In Proceedings of the 2nd international conference on Tangible and embedded interaction, TEI'08, pages 39-46, New York, NY, USA, 2008. ACM.
[44] S. Wesugi and Y. Miwa, Brick-building interface support for cocreative communication. Int. J. Hum. Comput. Interaction, 20(1) pp. 35-56, 2006.
[45] R. Ziola, S. Grampurohit, N. Landes, J. Fogarty, and B. Harrison, Oasis: Examining a framework for interacting with general-purpose object recognition. In Intel Labs Seattle Technical Report. Intel, 2010.

Index Terms:
user interfaces,cameras,data acquisition,solid modelling,on-screen silhouette,interactive 3D model,3D model acquisition,3D model tracking,building block structure,interactive construction,interactive modification,3D physical model,depth sensing camera,lattice-first algorithm,3D point lattice,visual feedback,user interaction,collaborative guided assembly system,user design,Solid modeling,Lattices,Three dimensional displays,Computational modeling,Cameras,Image color analysis,Visualization,building block structures.,Interactive physical model building,3D model acquisition,object tracking,depth cameras
A. Miller, B. White, E. Charbonneau, Z. Kanzler, J. J. LaViola, "Interactive 3D Model Acquisition and Tracking of Building Block Structures," IEEE Transactions on Visualization and Computer Graphics, vol. 18, no. 4, pp. 651-659, April 2012, doi:10.1109/TVCG.2012.48
Usage of this product signifies your acceptance of the Terms of Use.