The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.05 - May (2010 vol.32)
pp: 899-909
Liang Wang , University of Kentucky, Lexington
Jizhou Gao , University of Kentucky, Lexington
Jiejie Zhu , University of Central Florida, Orlando
ABSTRACT
Time-of-flight range sensors and passive stereo have complimentary characteristics in nature. To fuse them to get high accuracy depth maps varying over time, we extend traditional spatial MRFs to dynamic MRFs with temporal coherence. This new model allows both the spatial and the temporal relationship to be propagated in local neighbors. By efficiently finding a maximum of the posterior probability using Loopy Belief Propagation, we show that our approach leads to improved accuracy and robustness of depth estimates for dynamic scenes.
INDEX TERMS
Stereo, MRFs, time-of-flight sensor, data fusion, global optimization.
CITATION
Liang Wang, Jizhou Gao, Jiejie Zhu, "Spatial-Temporal Fusion for High Accuracy Depth Maps Using Dynamic MRFs", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.32, no. 5, pp. 899-909, May 2010, doi:10.1109/TPAMI.2009.68
REFERENCES
[1] A. Verri and V. Torre, "Absolute Depth Estimate in Stereopsis," Optical Soc. of Am. A, vol. 3, no. 3, pp. 297-299, 1986.
[2] K. Kanatani, "Detection of Surface Orientation and Motion from Texture by a Stereo-Logical Technique," Artificial Intelligence, vol. 23, no. 2, pp. 213-237, 1984.
[3] J. Michels, A. Saxena, and A.Y. Ng, "High Speed Obstacle Avoidance Using Monocular Vision and Reinforcement Learning," Proc. Int'l Conf. Machine Learning, 2005.
[4] D. Scharstein and R. Szeliski, "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," Int'l J. Computer Vision, vol. 47, no. 1, pp. 7-42, 2002.
[5] Swissranger, Inc., sr-3, http://www.csem.ch/fsimaging.htm, 2006.
[6] Canesta, Inc., Canestavision Electronic Perception Development Kit, http:/www.canesta.com/, 2006.
[7] Pmdtec, Inc., Photonix Mixer Device, http:/www.pmdtec.com, 2008.
[8] 3dv Systems, z-cam, http:/www.3dvsystems.com, 2004.
[9] Y. Boykov, O. Veksler, and R. Zabih, "Markov Random Fields with Efficient Approximations," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 648-655, 1998.
[10] V. Kolmogorov and R. Zabih, "Computing Visual Correspondence with Occlusions via Graph Cuts," Proc. Int'l Conf. Computer Vision, pp. 508-515, 2001.
[11] D. Scharsterin and R. Szeliski, vision.middlebury.edu/stereo/, 2006.
[12] J. Sun, N.N. Zeng, and H.Y. Shum, "Stereo Matching Using Belief Propagation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 7, pp. 787-800, July 2003.
[13] K.D. Kuhnert and M. Stommel, "Fusion of Stereo-Camera and pmd-Camera Data for Real-Time Suited Precise 3D Environment Reconstruction," Proc. IEEE/RSJ Int'l Conf. Intelligent Robots and Systems, pp. 4780-4785, 2006.
[14] S.A. GuÐmundsson, H. Aanæs, and R. Larsen, "Fusion of Stereo Vision and Time-of-Flight Imaging for Improved 3D Estimation," Proc. Int'l Workshop Dynamic 3D Imaging, 2007.
[15] J.J. Zhu, L. Wang, R.G. Yang, and J.E. Davis, "Fusion of Time-of-Flight Depth and Stereo for High Accuracy Depth Maps," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[16] J. Diebel and S. Thrun, "An Application of Markov Random Fields to Range Sensing," Proc. Advances in Neural Information Processing Systems 18, 2005.
[17] Q.X. Yang, R.G. Yang, J.E. Davis, and D. Nister, "Spatial-Depth Super Resolution for Range Images," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[18] O. Veksler, Y. Boykov, and R. Zabih, "Fast Approximate Energy Minimization via Graph Cuts," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222-1239, Nov. 2001.
[19] A. Blake, C. Rother, M. Brown, P. Perez, and P. Torr, "Interactive Image Segmentation Using an Adaptive Gmmrf Model," Proc. European Conf. Computer Vision, pp. 428-441, 2004.
[20] C. Rother, T. Minka, A. Blake, and V. Kolmogorov, "Cosegmentation of Image Pairs by Histogram Matching: Incorporating a Global Constraint into mrfs," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 993-1000, 2006.
[21] V. Kolmogorov, A. Criminisi, A. Blake, G. Gross, and C. Rother, "Probabilistic Fusion of Stereo with Color and Contrast for Bilayer Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 9, pp. 1480-1492, Sept. 2006.
[22] S. Roth and M. Black, "Fields of Experts: A Framework for Learning Image Priors," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 860-867, 2005.
[23] J. Wills, S. Agarwal, and S. Belongie, "What Went Where," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 37-44, 2003.
[24] S. Roth and M. Black, "On the Spatial Statistics of Optical Flow," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[25] S. Park, M. Park, and M. Kang, "Super-Resolution Image Reconstruction: A Technical Overview," IEEE Signal Processing Magazine, vol. 20, no. 3, pp. 21-36, May 2003.
[26] W.T. Freeman, E.C. Pasztor, and O.T. Carmichael, "Learning Low Level Vision," Int'l J. Computer Vision, vol. 40, no. 1, pp. 25-47, 2001.
[27] M.F. Tappen and W.T. Freeman, "Comparison of Graph Cuts with Belief Propagation for Stereo, Using Identical MRF Parameters," Proc. Int'l Conf. Computer Vision, pp. 900-907, 2003.
[28] P. Kohli and P.H. Torr, "Dynamic Graph Cuts for Efficient Inference in Markov Random Fields," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 12, pp. 2079-2088, Dec. 2007.
[29] O. Juan and Y. Boykov, "Active Graph Cuts," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2006.
[30] M. Isard and J. Maccormick, "Dense Motion and Disparity Estimation via Loopy Belief Propagation," Proc. Asia Conf. Computer Vision, 2006.
[31] Z.Z. Yin and R. Collins, "Belief Propagation in a 3D Spatio-Temporal mrf for Moving Object Detection," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[32] J. Chen and C.K. Tang, "Spatio-Temporal Markov Random Field for Video Denoising," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[33] K. Alahari, P. Kohli, and P.H.S. Torr, "Reduce, Reuse and Recycle: Efficiently Solving Multi-Label mrfs," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[34] K.J. Yoon and I.S. Kweon, "Locally Adaptive Support-Weight Approach for Visual Correspondence Search," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 924-931, 2005.
[35] A.T. Ihler, J.W. FisherIII, and A.S. Willsky, "Loopy Belief Propagation: Convergence and Effects of Message Errors," J. Machine Learning Research, vol. 6, pp. 905-936, 2005.
[36] D. Scharstein and R. Szeliski, "High-Accuracy Stereo Depth Maps Using Structured Light," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2003.
[37] P.F. Felzenszwalb and D.P. Huttenlocher, "Efficient Belief Propagation for Early Vision," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 261-268, 2004.
[38] Q.X. Yang, L. Wang, R.G. Yang, S.G. Wang, M. Liao, and D. Nister, "Real-Time Global Stereo Matching Using Hierarchical Belief Propagation," Proc. British Machine Vision Conf., 2006.
[39] Pointgrey Research, Inc., http:/www.pointgrey.com, 2006.
[40] A.S. Ogale and Y. Aloimonos, "Shape and the Stereo Correspondence Problem," Int'l J. Computer Vision, vol. 65, no. 3, pp. 147-162, 2005.
[41] Q.X. Yang, L. Wang, R.G. Yang, H. Stewenius, and D. Nister, "Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation and Occlusion Handling," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2347-2354, 2006.
[42] S. Birchfield and C. Tomasi, "A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 4, pp. 401-406, Apr. 1998.
[43] I. Schiller, C. Beder, and R. Koch, "Calibration of A PMD-Camera Using a Planar Calibration Pattern Together with a Multi-Camera Setup," Proc. Int'l Soc. for Photogrammetry and Remote Sensing, 2008.
[44] T. Kahlmann, F. Remondino, and H. Ingensand, "Calibration of the Fast Range Imaging Camera Swissranger for Use in the Surveillance of the Environment," Proc. SPIE: Electro-Optical Remote Sensing II, 2006.
[45] M. Linder, M. Lambers, and A. Kolb, "Sub-Pixel Data Fusion and Edge-Enhanced Distance Refinement for 2D/3D Images," Int'l J. Intelligent Systems Technologies and Applications, vol. 5, pp. 344-354, 2008.
[46] M. Lindner and A. Kolb, "Calibration of the Intensity-Related Distance Error of the PMD TOF-Camera," Proc. SPIE: Intelligent Robots and Computer Vision XXV, pp. 35-42, 2007.
[47] H. Gonzales-Banos and J.E. Davis, "Computing Depth under Ambient Illumination Using Multi-Shuttered Light," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 234-241, 2004.
[48] T. Kahlmann and H. Ingensand, "Increased Accuracy of 3D Range Imaging Camera by Means of Calibration," Proc. Conf. Optical 3D Measurement Techniques, July 2007.
[49] I. Schiller, C. Beder, and R. Koch, "Calibration of a PMD-Camera Using a Planar Calibration Pattern Together with a Multi-Camera Setup," Proc. XXXVII Int'l Soc. for Photogrammetry and Remote Sensing Congress, 2008.
[50] M. Linder, M. Lambers, and A. Kolb, "Sub-Pixel Data Fusion and Edge-Enhanced Distance Refinement for 2D/3D Images," Int'l J. Intelligent Systems Technologies and Applications, vol. 5, no. 3, pp. 344-354, 2008.
[51] L. Guan and M. Pollefeys, "A Unified Approach to Calibrate a Network of Camcorders and ToF Cameras," Proc. IEEE Workshop Multi-Camera and Multi-Model Sensor Fusion Algorithms and Applications, 2008.
19 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool