Issue No.01 - January (2011 vol.33)
pp: 158-171
Avinash Ravichandran , The Johns Hopkins University, Baltimore
René Vidal , The Johns Hopkins Univeristy, Baltimore
We consider the problem of spatially and temporally registering multiple video sequences of dynamical scenes which contain, but are not limited to, nonrigid objects such as fireworks, flags fluttering in the wind, etc., taken from different vantage points. This problem is extremely challenging due to the presence of complex variations in the appearance of such dynamic scenes. In this paper, we propose a simple algorithm for matching such complex scenes. Our algorithm does not require the cameras to be synchronized, and is not based on frame-by-frame or volume-by-volume registration. Instead, we model each video as the output of a linear dynamical system and transform the task of registering the video sequences to that of registering the parameters of the corresponding dynamical models. As these parameters are not uniquely defined, one cannot directly compare them to perform registration. We resolve these ambiguities by jointly identifying the parameters from multiple video sequences, and converting the identified parameters to a canonical form. This reduces the video registration problem to a multiple image registration problem, which can be efficiently solved using existing image matching techniques. We test our algorithm on a wide variety of challenging video sequences and show that it matches the performance of significantly more computationally expensive existing methods.
Dynamic textures, video registration, nonrigid dynamical scenes.
Avinash Ravichandran, René Vidal, "Video Registration Using Dynamic Textures", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.33, no. 1, pp. 158-171, January 2011, doi:10.1109/TPAMI.2010.61
[1] VideoAnalysis/ DemosSeq2S eq/, 2008.
[2] VideoAnalysis/ DemosTraj2Traj, 2010.
[3] tions.html , 2010.
[4] A. Agarwala, K.C. Zheng, C. Pal, M. Agrawala, M. Cohen, B. Curless, D. Salesin, and R. Szeliski, "Panoramic Video Textures," ACM Trans. Graphics, vol. 24, no. 3, pp. 821-827, 2005.
[5] Z. Bar-Joseph, R. El-Yaniv, D. Lischinski, and M. Werman, "Texture Mixing and Texture Movie Synthesis Using Statistical Learning," IEEE Trans. Visualization and Computer Graphics, vol. 7, no. 2, pp. 120-135, Apr.-June 2001.
[6] M. Brown, R. Szeliski, and S. Winder, "Multi-Image Matching Using Multi-Scale Oriented Patches," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 510-517, June 2005.
[7] Y. Caspi and M. Irani, "Spatio-Temporal Alignment of Sequences," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 11, pp. 1409-1424, Nov. 2002.
[8] Y. Caspi, D. Simakov, and M. Irani, "Feature-Based Sequence-to-Sequence Matching," Int'l J. Computer Vision, vol. 68, no. 1, pp. 53-64, 2006.
[9] A. Chan and N. Vasconcelos, "Classifying Video with Kernel Dynamic Textures," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-6, 2007.
[10] A. Chan and N. Vasconcelos, "Probabilistic Kernels for the Classification of Auto-Regressive Visual Processes," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 846-851, 2005.
[11] G. Doretto, A. Chiuso, Y. Wu, and S. Soatto, "Dynamic Textures," Int'l J. Computer Vision, vol. 51, no. 2, pp. 91-109, 2003.
[12] G. Doretto, D. Cremers, P. Favaro, and S. Soatto, "Dynamic Texture Segmentation," Proc. IEEE Conf. Computer Vision, pp. 44-49, 2003.
[13] G. Doretto and S. Soatto, "Dynamic Shape and Appearance Models," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 12, pp. 2006-2019, Dec. 2006.
[14] M.A. Fischler and R.C. Bolles, "RANSAC Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography," Comm. ACM, vol. 26, pp. 381-395, 1981.
[15] A. Fitzgibbon, "Stochastic Rigidity: Image Registration for Nowhere-Static Scenes," Proc. IEEE Int'l Conf. Computer Vision, pp. 662-669, 2001.
[16] C. Harris and M. Stephens, "A Combined Corner and Edge Detection," Proc. Fourth Alvey Vision Conf., 1988.
[17] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2000.
[18] V. Kwatra, A. Schödl, I. Essa, G. Turk, and A. Bobick, "Graphcut Textures: Image and Video Synthesis Using Graph Cuts," ACM Trans. Graphics, vol. 22, pp. 277-286, 2003.
[19] D. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," Int'l J. Computer Vision, vol. 20, pp. 91-110, 2003.
[20] Y. Ma, S. Soatto, J. Kosecka, and S. Sastry, An Invitation to 3D Vision: From Images to Geometric Models. Springer Verlag, 2003.
[21] P.V. Overschee and B.D. Moor, "Subspace Algorithms for the Stochastic Identification Problem," Automatica, vol. 29, no. 3, pp. 649-660, 1993.
[22] A. Rav-Acha, Y. Pritch, D. Lischinski, and S. Peleg, "Dynamosaics: Video Mosaics with Non-Chronological Time," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 58-65, 2005.
[23] A. Rav-Acha, Y. Pritch, and S. Peleg, "Online Registration of Dynamic Scenes Using Video Extrapolation," Proc. Workshop Dynamic Vision at IEEE Int'l Conf. Computer Vision, 2005.
[24] A. Ravichandran and R. Vidal, "Mosaicing Nonrigid Dynamical Scenes," Proc. Workshop Dynamic Vision, 2007.
[25] W.J. Rugh, Linear System Theory, second ed. Prentice Hall, 1996.
[26] P. Saisan, G. Doretto, Y.N. Wu, and S. Soatto, "Dynamic Texture Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 58-63, 2001.
[27] A. Schödl, R. Szeliski, D.H. Salesin, and I. Essa, "Video Textures," Proc. ACM SIGGRAPH, pp. 489-498, 2000.
[28] J. Shi and C. Tomasi, "Good Features to Track," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1994.
[29] R. Szeliski, "Image Alignment and Stitching: A Tutorial," Fundamental Trends in Computer Graphics and Vision, vol. 2, no. 1, pp. 1-104, 2006.
[30] M. Szummer and R.W. Picard, "Temporal Texture Modeling," Proc. IEEE Int'l Conf. Image Processing, vol. 3, pp. 823-826, 1996.
[31] Y. Ukrainitz and M. Irani, "Aligning Sequences and Actions by Maximizing Space-Time Correlations," Proc. European Conf. Computer Vision, pp. 538-550, 2006.
[32] R. Vidal and P. Favaro, "Dynamicboost: Boosting Time Series Generated by Dynamical Systems," Proc. IEEE Int'l Conf. Computer Vision, 2007.
[33] R. Vidal and A. Ravichandran, "Optical Flow Estimation and Segmentation of Multiple Moving Dynamic Textures," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 516-521, 2005.
[34] P.A. Viola, "Alignment by Maximization of Mutual Information," Technical Report AITR-1548, 1995.
[35] L. Wei and M. Levoy, "Fast Texture Synthesis Using Tree-Structured Vector Quantization," Proc. ACM SIGGRAPH, pp. 479-488, 2000.