This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Linear Sequence-to-Sequence Alignment
February 2010 (vol. 32 no. 2)
pp. 304-320
Flávio L.C. Pádua, Centro Federal de Educação Tecnológica de Minas Gerais , Belo Horizonte
Rodrigo L. Carceroni, Google Inc, Mountain View
Geraldo A.M.R. Santos, Universidade Federal de Minas Gerais, Belo Horizonte
Kiriakos N. Kutulakos, University of Toronto , Toronto
In this paper, we consider the problem of estimating the spatiotemporal alignment between N unsynchronized video sequences of the same dynamic 3D scene, captured from distinct viewpoints. Unlike most existing methods, which work for N=2 and rely on a computationally intensive search in the space of temporal alignments, we present a novel approach that reduces the problem for general N to the robust estimation of a single line in {{\hbox{\rlap{I}\kern 2.0pt{\hbox{R}}}}}^{N}. This line captures all temporal relations between the sequences and can be computed without any prior knowledge of these relations. Considering that the spatial alignment is captured by the parameters of fundamental matrices, an iterative algorithm is used to refine simultaneously the parameters representing the temporal and spatial relations between the sequences. Experimental results with real-world and synthetic sequences show that our method can accurately align the videos even when they have large misalignments (e.g., hundreds of frames), when the problem is seemingly ambiguous (e.g., scenes with roughly periodic motion), and when accurate manual alignment is difficult (e.g., due to slow-moving objects).

[1] S. Vedula, S. Baker, and T. Kanade, “Spatio-Temporal View Interpolation,” Proc. Eurographics Workshop Rendering, pp. 65-76, 2002.
[2] L. Zelnik-Manor and M. Irani, “Event-Based Analysis of Video,” Proc. IEEE Computer Vision and Pattern Recognition Conf., vol. 2, pp. II-123-II-130, 2001.
[3] Y. Caspi and M. Irani, “Alignment of Non-Overlapping Sequences,” Proc. Int'l Conf. Computer Vision, vol. 2, pp. 76-83, 2001.
[4] I. Reid and A. Zisserman, “Goal Directed Video Metrology,” Proc. European Conf. Computer Vision, pp. 647-658, 1996.
[5] D. Wedge, D. Huynh, and P. Kovesi, “Using Space-Time Interest Points for Video Sequence Synchronization,” Proc. IAPR Conf. Machine Vision Applications, pp. 190-194, 2007.
[6] L. Wolf and A. Zomet, “Wide Baseline Matching between Unsynchronized Video Sequences,” Int'l J. Computer Vision, vol. 68, no. 1, pp. 43-52, 2006.
[7] M. Ushizaki, T. Okatani, and K. Deguchi, “Video Synchronization Based on Co-Occurrence of Appearance Changes in Video Sequences,” Proc. Int'l Conf. Pattern Recognition, pp. 71-74, 2006.
[8] Y. Ukrainitz and M. Irani, “Aligning Sequences and Actions by Maximizing Space-Time Correlations,” Proc. European Conf. Computer Vision, pp. 538-550, 2006.
[9] O. Shakil, “An Efficient Video Alignment Approach for Non-Overlapping Sequences with Free Camera Movement,” Proc. Int'l Conf. Acoustics, Speech, and Signal Processing, vol. 2, pp. 257-260, 2006.
[10] C. Dai, Y. Zheng, and X. Li, “Accurate Video Alignment Using Phase Correlation,” IEEE Signal Processing Letters, vol. 13, no. 12, pp. 737-740, Dec. 2006.
[11] C. Dai, Y. Zheng, and X. Li, “Subframe Video Synchronization via 3d Phase Correlation,” Proc. Int'l Conf. Image Processing, pp. 501-504, 2006.
[12] K. Lee and R.D. Green, “Temporally Synchronising Image Sequences Using Motion Kinematics,” Proc. Image and Vision Computing New Zealand Conf., 2005.
[13] I. Laptev, S.J. Belongie, P. Perez, and J. Wills, “Periodic Motion Detection and Segmentation via Approximate Sequence Alignment,” Proc. Int'l Conf. Computer Vision, vol. 1, pp. 816-823, 2005.
[14] D. Wedge, P. Kovesi, and D. Huynh, “Trajectory Based Video Sequence Synchronization,” Proc. Digital Image Computing: Techniques and Applications Conf., pp. 79-86, 2005.
[15] J. Yan and M. Pollefeys, “Video Synchronization via Space-Time Interest Point Distribution,” Proc. Advanced Concepts for Intelligent Vision Systems, 2004.
[16] D.W. Pooley, M.J. Brooks, A.J. van den Hengel, and W. Chojnacki, “A Voting Scheme for Estimating the Synchrony of Moving-Camera Videos,” Proc. Int'l Conf. Image Processing, vol. 1, pp. 413-416, 2003.
[17] C. Rao, A. Gritai, M. Shah, and T.S. Mahmood, “View-Invariant Alignment and Matching of Video Sequences,” Proc. Int'l Conf. Computer Vision, vol. 2, pp. 939-945, 2003.
[18] Y. Caspi, D. Simakov, and M. Irani, “Feature-Based Sequence-to-Sequence Matching,” Int'l J. Computer Vision, vol. 68, no. 1, pp. 53-64, 2006.
[19] L. Wolf and A. Zomet, “Correspondence-Free Synchronization and Reconstruction in a Non-Rigid Scene,” Proc. Workshop Vision and Modelling of Dynamic Scenes, 2002.
[20] L. Wolf and A. Zomet, “Sequence to Sequence Self Calibration,” Proc. European Conf. Computer Vision, vol. 2, pp. 370-382, 2002.
[21] Y. Caspi and M. Irani, “A Step Towards Sequence-to-Sequence Alignment,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 682-689, 2000.
[22] L. Lee, R. Romano, and G. Stein, “Monitoring Activities from Multiple Video Streams: Establishing a Common Coordinate Frame,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 758-767, Aug. 2000.
[23] G. Stein, “Tracking from Multiple View Points: Self-Calibration of Space and Time,” Proc. DARPA Image Understanding Workshop, pp. 521-527, 1998.
[24] K. Raguse and C. Heipke, “Photogrammetric Synchronization of Image Sequences,” Proc. ISPRS Commission V Symp. Image Eng. and Vision Metrology, pp. 254-259, 2006.
[25] W. Anthony, L. Robert, and B. Prosenjit, “Temporal Synchronization of Video Sequences in Theory and in Practice,” Proc. Workshop Motion and Video Computing, vol. 2, pp. 132-137, 2005.
[26] E. Tola, V. Lepetit, and P. Fua, “A Fast Local Descriptor for Dense Matching,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[27] M. Fischler and R. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Comm. ACM, vol. 24, pp. 381-395, June 1981.
[28] J. Horst and I. Beichl, “A Simple Algorithm for Efficient Piecewise Linear Approximation of Space Curves,” Proc. Int'l Conf. Image Processing, vol. 2, pp. 744-747, 1997.
[29] W. Press, B. Flannery, S. Teukolsky, and W. Vetterling, Numerical Recipes in C: The Art of Scientific Computing. Cambridge Univ. Press, 1988.
[30] K. Atkinson, An Introduction to Numerical Analysis. John Wiley and Sons, 1989.
[31] C. Tomasi, “Mathematical Methods for Robotics and Vision,” Technical Report CS 205, Stanford Univ., 2000.
[32] J. Hefferon, Linear Algebra. Math. Dept. of Saint Michael's College, 2001.
[33] A. Jepson, D. Fleet, and T. El-Maraghi, “Robust Online Appearance Models for Visual Tracking,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1296-1311, Oct. 2003.
[34] FIFA, “FIFA World Cup Archives: Goal of the Century,” http://fifaworldcup.yahoo.com/02/en/pf/h/ gotclaunch.html, 2002.
[35] M. Brown and D. Lowe, “Recognizing Panoramas,” Proc. Int'l Conf. Computer Vision, pp. 1218-1225, 2003.
[36] R.L. Carceroni, F.L.C. Padua, G.A.M.R. Santos, and K.N. Kutulakos, “Linear Sequence-to-Sequence Alignment,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 746-753, 2004.

Index Terms:
Video synchronization, object tracking, epipolar geometry, spatiotemporal alignment, image and video registration.
Citation:
Flávio L.C. Pádua, Rodrigo L. Carceroni, Geraldo A.M.R. Santos, Kiriakos N. Kutulakos, "Linear Sequence-to-Sequence Alignment," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 2, pp. 304-320, Feb. 2010, doi:10.1109/TPAMI.2008.301
Usage of this product signifies your acceptance of the Terms of Use.