This Article 
 Bibliographic References 
 Add to: 
Compact Representations of Videos Through Dominant and Multiple Motion Estimation
August 1996 (vol. 18 no. 8)
pp. 814-830

Abstract—An explosion of on-line image and video data in digital form is already well underway. With the exponential rise in interactive information exploration and dissemination through the World-Wide Web (WWW), the major inhibitors of rapid access to on-line video data are costs and management of capture and storage, lack of real-time delivery, and nonavailability of content-based intelligent search and indexing techniques. The solutions for capture, storage, and delivery may be on the horizon or a little beyond. However, even with rapid delivery, the lack of efficient authoring and querying tools for visual content-based indexing may still inhibit as widespread a use of video information as that of text and traditional tabular data is currently.

In order to be able to nonlinearly browse and index into videos through visual content, it is necessary to develop authoring tools that can automatically separate moving objects and significant components of the scene, and represent these in a compact form. Given that video data comes in torrents—almost a megabyte every 30th of a second—it will be highly inefficient to search for objects and scenes in every frame of a video. In this paper, we present techniques to automatically derive compact representations of scenes and objects from the motion information.

Image motion is a significant cue in videos for the separation of scenes into their significant components and for the separation of moving objects. Motion analysis is useful in capturing the visual content of videos for indexing and browsing in two different ways. First, separation of the static scene from moving objects can be accomplished by employing dominant 2D/3D motion estimation methods. Alternatively, if the goal is to be able to represent the fixed scene too as a composition of significant structures and objects, then simultaneous multiple motion methods might be more appropriate. In either case, view-based summarized representations of the scene can be created by video compositing/mosaicing based on the estimated motions. We present robust algorithms for both kinds of representations: 1) dominant motion estimation based techniques which exploit a fairly common occurrence in videos that a mostly fixed background (scene) is imaged with or without independently moving objects, and 2) simultaneous multiple motion estimation and representation of motion video using layered representations. Ample examples of the representations achieved by each method are included in the paper.

[1] E.H. Adelson and P. Anandan, "Ordinal Characteristics of Transparency," Proc. AAAI Workshop Qualitative Vision, 1990.
[2] G. Adiv, "Determining 3D Motion and Structure from Optical Flows Generated by Several Moving Objects," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 7, no. 4, pp. 384-401, 1985.
[3] J. Ashley, M. Flickner, J. Hafner et al., , "Automatic and Semi-Automatic Methods for Image Annotation and Retrieval in QBIC," Image and Video Storage Retrieval III, vol. 2,420. San Jose, Calif.: SPIE, 1995.
[4] S. Ayer, "Sequential and Competitive Methods for the Estimation of Multiple Motions," PhD thesis, EPFL, Lausanne, 1995.
[5] S. Ayer and H. Sawhney, "Layered Representation of Motion Video Using Robust Maximum-Likelihood Estimation of Mixture Models and mdl Encoding," Int'l Conf. Computer Vision, pp. 777-784,Cambridge, Mass., June 1995.
[6] S. Ayer, P. Schroeter, and J. Bigün, "Segmentation of Moving Objects by Robust Motion Parameter Estimation Over Multiple Frames," Proc. ECCV,Stockholm, May 1994.
[7] J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hingorani, “Hiercharchical Model-Based Motion Estimation,” Proc. European Conf. Computer Vision, pp. 237-252, 1992.
[8] M.J. Black and P. Anandan, "The Robust Estimation of Multiple Motions: Affine and Piecewise-Smooth Flow Fields," Technical Report TR, Xerox PARC, CA, Dec. 1993.
[9] M. Bober and J. Kittler, "Robust Motion Analysis," Proc. CVPR, pp. 947-952,Seattle, June 1994.
[10] T. Darrell and A. Pentland, "Cooperative Robust Estimation Using Layers of Support," Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 5, pp. 474-487, May 1995.
[11] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker, “Query by Image and Video Content: The QBIC System,” IEEE Computer, 1995.
[12] F.R. Hampel, E.M. Ronchetti, P.J. Rousseeuw, and W.A. Stahel, Robust Statistics: The Approach Based on Influence Functions.New York: J. Wiley&Sons, 1986.
[13] K. Hanna, "Direct Multi-Resolution Estimation of Ego-Motion and Structure from Motion," IEEE Workshop Visual Motion, pp. 156-162,Princeton, N.J., Oct. 1991.
[14] R.I. Hartley, “Euclidean Reconstruction from Uncalibrated Views,” Proc. Workshop Applications of Invariants in Computer Vision, pp. 187-202, Oct. 1993.
[15] H. Hashihara, J. Takahashi, and J.-K. Hong, "Scene Retrieval Method for Motion Image Databases," technical report, IBM Tokyo Research Laboratory, 1991.
[16] V. Hasselblad, "Estimation of Parameters for a Mixture of Normal Distributions," Technometrics, vol. 8, no. 3, pp. 431-446, Aug. 1966.
[17] K. Hirata and T. Kato, "Rough Sketch-Based Image Information Retrieval," NEC R&D, vol. 34, no. 2, pp. 263-273, Apr. 1993.
[18] S. Hsu, P. Anandan, and S. Peleg, "Accurate Computation of Optical Flow by Using Layered Motion Representation," Proc. ICPR, pp. 743-746,Jerusalem, Oct. 1994.
[19] M. Irani, P. Anandan, and S. Hsu, “Mosaic Based Representations of Video Sequences and Their Applications,” Proc. Fifth Int'l Conf. Computer Vision, pp. 605-611, June 1995.
[20] M. Irani, B. Rousso, and S. Peleg, “Detecting and Tracking Multiple Moving Objects Using Temporal Integration,” Proc. European Conf. Computer Vision, pp. 282-287, May 1992.
[21] A.D. Jepson and M. Black, “Mixture Models for Optical Flow Computation,” Proc. Computer Vision and Pattern Recognition, pp. 760-761, June 1993.
[22] R. Kumar, P. Anandan, and K. Hanna, “Direct Recovery of Shape from Multiple Views: A Parallax Based Approach,” Proc. Int'l Conf. Pattern Recognition, pp. 685-688, Oct. 1994.
[23] R. Kumar, P. Anandan, M. Irani, J. Bergen, and K. Hanna, “Representation of Scenes from Collections of Images,“ Proc. IEEE Workshop Representation of Visual Scenes, pp. 10-17, June 1995.
[24] Y.G. Leclerc, "Constructing Simple Stable Descriptions for Image Partitioning," Int'l J. Computer Vision, vol. 3, no. 1, pp. 73-102, 1989.
[25] G. Li, "Robust Regression," Exploring Data Tables, Trends and Shapes, D.C. Hoaglin, F. Mosteller, and J.W. Tukey, eds., chap. 8. New York: John Wiley&Sons, 1985.
[26] W.J. MacLean, A.D. Jepson, and R.C. Frecker, "Recovery of Egomotion and Segmentation of Independent Object Motion Using the EM Algorithm," Proc. BMVC, 1994.
[27] S. Mann and R. Picard, “Virtual Bellows: Constructing High Quality Stills from Video,” Proc. First IEEE Int'l Conf. Image Processing, vol. I, pp. 363-367, Nov. 1994.
[28] G.J. McLachlan and K.E. Basford, Mixture Models Inference and Applications to Clustering.New York and Basel: Marcel Dekker, Inc., 1988.
[29] W. Niblack, R. Barber, W. Equitz et al., , "The QBIC Project: Querying Images by Content Using Color, Texture, and Shape," SPIE 1980, Storage and Retrieval for Image and Video Databases, pp. 173-187, Feb. 1993.
[30] J.M. Odobez and P. Bouthemy, "Detection of Multiple Moving Objects Using Multiscale mrf with Camera Motion Compensation," Proc. ICIP, pp. 257-261,Austin, Tex., Nov. 1994.
[31] J.M. Odobez and P. Bouthemy, "Robust Multiresolution Estimation of Parametric Motion Models in Complex Image Sequences," Proc. Seventh EUSIPCO European Conf. Signal Processing, pp. 411-414,Edinburgh, Sept. 1994.
[32] A. Pentland, R.W. Picard, and S. Sclaroff, "Photobook: Tools for Content-Based Manipulation of Image Databases," Proc. Storage and Retrieval for Image and Video Databases II. SPIE, 1994.
[33] J. Rissanen, "A Universal Prior for Integers and Estimation by Minimum Description Length," Annals of Statistics, vol. 11, no. 2, pp. 416-431, 1983.
[34] P. Rousseeuw and A. Leory, Robust Regression and Outlier Detection. Wiley Series in Probability and Statistics, 1987.
[35] H.S. Sawhney, "Simplifying Motion and Structure Analysis Using Planar Parallax and Image Warping," Proc. Int'l Conf. Pattern Recognition, 1994.
[36] H.S. Sawhney, S. Ayer, and M. Gorkani, “Model-Based 2D and 3D Dominant Motion Estimation for Mosaicing and Video Representation,” Proc. Fifth Int'l Conf. Computer Vision, pp. 583-590, June 1995.
[37] G.A.F. Seber and C.J. Wild, Nonlinear Regression.New York: Wiley, 1989.
[38] A. Shashua and N. Navab, "Relative Affine Structure: Theory and Application to 3D Reconstruction From Perspective Views," Proc. CVPR '94, pp. 483-489, 1994.
[39] R. Szeliski, “Image Mosaicing for Tele-Reality Applications,” IEEE Computer Graphics and Applications, 1996.
[40] R. Szeliski and S.B. Kang, “Direct Methods for Visual Scene Reconstruction,” Proc. Workshop Representations of Visual Scenes, 1995.
[41] L. Teodosio and W. Bender, "Salient Video Stills: Content and Context Preserved," Proc. ACM Multimedia 93, ACM Press, New York, 1993, pp. 39-46.
[42] Y.T.A. Akutsu, K. Otsuji, and T. Sadakata, “VideoMAP and VideoSpaceIcon: Tools for Anatomizing Video Content,” ACM INTERCHI, 1993, pp. 131-136.
[43] J.Y.A. Wang and E. Adelson, "Layered Representation for Motion Analysis," Proc. Computer Vision and Pattern Recognition Conf., 1993.

Index Terms:
Compact video representations, video motion analysis, video mosaics, video indexing, layered motion representations, motion segmentation, robust estimation, mixture models, expectation-maximization (EM) algorithm.
Harpreet S. Sawhney, Serge Ayer, "Compact Representations of Videos Through Dominant and Multiple Motion Estimation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 814-830, Aug. 1996, doi:10.1109/34.531801
Usage of this product signifies your acceptance of the Terms of Use.