The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - Oct. (2013 vol.19)
pp: 1677-1686
Shih-Syun Lin , Dept. of Comput. Sci. & Inf. Eng., Nat. Cheng Kung Univ., Tainan, Taiwan
Chao-Hung Lin , Dept. of Geomatics, Nat. Cheng Kung Univ., Tainan, Taiwan
I-Cheng Yeh , Dept. of Comput. Sci. & Eng., Yuan Ze Univ., Chungli, Taiwan
Shu-Huai Chang , Dept. of Comput. Sci. & Inf. Eng., Nat. Cheng Kung Univ., Tainan, Taiwan
Chih-Kuo Yeh , Dept. of Comput. Sci. & Inf. Eng., Nat. Cheng Kung Univ., Tainan, Taiwan
Tong-Yee Lee , Dept. of Comput. Sci. & Inf. Eng., Nat. Cheng Kung Univ., Tainan, Taiwan
ABSTRACT
A novel content-aware warping approach is introduced for video retargeting. The key to this technique is adapting videos to fit displays with various aspect ratios and sizes while preserving both visually salient content and temporal coherence. Most previous studies solve this spatiotemporal problem by consistently resizing content in frames. This strategy significantly improves the retargeting results, but does not fully consider object preservation, sometimes causing apparent distortions on visually salient objects. We propose an object-preserving warping scheme with object-based significance estimation to reduce this unpleasant distortion. In the proposed scheme, visually salient objects in 3D space-time space are forced to undergo as-rigid-as-possible warping, while low-significance contents are warped as close as possible to linear rescaling. These strategies enable our method to consistently preserve both the spatial shapes and temporal motions of visually salient objects and avoid overdeformations on low-significance objects, yielding a pleasing motion-aware video retargeting. Qualitative and quantitative analyses, including a user study and experiments on complex videos containing diverse cameras and dynamic motions, show a clear superiority of our method over related video retargeting methods.
INDEX TERMS
Coherence, Optimization, Shape, Weaving, Spatiotemporal phenomena, Three-dimensional displays, Motion segmentation,warping, Video retargeting, spatial and temporal coherence, optimization
CITATION
Shih-Syun Lin, Chao-Hung Lin, I-Cheng Yeh, Shu-Huai Chang, Chih-Kuo Yeh, Tong-Yee Lee, "Content-Aware Video Retargeting Using Object-Preserving Warping", IEEE Transactions on Visualization & Computer Graphics, vol.19, no. 10, pp. 1677-1686, Oct. 2013, doi:10.1109/TVCG.2013.75
REFERENCES
[1] G.-X. Zhang, M.-M. Cheng, S.-M. Hu, and R.R. Martin, "A Shape-Preserving Approach to Image Resizing," Computer Graphics Forum, vol. 28, no. 7, pp. 1897-1906, 2009.
[2] Y.-S. Wang, H. Fu, O. Sorkine, T.-Y. Lee, and H.-P. Seidel, "Motion-Aware Temporal Coherence for Video Resizing," ACM Trans. Graphics, vol. 28, no. 5, pp. 127:1-127:10, 2009.
[3] Y.-S. Wang, H.-C. Lin, O. Sorkine, and T.-Y. Lee, "Motion-Based Video Retargeting with Optimized Crop-And-Warp," ACM Trans. Graphics, vol. 29, no. 4, pp. 90:1-90:9, 2010.
[4] Y.-S. Wang, J.-H. Hsiao, O. Sorkine, and T.-Y. Lee, "Scalable and Coherent Video Resizing with Per-Frame Optimization," ACM Trans. Graphics, vol. 30, no. 4, pp. 88:1-88:8, 2011.
[5] Y.-S. Wang, C.-L. Tai, O. Sorkine, and T.-Y. Lee, "Optimized Scale-and-Stretch for Image Resizing," ACM Trans. Graphics, vol. 27, no. 5, pp. 118:1-118:8, 2008.
[6] S.-S. Lin, I.-C. Yeh, C.-H. Lin, and T.-Y. Lee, "Patch-Based Image Warping for Content-Aware Retargeting," IEEE Trans. Multimedia, vol. 15, no. 2, pp. 359-368, Feb. 2013.
[7] M. Rubinstein, A. Shamir, and S. Avidan, "Improved Seam Carving for Video Retargeting," ACM Trans. Graphics, vol. 27, no. 3, pp. 16:1-16:9, 2008.
[8] A. Shamir and O. Sorkine, "Visual Media Retargeting," Proc. ACM SIGGRAPH ASIA Courses, pp. 11:1-11:13, 2009.
[9] M. Rubinstein, D. Gutierrez, O. Sorkine, and A. Shamir, "A Comparative Study of Image Retargeting," ACM Trans. Graphics, vol. 29, no. 6, pp. 160:1-160:10, 2010.
[10] F. Liu and M. Gleicher, "Video Retargeting: Automating Pan and Scan," Proc. 14th Ann. ACM Int'l Conf. Multimedia, pp. 241-250, 2006.
[11] V. Setlur, T. Lechner, M. Nienhaus, and B. Gooch, "Retargeting Images and Video for Preserving Information Saliency," IEEE Computer Graphics Applications, vol. 27, no. 5, pp. 80-88, Sept./Oct. 2007.
[12] T. Lu, Z. Yuan, Y. Huang, D. Wu, and H. Yu, "Video Retargeting with Nonlinear Spatial-Temporal Saliency Fusion," Proc. IEEE Int'l Conf. Image Processing, pp. 1801-1804, 2010.
[13] Y. Luo, J. Yuan, P. Xue, and Q. Tian, "Salient Region Detection and Its Application to Video Retargeting," Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 1-6, 2011.
[14] M. Frankovich and A. Wong, "Enhanced Seam Carving via Integration of Energy Gradient Functionals," Signal Processing Letters, vol. 18, no. 6, pp. 375-378, 2011.
[15] S. Kopf, T. Haenselmann, J. Kiess, B. Guthier, and W. Effelsberg, "Algorithms for Video Retargeting," Multimedia Tools Applications, vol. 51, no. 2, pp. 819-861, 2011.
[16] A. Shamir and S. Avidan, "Seam Carving for Media Retargeting," Comm. ACM, vol. 52, no. 1, pp. 77-85, 2009.
[17] C.-K. Chiang, S.-F. Wang, Y.-L. Chen, and S.-H. Lai, "Fast JND-Based Video Carving with GPU Acceleration for Real-Time Video Retargeting," IEEE Trans. Circuits and Systems for Video Technology, vol. 19, no. 11, pp. 1588-1597, Nov. 2009.
[18] M. Grundmann, V. Kwatra, M. Han, and I. Essa, "Discontinuous Seam-Carving for Video Retargeting," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 569-576, 2010.
[19] Y. Hu and D. Rajan, "Hybrid Shift Map for Video Retargeting," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 577-584, 2010.
[20] H.-M. Nam, K.-Y. Byun, J.-Y. Jeong, and K.-S. Choi, "Low Complexity Content-Aware Video Retargeting for Mobile Devices," IEEE Trans. Consumer Electronics, vol. 56, no. 1, pp. 182-189, Feb. 2010.
[21] W.-L. Chao, H.-H. Su, S.-Y. Chien, W.H. Hsu, and J.-J. Ding, "Coarse-to-Fine Temporal Optimization for Video Retargeting Based on Seam Carving," Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 1-6, 2011.
[22] L. Wolf, M. Guttmann, and D. Cohen-Or, "Non-Homogeneous Content-Driven Video-Retargeting," Proc. 11th IEEE Int'l Conf. Computer Vision, pp. 1-6, 2007.
[23] Y.-F. Zhang, S.-M. Hu, and R.R. Martin, "Shrinkability Maps for Content-Aware Video Resizing," Computer Graphics Forum, vol. 27, no. 7, pp. 1797-1804, 2008.
[24] P. Krähenbühl, M. Lang, A. Hornung, and M. Gross, "A System for Retargeting of Streaming Video," ACM Trans. Graphics, vol. 28, no. 5, pp. 126:1-126:10, 2009.
[25] Y. Niu, F. Liu, X. Li, and M. Gleicher, "Warp Propagation for Video Resizing," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 537-544, 2010.
[26] C. Tao, J. Jia, and H. Sun, "Active Window Oriented Dynamic Video Retargeting," Proc. Workshop Dynamical Vision, 2007.
[27] J. Sun and H. Ling, "Scale and Object Aware Image Retargeting for Thumbnail Browsing," Proc. Int'l Conf. Computer Vision (ICCV), pp. 1511-1518, 2011.
[28] S. Goferman, L. Zelnik-Manor, and A. Tal, "Context-Aware Saliency Detection," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2376-2383, 2010.
[29] M. Grundmann, V. Kwatra, M. Han, and I. Essa, "Efficient Hierarchical Graph-Based Video Segmentation," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2141-2148, 2010.
126 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool