This Article 
 Bibliographic References 
 Add to: 
Fast Exact Nearest Patch Matching for Patch-Based Image Editing and Processing
August 2011 (vol. 17 no. 8)
pp. 1122-1134
Chunxia Xiao, Wuhan University, Wuhan
Meng Liu, Wuhan University, Wuhan
Yongwei Nie, Wuhan University, Wuhan
Zhao Dong, MPI Informatik, Saarbrücken
This paper presents an efficient exact nearest patch matching algorithm which can accurately find the most similar patch-pairs between source and target image. Traditional match matching algorithms treat each pixel/patch as an independent sample and build a hierarchical data structure, such as kd-tree, to accelerate nearest patch finding. However, most of these approaches can only find approximate nearest patch and do not explore the sequential overlap between patches. Hence, they are neither accurate in quality nor optimal in speed. By eliminating redundant similarity computation of sequential overlap between patches, our method finds the exact nearest patch in brute-force style but reduces its running time complexity to be linear on the patch size. Furthermore, relying on recent multicore graphics hardware, our method can be further accelerated by at least an order of magnitude ({\ge} 10{\times}). This greatly improves performance and ensures that our method can be efficiently applied in an interactive editing framework for moderate-sized image even video. To our knowledge, this approach is the fastest exact nearest patch matching method for high-dimensional patch and also its extra memory requirement is minimal. Comparisons with the popular nearest patch matching methods in the experimental results demonstrate the merits of our algorithm.

[1] D. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” Int'l J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
[2] J. Sivic and A. Zisserman, “Video Google: A Text Retrieval Approach to Object Matching in Videos,” Proc. Ninth IEEE Int'l Conf. Computer Vision (ICCV), vol. 2, pp. 1470-1477, 2003.
[3] A. Efros and W. Freeman, “Image Quilting for Texture Synthesis and Transfer,” Proc. ACM SIGGRAPH, pp. 341-346, 2001.
[4] L.-Y. Wei and M. Levoy, “Fast Texture Synthesis Using Tree-Structured Vector Quantization,” Proc. ACM SIGGRAPH, pp. 479-488, 2000.
[5] J. Kopf, C.W. Fu, D. Cohen-Or, O. Deussen, D. Lischinski, and T.T. Wong, “Solid Texture Synthesis from 2D Exemplars,” ACM Trans. Graphics, vol. 26, no. 3, pp. 21-29, 2007.
[6] W. Freeman, T. Jones, and E. Pasztor, “Example-Based Super-Resolution,” IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56-65, Mar./Apr. 2002.
[7] A. Buades, B. Coll, and J. Morel, “A Non-Local Algorithm for Image Denoising,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR), vol. 2, p. 60, 2005.
[8] D. Simakov, Y. Caspi, E. Shechtman, and M. Irani, “Summarizing Visual Data Using Bidirectional Similarity,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1-8, 2008.
[9] J. Bentley, “Multidimensional Binary Search Trees Used for Associative Searching,” Comm. ACM, vol. 18, no. 9, pp. 509-517, 1975.
[10] S. Arya, D.M. Mount, N.S. Netanyahu, and R. Silverman, and A. Wu, “An Optimal Algorithm for Approximate Nearest Neighbor Searching,” J. ACM, vol. 45, no. 6, pp. 891-923, 1998.
[11] M. Ashikhmin, “Synthesizing Natural Textures,” Proc. ACM Symp. Interactive 3D Graphics (I3D), pp. 217-226, 2001.
[12] C. Barnes, E. Shechtman, A. Finkelstein, and D. Goldman, “Patchmatch: A Randomized Correspondence Algorithm for Structural Image Editing,” Proc. ACM SIGGRAPH, 2009.
[13] X. Tong, J. Zhang, L. Liu, X. Wang, B. Guo, and H. Shum, “Synthesis of Bidirectional Texture Functions on Arbitrary Surfaces,” ACM Trans. Graphics, vol. 21, no. 3, pp. 665-672, 2002.
[14] J. Hays and A.A. Efros, “Scene Completion Using Millions of Photographs,” Proc. ACM SIGGRAPH, 2007.
[15] T. Huang, Two-Dimensional Signal Processing II: Transforms and Median Filters, Springer-Verlag, 1981.
[16] B. Weiss, “Fast Median and Bilateral Filtering,” ACM Trans. Graphics, vol. 25, pp. 519-526, 2006.
[17] P. Indyk and R. Motwani, “Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality,” Proc. 30th Ann. ACM Symp. Theory of Computing, pp. 604-613, 1998.
[18] E. Chávez, G. Navarro, R. Baeza-Yates, and J. Marroquín, “Searching in Metric Spaces,” ACM Computing Surveys, vol. 33, no. 3, pp. 273-321, 2001.
[19] G. Shakhnarovich, T. Darrell, and P. Indyk, Nearest-Neighbor Methods in Learning and Vision: Theory and Practice. MIT Press, 2005.
[20] N. Kumar, L. Zhang, and S. Nayar, “What is a Good Nearest Neighbors Algorithm for Finding Similar Patches in Images?” Proc. European Conf. Computer Vision (ECCV), pp. 364-378, 2008.
[21] R. Sproull, “Refinements to Nearest-Neighbor Searching Ink-Dimensional Trees,” Algorithmica, vol. 6, no. 1, pp. 579-589, 1991.
[22] J. MacQueen, “Some Methods for Classification and Analysis of Multivariate Observations,” Proc. Symp. Math. Statistics and Probability, pp. 281-297, 1967.
[23] Y. Linde, A. Buzo, and R. Gray, “An Algorithm for Vector Quantizer Design,” IEEE Trans. Comm., vol. 28, no. 1, pp. 84-95, Jan. 1980.
[24] P. Yianilos, “Data Structures and Algorithms for Nearest Neighbor Search in General Metric Spaces,” Proc. Fourth Ann. ACM-SIAM Symp. Discrete Algorithms, pp. 311-321, 1993.
[25] C. Xiao and M. Liu, “Efficient Mean-Shift Clustering Using Gaussian KD-Tree,” Computer Graphics Forum, vol. 29, no. 7, pp. 2065-2074, 2010.
[26] S. Lefebvre and H. Hoppe, “Appearance-Space Texture Synthesis,” ACM Trans. Graphics, vol. 25, no. 3, pp. 541-548, 2006.
[27] S. Kilthau, M. Drew, and T. Moller, “Full Search Content Independent Block Matching Based on the Fast Fourier Transform,” Proc. Int'l Conf. Image Processing (ICIP), vol. 1, pp. 669-672, 2002.
[28] C. Soler, M. Cani, and A. Angelidis, “Hierarchical Pattern Mapping,” Proc. ACM SIGGRAPH, pp. 673-680, 2002.
[29] F. Crow, “Summed-Area Tables for Texture Mapping,” Computer Graphics, vol. 18, no. 3, pp. 207-212, 1984.
[30] A. Rivers and D. James, “FastLSM: Fast Lattice Shape Matching for Robust Real-Time Deformation,” ACM Trans. Graphics, vol. 26, no. 3, 2007.
[31] A. Lefohn, M. Houston, C. Boyd, K. Fatahalian, T. Forsyth, D. Luebke, and J. Owens, “Beyond Programmable Shading: Fundamentals,” Proc. ACM SIGGRAPH: ACM SIGGRAPH '08 classes, pp. 1-21, 2008.
[32] C. NVIDIA, “Compute Unified Device Architecture Programming Guide, Version 2.2,” NVIDIA, 2009.
[33] V. Kwatra, I. Essa, A. Bobick, and N. Kwatra, “Texture Optimization for Example-Based Synthesis,” Proc. ACM SIGGRAPH, pp. 795-802, 2005.
[34] Y. Wexler, E. Shechtman, and M. Irani, “Space Time Completion of Video,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp. 463-476, Mar. 2007.
[35] C. Xiao, Y. Nie, W. Hua, and W. Zheng, “Fast Multi-Scale Joint Bilateral Texture Upsampling,” The Visual Computer, vol. 26, no. 4, pp. 263-275, 2010.

Index Terms:
Nearest patch search, texture synthesis, image completion, image denoising, image summarization.
Chunxia Xiao, Meng Liu, Yongwei Nie, Zhao Dong, "Fast Exact Nearest Patch Matching for Patch-Based Image Editing and Processing," IEEE Transactions on Visualization and Computer Graphics, vol. 17, no. 8, pp. 1122-1134, Aug. 2011, doi:10.1109/TVCG.2010.226
Usage of this product signifies your acceptance of the Terms of Use.