This Article 
 Bibliographic References 
 Add to: 
The Patch Transform
August 2010 (vol. 32 no. 8)
pp. 1489-1501
Taeg Sang Cho, Massachusetts Institute of Technology, Cambridge
Shai Avidan, Adobe Systems Incorporated, Auburndale
William T. Freeman, Massachusetts Institute of Technology, Cambridge and Adobe Systems Incorporated, Auburndale
The patch transform represents an image as a bag of overlapping patches sampled on a regular grid. This representation allows users to manipulate images in the patch domain, which then seeds the inverse patch transform to synthesize modified images. Possible modifications include the spatial locations of patches, the size of the output image, or the pool of patches from which an image is reconstructed. When no modifications are made, the inverse patch transform reduces to solving a jigsaw puzzle. The inverse patch transform is posed as a patch assignment problem on a Markov random field (MRF), where each patch should be used only once and neighboring patches should fit to form a plausible image. We find an approximate solution to the MRF using loopy belief propagation, introducing an approximation that encourages the solution to use each patch only once. The image reconstruction algorithm scales well with the total number of patches through label pruning. In addition, structural misalignment artifacts are suppressed through a patch jittering scheme that spatially jitters the assigned patches. We demonstrate the patch transform and its effectiveness on natural images.

[1] T.S. Cho, M. Butman, S. Avidan, and W.T. Freeman, "The Patch Transform and Its Applications to Image Editing," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[2] E.D. Demaine and M.L. Demaine, "Jigsaw Puzzles, Edge Matching, and Polyomino Packing: Connections and Complexity," Graphs and Combinatorics, vol. 23, pp. 195-208, 2007.
[3] M.G. Chung, M.M. Fleck, and D.A. Forsyth, "Jigsaw Puzzle Solver Using Shape and Color," Proc. Int'l Conf. Signal Processing, 1998.
[4] T.R. Nielsen, P. Drewsen, and K. Hansen, "Solving Jigsaw Puzzles Using Image Features," Pattern Recognition Letters, vol. 29, pp. 1924-1933, 2008.
[5] B.J. Brown, C. Toler-Franklin, D. Nehab, M. Burns, D. Dobkin, A. Vlachopoulos, C. Doumas, S. Rusinkiewicz, and T. Weyrich, "A System for High-Volume Acquisition and Matching of Fresco Fragments: Reassembling Theran Wall Paintings," Proc. ACM SIGGRAPH, 2008.
[6] H.C. da Gama Leitao and J. Stolfi, "A Multiscale Method for the Reassembly of Two-Dimensional Fragmented Objects," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 9, pp. 1239-1251, Sept. 2002.
[7] D. Koller and M. Levoy, "Computer-Aided Reconstruction and New Matches in the Forma Urbis Romae," Bullettino Della Commissione Archeologica Comunale di Roma, 2006.
[8] C.-S. Wang, "Determining Molecular Conformation from Distance or Density Data," PhD dissertation, Massachusetts Inst. of Tech nology, 2000.
[9] M. Levison, "The Computer in Literary Studies," Machine Translation, pp. 173-194, 1967.
[10] L. Zhu, Z. Zhou, and D. Hu, "Globally Consistent Reconstruction of Ripped-Up Documents," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 1, pp. 1-13, Jan. 2008.
[11] Y.-X. Zhao, M.-C. Su, Z.-L. Chou, and J. Lee, "A Puzzle Solver and Its Application in Speech Descrambling," Proc. 2007 WSEAS Int'l Conf. Computer Eng. and Applications, 2007.
[12] N. Jojic, B.J. Frey, and A. Kannan, "Epitomic Analysis of Appearance and Shape," Proc. IEEE Int'l Conf. Computer Vision, 2003.
[13] A. Kannan, J. Winn, and C. Rother, "Clustering Appearance and Shape by Learning Jigsaws," Advances in Neural Information Processing Systems 19, MIT Press, 2006.
[14] A. Criminisi, P. Pérez, and K. Toyama, "Object Removal by Exemplar-Based Inpainting," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2003.
[15] Y. Wexler, E. Shechtman, and M. Irani, "Space-Time Video Completion," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2004.
[16] J. Sun, L. Yuan, J. Jia, and H.-Y. Shum, "Image Completion with Structure Propagation," Proc. ACM SIGGRAPH, 2005.
[17] J. Kopf, C.-W. Fu, D. Cohen-Or, O. Deussen, D. Lischinski, and T.-T. Wong, "Solid Texture Synthesis from 2D Exemplars," Proc. ACM SIGGRAPH, 2007.
[18] E.N. Mortensen and W.A. Barrett, "Intelligent Scissors for Image Composition," ACM Trans. Graphics, 1995.
[19] J.-F. Lalonde, D. Hoiem, A.A. Efros, C. Rother, J. Winn, and A. Criminisi, "Photo Clip Art," ACM Trans. Graphics, 2007.
[20] P. Pérez, M. Gangnet, and A. Blake, "Poisson Image Editing," ACM Trans. Graphics, vol. 22, pp. 313-318, 2003.
[21] J. Wang and M.F. Cohen, "An Iterative Optimization Approach for Unified Image Segmentation and Matting," Proc. IEEE Int'l Conf. Computer Vision, 2005.
[22] A. Levin, A. Rav-Acha, and D. Lischinski, "Spectral Matting," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[23] D. Simakov, Y. Caspi, E. Shechtman, and M. Irani, "Summarizing Visual Data Using Bidirectional Similarity," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[24] C. Rother, L. Bordeaux, Y. Hamadi, and A. Blake, "Autocollage," ACM Trans. Graphics, 2006.
[25] M. Brown and D. Lowe, "Recognising Panoramas," Proc. IEEE Int'l Conf. Computer Vision, 2003.
[26] J.D. Bonet, "Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images," ACM Trans. Graphics, 1997.
[27] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, "Image Inpainting," ACM Trans. Graphics, 2000.
[28] A.A. Efros and T.K. Leung, "Texture Synthesis by Non-Parametric Sampling," Proc. IEEE Int'l Conf. Computer Vision, 1999.
[29] A.A. Efros and W.T. Freeman, "Image Quilting for Texture Synthesis and Transfer," Proc. ACM SIGGRAPH, 2001.
[30] L. Liang, C. Liu, Y.-Q. Xu, B. Guo, and H.-Y. Shum, "Real-Time Texture Synthesis by Patch-Based Sampling," ACM Trans. Graphics, vol. 20, no. 3, pp. 127-150, July 2001.
[31] W.T. Freeman, E.C. Pasztor, and O.T. Carmichael, "Learning Low-Level Vision," Int'l J. Computer Vision, vol. 40, pp. 25-47, 2000.
[32] V. Kwatra, A. Schodl, I. Essa, G. Turk, and A. Bobick, "Graphcut Textures: Image and Video Synthesis Using Graph Cuts," Proc. ACM SIGGRAPH, 2003.
[33] G. Ramanarayanan and K. Bala, "Constrained Texture Synthesis via Energy Minimization," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 1, pp. 167-178, Jan./Feb. 2007.
[34] N. Komodakis and G. Tziritas, "Image Completion Using Efficient Belief Propagation via Priority Scheduling and Dynamic Pruning," IEEE Trans. Image Processing, vol. 16, no. 11, pp. 2649-2661, Nov. 2007.
[35] A. Agrawal, R. Raskar, and R. Chellappa, "What Is the Range of Surface Reconstructions from a Gradient Field?" Proc. European Conf. Computer Vision, 2006.
[36] J.S. Yedidia, W.T. Freeman, and Y. Weiss, "Understanding Belief Propagation and Its Generalizations," Exploring Artificial Intelligence in the New Millennium, pp. 239-269, http://portal.acm.orgcitation.cfm?id=779352 , Morgan Kaufmann Publishers, 2003.
[37] S. Avidan and A. Shamir, "Seam Carving for Content-Aware Image Resizing," ACM Trans. Graphics, vol. 26, 2007.
[38] J.M. Coughlan and S.J. Ferreira, "Finding Deformable Shapes Using Loopy Belief Propagation," Proc. European Conf. Computer Vision, 2002.
[39] M.P. Kumar and P. Torr, "Fast Memory-Efficient Generalized Belief Propagation," Proc. European Conf. Computer Vision, 2006.
[40] J. Lasserre, A. Kannan, and J. Winn, "Hybrid Learning of Large Jigsaws," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[41] J. Sun, N.-N. Zheng, and H.-Y. Shum, "Stereo Matching Using Belief Propagation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 7, pp. 787-800, July 2003.
[42] M. Pechaud, R. Keriven, T. Papadopoulo, and J.-M. Badier, "Combinatorial Optimization for Electrode Labeling of Eeg Caps," Medical Image Computing and Computer-Assisted Intervention, Springer, 2007.
[43] C.M. Bishop, D. Spiegelhalter, and J. Winn, "Vibes: A Variational Inference Engine for Bayesian Networks," Proc. Conf. Neural Information Processing Systems, 2003.
[44] D. Koller, U. Lerner, and D. Angelov, "A General Algorithm for Approximate Inference and Its Application to Hybrid Bayes Nets," Proc. Ann. Conf. Uncertainty in Artificial Intelligence, 1998.

Index Terms:
Image models, statistical, applications, image-based rendering.
Taeg Sang Cho, Shai Avidan, William T. Freeman, "The Patch Transform," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 8, pp. 1489-1501, Aug. 2010, doi:10.1109/TPAMI.2009.133
Usage of this product signifies your acceptance of the Terms of Use.