This Article 
 Bibliographic References 
 Add to: 
Generalized Mosaicing: Wide Field of View Multispectral Imaging
October 2002 (vol. 24 no. 10)
pp. 1334-1348

Abstract—We present an approach to significantly enhance the spectral resolution of imaging systems by generalizing image mosaicing. A filter transmitting spatially varying spectral bands is rigidly attached to a camera. As the system moves, it senses each scene point multiple times, each time in a different spectral band. This is an additional dimension of the generalized mosaic paradigm, which has recently demonstrated yielding high radiometric dynamic range images in a wide field of view, using a spatially varying density filter. The resulting mosaic represents the spectrum at each scene point. The image acquisition is as easy as in traditional image mosaics. We derive an efficient scene sampling rate, and use a registration method that accommodates the spatially varying properties of the filter. Using the data acquired by this method, we demonstrate scene rendering under different simulated illumination spectra. We are also able to infer information about the scene illumination. The approach was tested using a standard 8-bit black/white video camera and a fixed spatially varying spectral (interference) filter.

[1] A. Abrardo, L. Alparone, V. Cappellini, and A. Prosperi, “Color Constancy from Multispectral Images,” Proc. IEEE Int'l Conf. Image Processing, vol. 3, pp. 570-574, 1999.
[2] E.H. Adelson and J.R. Bergen, “The Plenoptic Function and the Elements of Early Vision,” Computational Models of Visual Processing, M. Landy and J.A. Movshon, eds., pp. 3-20, MIT Press, 1991.
[3] M. Aggarwal and N. Ahuja, “High Dynamic Range Panoramic Imaging,” Proc. Int'l Conf. Computer Vision, vol. I, pp. 2-9, 2001.
[4] M.J. Black and A. Rangarajan, “On the Unification of Line Processes, Outlier Rejection, and Robust Statistics with Applications in Early Vision,” Int'l J. Computer Vision, vol. 19, no. 1, pp. 57-91, 1996.
[5] P.J. Burt and E.H. Adelson, “A Multiresolution Spline with Application to Image Mosaics,” ACM Trans. Graphics, vol. 2, no. 4, pp. 217-236, Oct. 1983.
[6] R. Bernstein, “Digital Image Processing of Earth Observation Sensor Data,” IBM J. Research and Development, vol. 20, no. 1, pp. 40-57, 1976.
[7] D. Capel and A. Zisserman, “Automated Mosaicing with Super-Resolution Zoom,” Proc. IEEE Computer Vision and Pattern Recognition, pp. 885-891, 1998.
[8] S. Coorg, N. Master, and S. Teller, Acquisition of a Large Pose-Mosaic Dataset Proc. IEEE Computer Soc. Conf. Computer Vision and Pattern Recognition, 1998.
[9] M.L. Duplaquet, “Building Large Image Mosaics with Invisible Seam Lines,” Proc. SPIE Visual Information Processing VII, vol. 3387, pp. 369-377, 1998.
[10] D. Dykaar and G. Allan, “Capturing Color Information with Linear CCDs: The Case for Three-Chip Cameras,” Photonics Spectra, vol. 34, no. 5, pp. 183-186, 2000.
[11] 2000 Optics and Optical Instruments Catalog, Edmund Industrial Optics, stock nos. K45-645, p. 67, 2000.
[12] H. Farid and E.P. Simoncelli, “Range Estimation by Optical Differentiation,” J. Optical Soc. Am. A, vol. 15, pp. 1777-1786, 1998.
[13] E. Fernandez, R. Garfinkel, and R. Arbiol, “Mosaicking of Aerial Photographic Maps via Seams Defined by Bottleneck Shortest Paths,” Operations Research, vol. 46, no. 3, pp. 293-304, 1998.
[14] N. Gat, “Spectrometer Apparatus,” US Patent 5166755, 1992.
[15] N. Gat, “Imaging Spectroscopy Using Tunable Filters: A Review,” Proc. SPIE Wavelet Applications VII, vol. 4056, pp. 50-64, 2000.
[16] A. Glassner, Principles of Digital Image Synthesis.San Francisco: Morgan Kaufman, 1995.
[17] R. Gore, “Ancient Ashkelon,” Nat'l Geographic, vol. 199, no. 1, pp. 66-93, 2001.
[18] S. Gumustekin and R.W. Hall, “Mosaic Image Generation on a Flattened Sphere,” Proc. IEEE Workshop Applications of Computer Vision, pp. 50-55, 1996.
[19] M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “A Prototype of the Spectral Vision System,” Proc. 11th Scandinavian Conf. Image Analysis, vol. 1, pp. 79-86, 1999.
[20] M. Irani, P. Anandan, J. Bergen, R. Kumar, and S. Hsu, “Efficient Representations of Video Sequences and Their Application,” Signal Processing: Image Comm., vol. 8, pp. 327-351, 1996.
[21] M. Irani and P. Anandan, “Robust Multi-Sensor Image Alignment,” Proc. Int'l Conf. Computer Vision, pp. 959-966, 1998.
[22] A.K. Jain, Fundamentals of Digital Image Processing. Prentice Hall, 1989.
[23] R.B. Kerr, “Filters Expand Capabilities of Infrared Imaging,” Laser Focus World, vol. 37, no. 1, pp. 151-158, 2001.
[24] R. Kwok, J.C. Curlander, and S. Pang, “An Automated System for Mosaicking Spaceborne SAR Imagery,” Int'l J. Remote Sensing, vol. 11, pp. 209-223, 1990.
[25] S.G. Lipson, H. Lipson, and D.S. Tannhauser, Optical Physics, third ed. pp. 305-311, Cambridge: UK, 1998.
[26] C.J. Lada, D.L. DePoy, K.M. Merrill, and I. Gatley, “Infrared Images of M17,” The Astronomical J., vol. 374, pp. 533-539, 1991.
[27] C. Mahoney, “Thermal Infrared Imaging Spectrometer: An Advanced Optics Technology Instrument,” Proc. SPIE Imaging Spectroscopy of the Terrestrial Environment, vol. 1298, pp. 87-92, 1990.
[28] S. Mann, Pencigraphy with AGC: Joint Parameter Estimation in Both Domain and Range of Functions in Same Orbit of the Projective-Wyckoff Group Proc. Int'l Conf. Image Processing, vol. 3, pp. 193-196, 1996.
[29] Melles Griot Catalog, pp. 13.27-13.29, 2000.
[30] A.M. Mika, “Linear-Wedge Spectrometer,” Proc. SPIE Imaging Spectroscopy of the Terrestrial Environment, vol. 1298, pp. 127-131, 1990.
[31] R. Miskelly and C. Seymour, “Capturing Color Information with Linear CCDs: The Case for Trilinear Cameras,” Photonics Spectra, vol. 34, no. 5, pp. 183-184, 2000.
[32] G. Monnet, “3D Spectroscopy with Large Telescopes: Past, Present and Prospects,” Tridimensional Optical Spectroscopic Methods in Astronomy, vol. 71, pp. 12-17, 1995.
[33] M.R. Nagel, H. Quenzel, W. Kweta, and R. Wendling, Daylight Illumination—Color-Contrast Tables, pp. 78-87, Academic Press, 1978.
[34] S. Peleg and J. Herman, “Panoramic Mosaics by Manifold Projection,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 338-343, June 1997.
[35] S.F. Ray, Applied Photographic Optics: Lenses and Optical Systems for Photography, Film, Video and Electronic Imaging, second ed. pp. 559-563, Oxford: Focal Press, 1994.
[36] E.M. Reynoso, G.M. Dubner, W.M. Goss, and E.M. Arnal, “VLA Observations of Neutral Hydrogen in the Direction of Puppis A,” The Astronomical J., vol. 110, pp. 318-324, 1995.
[37] I. Sato, Y. Sato, and K. Ikeuchi, “Illumination Distribution from Brightness in Shadows: Adaptive Estimation of Illumination Distribution with Unknown Reflectance Properties in Shadow Regions,” Proc. Int'l Conf. Computer Vision, pp. 875-883, 1999.
[38] H.S. Sawhney and S. Ayer, "Compact Representations of Videos Through Dominant and Multiple Motion Estimation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 814-831, 1996.
[39] H.S. Sawhney, R. Kumar, G. Gendel, J. Bergen, D. Dixon, and V. Paragano, “VideoBrushTM: Experiences with Consumer Video Mosaicing,” Proc. IEEE Workshop Applications of Computer Vision, pp. 52-62, 1998.
[40] Y.Y. Schechner and S.K. Nayar, “Generalized Mosaicing,” Proc. Int'l Conf. Computer Vision, vol. I, pp. 17-24, 2001.
[41] Y.Y. Schechner and S.K. Nayar, “Generalized Mosaicing: High Dynamic Range in a Wide Field of View,” pending publication.
[42] R.K. Sharma and M. Pavel, “Multisensor Image Registration,” Soc. for Information Display, vol. XXVIII, pp. 951-954, 1997.
[43] H.Y. Shum and R. Szeliski, “Construction of Panoramic Image Mosaics with Global and Local Alignment,” Int'l J. Computer Vision, vol. 36, no. 2, pp. 101-130, Feb. 2000.
[44] D. Slater and G. Healey, “Material Classification for 3D Objects in Aerial Hyperspectral Images,” Proc. Computer Vision and Pattern Recognition, vol. 2, pp. 268-273, 1999.
[45] A. Smajkiewicz, “An Argument for a Filter Array vs. Linear Variable Filter in Precision Analytical Instruments Applications,” Barr Assoc. Inc., Document no. P95081.
[46] L.A. Soderblom, K. Edwards, E.M. Eliason, E.M. Sanchez, and M.P. Charette, “Global Color Variations on the Martian Surface,” Icarus, vol. 34, pp. 446-464, 1978.
[47] H.M.G. Stokman, T. Gevers, and J.J. Koenderink, “Color Measurement by Imaging Spectrometry,” Computer Vision and Image Understanding, vol. 79, no. 2, pp. 236-249, 2000.
[48] X. Sun and J.M. Anderson, “A Spatially Variable Light-Frequency-Selective Component-Based, Airborne Pushbroom Imaging Spectrometer for the Water Environment,” Photogrammetric Eng.&Remote Sensing, vol. 59, no. 3, pp. 399-406, 1993.
[49] Osram Sylvania Corp., “Fluorescent Lamps,” Eng. Bulletin 0-341, Danvers, Mass.
[50] R. Szeliski, “Image Mosaicing for Tele-Reality Applications,” IEEE Computer Graphics and Applications, 1996.
[51] C. Tomasi and T. Kanade, "Shape and Motion From Image Streams Under Orthography: A Factorization Method," Int'l J. Computer Vision, vol. 9, no. 2, pp. 137-154, 1992.
[52] J.M. Uson, S.P. Boughn, and J.R. Kuhn, “The Central Galaxy in Abell 2029: An Old Supergiant,” Science, vol. 250, pp. 539-540, 1990.
[53] A.R. Vasavada, A.P. Ingersoll, D. Banfield, M. Bell, P.J. Gierasch, and M.J.S. Belton, “Galileo Imaging of Jupiter's Atmosphere: The Great Red Spot, Equatorial Region, and White Ovals,” Icarus, vol. 135, pp. 265-275, 1998.
[54] A. Villemaire, S. Fortin, C. Lafond, M.A. Soucy, J.F. Legault, J. Giroux, S.M. Goodrich, R.S. Bauldree, and R.J. Rapp, “A High Resolution Airborne Imaging Spectrometer,” Proc. SPIE Infrared Technology and Applications XXIV, vol. 3436, pp. 924-930, 1996.
[55] J.B. Wellman, “Multispectral Mapper: Imaging Spectroscopy as Applied to the Mapping of Earth Resources,” Proc. SPIE Imaging Spectroscopy, vol. 268, pp. 64-73, 1981.

Index Terms:
Multispectral, hyperspectral imaging, color balance, enhancement, image fusion, physics-based vision, panorama, mosaic, mosaicing, illumination, image-based rendering.
Yoav Y. Schechner, Shree K. Nayar, "Generalized Mosaicing: Wide Field of View Multispectral Imaging," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 10, pp. 1334-1348, Oct. 2002, doi:10.1109/TPAMI.2002.1039205
Usage of this product signifies your acceptance of the Terms of Use.