This Article 
 Bibliographic References 
 Add to: 
Multimodal Data Fusion Based on Mutual Information
Sept. 2012 (vol. 18 no. 9)
pp. 1574-1587
M. Feixas, Univ. of Girona, Girona, Spain
J. Rodriguez, Univ. of Girona, Girona, Spain
A. Bardera, Univ. of Girona, Girona, Spain
I. Boada, Univ. of Girona, Girona, Spain
R. Bramon, Univ. of Girona, Girona, Spain
J. Puig, Hosp. Josep Trueta of Girona, Girona, Spain
M. Sbert, Univ. of Girona, Girona, Spain
Multimodal visualization aims at fusing different data sets so that the resulting combination provides more information and understanding to the user. To achieve this aim, we propose a new information-theoretic approach that automatically selects the most informative voxels from two volume data sets. Our fusion criteria are based on the information channel created between the two input data sets that permit us to quantify the information associated with each intensity value. This specific information is obtained from three different ways of decomposing the mutual information of the channel. In addition, an assessment criterion based on the information content of the fused data set can be used to analyze and modify the initial selection of the voxels by weighting the contribution of each data set to the final result. The proposed approach has been integrated in a general framework that allows for the exploration of volumetric data models and the interactive change of some parameters of the fused data set. The proposed approach has been evaluated on different medical data sets with very promising results.

[1] W. Cai and G. Sakas, "Data Intermixing and Multi-volume Rendering," Computer Graphics Forum, vol. 18, pp. 359-368, 1999.
[2] D.L.G. Hill, P.G. Batchelor, M. Holden, and D.J. Hawkes, "Medical Image Registration," Physics in Medicine and Biology, vol. 46, pp. R1-R45, 2001.
[3] P.A. Viola, "Alignment by Maximization of Mutual Information," PhD dissertation, MIT Artificial Intelligence Laboratory (TR 1548), 1995.
[4] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens, "Multimodality Image Registration by Maximization of Mutual Information," IEEE Trans. Medical Imaging, vol. 16, no. 2, pp. 187-198, Apr. 1997.
[5] J.P. Pluim, J. Maintz, and M. Viergever, "Mutual-information-Based Registration of Medical Images: A Survey," IEEE Trans. Medical Imaging, vol. 22, no. 8, pp. 986-1004, July 2003.
[6] B. Wilson, E.B. Lum, and K.-L. Ma, "Interactive Multi-volume Visualization," Proc. Int'l Conf. Computational Science-Part II (ICCS '02), pp. 102-110, 2002.
[7] R. Fuchs and H. Hauser, "Visualization of Multivariate Scientific Data," Computer Graphics Forum, vol. 28, no. 6, pp. 1670-1690, 2009.
[8] S. Gupta, K. Ramesh, and E. Blasch, "Mutual Information Metric Evaluation for PET/MRI Image Fusion," Proc. IEEE Nat'l Aerospace and Electronics Conf., pp. 305-311, 2008.
[9] E. Gulsoy, J. Simmons, and M.D. Graef, "Application of Joint Histogram and Mutual Information to Registration and Data Fusion Problems in Serial Sectioning Microstructure Studies," Scripta Materialia, vol. 60, no. 6, pp. 381-384, 2009.
[10] L. Leung, B. King, and V. Vohora, "Comparison of Image Data Fusion Techniques Using Entropy and INI," Proc. Asian Conf. Remote Sensing, vol. 1, pp. 152-157, 2001.
[11] A.A. Goshtasby, "Fusion of Multi-exposure Images," Image and Vision Computing, vol. 23, pp. 611-618, 2005.
[12] P.P. Vázquez, M. Feixas, M. Sbert, and W. Heidrich, "Viewpoint Selection Using Viewpoint Entropy," Proc. Vision, Modeling, and Visualization, pp. 273-280, 2001.
[13] U.D. Bordoloi and H.-W. Shen, "Viewpoint Evaluation for Volume Rendering," Proc. IEEE Visualization, pp. 487-494, 2005.
[14] S. Takahashi, I. Fujishiro, Y. Takeshima, and T. Nishita, "A Feature-Driven Approach to Locating Optimal Viewpoints for Volume Visualization," Proc. IEEE Visualization, pp. 495-502, 2005.
[15] M. Feixas, M. Sbert, and F. González, "A Unified Information-Theoretic Framework for Viewpoint Selection and Mesh Saliency," ACM Trans. Applied Perception, vol. 6, no. 1, pp. 1-23, 2009.
[16] I. Viola, M. Feixas, M. Sbert, and M.E. Gröller, "Importance-Driven Focus of Attention," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 5, pp. 933-940, Sept. 2006.
[17] M. Ruiz, I. Boada, M. Feixas, and M. Sbert, "Viewpoint Information Channel for Illustrative Volume Rendering," Computers & Graphics, vol. 34, no. 4 pp. 351-360, 2010.
[18] C. Wang, H. Yu, and K.-L. Ma, "Importance-Driven Time-Varying Data Visualization," IEEE Trans. Visualization and Computer Graphics, vol. 14, no. 6, pp. 1547-1554, Nov./Dec. 2008.
[19] L. Xu, T.-Y. Lee, and H.-W. Shen, "An Information-theoretic Framework for Flow Visualization," IEEE Trans. Visualization and Computer Graphics, vol. 16, no. 6, pp. 1216-1224, Nov./Dec. 2010.
[20] S. Bruckner and T. Möller, "Isosurface Similarity Maps," Computer Graphics Forum, vol. 29, no. 3, pp. 773-782, 2010.
[21] M. Haidacher, S. Bruckner, A. Kanitsar, and M.E. Gröller, "Information-Based Transfer Functions for Multimodal Visualization," Proc. Visual Computing for Biology and Medicine, pp. 101-108, 2008.
[22] J. Beyer, M. Hadwiger, S. Wolfsberger, and K. Buhle, "High-Quality Multimodal Volume Rendering for Preoperative Planning of Neurosurgical Interventions," IEEE Trans. Visualization and Computer Graphics, vol. 3, no. 6, pp. 1696-1703, Nov./Dec. 2007.
[23] D. Levin, X. Hu, K. Tan, S. Galhotra, C. Pelizzari, G. Chen, R. Beck, C. Chen, M. Cooper, and J. Mullan, "The Brain: Integrated Three-Dimensional Display of MR and PET Images," Radiology, vol. 172, pp. 783-789, 1989.
[24] J. Jacq and C. Roux, "A Direct Multi-Volume Rendering Method Aiming at Comparisons of 3D Images and Models," IEEE Trans. Information Technology in Biomedicine, vol. 1, no. 1, pp. 30-43, Mar. 1997.
[25] M. Levoy, "Display of Surfaces from Volume Data," IEEE Computer Graphics Applications, vol. 8, no. 3, pp. 29-37, May 1988.
[26] G. Kindlmann and J.W. Durkin, "Semi-Automatic Generation of Transfer Functions for Direct Volume Rendering," Proc. IEEE Symp. Volume Visualization, pp. 79-86, 1998.
[27] J. Kniss, S. Premoze, C. Hansen, and D. Ebert, "Interactive Translucent Volume Rendering and Procedural Modeling," Proc. IEEE Visualization, pp. 109-116, 2002.
[28] J. Kniss, S. Premoze, M. Ikits, A. Lefohn, C. Hansen, and E. Praun, "Gaussian Transfer Functions for Multi-field Volume Visualization," Proc. IEEE Visualization, pp. 497-504, 2003.
[29] M. Tory, S. Potts, and T. Möller, "A Parallel Coordinates Style Interface for Exploratory Volume Visualization," IEEE Trans. Visualization and Computer Graphics, vol. 11, no. 1, pp. 71-80, Jan./Feb. 2005.
[30] A. Ghosh, P. Prabhu, A.E. Kaufman, and K. Mueller, "Hardware Assisted Multichannel Volume Rendering," Proc. Int'l Computer Graphics, pp. 2-7, 2003.
[31] F. Robler, E. Tejada, T. Fangmeier, T. Ertl, and M. Knauff, "GPU-Based Multi-Volume Rendering for the Visualization of Functional Brain Images," Proc. SimVis, pp. 305-318, 2006.
[32] R. Brecheisen, A. Vilanova, B. Platel, and B.M. ter Haar Romeny, "Flexible GPU-Based Multi-Volume Ray-casting," Proc. Vision, Modelling and Visualization, pp. 303-312, 2008.
[33] J. Plate, T. Holtkaemper, and B. Froehlich, "A Flexible Multi-Volume Shader Framework for Arbitrarily Intersecting Multi-Resolution Datasets," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1584-1591, Nov. 2007.
[34] V. Tsagaris and V. Anastassopoulos, "An Information Measure for Assessing Pixel-Level Fusion Methods," Proc. SPIE Image and Signal Processing for Remote Sensing, vol. 5573, pp. 64-71, 2004.
[35] T.M. Cover and J.A. Thomas, Elements of Information Theory. Wiley Series in Telecomm, John Wiley and Sons, 1991.
[36] M.R. DeWeese and M. Meister, "How to Measure the Information Gained from One Symbol," Network: Computation in Neural Systems, vol. 10, no. 4, pp. 325-340, 1999.
[37] M. Bezzi, "Quantifying the Information Transmitted in a Single Stimulus," Biosystems, vol. 89, nos. 1-3, pp. 4-9, 2007.
[38] D.A. Butts, "How Much Information is Associated with a Particular Stimulus?" Network: Computation in Neural Systems, vol. 14, pp. 177-187, 2003.
[39] Nat'l Inst. of Health, Retrospective Image Registration Evaluation, Vanderbilt Univ., Nashville (TN), USA, Project Number 8R01EB002124-03, Principal Investigator J. Michael Fitzpatrick, http://www.insight-journal.orgrire/, 2003.

Index Terms:
sensor fusion,content management,data visualisation,medical computing,medical data sets,mutual information-based multimodal data fusion,multimodal data visualization,information-theoretic approach,informative voxels,information channel,intensity value,mutual information decomposition,assessment criterion,information content,volumetric data models exploration,Transfer functions,Data visualization,Rendering (computer graphics),Mutual information,Biomedical imaging,Image color analysis,Data models,mutual information.,Multimodal visualization,image fusion,information theory
M. Feixas, J. Rodriguez, A. Bardera, I. Boada, R. Bramon, J. Puig, M. Sbert, "Multimodal Data Fusion Based on Mutual Information," IEEE Transactions on Visualization and Computer Graphics, vol. 18, no. 9, pp. 1574-1587, Sept. 2012, doi:10.1109/TVCG.2011.280
Usage of this product signifies your acceptance of the Terms of Use.