This Article 
 Bibliographic References 
 Add to: 
A User Study to Compare Four Uncertainty Visualization Methods for 1D and 2D Datasets
November/December 2009 (vol. 15 no. 6)
pp. 1209-1218
Jibonananda Sanyal, Mississippi State University
Song Zhang, Mississippi State University
Gargi Bhattacharya, Southern Illinois University
Phil Amburn, Mississippi State University
Robert Moorhead, Mississippi State University
Many techniques have been proposed to show uncertainty in data visualizations. However, very little is known about their effectiveness in conveying meaningful information. In this paper, we present a user study that evaluates the perception of uncertainty amongst four of the most commonly used techniques for visualizing uncertainty in one-dimensional and two-dimensional data. The techniques evaluated are traditional errorbars, scaled size of glyphs, color-mapping on glyphs, and color-mapping of uncertainty on the data surface. The study uses generated data that was designed to represent the systematic and random uncertainty components. Twenty-seven users performed two types of search tasks and two types of counting tasks on 1D and 2D datasets. The search tasks involved finding data points that were least or most uncertain. The counting tasks involved counting data features or uncertainty features. A 4x4 full-factorial ANOVA indicated a significant interaction between the techniques used and the type of tasks assigned for both datasets indicating that differences in performance between the four techniques depended on the type of task performed. Several one-way ANOVAs were computed to explore the simple main effects. Bonferronni’s correction was used to control for the family-wise error rate for alpha-inflation. Although we did not find a consistent order among the four techniques for all the tasks, there are several findings from the study that we think are useful for uncertainty visualization design. We found a significant difference in user performance between searching for locations of high and searching for locations of low uncertainty. Errorbars consistently underperformed throughout the experiment. Scaling the size of glyphs and color-mapping of the surface performed reasonably well. The efficiency of most of these techniques were highly dependent on the tasks performed. We believe that these findings can be used in future uncertainty visualization design. In addition, the framework developed in this user study presents a structured approach to evaluate uncertainty visualization techniques, as well as provides a basis for future research in uncertainty visualization.

[1] J. Bertin, Semiology of graphics: Diagrams, networks, maps. (Translation from a French 1967 edition by W. Berg, editor). Madison, WI: University of Wisconsin Press. 1983.
[2] S. Blenkinsop, P. Fisher, L. Bastin, and J. Wood., Evaluating the Perception of Uncertainty in Alternative Visualization Strategies. Cartographica 37 (1): 1-13. 2000.
[3] B.P. Buttenfield, Representing data quality. Cartographica (special content, Mapping Data Quality) 30 (2&3): 1-7. 1993.
[4] A. Cedilnik and P. Rheingans, Procedural annotation of uncertain information. In Proc., Visualization 2000, Salt Lake City, 77-84, 2000.
[5] CIPM, BIPM. In Proc., Verb. Com. Int. Poids et Mesures (in French). 1982.
[6] H. Couclelis, The certainty of uncertainty: GIS and the limits of geographic knowledge. In Trans. in GIS 7 (2): 165-75, 2003.
[7] G. Cumming, F. Fidler, and D.L. Vaux, Error bars in experimental biology. Journal of Cell Biology, 177: 7-11. 2007.
[8] T.J. Davis and C.P. Keller, Modeling and visualizing multiple spatial uncertainties. Computers & Geosciences 23 (4): 397-408. 1997.
[9] N. Gershon, Short note: Visualization of an imperfect world. In IEEE Computer Graphics and Applications, 18: 43–45, Jul.-Aug. 1998.
[10] M. Harrower, Representing uncertainty: Does it help people make better decisions? In UCGIS Workshop: Geospatial Visualization and Knowledge Discovery Workshop, National Conference Center, Landsdowne, VA., 2003, (invited white paper).
[11] T. Hengl, Visualisation of uncertainty using the HIS colour model: Computations with colours. In Proc., 7th International Conference on GeoComputation, Southampton, United Kingdom. 8-17. 2003.
[12] T. Hengl and N. Toomanian, Maps are not what they seem: representing uncertainty in soil property maps. In Proc., 7th International Symposium on Spatial Accuracy Assessment in Natural Resources and Environmental Sciences, 805-813. 2006.
[13] S. Huang, Exploratory Visualization of Data with Variable Quality. Masters Thesis, Worcester Polytechnic Institute, Jan. 2005.
[14] G.J. Hunter and M.F. Goodchild, Managing uncertainty in spatial databases: Putting theory into practice. In Proc., URISA, Atlanta, July 25–29. pages 1–14. 1993.
[15] B. Jiang, F. Ormeling, and W. Kainz, Visualization support for fuzzy spatial analysis. In Proc,. ACSM/ASPRS Conference, Charlotte, North Carolina. 291-300, 1995.
[16] C.R. Johnson ad, A.R. Sanderson, A Next Step: Visualizing Errors and Uncertainty. Visualization Viewpoints, Sep.-Oct. 2003.
[17] D. Kobus, S. Proctor, and S. Holste, Effects of experience and uncertainty during dynamic decision making. International Journal of Industrial Ergonomics. 28: 275-90. 2001.
[18] D.H. Laidlaw, R.M. Kirby, C.D. Jackson, J.S. Davidson, T.S. Miller, M. da Silva, W.H. Warren, and M.J. Tarr In , IEEE Trans. Visualization and Computer Graphics, 11 (1), Jan/Feb 2005.
[19] M. Leitner and B.P. Buttenfield, Guidelines for the display of attribute certainty. Cartography and Geographic Information Science. 27 (1): 3–14. 2000.
[20] H. Levkowitz and G.T. Herman, Color scales for image data. In IEEE Computer Graphics and Applications, 12 (1), pages 72-80, Jan 1992.
[21] H. Li, C.W. Fu, Y. Li, and A. Hanson, Visualizing Large-Scale Uncertainty in Astrophysical Data. In IEEE Trans. on Visualization and Computer Graphics, 13 (6): 1640-1647, Nov.-Dec. 2007.
[22] S.K. Lodha, C.M. Wilson, and R.E. Sheehan, LISTEN: Sounding uncertainty visualization. In Proc., IEEE Visualization '96, San Francisco, California. 189-95. 1996.
[23] K. Lowell, Outside-in, inside-out: Two methods of generating spatial certainty maps. In Proc., Second Annual Conference of GeoComputation, University of Otago, Dunedin, New Zealand, 15-25. 1997.
[24] C. Lundstrom, P. Ljung, A. Persson, and A. Ynnerman, Uncertainty Visualization in Medical Volume Rendering Using Probabilistic Animation. In IEEE Trans. Visualization and Computer Graphics, 13 (6): 1648-1655, Nov.-Dec. 2007.
[25] A.M. MacEachren, Visualizing Uncertain Information. Cartographic Perspective, 13: 10-19, 1992.
[26] A.M. MacEachren, C.A. Brewer, and L.W. Pickle, Visualizing georeferenced data: Representing reliability of health statistics. Environment and Planning: A30:1547-61. 1998.
[27] J.P. Martin, J.E. Swan II, R.J. Moorhead II, Z. Liu, and S. Cai, Results of a User Study on 2D Hurricane Visualization. Computer Graphics Forum (Special Issue on EuroGraphics / IEEE VGTC EuroVis'08, formerly VisSym). 27 (3): 991-998. 2008.
[28] Matlab, The Mathworks Inc., R2008a, Natick, Massachusetts, USA.
[29] S.E. Maxwell and H.D. Delaney, Designing experiments and analyzing data: A model comparison perspective. Lawrence Erlbaum Associates Inc. Publishers, New Jersey. 2004.
[30] C. Olston and J.D. Mackinlay, Visualizing Data with Bounded Uncertainty. In Proc., IEEE Symposium on Information Visualization 2002 (InfoVis'02). 37-40. 2002.
[31] A.T. Pang, C.M. Wittenbrink, and S.K. Lodha, Approaches to uncertainty visualization, The Visual Computer, 13 (8): 370–390, 1997.
[32] P. Rheingans and S. Joshi, Visualization of Molecules with Positional Uncertainty. In Proc., Data Visualization '99. 299-306. 1999.
[33] G.S. Schmidt, S.L. Chen, A.N. Bryden, M.A. Livingston, L.J. Rosenblum, and B.R. Osborn, Multidimensional Visual Representations for Underwater Environmental Uncertainty. IEEE Computer Graphics and Applications 24 (5): 56-65. 2004.
[34] D.M. Schweizer and M.F. Goodchild, Data quality and choropleth maps: An experiment with the use of color. In Proc., GIS/LIS '92, San Jose, California. ACSM and ASPRS, Washington, D.C., 686-99. 1992.
[35] T. Strothotte, M. Masuch, and T. Isenberg, Visualizing Knowledge about Virtual Reconstructions of Ancient Architecture. In Proc., International Conference on Computer Graphics, IEEE Computer Society. 36–43. 1999.
[36] B.N. Taylor and C.E. Kuya, Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results. NIST Technical Note 1297, Physics Laboratory, National Institute of Standards and Technology, Gaithersburg, MD, 1994.
[37] J. Thomson, B. Hetzler, A. MacEachren, M. Gahegan, and Misha Pavel, A Typology for Visualizing Uncertainty. In Proc., Conference on Visualization and Data Analysis 2005 (part of the IS&T/SPIE Symposium on Electronic Imaging 2005). San Jose, California. Jan. pages 16-20. 2005.
[38] M. Tory and T. Möller, Rethinking Visualization: a High-Level Taxonomy. In Proc., IEEE Symposium on Information Visualization 2004 (Infovis '04). 151-158. 2004.
[39] E.R. Tufte, The Visual Display of Quantitative Information. Graphics Press. 1986.
[40] C. Ware, Information Visualization: Perception for Design. Morgan Kaufmann Publishers, 2000.
[41] S.P. Wechsler, Perceptions of Digital Elevation Model Uncertainty by DEM Users. URISA Journal, 15: 57-64. 2003.
[42] T. Zuk and M. S.T. Carpendale, Theoretical Analysis of Uncertainty Visualizations. In Proc., SPIE & IS&T Conf. Electronic Imaging, Vol. 6060: Visualization and Data Analysis 2006, 606007. 2006.

Index Terms:
User study, uncertainty visualization
Jibonananda Sanyal, Song Zhang, Gargi Bhattacharya, Phil Amburn, Robert Moorhead, "A User Study to Compare Four Uncertainty Visualization Methods for 1D and 2D Datasets," IEEE Transactions on Visualization and Computer Graphics, vol. 15, no. 6, pp. 1209-1218, Nov.-Dec. 2009, doi:10.1109/TVCG.2009.114
Usage of this product signifies your acceptance of the Terms of Use.