This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Empirical Studies in Information Visualization: Seven Scenarios
Sept. 2012 (vol. 18 no. 9)
pp. 1520-1536
H. Lam, Google, Inc., Mountain View, CA, USA
E. Bertini, Dept. of Comput. & Inf. Sci., Univ. of Konstanz, Konstanz, Germany
P. Isenberg, INRIA, Univ. Paris-Sud, Orsay, France
C. Plaisant, Univ. of Maryland, College Park, MD, USA
S. Carpendale, Dept. of Comput. Sci., Univ. of Calgary, Calgary, AB, Canada
We take a new, scenario-based look at evaluation in information visualization. Our seven scenarios, evaluating visual data analysis and reasoning, evaluating user performance, evaluating user experience, evaluating environments and work practices, evaluating communication through visualization, evaluating visualization algorithms, and evaluating collaborative data analysis were derived through an extensive literature review of over 800 visualization publications. These scenarios distinguish different study goals and types of research questions and are illustrated through example studies. Through this broad survey and the distillation of these scenarios, we make two contributions. One, we encapsulate the current practices in the information visualization research community and, two, we provide a different approach to reaching decisions about what might be the most effective evaluation of a given information visualization. Scenarios can be used to choose appropriate research questions and goals and the provided examples can be consulted for guidance on how to design one's own study.

[1] D.L. Alonso, A. Rose, C. Plaisant, and K.L. Norman, "Viewing Personal History Records: A Comparison of Tabular Format and Graphical Presentation Using Lifelines," Behaviour & Information Technology, vol. 17, no. 5, pp. 249-262, 1998.
[2] K. Andrews, "Evaluation Comes in Many Guises," Proc. CHI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), pp. 7-8, 2008.
[3] A.O. Artero, M.C.F. de Oliveira, and H. Levkowitz, "Uncovering Clusters in Crowded Parallel Coordinates Visualizations," Proc. IEEE Symp. Information Visualization (InfoVis), pp. 81-88, 2004.
[4] K. Baker, S. Greenberg, and C. Gutwin, "Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration," Proc. Int'l Conf. Eng. for Human-Computer Interaction, pp. 123-139, 2001.
[5] L. Barkhuus and J.A. Rode, "From Mice to Men: 24 Years of Evaluation in CHI," Proc. alt.chi: Extended Abstracts of the Conf. Human Factors in Computing Systems (CHI), 2007.
[6] L. Bartram and C. Ware, "Filtering and Brushing with Motion," Information Visualization, vol. 1, no. 1, pp. 66-79, 2002.
[7] B.B. Bederson, B. Shneiderman, and M. Wattenberg, "Ordered and Quantum Treemaps: Making Effective Use of 2D Space to Display Hierarchies," ACM Trans. Graphics, vol. 21, no. 4, pp. 833-854, 2002.
[8] E. Bertini, A. Perer, C. Plaisant, and G. Santucci, "Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (beliv)," Proc. Extended Abstracts of the Conf. Human Factors in Computing Systems (CHI), pp. 3913-3916, 2008.
[9] I. Brewer, A.M. MacEachren, H. Abdo, J. Gundrum, and G. Otto, "Collaborative Geographic Visualization: Enabling Shared Understanding of Environmental Processes," Proc. IEEE Symp. Information Visualization (InfoVis), pp. 137-141, 2000.
[10] S. Carpendale, "Evaluating Information Visualizations," Information Visualization: Human-Centered Issues and Perspectives, A. Kerren, J.T. Stasko, J.-D. Fekete, and C. North, eds., pp. 19-45, Springer, 2007.
[11] C. Chen and Y. Yu, "Empirical Studies of Information Visualization: A Meta-Analysis," Int'l J. Human-Computer Studies, vol. 53, no. 5, pp. 851-866, 2000.
[12] J. Chen, A.M. MacEachren, and D.J. Peuquet, "Constructing Overview + Detail Dendrogram-Matrix Views," IEEE Trans. Visualization and Computer Graphics, vol. 15, no. 6, pp. 889-896, Nov. 2009.
[13] Y. Chen, L. Wang, M. Dong, and J. Hua, "Exemplar-Based Visualization of Large Document Corpus," IEEE Trans. Visualization and Computer Graphics, vol. 15, no. 6, pp. 1161-1168, Nov./Dec. 2009.
[14] R.K. Christopher, C.G. Healey, V. Interrante, D.H. Laidlaw, and C. Ware, "Thoughts on User Studies: Why, How, and When," IEEE Computer Graphics and Applications, vol. 23, no. 4, pp. 20-25, July 2003.
[15] J. Corbin and A. Strauss, Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, 2008.
[16] L. Costello, G. Grinstein, C. Plaisant, and J. Scholtz, "Advancing User-Centered Evaluation of Visual Analytic Environments through Contests," Information Visualization, vol. 8, pp. 230-238, 2009.
[17] J.W. Creswell, Research Design, Qualitative, Quantitative, and Mixed Methods Approaches, second ed. Sage Publications, Inc., 2002.
[18] M.P. Czerwinski, M.V. Dantzich, G. Robertson, and H. Hoffman, "The Contribution of Thumbnail Image, Mouse-over Text and Spatial Location Memory to Web Page Retrieval in 3D," Proc. Int'l Conf. INTERACT, pp. 163-170, 1999.
[19] J. Drury and M.G. Williams, "A Framework for Role-Based Specification and Evaluation of Awareness Support in Synchronous Collaborative Applications," Proc. Workshops Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), pp. 12-17, 2002.
[20] T. Dwyer and D.R. Gallagher, "Visualising Changes in Fund Manager Holdings in Two and a Half-Dimensions," Information Visualization, vol. 3, no. 4, pp. 227-244, 2004.
[21] R. Eccles, T. Kapler, R. Harper, and W. Wright, "Stories in Geotime," Information Visualization, vol. 7, no. 1, pp. 3-17, 2008.
[22] G. Ellis and A. Dix, "An Exploratory Analysis of User Evaluation Studies in Information Visualization," Proc. AVI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), 2006.
[23] L. Grammel, M. Tory, and M.-A. Storey, "How Information Visualization Novices Construct Visualizations," IEEE Trans. Visualization and Computer Graphics, vol. 16, no. 6, pp. 943-952, Nov./Dec. 2010.
[24] S. Greenberg and B. Buxton, "Usability Evaluation Considered Harmful (Some of the Time)," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 217-224, 2008.
[25] S. Greenberg, G. Fitzpatrick, C. Gutwin, and S. Kaplan, "Adapting the Locales Framework for Heuristic Evaluation of Groupware," Australian J. Information Systems, vol. 7, no. 2, pp. 102-108, 2000.
[26] S. Greenberg, "Observing Collaboration: Group-Centered Design," HCI Remixed: Reflections on Works That Have Influenced the HCI Community, T. Erickson, and D.W. McDonald, eds., chapter 18, pp. 111-118, MIT Press, 2008.
[27] J. Grudin, "Why Cscw Applications Fail: Problems in the Design and Evaluation of Organizational Interfaces," Proc. Conf. Computer-Supported Cooperative Work (CSCW), pp. 85-93, 1988.
[28] C. Gutwin and S. Greenberg, "The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces," Proc. Workshops Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), pp. 98-103, 2000.
[29] M. Hao, D.A. Keim, U. Dayal, D. Oelke, and C. Tremblay, "Density Displays for Data Stream Monitoring," Computer Graphics Forum, vol. 27, no. 3, pp. 895-902, 2008.
[30] J. Heer and M. Bostock, "Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 203-212, 2010.
[31] J. Heer and D. Boyd, "Vizster: Visualizing Online Social Networks," Proc. Symp. Information Visualization (InfoVis), pp. 33-40, 2005.
[32] J. Heer and G. Robertson, "Animated Transitions in Statistical Data Graphics," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1240-1247, Nov./Dec. 2007.
[33] J. Heer, F.B. Viégas, and M. Wattenberg, "Voyagers and Voyeurs: Supporting Asynchronous Collaborative Information Visualization," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 1029-1038, 2007.
[34] D.M. Hilbert and D.F. Redmiles, "Extracting Usability Information from User Interface Events," ACM Computing Survey, vol. 32, no. 4, pp. 384-421, 2000.
[35] U. Hinrichs, H. Schmidt, and S. Carpendale, "EMDialog: Bringing Information Visualization into the Museum," IEEE Trans. Visualization and Computer Graphics, vol. 14, no. 6, pp. 1181-1188, Nov./Dec. 2008.
[36] K. Holtzblatt and S. Jones, Contextual Inquiry: A Participatory Technique for Systems Design. Lawrence Earlbaum, 1993.
[37] K. Hornbæk and E. Frokjær, "Reading of Electronic Documents: The Usability of Linear, Fisheye and Overview+Detail Interfaces," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 293-300, 2001.
[38] P. Isenberg, A. Bezerianos, N. Henry, S. Carpendale, and J.-D. Fekete, "CoCoNutTrix: Collaborative Retrofitting for Information Visualization," Computer Graphics and Applications: Special Issue on Collaborative Visualization, vol. 29, no. 5, pp. 44-57, 2009.
[39] P. Isenberg, D. Fisher, M. Ringel Morris, K. Inkpen, and M. Czerwinski, "An Exploratory Study of Co-Located Collaborative Visual Analytics around a Tabletop Display," Proc. Conf. Visual Analytics Science and Technology (VAST), pp. 179-186, 2010.
[40] P. Isenberg, A. Tang, and S. Carpendale, "An Exploratory Study of Visual Information Analysis," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 1217-1226, 2008.
[41] P. Isenberg, T. Zuk, C. Collins, and S. Carpendale, "Grounded Evaluation of Information Visualizations," Proc. CHI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), pp. 56-63, 2008.
[42] M.Y. Ivory and M.A. Hearst, "The State of the Art in Automating Usability Evaluation of User Interfaces," ACM Computing Surveys, vol. 33, no. 4, pp. 470-516, 2001.
[43] F. Janoos, S. Singh, O. Irfanoglu, R. Machiraju, and R. Parent, "Activity Analysis Using Spatio-Temporal Trajectory Volumes in Surveillance Applications," Proc. Symp. Visual Analytics Science and Technology (VAST), pp. 3-10, 2007.
[44] Y.-A. Kang, C. Görg, and J. Stasko, "Evaluating Visual Analytics Systems for Investigative Analysis: Deriving Design Principles from a Case Study," Proc. Symp. Visual Analytics Science and Technology (VAST), pp. 139-146, 2009.
[45] A. Kittur, E.H. Chi, and B. Suh, "Crowdsourcing User Studies with Mechanical Turk," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 453-456, 2008.
[46] O. Kulyk, R. Kosara, J. Urquiza, and I. Wassin, "Human-Centered Aspects," Human-Centered Visualization Environments, G. Goos, J. Hartmanis, and J. van Leeuwen, eds., pp. 13-76, Springer, 2007.
[47] H. Lam and T. Munzner, "Increasing the Utility of Quantitative Empirical Studies for Meta-Analysis," Proc. CHI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), pp. 21-27, 2008.
[48] H. Lam, R.A. Rensink, and T. Munzner, "Effects of 2D Geometric Transformations on Visual Memory," Proc. Symp. Applied Perception in Graphics and Visualization (APGV), pp. 119-126, 2006.
[49] J.D. Mackinlay, P. Hanrahan, and C. Stolte, "Show Me: Automatic Presentation for Visual Analysis," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1137-1144, Nov./Dec. 2007.
[50] N. Mahyar, A. Sarvghad, and M. Tory, "A Closer Look at Note Taking in the Co-Located Collaborative Visual Analytics Process," Proc. Conf. Visual Analytics Science and Technology (VAST), pp. 171-178, 2010.
[51] R. Mandryk, "Modeling User Emotion in Interactive Play Environments: A Fuzzy Physiological Approach," PhD thesis, Simon Fraser Univ., 2005.
[52] G. Mark and A. Kobsa, "The Effects of Collaboration and System Transparency on CIVE Usage: An Empirical Study and Model," Presence, vol. 14, no. 1, pp. 60-80, 2005.
[53] J. McGrath, "Methodology Matters: Doing Research in the Behavioral and Social Sciences," Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, 1994.
[54] T. Munzner, "A Nested Process Model for Visualization Design and Validation," IEEE Trans. Visualization and Computer Graphics, vol. 15, no. 6, pp. 921-928, Nov./Dec. 2009.
[55] D.C. Neale, J.M. Carroll, and M.B. Rosson, "Evaluating Computer-Supported Cooperative Work: Models and Frameworks," Proc. Conf. Computer-Supported Cooperative Work (CSCW), pp. 112-121, 2004.
[56] W. Peng, M.O. Ward, and E.A. Rundensteiner, "Clutter Reduction in Multi-Dimensional Data Visualization Using Dimension Reordering," Proc. Symp. Information Visualization (InfoVis), pp. 89-96, 2004.
[57] A. Perer and B. Shneiderman, "Integrating Statistics and Visualization for Exploratory Power: From Long-Term Case Studies to Design Guidelines," IEEE Computer Graphics and Applications, vol. 29, no. 3, pp. 39-51, May/June 2009.
[58] D. Pinelle and C. Gutwin, "A Review of Groupware Evaluations," Proc. Workshops Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), pp. 86-91, 2000.
[59] D. Pinelle and C. Gutwin, "Groupware Walkthrough: Adding Context to Groupware Usability Evaluation," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 455-462, 2002.
[60] D. Pinelle, C. Gutwin, and S. Greenberg, "Task Analysis for Groupware Usability Evaluation: Modeling Shared-Workspace Tasks with the Mechanics of Collaboration," ACM Trans. Human Computer Interaction, vol. 10, no. 4, pp. 281-311, 2003.
[61] P. Pirolli and S. Card, "The Sensemaking Process and Leverage Points for Analyst Technology as Identified through Cognitive Task Analysis," Proc. Int'l Conf. Intelligence Analysis, 2005.
[62] C. Plaisant, J. Grosjean, and B. Bederson, "Spacetree: Supporting Exploration in Large Node Link Tree, Design Evolution and Empirical Evaluation," Proc. Symp. Information Visualization (InfoVis), pp. 57-64, 2007.
[63] C. Plaisant, "The Challenge of Information Visualization Evaluation," Proc. Working Conf. Advanced Visual Interfaces (AVI), pp. 109-116, 2004.
[64] Z. Pousman, J. Stasko, and M. Mateas, "Casual Information Visualization: Depictions of Data in Everyday Life," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1145-1152, Nov./Dec. 2007.
[65] A.J. Pretorius and J.J. van Wijk, "Visual Inspection of Multivariate Graphs," Computer Graphics Forum, vol. 27, no. 3, pp. 967-974, 2008.
[66] G. Robertson, M. Czerwinski, K. Larson, D.C. Robbins, D. Thiel, and M. van Dantzich, "Data Mountain: Using Spatial Memory for Document Management," Proc. Symp. User Interface Software and Technology, pp. 153-162, 1998.
[67] A. Robinson, "Collaborative Synthesis of Visual Analytic Results," Proc. Symp. Visual Analytics Science and Technology (VAST), pp. 67-74, 2008.
[68] J. Sajaniemi and M. Kuittinen, "Visualizing Roles of Variables in Program Animation," Information Visualization, vol. 3, no. 3, pp. 137-153, 2004.
[69] P. Saraiya, C. North, and K. Duca, "An Evaluation of Microarray Visualization Tools for Biological Insight," Proc. Symp. Information Visualization (InfoVis), pp. 1-8, 2004.
[70] P. Saraiya, C. North, V. Lam, and K.A. Duca, "An Insight-Based Longitudinal Study of Visual Analytics," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 6, pp. 1511-1522, Nov./Dec. 2006.
[71] K. Sedig, S. Rowhani, J. Morey, and H.-N. Liang, "Application of Information Visualization Techniques to the Design of a Mathematical Mindtool: A Usability Study," Information Visualization, vol. 2, no. 3, pp. 142-159, 2003.
[72] M. Sedlmair, D. Baur, S. Boring, P. Isenberg, M. Jurmu, and A. Butz, "Requirements for a Mde System to Support Collaborative In-Car Communication Diagnostics," Proc. CSCW Workshop Beyond the Laboratory: Supporting Authentic Collaboration with Multiple Displays, 2008.
[73] M. Sedlmair, P. Isenberg, D. Baur, and A. Butz, "Information Visualization Evaluation in Large Companies: Challenges, Experiences and Recommendations," Information Visualization J., vol. 10, no. 3, pp. 248-266, 2011.
[74] J. Seo and B. Shneiderman, "Knowledge Discovery in High-Dimensional Data: Case Studies and a User Survey for the Rank-by-Feature Framework," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 3, pp. 311-322, May/June 2006.
[75] W. Shadish, T. Cook, and D. Campbell, Experimental and Quasi-Experimental Designs. Houghton Mifflin Company, 2002.
[76] B. Shneiderman and C. Plaisant, "Strategies for Evaluating Information Visualization Tools: Multi-Dimensional In-Depth Long-Term Case Studies," Proc. AVI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), 2006.
[77] B. Shneiderman and C. Plaisant, Designing the User Interface: Strategies for Effective Human-Computer Interaction, fifth ed. Addison-Wesley, 2009.
[78] T. Skog, S. Ljungblad, and L.E. Holmquist, "Between Aesthetics and Utility: Designing Ambient Information Visualizations," Proc. Symp. Information Visualization (InfoVis), pp. 233-240, 2003.
[79] H. Song, E. Curran, and R. Sterritt, "Multiple Foci Visualisation of Large Hierarchies with Flextree," Information Visualization, vol. 3, no. 1, pp. 19-35, 2004.
[80] G. Stahl, Group Cognition. MIT Press, 2006.
[81] G. Symon, Qualitative Research Diaries, pp. 94-117. Sage Publications, 1998.
[82] Illuminating the Path: The Research and Development Agenda for Visual Analytics, J. Thomas, and K.A. Cook, eds. IEEE CS Press, 2005.
[83] M. Tory and T. Möller, "Evaluating Visualizations: Do Expert Reviews Work?" Computer Graphics and Applications, vol. 25, no. 5, pp. 8-11, 2005.
[84] M. Tory, D.W. Sprague, F. Wu, W.Y. So, and T. Munzner, "Spatialization Design: Comparing Points and Landscapes," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1262-1285, Nov./Dec. 2007.
[85] M. Tory and S. Staub-French, "Qualitative Analysis of Visualization: A Building Design Field Study," Proc. CHI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), 2008.
[86] J.G. Trafton, S.S. Kirschenbaum, T.L. Tsui, R.T. Miyamoto, J.A. Ballas, and P.D. Raymond, "Turning Pictures into Numbers: Extracting and Generating Information from Complex Visualizations," Int'l J. Human-Computer Studies, vol. 53, no. 5, pp. 827-850, 2000.
[87] M. Twidale, D. Randall, and R. Bentley, "Situated Evaluation for Cooperative Systems," Proc. Conf. Computer-Supported Cooperative Work (CSCW), pp. 441-452, 1994.
[88] Usability.net. Usability.net Methods, http://www.usabilitynet. org/toolsmethods.htm , 2009.
[89] F. Viégas, E. Perry, E. Howe, and J. Donath, "Artifacts of the Presence Era: Using Information Visualization to Create an Evocative Souvenir," Proc. Symp. Information Visualization (InfoVis), pp. 105-111, 2004.
[90] F. Viégas, M. Wattenberg, F. van Ham, J. Kriss, and M. McKeon, "Many Eyes: A Site for Visualization at Internet Scale," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 5, pp. 1121-1128, Nov./Dec. 2007.
[91] C. Weaver, D. Fyfe, A. Robinson, D. Holdsworth, D. Peuquet, and A.M. MacEachren, "Visual Exploration and Analysis of Historic Hotel Visits," Information Visualization, vol. 6, no. 1, pp. 89-103, 2007.
[92] W. Willett, J. Heer, and M. Agrawala, "Scented Widgets: Improving Navigation Cues with Embedded Visualizations," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 5, pp. 1129-1136, Nov./Dec. 2007.
[93] K. Wongsuphasawat and B. Shneiderman, "Finding Comparable Temporal Categorical Records: A Similarity Measure with an Interactive Visualization," Proc. Symp. Visual Analytics Science and Technology (VAST), pp. 27-34, 2009.
[94] B. Yost and C. North, "The Perceptual Scalability of Visualization," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 837-844, Sept./Oct. 2007.
[95] T. Zuk, L. Schlesier, P. Neumann, M.S. Hancock, and S. Carpendale, "Heuristics for Information Visualization Evaluation," Proc. AVI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), pp. 55-60, 2006.

Index Terms:
data visualisation,visualization publications,empirical studies,information visualization,seven scenarios,visual data analysis,visual data reasoning,user performance evaluation,evaluating user experience,evaluating environments,evaluating communication,evaluating visualization algorithms,evaluating collaborative data analysis,Data visualization,Visualization,Data analysis,Encoding,Systematics,Taxonomy,Electronic mail,evaluation.,Information visualization
Citation:
H. Lam, E. Bertini, P. Isenberg, C. Plaisant, S. Carpendale, "Empirical Studies in Information Visualization: Seven Scenarios," IEEE Transactions on Visualization and Computer Graphics, vol. 18, no. 9, pp. 1520-1536, Sept. 2012, doi:10.1109/TVCG.2011.279
Usage of this product signifies your acceptance of the Terms of Use.