The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.12 - Dec. (2012 vol.18)
pp: 2431-2440
Michael Sedlmair , University of British Columbia
Miriah Meyer , University of Utah
Tamara Munzner , University of British Columbia
ABSTRACT
Design studies are an increasingly popular form of problem-driven visualization research, yet there is little guidance available about how to do them effectively. In this paper we reflect on our combined experience of conducting twenty-one design studies, as well as reading and reviewing many more, and on an extensive literature review of other field work methods and methodologies. Based on this foundation we provide definitions, propose a methodological framework, and provide practical guidance for conducting design studies. We define a design study as a project in which visualization researchers analyze a specific real-world problem faced by domain experts, design a visualization system that supports solving this problem, validate the design, and reflect about lessons learned in order to refine visualization design guidelines. We characterize two axes — a task clarity axis from fuzzy to crisp and an information location axis from the domain expert’s head to the computer — and use these axes to reason about design study contributions, their suitability, and uniqueness from other approaches. The proposed methodological framework consists of 9 stages: learn, winnow, cast, discover, design, implement, deploy, reflect, and write. For each stage we provide practical guidance and outline potential pitfalls. We also conducted an extensive literature survey of related methodological approaches that involve a significant amount of qualitative field work, and compare design study methodology to that of ethnography, grounded theory, and action research.
INDEX TERMS
Data visualization, Collaboration, Design methodology, Visualization, Algorithm design and analysis, Logic gates, framework, Design study, methodology, visualization
CITATION
Michael Sedlmair, Miriah Meyer, Tamara Munzner, "Design Study Methodology: Reflections from the Trenches and the Stacks", IEEE Transactions on Visualization & Computer Graphics, vol.18, no. 12, pp. 2431-2440, Dec. 2012, doi:10.1109/TVCG.2012.213
REFERENCES
[1] M. Agrawala, W. Li, and F. Berthouzoz, Design principles for visual communication Communications of the ACM, pages 60-69, Apr. 2011.
[2] R. Amar, J. Eagan, and J. Stasko, Low-level components of analytic activity in information visualization IEEE Symp. Information Visualization (Info Vis), pages 111-117, 2005.
[3] R. Amar and J. Stasko, Knowledge precepts for design and evaluation of information visualizations IEEE Trans. Visualization and Computer Graphics (TVCG), 11(4):432-442, 2005.
[4] A. Barsky, T. Munzner, J. Gardy, and R. Kincaid, Cerebral: visualizing multiple experimental conditions on a graph with biological context. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2008),14(6):1253-60, 2008.
[5] D. Baur, F. Seiffert, M. Sedlmair, and S. Boring, The streams of our lives: Visualizing listening histories in context. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2010). 16(6):1119-1128. 2010.
[6] H. R, Bernard Research Methods in Anthropology: Qualitative and Quantitative Approaches. AltaMira Press, 5th edition, 2011.
[7] H. Beyer and K. Holtzblatt, Contextual design: defining customer-centered systems. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1998.
[8] J. Blomberg, J. Giacomi, A. Mosher, and P. Swenton-Wall, Ethnographic field methods and their relation to design. Participatory design: Principles and practices, pages 123-155, 1993.
[9] K. Bodker, F. Kensing, and J. Simonsen, Participatory IT design: Designing for business and workplace realities. The MIT Press, 2004.
[10] M. Booshehrian, T. Moller, R. M. Peterman, and T. Munzner, Vismon: Facilitating analysis of trade-offs, uncertainty, and sensitivity in fisheries management decision making. Computer Graphics Forum (Proc. EuroVis 2012),2012. in press.
[11] M. A. Borkin, K. Z. Gajos, A. Peters, D. Mitsouras, S. Melchionna, F. J. Rybicki, C. L. Feldman, and H. Pfister, Evaluation of artery visualizations for heart disease diagnosis IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2011), pages 2479-2488, 2011.
[12] B. Brown, S. Reeves, and S. Sherwood, Into the wild: Challenges and opportunities for field trial methods. In Proc. ACM Conf Human Factors in Computing Systems (CHI), pages 1657-1666, 2011.
[13] S. Carpendale, Evaluating information visualizations. In A. Kerren, J. T. Stasko, J.-D. Fekete and C. North, editors, Information Visualization: Human-Centered Issues and Perspectives, volume 4950, pages 19-45. Springer LNCS, 2008.
[14] K. Charmaz, Constructing grounded theory. A practical guide through qualitative analysis. Sage Publications, Inc, 2006.
[15] A. Cockburn, Agile Software Development: The Cooperative Game. Addison-Wesley Professional, 2006.
[16] A. Crabtree, T. Rodden, P. Tolmie, and G. Button, Ethnography considered harmful. In Proc. ACM Conf Human Factors in Computing Systems (CHI), pages 879-888, 2009.
[17] P. Craig and J. Kennedy, Coordinated graph and scatter-plot views for the visual exploration of micro array time-series data. In Proc. IEEE Symp. Information Visu-alization (InfoVis), pages 173-180, 2003.
[18] M. Crotty, The foundations of social research: Meaning and perspective in the research process - Chapter 1. Sage Publications Ltd, 1998.
[19] A. Dix, Theoretical analysis and theory creation. In Research Methods in Human-Computer Interaction, pages 175-195. Cambridge Univ. Press, 2008.
[20] P. Dourish, Implications for design. In Proc. ACM Con! Human Factors in Computing Systems (CHI), pages 541-550, 2006.
[21] P. Dourish, Responsibilities and implications: further thoughts on ethnography and design. In Proc. ACM Conf Designing for User eXperiences (DUX), pages 2-16, 2007.
[22] S. P. Dow, A. Glassco, J. Kass, M. Schwarz, D. L. Schwartz, and S. R. Klemmer, Parallel prototyping leads to better design results, more divergence, and increased self-efficacy ACMTrans. Computer-Human Interaction (To CHI), 17(4):1-24, 2010.
[23] K. A. Ericsson and H. A. Simon, Verbal reports as data Psychological Review, 87(3):215-251, 1980.
[24] D. M. Fetterman, Ethnography: Step by Step. Sage Publications, 2nd edition, 1998.
[25] D. Furniss, A. Blandford, and P. Curzon, Confessions from a Grounded Theory PhD: Experiences and Lessons Learnt. In ACM Trans. Computer-Human Interaction (ToCHI), pages 113-122, 2011.
[26] S. Greenberg and B. Buxton, Usability evaluation considered harmful (some of the time). In Proc. ACM Con! Human Factors in Computing Systems (CHI), pages 111-120, 2008.
[27] G. R. Hayes, The relationship of action research to human-computer interaction ACM Trans. Computer-Human Interaction (To CHI), 18(3):1-20, 2011.
[28] K. Holtzblatt and S. Jones, Contextual inquiry: A participatory technique for system design. In D. Schuler, A. Namioka (eds.): Participatory Design: Principles and Practices, pages 177-210. Lawrence Erlbaum Associates, 1993.
[29] J. Hughes, V. King, and T. Rodden, Moving out from the control room: ethnography in system design Proc. ACM Conf Computer Supported Cooperative Work (CSCW), pages 429-439, 1994.
[30] J. Hughes, J. O'Brien, and T. Rodden, Designing with ethnography: a presentation framework for design. In Proc. Conf Designing Interactive Systems (DIS), pages 147-158. 1997.
[31] H. Hutchinson et al. Technology probes: inspiring design for and with families. In Proc. ACM Con! Human Factors in Computing Systems (CHI), pages 17-24, 2003.
[32] M. J, Intons-Peterson. Imagery paradigms: How vulnerable are they to experimenters’ expectations? Journ. Experimental Psychology - Human Perception and Performance, 9: 394-412, 1983.
[33] P. Isenberg, T. Zuk, C. Collins, and S. Carpendale, Grounded evaluation of information visualizations. In Proc. ACM CHI Wkshp. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV). 2008.
[34] D. F. Keefe, M. Ewert, W. Ribarsky, and R. Chang, Interactive coordinated multiple-view visualization of biomechanical motion data IEEE Trans. Visualization and Computer Graphics (Proc. Vis 2009), 15(6):1383-1390, 2009.
[35] R. Kincaid, SignalLens: Focus+Context Applied to Electronic Time Series IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2010), 16(6):900-907, 2010.
[36] B. L. Kovitz, Practical Sofware Requirements: A Manual of Content & Style. Man-ning, 1999.
[37] J. Kruger and D. Dunning, Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journ. Personality and Social Psychology, 77(6):1121-1134, 1999.
[38] M. Kuniavsky, Observing the User Experience: A Practitioner's Guide To User Research. Morgan Kaufmann, 2003.
[39] H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale, Empirical studies in information visualization: Seven scenarios IEEE Trans. Visualization and Computer Graphics (TVCG), 2012. Appeared online: 30 Nov. 2011.
[40] H. Lam, D. Russell, D. Tang, and T. Munzner, Session Viewer: Visual Exploratory Analysis of Web Session Logs. In Proc. IEEE Symp. Visual Analytics Science and Technology (VAST), pages 147-154, 2007.
[41] C. Larman and V. R, Basili. Iterative and incremental development: A brief history IEEE Computer, 36(6):47-56, 2003.
[42] D. Lloyd and J. Dykes, Human-centered approaches in geovisualization design: investigating multiple methods through a long-term case study IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2011), 17(12):498-507, 2011.
[43] P. McLachlan, T. Munzner, E. Koutsofios, and S. North, LiveRAC: interactive visual exploration of system management time-series data. In Proc. ACM Con! on Human Factors in Computing Systems (CHI), pages 1483-1492, 2008.
[44] M. Meyer, T. Munzner, A. dePace, and H. Pfister, MulteeSum: A tool for exploring space-time expression data. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2010), 16(6):908-917, 2010.
[45] M. Meyer, T. Munzner, and H. Pfister, MizBee: A multiscale synteny browser IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2009), 15(6):897-904, 2009.
[46] M. Meyer, B. Wong, M. Styczynski, T. Munzner, and H. Pfister, Pathline: A tool for comparative functional genomics Computer Graphics Forum (Proc. EuroVis 2010), 29(3):1043-1052, 2010.
[47] D. Millen, Rapid ethnography: Time deepening strategies for HCI field research. In Proc. ACM Con! Designing Interactive Systems (DIS), pages 280-286, 2000.
[48] T. Munzner, Interactive Visualization of Large Graphs and Networks. PhD thesis, Stanford University Department of Computer Science, 2000.
[49] T. Munzner, Process and pitfalls in writing information visualization research papers. In Information Visualization: Human-Centered Issues and Perspectives, pages 134-153. Springer LNCS 4950, 2008.
[50] T. Munzner, A Nested Model for Visualization Design and Validation IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2009), 15(6):921-928, 2009.
[51] T. Munzner, A. Barsky, and M. Williams, Reflections on QuestVis: A visualization system for an environmental sustainability model. In Scientific Visualization: Interactions, Features, Metaphors, volume 2, pages 240-259. Dagstuhl Follow-Ups, 2011.
[52] T. Munzner, F. Guimbretiere, and G. Robertson, Constellation: a visualization tool for linguistic queries from MindNet. In Proc. IEEE Symp. Information Visualization (Info Vis), pages 132-135, 1999.
[53] T. Munzner, E. Hoffman, K. Claffy, and B. Fenner, Visualizing the Global Topology of the MBone. In Proc. IEEE Symp. Information Visualization (Info Vis), pages 85-92, 1996.
[54] T. Munzner, Q. Kong, R. T. Ng, J. Lee, J. Klawe, D. Radulovic, and C. K. Leung, Visual mining of power sets with large alphabets. Technical report, Univ. British Columbia Computer Science TR-2005-25, Dec. 2005.
[55] T. Munzner and S. North, InfoVis 2003 Call for Participation: Papers. http://www.infovis.org/infovis/2003/CFP#papers.
[56] C. B. Nielsen, S. D. Jackman, I. Birol, and S. J. Jones, ABySS-Explorer: Visualizing genome sequence assemblies IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2009), 15(6):881-8, 2009.
[57] J. Nielsen, Guerrilla HCI: Using discount usability engineering to penetrate the intimidation barrier. In Cost-Justifying Usability, pages 245-272. Academic Press, 1994.
[58] A. Perer, I. Guy, E. Uziel, I. Ronen, and M. Jacovi, Visual Social Network Analytics for Relationship Discovery in the Enterprise. In Proc. IEEE Conf Visual Analytics Science and Technology (VAST), pages 71-79, 2011.
[59] C. Plaisant, The challenge of information visualization evaluation. In Proc. Advanced Visual Interfaces (AVI), pages 109-116. ACM Press, 2004.
[60] J. Preece, Y. Rogers, and H. Sharp, Interaction Design: Beyond Human-Computer Interaction. John Wiley & Sons, 2002.
[61] A. J. Pretorius and J. J. van Wijk, What does the user want to see? what do the data want to be? Information Visualization, 8(3):153-166, 2009.
[62] L. Richardson and E. A. S. Pierre, Writing: A method for inquiry. In Handbook of Qualitative Research, pages 959-978. Sage, 3rd edition, 2005.
[63] J. Rode, Reflexivity in Digital Anthropology. In ACM Trans. Computer-Human Interaction (To CHI), pages 123-132, 2011.
[64] Y. Rogers, New theoretical approaches for HCI Annual review of information science and technology, 38(1):87-143, 2004.
[65] Y. Rogers, K. Connelly, L. Tedesco, W. Hazlewood, A. Kurtz, R. Hall, J. Hursey, and T. Toscos, Why it's worth the hassle: the value of in-situ studies when designing Ubicomp. In Proc. Intl. Con! on Ubiquitous Computing (UbiComp), pages 336-353. Springer-Verlag, 2007.
[66] B. E. Rogowitz, L. A. Treinish, and S. Bryson, How not to lie with visualization. Comput. Phys. 10(3):268-273, June 1996.
[67] K. Ruhland, M. Sedlmair, S. Bioletti, and C. O'sullivan, LibViz: Data Visualisation of the Old Library. Intl. Journ. Architectural Computing, 7(1):177-192, 2009.
[68] P. Saraiya, C. North, and K. Duca, An insight-based methodology for evaluating bioinformatics visualizations IEEE Trans. Visualization and Computer Graphics, 11(4):443-456, 2005.
[69] P. Saraiya, C. North, V. Lam, and K. Duca, An insight-based longitudinal study of visual analytics. IEEE Trans. Visualization and Computer Graphics, 12(6):1511-1522. 2006.
[70] M. Sedlmair, Car-x-ray: Closely Integrating Both Worlds. In Visual Analysis of In-Car Communication Networks (PhD Thesis). LMU, 2010.
[71] M. Sedlmair, ProgSpy20 1 0: Visualizing Flash Activities. In Visual Analysis of In-Car Communication Networks (PhD Thesis). LMU, 2010.
[72] M. Sedlmair, WiKeVis: Visualizing Functional Networks. In Visual Analysis of In-Car Communication Networks (PhD Thesis). LMU, 2010.
[73] M. Sedlmair, C. Bernhold, D. Herrscher, S. Boring, and A. Butz, MostVis: An Interactive Visualization Supporting Automotive Engineers in MOST Catalog Ex-ploration. In Proc. Intl. Conf. Information Visualization (IV), pages 173-182, 2009.
[74] M. Sedlmair, A. Frank, T. Munzner, and A. Butz, RelEx: A design study of visual network analysis in the automotive domain. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2012),2012. In press.
[75] M. Sedlmair, W. Hintermaier, K. Stocker, T. Boring, and A. Butz, A dual-view visualization of in-car communication processes. In Proc. Intl. Conf Information Visualization (IV), pages 157-162, 2008.
[76] M. Sedlmair, P. Isenberg, D. Baur, and A. Butz, Information Visualization Evaluation in Large Companies: Challenges, Experiences and Recommendations. Information Visualization, 10(3):248-266, 2011.
[77] M. Sedlmair, P. Isenberg, D. Baur, M. Mauerer, C. Pigorsch, and A. Butz, Car-diogram: Visual Analytics for Automotive Engineers. In Proc. ACM Con! Human Factors in Computing Systems (CHI), pages 1727-1736, 2011.
[78] M. Sedlmair, B. Kunze, W. Hintermaier, and A. Butz, User-centered development of a visual exploration system for in-car communication. In Proc. Intl. Symp. Smart Graphics (SG), pages 105-116, 2009.
[79] B. Shneiderman, The eyes have it: A task by data type taxonomy for information visualizations. In Proc. IEEE Symp. Visual Languages (VL), pages 336-343. IEEE, 1996.
[80] B. Shneiderman and C. Plaisant, Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies. In Proc. AVI Wkshp. on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV). ACM, 2006.
[81] D. Silverman, A Very Short, Fairly Interesting and Reasonably Cheap Book About Qualitative Research. Sage Publications Ltd., 2007.
[82] A. Slingsby, J. Dykes, and J. Wood, Configuring hierarchical layouts to address research questions IEEE Trans. Vis. Comput. Graph., 15(6):977-984, 2009.
[83] H. Thimbleby, Write now! In Research Methods in Human-Computer Interaction, pages 196-211. Cambridge Univ. Press, 2008.
[84] J. Thomas and K. Cook, Illuminating the path: The research and development agenda for visual analytics National Visualization and Analytics Center, 2005.
[85] M. Tory and T. Möller, Human factors in visualization research. IEEE Trans. Visu-alization and Computer Graphics (TVCG), 10(1):72-84, 2004.
[86] M. Tory and T. Möller, Evaluating visualizations: do expert reviews work? IEEE Computer Graphics and Applications, 25(5):8-11, 2005.
[87] J. van Niekerk and J. Roode, Glaserian and Straussian grounded theory: Similar or completely different? In Proc. South African Institute of Computer Scientists and Information Technologists (SAICSIT), pages 96-103, 2009.
[88] J. J. van Wijk, Views on visualization IEEE Trans. Visualization and Computer Graphics (TVCG), 12(4):421-432, 2006.
[89] K. L. Wagstaff, Machine Learning that Matters. In Proceedings of the 29th Inter-national Conference on Machine Learning. Copyright 2012 California Institute of Technology, June 2012.
[90] M. Q. Wang Baldonado, A. Woodruff, and A. Kuchinsky, Guidelines for using multiple views in information visualization. In Proc. ACM Advanced Visual Interfaces (AVI), pages 110-119, 2000.
[91] C. Ware, Information Visualization: Perception for Design. Morgan Kaufmann, 2004.
[92] W. Willinger, D. Alderson, and J. C, Doyle. Mathematics and the Internet: A source of enormous confusion and great potential Notices of the American Mathemematical Society, 56(5):586-599, 2009.
[93] H. Wolcott, Writing up qualitative research, volume 20 of Qualitative Research Methods. Sage, 3rd edition, 2009.
[94] J. Wood, D. Badawood, J. Dykes, and A. Slingsby, BallotMaps: Detecting name bias in alphabetically ordered ballot papers. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2011), 17(12):2384-2391, 2011.
[95] T. Zuk, L. Schlesier, P. Neumann, M. S. Hancock, and S. Carpendale, Heuristics for information visualization evaluation. In Proc. ACM CHI Wkshp. on BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV). 2008.
6 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool