The Community for Technology Leaders
RSS Icon
Issue No.06 - November/December (2010 vol.16)
pp: 999-1008
Zhicheng Liu , Georgia Institute of Technology
John Stasko , Georgia Institute of Technology
Although previous research has suggested that examining the interplay between internal and external representations can benefit our understanding of the role of information visualization (InfoVis) in human cognitive activities, there has been little work detailing the nature of internal representations, the relationship between internal and external representations and how interaction is related to these representations. In this paper, we identify and illustrate a specific kind of internal representation, mental models, and outline the high-level relationships between mental models and external visualizations. We present a top-down perspective of reasoning as model construction and simulation, and discuss the role of visualization in model based reasoning. From this perspective, interaction can be understood as active modeling for three primary purposes: external anchoring, information foraging, and cognitive offloading. Finally we discuss the implications of our approach for design, evaluation and theory development
mental model, model-based reasoning, distributed cognition, interaction, theory, information visualization
Zhicheng Liu, John Stasko, "Mental Models, Visual Reasoning and Interaction in Information Visualization: A Top-down Perspective", IEEE Transactions on Visualization & Computer Graphics, vol.16, no. 6, pp. 999-1008, November/December 2010, doi:10.1109/TVCG.2010.177
[1] Gapminder. http:/
[2] Tableausoftware. http:/
[3] D. H. Ballard, M. M. Hayhoe, P. K. Pook, and R. P. N. Rao, Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences, 20 (04): 723–742, 1997.
[4] R. A. Brooks, Intelligence without reason. Artificial intelligence: critical concepts, 3, 1991.
[5] S. K. Card, J. D. Mackinlay, and B. Shneiderman, Readings in information visualization: using vision to think. pages 1–34. Morgan Kaufmann Publishers Inc, Jan. 1999.
[6] C. Conati and H. Maclaren, Exploring the role of individual differences in information visualization. In Proceedings of the working conference on Advanced visual interfaces, pages 199–206, Napoli, Italy, 2008. ACM.
[7] K. Craik, The Nature of Explanation. Cambridge University Press, Cambridge, 1943.
[8] A. Dix and G. Ellis, Starting simple: adding value to static visualisation through simple interaction. In Proceedings of the working conference on Advanced Visual Interfaces, pages 124–134, 1998.
[9] T. Dwyer, B. Lee, D. Fisher, K. I. Quinn, P. Isenberg, G. Robertson, C. North, and K. Inkpen, A comparison of User-Generated and automatic graph layouts. IEEE Transactions on Visualization and Computer Graphics, 15 (6): 961–968, 2009.
[10] K. A. Ericsson and H. A. Simon, Protocol analysis: Verbal reports as data. MIT Press, 1993.
[11] E. S. Ferguson, Engineering and the Mind's Eye. The MIT Press, 1994.
[12] J. J. Gibson, The Ecological Approach to Visual Perception. Lawrence Erlbaum Associates Inc, US, 1986.
[13] V. Gottschling, Visual imagery, mental models, and reasoning. Advances in Psychology, 138 :211–235, 2006.
[14] D. Gotz and M. X. Zhou, Characterizing users visual analytic activity for insight provenance. Information Visualization, 8 (1): 42–55, 2009.
[15] M. Hegarty, Diagrams in the mind and in the world: Relations between internal and external visualizations. Lecture notes in computer science, pages 1–13, 2004.
[16] E. Hutchins, Material anchors for conceptual blends. Journal of Pragmatics, 37 (10): 1555–1577, 2005.
[17] E. L. Hutchins, J. D. Hollan, and D. A. Norman, Direct manipulation interfaces. Human-Computer Interaction, 1 (4): 311–338, 1985.
[18] V. John-Steiner, Notebooks of the Mind: Explorations of Thinking. Oxford University Press, USA, revised edition, 1997.
[19] P. N. Johnson-Laird, Mental models: Towards a cognitive science of language, inference and consciousness. Harvard Univ Pr, 1983.
[20] P. N. Johnson-Laird, Imagery, visualization, and thinking. Perception and cognition at centurys end, pages 441–467, 1998.
[21] D. E. Kieras and S. Bovair, The role of a mental model in learning to operate a device. Cognitive science, 8 (3): 255–273, 1984.
[22] D. Kirsh, Projection, problem space and anchoring. In Proceedings of the 31st Annual Conference of the Cognitive Science Society, pages 2310–2315, 2009.
[23] G. Klein, J. K. Phillips, E. L. Rall, and D. A. Peluso, A data-frame theory of sensemaking. In Expertise Out of Context: Proc. 6th Intl Conf. Naturalistic Decision Making, 2006.
[24] M. Knauff and P. N. Johnson-Laird, Visual imagery can impede reasoning. Memory & Cognition, 30 (3): 363–371, 2002.
[25] A. Kobsa, An empirical comparison of three commercial information visualization systems. In IEEE Symposium on Information Visualization, pages 123–130, 2001.
[26] S. M. Kosslyn, W. L. Thompson, and G. Ganis, The case for mental imagery. Oxford University Press, USA, 2006.
[27] G. Lakoff and M. Johnson, Metaphors We Live by. University of Chicago Press, 1980.
[28] H. Lam, A framework of interaction costs in information visualization. IEEE transactions on visualization and computer graphics, 14 (6): 1149–1156, 2008.
[29] J. H. Larkin and H. A. Simon, Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11 (1): 65–100, 1987.
[30] Z. Liu, N. J. Nersessian, and J. T. Stasko, Distributed cognition as a theoretical framework for information visualization. IEEE Transactions on Visualization and Computer Graphics, 14 (6): 1173–1180, Dec. 2008.
[31] Z. Liu, J. Stasko, and T. Sullivan, SellTrend: Inter-Attribute visual analysis of temporal transaction data. IEEE Transactions on Visualization and Computer Graphics, 15 (6): 1025–1032, 2009.
[32] R. H. Logie, Visuo-spatial working memory. Psychology Press, 1995.
[33] L. Magnani and N. J. Nersessian, Model-based reasoning: science, technology, values. Springer, 2002.
[34] J. M. Mandler, Stories, Scripts, and Scenes: Aspects of Schema Theory. Psychology Press, 1st edition, 1984.
[35] B. A. Nardi and C. L. Zarmer, Beyond models and metaphors: Visual formalisms in user interface design. Journal of Visual Languages & Computing, 4 (1): 5–33, 1993.
[36] N. Nersessian, Creating scientific concepts. The MIT Press, 2008.
[37] A. Noe, Action in Perception. The MIT Press, 2005.
[38] D. A. Norman, Some observations on mental models. Mental models, pages 7–14, 1983.
[39] C. North, Visualization viewpoints: Toward measuring visualization insight. IEEE Computer Graphics and Applications, 26 (3): 6–9, 2006.
[40] C. North and B. Shneiderman, Snap-together visualization: a user interface for coordinating visualizations via relational schemata. In Proceedings of the working conference on Advanced visual interfaces, pages 128–135, 2000.
[41] C. North and B. Shneiderman, Snap-together visualization: can users construct and operate coordinated visualizations? International Journal of Human Computer Studies, 53 (5): 715–740, 2000.
[42] S. J. Payne, Users mental models: The very ideas. HCI Models, theories and frameworks: Toward a multidisciplinary science, pages 135–156, 2003.
[43] W. A. Pike, J. Stasko, R. Chang, and T. A. O'Connell, The science of interaction. Information Visualization, 8 (4): 263–274, 2009.
[44] P. Pirolli and S. Card, The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of International Conference on Intelligence Analysis, pages 2–4, 2005.
[45] H. Purchase, E. Hoggan, and C. Görg, How important is the "Mental map"? an empirical investigation of a dynamic graph layout algorithm. In Graph Drawing, pages 184–195, 2006.
[46] Z. W. Pylyshyn, What the mind's eye tells the mind's brain: A critique of mental imagery. Psychological bulletin, 80 (1): 1–24, 1973.
[47] W B. Rouse and N. M. Morris, On looking into the black box: Prospects and limits in the search for mental models. Psychological bulletin, 100 (3): 349–363, 1986.
[48] M. Scaife and Y. Rogers, External cognition : how do graphical representations work? Int. J. Human-Computer Studies, 45 (2): 185–213, 1996.
[49] B. Shneiderman, The eyes have it: A task by data type taxonomy of information visualizations. In Proc. IEEE Visual Languages `96, pages 336–343, 1996.
[50] B. Shneiderman and C. Plaisant, Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies. In BELIV : BEyond time and errors: novel evaLuation methods for Information Visualization, Venice, Italy, 2006.
[51] B. Shore, Culture in Mind: Cognition, Culture, and the Problem of Meaning. Oxford University Press US, 1996.
[52] R. Spence, Information Visualization: Design for Interaction. Prentice Hall, 2nd edition, 2007.
[53] D. Sperber, Anthropology and psychology: Towards an epidemiology of representations. Man, 20 (1): 73–89, 1985.
[54] M. Spivey, The Continuity of Mind. Oxford University Press, USA, 2008.
[55] C. R. Stolte, Query, analysis, and visualization of multidimensional databases. PhD thesis, Stanford University, 2003.
[56] J. J. Thomas and K. A. Cook, Illuminating the path: The research and development agenda for visual analytics. IEEE Computer Society, 2005.
[57] J. G. Trafton, S. B. Trickett, and F. E. Mintz, Connecting internal and external representations: Spatial transformations of scientific visualizations. Foundations of Science, 10 (1): 89–106, 2005.
[58] E. R. Tufte, Visual Explanations: Images and Quantities, Evidence and Narrative. Graphics Press, 4th printing with revisions edition, Feb. 1997.
[59] B. Tversky, Cognitive maps, cognitive collages, and spatial mental models. Spatial Information Theory - A Theoretical Basis for GIS: Proc. European Conference COSIT, pages 14–24, 1993.
[60] L. Tweedie, Characterizing interactive externalizations. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 375–382, 1997.
[61] F. van Ham and B. Rogowitz, Perceptual organization in user-generated graph layouts. IEEE Transactions on Visualization and Computer Graphics, 14 (6): 1333–1339, 2008.
[62] L. S. Vygotsky, Thought and Language. The MIT Press, 1986.
[63] T. D. Wang, C. Plaisant, B. Shneiderman, N. Spring, D. Roseman, G. Marchand, V. Mukherjee, and M. Smith, Temporal summaries: Supporting temporal categorical searching, aggregation and comparison. IEEE Transactions on Visualization and Computer Graphics, 15 (6): 1049–1056, 2009.
[64] J. V. Wertsch, Vygotsky and the Social Formation of Mind. Harvard University Press, 1988.
[65] L. Wilkinson, The grammar of graphics. Springer Verlag, 2005.
[66] J. S. Yi, Y. ah Kang, J. T. Stasko, and J. A. Jacko, Toward a deeper understanding of the role of interaction in information visualization. IEEE Transactions on Visualization and Computer Graphics, 13 (6): 1224–1231, 2007.
[67] C. Ziemkiewicz and R. Kosara, The shaping of information by visual metaphors. IEEE Transactions on Visualization and Computer Graphics, 14 (6), Dec. 2008.
27 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool