This Article 
 Bibliographic References 
 Add to: 
Visual Perception and Mixed-Initiative Interaction for Assisted Visualization Design
March/April 2008 (vol. 14 no. 2)
pp. 396-411
This paper describes the integration of perceptual guidelines from human vision with an AI-based mixed-initiative search strategy. The result is a visualization assistant called ViA, a system that collaborates with its users to identify perceptually salient visualizations for large, multidimensional datasets. ViA applies knowledge of low-level human vision to: (1) evaluate the effectiveness of a particular visualization for a given dataset and analysis tasks; and (2) rapidly direct its search towards new visualizations that are most likely to offer improvements over those seen to date. Context, domain expertise, and a high-level understanding of a dataset are critical to identifying effective visualizations. We apply a mixed-initiative strategy that allows ViA and its users to share their different strengths and continually improve ViA's understanding of a user's preferences. We visualize historical weather conditions to compare ViA's search strategy to exhaustive analysis, simulated annealing, and reactive tabu search, and to measure the improvement provided by mixed-initiative interaction. We also visualize intelligent agents competing in a simulated online auction to evaluate ViA's perceptual guidelines. Results from each study are positive, suggesting that ViA can construct high-quality visualizations for a range of real-world datasets.

[1] J. Allen, C.I. Guinn, and E. Horowitz, “Mixed-Initiative Interaction,” IEEE Intelligent Systems, vol. 14, no. 5, pp. 14-23, 1999.
[2] R. Battiti, “Reactive Search: Toward Self-Tuning Heuristics,” Modern Heuristic Search Models, V.J. Rayward-Smith, I.H. Osman, C.R. Reeves, and G.D. Smith, eds. John Wiley & Sons, pp. 61-83, 1996.
[3] L.D. Bergman, B.E. Rogowitz, and L.A. Treinish, “A Rule-Based Tool for Assisting Colormap Selection,” Proc. Visualization, pp.118-125, 1995.
[4] C. Beshers and S. Feiner, “AutoVisual: Rule-Based Design of Interactive Multivariate Visualizations,” IEEE Computer Graphics and Applications, vol. 13, no. 4, pp. 41-49, July 1993.
[5] M. Brown and J. Chu-Carroll, “An Evidential Model for Tracking Initiative in Collaborative Dialog Interactions,” User Modeling and User-Adapted Interaction, vol. 8, no. 3, pp. 215-253, 1998.
[6] T.C. Callaghan, “Dimensional Interaction of Hue and Brightness in Preattentive Field Segregation,” Perception and Psychophysics, vol. 36, no. 1, pp. 25-34, 1984.
[7] T.C. Callaghan, “Interference and Dominance in Texture Segregation,” Proc. Visual Search, D. Brogan, ed., pp. 81-87, 1990.
[8] CIE, Official Recommendations on Uniform Color Spaces, Color-Difference Equations, and Metric Color Terms, CIE Publication No. 15, Supplement Number 2 (E-1.3.1, 1971), Commission Int'l e de L'Èclairge, 1978.
[9] P.R. Cohen, Empirical Methods for Artificial Intelligence. MIT Press, 1995.
[10] P.R. Cohen, C. Allaby, C. Cumbaa, M. Fitzgerald, K. Ho, B. Hui, C. Latulipe, F. Lu, N. Moussa, D. Pooley, A. Qian, and S. Dissiqi, “What Is Initiative,” User Modeling and User-Adapted Interaction, vol. 8, no. 3, pp. 171-214, 1998.
[11] G. Ferguson, J.F. Allen, and B. Miller, “TRAINS-95: Towards a Mixed-Initiative Planning Assistant,” Proc. Third Int'l Conf. Artificial Intelligence Planning Systems, pp. 70-77, 1996.
[12] J. Gallop, “Underlying Data Models and Structures for Visualization,” Scientific Visualization: Advances and Challenges, L. Rosenblum, ed., Academic Press, pp. 87-102, 1994.
[13] F. Glover and M. Laguna, “Tabu Search,” Modern Heuristic Techniques for Combinatorial Problems, C.R. Reeves, ed., Blackwell Scientific Publishing, pp. 70-150, 1993.
[14] P. Gray, W. Hart, L. Painton, C. Phillips, M. Trahan, and J. Wagner, A Survey of Global Optimization Methods, , 1997.
[15] C.G. Healey, K.S. Booth, and J.T. Enns, “Harnessing Preattentive Processes for Multivariate Data Visualization,” Proc. Graphics Interface, pp. 107-117, 1993.
[16] C.G. Healey and J.T. Enns, “Large Datasets at a Glance: Combining Textures and Colors in Scientific Visualization,” IEEE Trans. Visualization and Computer Graphics, vol. 5, no. 2, pp. 145-167, Apr.-June 1999.
[17] C.G. Healey, J.T. Enns, L.G. Tateosian, and M. Remple, “Perceptually-Based Brush Strokes for Nonphotorealistic Visualization,” ACM Trans. Graphics, vol. 23, no. 1, pp. 64-96, 2004.
[18] C.G. Healey, R. St. Amant, and J. Chang, “Assisted Visualization of E-Commerce Auction Agents,” Proc. Graphics Interface, pp. 201-208, 2001.
[19] C.G. Healey, R. St. Amant, and M. Elhaddad, “ViA: A Perceptual Visualization Assistant,” Proc. 28th Workshop Advanced Imagery Pattern Recognition (AIPR '99), pp. 1-11, 1999.
[20] C.G. Healey and P.R. Wurman, “Visualizing Market Data,” IEEE Internet Computing, vol. 5, no. 2, p. 88, 2001.
[21] B. Hibbard and D. Santek, “The VIS-5D System for Easy Interactive Visualization,” Proc. Visualization, pp. 28-35, 1990.
[22] E. Horowitz, “Uncertainty, Action and Interaction: In Pursuit of Mixed-Initiative Computing,” IEEE Intelligent Systems, vol. 14, no. 5, pp. 17-20, 1990.
[23] E. Horowitz, “Principles of Mixed Initiative User Interfaces,” Proc. SIGCHI '99, pp. 159-166, 1999.
[24] D.E. Huber and C.G. Healey, “Visualizing Data with Motion,” Proc. Visualization, pp. 527-534, 2005.
[25] V. Interrante, “Illustrating Surface Shape in Volume Data via Principle Direction-Driven 3D Line Integral Convolution,” Proc. SIGGRAPH '97, T. Whitted, ed., pp. 109-116, 1997.
[26] V. Interrante, “Harnessing Natural Textures for Multivariate Visualization,” IEEE Computer Graphics and Applications, vol. 20, no. 6, pp. 6-11, 2000.
[27] C.R. Johnson, “Top Scientific Visualization Research Problems,” IEEE Computer Graphics and Applications 24, vol. 4, pp. 13-17, 2004.
[28] C.R. Johnson, R. Moorhead, T. Munzner, H. Pfsiter, P. Rheingans, and T.S.e. Yoo, NIH/NSF Visualization Research Challenges Report. IEEE Press, 2006.
[29] B. Julész, E.N. Gilbert, and L.A. Shepp, “Inability of Humans to Discriminate between Visual Textures that Agree in Second-Order Statistics—Revisited,” Perception, vol. 2, pp. 391-405, 1973.
[30] D.H. Laidlaw, R.M. Kirby, C.D. Jackson, J.S. Davidson, T.S. Miller, M. da Silva, W.H. Warren, and M.J. Tarr, “Comparing 2D Vector Field Visualization Methods: A User Study,” IEEE Trans. Visualization and Computer Graphics, vol. 11, no. 1, pp.59-70, Jan.-Feb. 2005.
[31] J.C. Lester, B.A. Stone, and G.D. Stelling, “Lifelike Pedagogical Agents for Mixed-Initiative Problem Solving in Constructivist Learning Environments,” User Modeling and User-Adapted Interaction, vol. 9, no. 1, pp. 1-44, 1999.
[32] H. Levkowitz and G.T. Herman, “Color Scales for Image Data,” IEEE Computer Graphics and Applications, vol. 12, no. 1, pp. 72-80, 1992.
[33] J. Lohse, H. Rueter, K. Biolsi, and N. Walker, “Classifying Visual Knowledge Representations: A Foundation for Visualization Research,” Proc. IEEE Visualization Conf., pp. 131-138, 1990.
[34] J. Mackinlay, “Automating the Design of Graphical Presentations of Relational Information,” ACM Trans. Graphics, vol. 5, no. 2, pp.110-141, 1986.
[35] J. Malik and P. Perona, “Preattentive Texture Discrimination with Early Vision Mechanisms,” J. Optical Soc. of Am. A, vol. 7, no. 5, pp.923-932, 1990.
[36] J. Marks, B. Andalman, P.A. Beardsley, W. Freeman, S. Gibson, J. Hodgins, and T. Kang, “Design Galleries: A General Approach to Setting Parameters for Computer Graphics and Animation,” Proc. SIGGRAPH '97, T. Whitted, ed., pp. 389-400, 1997.
[37] B.H. McCormick, T.A. DeFanti, and M.D. Brown, “Visualization in Scientific Computing,” Computer Graphics, no. 21, vol. 6, pp. 1-14, 1987.
[38] B. Miller, “Is Explicit Representation of Initiative Desirable?” Proc. Working Notes of AAAI 97 Spring Symp. Mixed-Initiative Interaction, pp. 105-110, 1997.
[39] A.H. Munsell, A Color Notation. Munsell Color Co., 1905.
[40] A.R. Rao and G.L. Lohse, “Towards a Texture Naming System: Identifying Relevant Dimensions of Texture,” Proc. IEEE Visualization Conf., pp. 220-227, 1993.
[41] T.R. Reed and J.M. Hans Du Buf, “A Review of Recent Texture Segmentation and Feature Extraction Techniques,” CVGIP: Image Understanding, vol. 57, no. 3, pp. 359-372, 1993.
[42] P. Rheingans and B. Tebbs, “A Tool for Dynamic Explorations of Color Mappings,” Computer Graphics, vol. 24, no. 2, pp. 145-146, 1990.
[43] P.K. Robertson, “A Methodology for Scientific Data Visualisation: Choosing Representations Based on a Natural Scene Paradigm,” Proc. Visualization, pp. 114-123, 1990.
[44] P.K. Robertson and L. De Ferrari, “Systematic Approaches to Visualization: Is a Reference Model Needed?” Scientific Visualization: Advances and Challenges, L. Rosenblum, ed., Academic Press, pp. 239-250, 1994.
[45] B.E. Rogowitz and L.A. Treinish, “An Architecture for Perceptual Rule-Based Visualization,” Proc. Visualization, pp. 236-243, 1993.
[46] L.J. Rosenblum, “Research Issues in Scientific Visualization,” IEEE Computer Graphics and Applications, vol. 14, no. 2, pp. 61-85, 1994.
[47] S.J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, second ed. Prentice Hall, 2003.
[48] W. Schroeder, K. Martin, and B. Lorensen, The Visualization Toolkit. Prentice Hall, 1998.
[49] H. Senay and E. Ignatius, “A Knowledge-Based System for Visualization Design,” IEEE Computer Graphics and Applications, vol. 14, no. 6, pp. 36-47, 1994.
[50] R.J. Snowden, “Texture Segregation and Visual Search: A Comparison of the Effects of Random Variations along Irrelevant Dimensions,” J. Experimental Psychology: Human Perception and Performance, vol. 14, no. 5, pp. 1354-1367, 1998.
[51] R. St. Amant and P.R. Cohen, “Interaction with a Mixed-Initiative System for Exploratory Data Analysis,” Proc. Second Int'l Conf. Intelligent User Interfaces (IUI '97), pp. 15-22, 1997.
[52] L.G. Tateosian and C.G. Healey, “Engaging Viewers with Aesthetic Visualizations,” Proc. Fifth Int'l Symp. Non-Photorealistic Animation and Rendering (NPAR '07), to appear.
[53] J.J. Thomas and K.A Cook, Illuminating the Path: Research and Development Agenda for Visual Analytics. IEEE Press, 2005.
[54] A. Triesman, “Search, Similarity, and Integration of Features between and within Dimensions,” J. Experimental Psychology: Human Perception and Performance, vol. 17, no. 3, pp. 652-676, 1991.
[55] C. Ware, “Color Sequences for Univariate Maps: Theory, Experiments, and Principles,” IEEE Computer Graphics and Applications, vol. 8, no. 5, pp. 41-49, 1988.
[56] C. Ware and W. Knight, “Using Visual Texture for Information Display,” ACM Trans. Graphics, vol. 14, no. 1, pp. 3-20, 1995.
[57] S. Wehrend and C. Lewis, “A Problem-Oriented Classification of Visualization Techniques,” Proc. Visualization, pp. 139-143, 1990.
[58] C. Weigle, W.G. Emigh, G. Liu, R.M. Taylor, J.T. Enns, and C.G. Healey, “Oriented Texture Slivers: A Technique for Local Value Estimation of Multiple Scalar Fields,” Proc. Graphics Interface, pp.163-170, 2000.
[59] J.M. Wolfe, “Guided Search 2.0: A Revised Model of Visual Search,” Psychonomic Bull. and Rev., vol. 1, no. 2, pp. 202-238, 1994.
[60] G. Wyszecki and W.S. Stiles, Color Science: Concepts and Methods, Quantitative Data and Formulae, second ed. John Wiley & Sons, 1982.

Index Terms:
Multivariate visualization, Display algorithms, Interaction techniques, Human information processing
Christopher Healey, Sarat Kocherlakota, Vivek Rao, Reshma Mehta, Robert St. Amant, "Visual Perception and Mixed-Initiative Interaction for Assisted Visualization Design," IEEE Transactions on Visualization and Computer Graphics, vol. 14, no. 2, pp. 396-411, March-April 2008, doi:10.1109/TVCG.2007.70436
Usage of this product signifies your acceptance of the Terms of Use.