The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.01 - January/February (2008 vol.14)
pp: 120-134
ABSTRACT
<p><b>Abstract</b>—Information Visualization (InfoVis) is now an accepted and growing field but questions remain about the best uses for and the maturity of novel visualizations. Usability studies and controlled experiments are helpful but generalization is difficult. We believe that the systematic development of benchmarks will facilitate the comparison of techniques and help identify their strengths under different conditions. We were involved in the organization and management of three information visualization contests for the 2003, 2004 and 2005 IEEE Information Visualization Symposia, which requested teams to report on insights gained while exploring data. We give a summary of the state of the art of evaluation in information visualization, describe the three contests, summarize their results, discuss outcomes and lessons learned, and conjecture the future of visualization contests. All materials produced by the contests are archived in the Information Visualization Benchmark Repository.</p>
INDEX TERMS
Visualization, information, competition, contest, benchmark, repository, measure, metrics
CITATION
Catherine Plaisant, Jean-Daniel Fekete, Georges Grinstein, "Promoting Insight-Based Evaluation of Visualizations: From Contest to Benchmark Repository", IEEE Transactions on Visualization & Computer Graphics, vol.14, no. 1, pp. 120-134, January/February 2008, doi:10.1109/TVCG.2007.70412
REFERENCES
[1] A. Ahmed, T. Dwyer, C. Murray, L. Song, and Y.X. Wu, “WilmaScope,” Poster Compendium of IEEE Information Visualization, 2004.
[2] D. Auber, M. Delest, J.-P. Domenger, P. Ferraro, and R. Strandh, “EVAT—Environment for Visualization and Analysis of Trees,” Poster Compendium of IEEE Information Visualization, 2003.
[3] Proc. Workshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV '06), http://www.dis.uniroma1. it~beliv06/, 2007.
[4] Information Visualization Benchmark Repository, www.cs.umd. edu/hcilInfovisRepository, 2007.
[5] InfoVis CyberInfrastructure, http:/iv.slis.indiana.edu, 2007.
[6] K. Börner, L. Dall'Asta, W. Ke, and A. Vespignani, “Studying the Emerging Global Brain: Analyzing and Visualizing the Impact of Co-Authorship Teams,” Complexity, special issue on understanding complex systems, vol. 10, no. 4, pp. 58-67, 2005.
[7] Proc. Sixth Int'l Conf. Critical Assessment of Microarray Data Analysis (CAMDA '06), http://www.camda.duke.educamda06, 2006.
[8] Council of European Social Science Data Archives (CESSDA), http://www.nsd.uib.nocessda, 2007.
[9] “Introduction to the Special Issue on Empirical Evaluation of Information Visualizations,” Int'l J. Human-Computer Studies, C.Chen and M. Czerwinski, eds., vol. 53, no. 5, pp. 631-635, 2000.
[10] C. Chen, “Citation and Co-Citation Perspective,” Poster Compendium of IEEE Information Visualization, 2004.
[11] J. Chen, D. Guo, and A.M. MacEachren, “Space-Time-Attribute Analysis and Visualization of US Company Data,” Poster Compendium of IEEE Information Visualization, 2005.
[12] J. Chen, A.M. MacEachren, and D. Guo, “Visual Inquiry Toolkit—An Integrated Approach for Exploring and Interpreting Space-Time, Multivariate Patterns,” Proc. AutoCarto '06, CD-ROM, June 2006.
[13] J. Chen, A.M. MacEachren, and D. Guo, “Supporting the Process of Exploring and Interpreting Space-Time, Multivariate Patterns: The Visual Inquiry Toolkit,” Cartography and Geographic Information Science, to be published.
[14] N. Chinchor and L. Hirschman, “Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3),” Computational Linguistics, vol. 19, no. 3, pp. 409-449, 1993.
[15] P. Cowley, L. Nowell, and J. Scholtz, “Glassbox: An Instrumented Infrastructure for Supporting Human-Interaction with Information,” Proc. 38th Ann. Hawaii Int'l Conf. System Sciences (HICSS '05), p. 296.3, 2005.
[16] M. Delest, T. Munzner, D. Auber, and J.-P. Domenger, “Tulip,” Poster Compendium of IEEE Information Visualization, 2004.
[17] S. Faisal, P. Cairns, and A. Blandford, “Subjective Information Visualisations,” Proc. Workshop Combining Visualisation and Interaction to Facilitate Scientific Exploration and Discovery, 2006.
[18] J.-D. Fekete, “The InfoVis Toolkit,” Proc. IEEE Symp. Information Visualization (INFOVIS '04), pp. 167-174, 2004.
[19] J. Gehre, P. Ginsparg, and J. Kleinburg, “Overview of the 2003 KDD Cup,” Proc. SIGKDD Explorations, vol. 5, no. 2, pp. 149-151, 2004.
[20] V. Gonzales and A. Kobsa, “Benefits of Information Visualization for Administrative Data Analysts,” Proc. Seventh Int'l Conf. Information Visualization (IV '03), pp. 331-337, 2003.
[21] G. Grinstein, P. Hoffman, R. Pickett, and S. Laskowski, “Benchmark Development for the Evaluation of Visualization for Data Mining,” Information Visualization in Data Mining and Knowledge Discovery, U. Fayyad, G. Grinstein, and A. Wierse, eds., pp. 129-176, Morgan Kaufmann, 2002.
[22] G. Grinstein, T. O'Connell, S. Laskowski, C. Plaisant, J. Scholtz, and M. Whiting, “VAST 2006 Contest: A Tale of Alderwood,” Proc. First IEEE Int'l Symp. Visual Analytics Science and Technology (VAST '06), pp. 215-216, 2006.
[23] D. Guo, J. Chen, A.M. MacEachren, and K. Liao, “A Visual Inquiry System for Space-Time and Multivariate Patterns (VIS-STAMP),” IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 6, pp.1461-1474, Nov./Dec. 2006.
[24] F. van Ham, “Technische Universiteit Eindhoven Contest Submission,” Poster Compendium of IEEE Information Visualization, 2004.
[25] H. Hofmann, H. Wickham, D. Cook, J. Sun, and C. Röttger, “Boom and Bust of Technology Companies at the Turn of the 21st Century,” Poster Compendium of IEEE Information Visualization, 2005.
[26] J.Y. Hong, J. D'Andries, M. Richman, and M. Westfall, “Zoomology: Comparing Two Large Hierarchical Trees,” Poster Compendium of IEEE Information Visualization, 2003.
[27] B. Hoser, M. Blume, J. Schröder, and M. Franke, “CPS- Company Positioning System: Visualizing the Economic Environment,” Poster Compendium of IEEE Information Visualization, 2005.
[28] H. Tzu-Wei, L.I. Farabaugh, D. McColgin, and K. Stamper, “MonkEllipse,” Poster Compendium of IEEE Information Visualization, 2004.
[29] J.-D. Fekete, G. Grinstein, and C. Plaisant, 10th IEEE Symp. Information Visualization (InfoVis '04) Contest, www.cs.umd.edu/hciliv04contest, 2004.
[30] G. Grinstein, U. Cvek, M. Derthick, and M. Trutschl, Proc. 11th IEEE Symp. Information Visualization (InfoVis '05) Contest, Technology Data in the US, http://ivpr.cs.uml.eduinfovis05, 2005.
[31] http://sun.cs.lsus.eduiv06/, 2006.
[32] D. Keim, C. Panse, M. Sips, J. Schneidewind, and H. Barro Univ. of Konstanz Contest Submission, Poster Compendium of IEEE Information Visualization, 2004.
[33] D. Keim, R.D. Bergeron, and R. Pickett, “Test Datasets for Evaluating Data Visualization Techniques,” Perceptual Issues in Visualization, G. Grinstein and H. Levkowitz, eds., pp. 9-22, Springer, 1995.
[34] A. Komlodi, A. Sears, and E. Stanziola, ”InformationVisualization Evaluation Rev.,” ISRC Technical Report UMBC-ISRC-2004-1, Dept. of Information Systems, Univ. of Maryland, Baltimore County, http://www.research.umbc.edu/~komlodi IV_eval , 2004.
[35] L. Bongshin, M. Czerwinski, G. Robertson, and B.B. Bederson, “PaperLens,” Poster Compendium of IEEE Information Visualization, 2004.
[36] L. Xia, J. Buzydlowski, and H.D. White, “Associative Information Visualizer,” Poster Compendium of IEEE Information Visualization, 2004.
[37] S. Laskowski and C. Plaisant, “Evaluation Methodologies for Visual Analytics,” Illuminating the Path—the Research and Development Agenda for Visual Analytics, J. Thomas and K. Cook, eds., pp.150-157, IEEE Press, 2005.
[38] J. Mackinlay, “Automating the Design of Graphical Presentations of Relational Information,” ACM Trans. Graphics, vol. 5, no. 2, pp.110-141, 1986.
[39] R. Minghim, F. Paulovich, and A. Lopes, “Content-Based Text Mapping Using Multi-Dimensional Projections for Exploration of Document Collections,” Proc. Conf. Visualization and Data Analysis (VDA '06), 2006.
[40] Univ. of California, Irvine, Machine Learning Repository, http://www.ics.uci.edu/~mlearnMLRepository.html , 2007.
[41] D.R. Morse, N. Ytow, D.M. Roberts, and A. Sato, “Comparison of Multiple Taxonomic Hierarchies Using TaxoNote,” Poster Compendium of IEEE Information Visualization, 2003.
[42] K. Mullet, C. Fry, and D. Schiano, “On Your Marks, Get Set, Browse! (The Great CHI '97 Browse Off),” Panel Description in ACM Conf. Human Factors in Computing Systems (CHI '97) Extended Abstracts, pp. 113-114, 1997.
[43] J. Slack, T. Munzner, and F. Guimbretiere, “TreeJuxtaposer: InfoVis03 Contest Entry,” Poster Compendium of IEEE Information Visualization, 2003.
[44] T. Munzner, F. Guimbretière, S. Tasiran, L. Zhang, and Y. Zhou, “TreeJuxtaposer: Scalable Tree Comparison Using Focus+Context with Guaranteed Visibility,” ACM Trans. Graphics—Proc. ACM SIGGRAPH '03, pp. 453-462, 2003.
[45] North American Industry Classification System, www.census. gov/epcd/wwwnaics.html, 2007.
[46] New York Times—Election 2004, http://www.nytimes.com/packages/html/politics/ 2004_ELECTIONGUIDE_GRAPHIC?oref=login , (retrieved June 2005).
[47] D. Pallett, J. Garofolo, and J. Fiscus, “Measurement in Support of Research Accomplishments,” Comm. ACM, vol. 43, no. 2, pp. 75-79, 2000.
[48] C. Plaisant, J. Grosjean, and B.B. Bederson, “SpaceTree: Supporting Exploration in Large Node-Link Tree: Design Evolution and Empirical Evaluation,” Proc. IEEE Symp. Information Visualization (InfoVis '02), pp. 57-64, 2002.
[49] C. Plaisant, “The Challenge of Information Visualization Evaluation,” Proc. Working Conf. Advanced Visual Interfaces (AVI '04), pp.109-116, 2004.
[50] P. Saraiya, C. North, and K. Duca, “An Evaluation of Microarray Visualization Tools for Biological Insight,” Proc. IEEE Symp. Information Visualization (InfoVis '04), pp. 1-8, 2004.
[51] J. Scholtz, L. Arnstein, M. Kim, T. Kindberg, and S. Consolvo, User-Centered Evaluations of Ubicomp Applications, Technical Report IRS-TR-02-006, Intel Corp., May 2002.
[52] J. Scholtz and M.P. Steves, “A Framework for Evaluating Collaborative Systems in the Real World,” Proc. 38th Hawaii Int'l Conf. System Sciences (HICSS '05), 2005.
[53] N. Sheth, K. Börner, J. Baumgartner, K. Mane, and E. Wernert, “Treemap, Radial Tree, and 3D Tree Visualizations,” Poster Compendium of IEEE Information Visualization, 2003.
[54] B. Shneiderman and C. Plaisant, “Strategies for Evaluating Information Visualization Tools: Multidimensional In-Depth Long-Term Case Studies,” Proc. Workshop BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization (BELIV '06), pp. 38-43, 2006.
[55] Smart Money Map of the Market, www.smartmoney.com, (retrieved June 2005).
[56] M. Spenke and C. Beilken, “Analysing Formula One Racing Results with an Interactive Data Mining and Visualization Tool,” Data Mining II, N. Ebecken, ed., pp. 455-464, 2000.
[57] S.T. Teoh and K.-L. Ma One-for-All—Univ. of California, Davis, Contest Submission, Poster Compendium of IEEE Information Visualization, 2004.
[58] J. Trafton, T. Tsui, R. Miyamoto, J. Ballas, and P. Raymond, “Turning Pictures into Numbers: Extracting and Generating Information from Complex Visualizations,” Int'l J. Human Computer Studies, vol. 53, no. 5, pp. 827-850, 2000.
[59] J. Thomas and K. Cook, Illuminating the Path: The Research and Development Agenda for Visual Analytics, CS Press, http://nvac.pnl.govagenda.stm, 2005.
[60] Proc. Text REtrieval Conf. (TREC), http:/trec.nist.gov, 2007.
[61] J. Tyman, G.P. Gruetzmacher, and J. Stasko, “InfoVisExplorer,” Poster Compendium of IEEE Information Visualization, 2004.
[62] A.R. Unwin, M. Theus, and H. Hofmann, Graphics of Large Datasets. Springer, 2006.
[63] www.cs.umd.edu/hcilVASTcontest06, 2006.
[64] E. Voorhees and D. Harman, “Overview of the Sixth Text Retrieval Conf. (TREC-6),” Information Processing and Management, vol. 36, pp. 3-35, 2000.
[65] W. Ke, K. Borner, and L. Viswanath, “Indiana Univ. Contest Submission,” Poster Compendium of IEEE Information Visualization, 2004.
[66] M.A. Whiting, W. Cowley, J. Haack, D. Love, S. Tratz, C. Varley, and K. Wiessner, “Threat Stream Data Generator: Creating the Known Unknowns for Test and Evaluation of Visual Analytics Tools,” Proc. Workshop BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization (BELIV '06), pp. 1-3, 2006.
[67] W.P. Chung, B. Hetzler, C. Posse, M. Whiting, S. Havre, N. Cramer, A. Shah, M. Singhal, A. Turner, and J. Thomas, “IN-SPIRE,” Poster Compendium of IEEE Information Visualization, 2004.
[68] Z. Annerose, S. Potapov, M. Theus, and A. Unwin, “Analyzing Company Data with Interactive Statistical Graphics,” Poster Compendium of IEEE Information Visualization, 2005.
[69] C. North, “Visualization Viewpoints: Toward Measuring Visualization Insight,” IEEE Computer Graphics and Applications, vol. 26, no. 3, pp. 6-9, May/June 2006.
[70] J.-D. Fekete and C. Plaisant InfoVis 2003 Contest, www.cs.umd. edu/hciliv03contest, 2003.
5 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool