Issue No.04 - Oct.-Dec. (2012 vol.11)
pp: 60-67
Antti Oulasvirta , Max Planck Institute for Informatics
Laboratory-based usability evaluations are often inadequate for the post-desktop-computer era. Field studies challenge us to design evaluations as quasi-experiments that don't assume perfect control and randomization. Such evaluations are expensive, but there are ways to reduce costs.
Usability, Mobile communication, Performance evaluation, Prototypes, Three dimensional displays, Context awareness, Laboratories, quasi-experimentation, usability evaluation, human-computer interaction, mobile applications, field evaluation
Antti Oulasvirta, "Rethinking Experimental Designs for Field Evaluations", IEEE Pervasive Computing, vol.11, no. 4, pp. 60-67, Oct.-Dec. 2012, doi:10.1109/MPRV.2011.84
1. J. Nielsen, Usability Engineering, Morgan Kaufmann, 1993.
2. J. Dumas, “The Great Leap Forward: The Birth of the Usability Profession (1988–1993),” J. Usability Studies, vol. 2, no. 2, 2007, pp. 54–60.
3. J. Kjeldskov, “Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field,” Proc. Mobile Human-Computer Interaction (MobileHCI 04), ACM, 2004, pp. 61–73.
4. J. Kjeldskov and C. Graham, “A Review of Mobile HCI Research Methods,” Proc. Mobile Human-Computer Interaction (MobileHCI 03), LNCS 2795, Springer, 2003, pp. 317–335.
5. G. Abowd and E. Mynatt, “Charting Past, Present, and Future Research in Ubiquitous Computing,” ACM Trans. Computer-Human Interaction, vol. 7, no. 1, 2000, pp. 29–58.
6. S. Carter et al., “Exiting the Cleanroom: On Ecological Validity and Ubiquitous Computing,” Human-Computer Interaction, vol. 23, no. 1, 2008, pp. 47–99.
7. Y. Rogers et al., “Why It's Worth the Hassle: A Case Study in Using In-Situ Studies during Design,” Proc. 9th Int'l Conf. Ubiquitous Computing (Ubicomp 07), LNCS 4717, Springer, 2007, pp. 336–353.
8. S. Consolvo et al., “Conducting In Situ Evaluations for and with Ubiquitous Computing Technologies,” Int'l J. Human-Computer Interaction, vol. 22, no. 1, 2007, pp. 107–122.
9. A. Morrison et al., “Like Bees Around the Hive: A Comparative Study of a Mobile Augmented Reality Map,” Proc. 27th Int'l Conf. Human Factors in Computing Systems (CHI 09), ACM, 2009, pp. 1889–1898.
10. W. Shadish, T. Cook, and D. Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Houghton-Mifflin, 2002.
11. A. Oulasvirta, S. Estlander, and A. Nurminen, “Embodied Interaction with a 3D versus 2D Mobile Map,” J. Personal and Ubiquitous Computing, vol. 13, no. 4, 2009, pp. 303–320.
12. A. Oulasvirta and J. Bergstrom-Lehtovirta, “A Simple Index for Multimodal Flexibility,” Proc. 28th Int'l Conf. Human Factors in Computing Systems (CHI 10), ACM, 2010, pp. 1475–1484.
13. A. Oulasvirta and J. Bergstrom-Lehtovirta, “Ease of Juggling: Studying the Effects of Manual Multitasking,” Proc. 2011 Ann. Conf. Human Factors in Computing Systems (CHI 11), ACM, 2011, pp. 3103–3112.
14. M. Rohs, A. Oulasvirta, and T. Suomalainen, “Interaction with Magic Lenses: Real-World Validation of a Fitts' Law Model,” Proc. 2011 Ann. Conf. Human Factors in Computing Systems (CHI 11), ACM, 2011, pp. 2725–2728.
15. A. Sellen et al., “Reflecting Human Values in the Digital Age,” Comm. ACM, vol. 52, no. 3, 2009, pp. 58–66.
16. M. Raento, A. Oulasvirta, and N. Eagle, “Smartphones: An Emerging Tool for Social Scientists,” Sociological Methods and Research, vol. 27, no. 3, 2009, pp. 221–237.
17. S. Neely et al., “Evaluating Pervasive and Ubiquitous Systems,” IEEE Pervasive Computing, vol. 7, no. 3, 2007, pp. 85–88.
18. F.H. Previc, “The Neuropsychology of 3D Space,” Psychological Bulletin, vol. 124, no. 2, 1998, p. 123.