The Community for Technology Leaders
RSS Icon
Issue No.01 - First Quarter (2013 vol.6)
pp: 46-57
Dawei Jia , Centre for Intell. Syst. Res., Deakin Univ., Geelong, VIC, Australia
A. Bhatti , Centre for Intell. Syst. Res. (CISR), Deakin Univ., Geelong, VIC, Australia
S. Nahavandi , Centre for Intell. Syst. Res. (CISR), Deakin Univ., Geelong, VIC, Australia
B. Horan , Sch. of Eng., Deakin Univ., Geelong, VIC, Australia
Virtual reality and simulation are becoming increasingly important in modern society and it is essential to improve our understanding of system usability and efficacy from the users' perspective. This paper introduces a novel evaluation method designed to assess human user capability when undertaking technical and procedural training using virtual training systems. The evaluation method falls under the user-centered design and evaluation paradigm and draws on theories of cognitive, skill-based and affective learning outcomes. The method focuses on user interaction with haptic-audio-visual interfaces and the complexities related to variability in users' performance, and the adoption and acceptance of the technologies. A large scale user study focusing on object assembly training tasks involving selecting, rotating, releasing, inserting, and manipulating three-dimensional objects was performed. The study demonstrated the advantages of the method in obtaining valuable multimodal information for accurate and comprehensive evaluation of virtual training system efficacy. The study investigated how well users learn, perform, adapt to, and perceive the virtual training. The results of the study revealed valuable aspects of the design and evaluation of virtual training systems contributing to an improved understanding of more usable virtual training systems.
Haptic interfaces, User centered design, Training, Ergonomics, Virtual reality, User interfaces,user interaction, Human haptics, user-centered design, virtual training evaluation, ergonomics
Dawei Jia, A. Bhatti, S. Nahavandi, B. Horan, "Human Performance Measures for Interactive Haptic-Audio-Visual Interfaces", IEEE Transactions on Haptics, vol.6, no. 1, pp. 46-57, First Quarter 2013, doi:10.1109/TOH.2012.41
[1] B. Horan et al., "Towards Haptic Microrobotic Intracellular Injection," Proc. IEEE/ASME Int'l Conf. Mechatronic and Embedded Systems and Applications, 2009.
[2] T.R. Coles, D. Meglan, and N.W. John, "The Role of Haptics in Medical Training Simulators: A Survey of the State of the Art," IEEE Trans. Haptics, vol. 4, no. 1, pp. 51-66, Jan./Feb. 2011.
[3] F. Crison et al., "Virtual Technical Trainer: Learning How to Use Milling Machines with Multi-Sensory Feedback in Virtual Reality," Proc. IEEE Conf. Virtual Reality, pp. 139-145, 2005.
[4] O. Goksel, K. Sapchuk, and S.E. Salcudean, "Haptic Simulator for Prostate Brachytherapy with Simulated Needle and Probe Interaction," IEEE Trans. Haptics, vol. 4, no. 3, pp. 188-198, May/June 2011.
[5] K.M. Stanney et al., "Usability Engineering of Virtual Environments (VEs): Identifying Multiple Criteria that Drive Effective VE System Design," J. Human-Computer Studies, vol. 58, pp. 447-481, 2003.
[6] R. Kennedy, K. Stanney, and W. Dunlap, "Duration and Exposure to Virtual Environments: Sickness Curves during and across Sessions," Presence: Teleoperators and Virtual Environments, vol. 9, no. 5, pp. 463-472, 2000.
[7] M.C. Salzman et al., "The Design and Evaluation of Virtual Reality-Based Learning Enviornments," Presence: Teleoperators and Virtual Environments, fanliangchen-huangjing/xxzy/sjkf/sj 9.pdf, 1993.
[8] D. Stone et al., User Interface Design and Evaluation. Morgan Kaufmann Publishers, 2005.
[9] Y.-L. Theng et al., "Mixed Reality Systems for Learning: A Pilot Study Understanding User Perceptions and Acceptance," Proc. 12th Int'l Conf. Human-Computer Interaction, 2007.
[10] T. Whalen, S. Noel, and J. Stewart, "Measuring the Human Side of Virtual Reality," Proc. IEEE Int'l Symp. Virtual Environments, Human-Computer Interfaces and Measurement Systems (VECIMS '03), 2003.
[11] M. Steffin, "Visual-Haptic Interfaces, Modification of Motor and Cognitive Performance," , 2009.
[12] K.S. Hale, K.M. Stanney, and L. Malone, "Enhancing Virtual Environment Spatial Awareness Training and Transfer through Tactile and Vestibular Cues. Ergonomics," vol. 52, no. 2, pp. 187-203, 2009.
[13] A. Bicchi et al., "An Atias of Physical Human-Robot Interaction," J. Mechanism and Machine Theory, vol. 43, pp. 253-270, 2008.
[14] S. Ricciardi et al., "Dependability Issues in Visual-Haptic Interfaces," J. Visual Languages and Computing, vol. 21, pp. 33-40, 2010.
[15] W. Winn, "Learning in Artifical Environments: Embodiment, Embeddedness and Dynamic Adaption," Technology, Instruction, Cognition & Learning, vol. 1, no. 1, pp. 87-114, 2002.
[16] R.A. Burkhard, "Towards a Framework and a Model for Knowledge Visualization: Synergies between Information and Knowledge Visualization," Knowledge and Information Visualization, S.O. Tergan and T. Keller, ed., pp. 238-255, Springer-Verlag, 2005.
[17] G.A. Miller, "The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information," Psychological Rev., vol. 63, pp. 81-97, 1956.
[18] J.F. Jensen, "Virtual Inhabited 3D Worlds: Interactivity and Interaction between Avatars, Autonomous Agents and USers," Virtual Interaction: Interaction in Virtual Inhabited 3D Worlds, L. Ovortrup, ed., pp. 23-47, Springer-Verlag, 2001.
[19] M. De Marsico and G. Vitiello, "Introduction to the Special Issue on Multimodal Interaction through Haptic Feedback," J. Visual Languages and Computing, vol. 20, no. 5, pp. 285-286, 2009.
[20] M. Tavakoli, R.V. Patel, and M. Moallem, "A Haptic Interface for Computer-Integrated Endoscopic Surgery and Training Virtual Reality," vol. 9, pp. 160-176, 2006.
[21] S. Okamoto et al., "Virtual Active Touch: Perception of Virtual Gratings Wavelength through Pointing-Stick Interface," IEEE Trans. Haptics, vol. 5, no. 1, pp. 85-93, Jan.-Mar. 2012.
[22] Y.H. Yoo and W. Bruns, "Motor Skill Learning with Force-feedback in Mixed Reality," Proc. 16th Int'l Federation of Automatic Control (IFAC) World Congress, 2005.
[23] A. Seth, H.-J. Su, and J.M. Vance, "Development of a Dual-Handed Haptic Assembly System: SHARP," J. Computing and Information Science in Eng., vol. 8, no. 4, pp. 044-502, 2008.
[24] A. Tang et al., "Comparative Effectiveness of Augmented Reality in Object Assembly," Computer Human Interaction, vol. 5, no. 1, pp. 73-80, 2003.
[25] A. Bhatti et al., "Haptically Enable Interactive Virtual Assembly Training System Development and Evalaution," Proc. Ann. Simulation Technology and Training Conf. (SimTecT '09). 2009.
[26] P. Boulanger et al., "Hapto-Audio-Visual Environments for Collaborative Training of Ophthalmic Surgery over Optical Network," Proc. IEEE Int'l Workshop Haptic Audio Visual Environments and their Applications (HAVE), 2006.
[27] H. Bleuler et al., "Generic and Systematic Evaluation of Haptic Interfaces Based on Testbeds," Proc. IEEE/RSJ Int'l Conf. Intelligent Robots and Systems, 2007.
[28] S. Ricciardi et al., "Dependability Issues in Visual-Haptic Interfaces," J. Visual Languages and Computing, vol. 21, no. 1, pp. 33-40, 2010.
[29] A. Sutcliffe, B. Gault, and J.-E. Shin, "Presence, Memory and Interaction in Virtual Environments," Int'l J. Human-Computer Studies, vol. 62, no. 3, pp. 307-327, 2005.
[30] D.M. Popovici and A.-M. Marhan, "Virtual Reality-Based Environments for Learning and Training," Product Engineering: Tools and Methods Based on Virtual Reality, D. Talaba and A. Amditis, ed., pp. 123-142, Springer, 2008.
[31] J.-W.J. Lin, "User Experience Modeling and Enhancement for Virtual Environment that Employ Wide-Field Displays," Digital Human Modeling, V.G. Duffy, ed., pp. 423-433, Springer, 2007.
[32] Y.L. Theng et al., "Mixed Reality System for Learning: A Pilot Study Understanding User Perceptions and Acceptance," Proc. Second Int'l Conf. Virtual Reality, 2007.
[33] J.L. Gabbard, D. Hix, and J.E. Swan, "User-Centered Design and Evaluation of Virtual Environments," IEEE Computer Graphics and Applications, vol. 19, no. 6, pp. 51-59, Nov./Dec. 1999.
[34] D.A. Bowman et al., 3D User Interfaces: Theory and Practice. Addison-Wesley/Pearson Education, 2004.
[35] A. Seffah et al., "Usability Measurement and Metrics: A Consolidated Model," Software Quality J., vol. 14, no. 2, pp. 159-178, 2006.
[36] K. Kraiger, J.K. Ford, and E. Salas, "Application of Cognitive, Skill-Based, and Affective Theories of Learning Outcomes to New Methods of Training Evaluation," J. Applied Psychology, vol. 78, no. 2, pp. 311-328, 1993.
[37] M.C. Salzman et al., "A Model for Understanding How Virtual Reality Aids Complex Conceptual Learning," Presence: Teleoperators and Virtual Environments, vol. 8, no. 3, pp. 293-316, 1999.
[38] M. Good et al., "User-Derived Impact Analysis as a Tool for Usability Engineering," Proc. SIGCHI Conf. Human Factors in Computing Systems, 1986.
[39] G. Lindgaard, Usability Testing and System Evaluation. Chapman & Hall, 1994.
[40] B. Riecke et al., "Cognitive Factors Can Influence Self-Motion Perception (Vection) in Virtual Reality," ACM Trans. Applied Perception, vol. 3, no. 3, pp. 194-216, 2006.
[41] E. Hudicka, "To Feel or Not to Feel: The Role of Affect in Human-Computer Interaction," Int'l J. Human-Computer Studies, vol. 59, pp. 1-32, 2003.
[42] C. Coelho et al., "Media Presence and Inner Presence: The Sense of Presence in Virtual Reality Technologies," From Communication to Presence: Cognition, Emotions and Culture Towards the Ultimate Communicative Experience, G. Riva, et al., ed., IOS Press, 2006.
[43] J.-W.J. Lin, Enhancement of User-Experiences in Immersive Virtual Environments that Employ Wide-Field Displays, p. 207. Univ. of Washington, 2004.
[44] D.A. Bowman, J.L. Gabbard, and D. Hix, "A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods," Presence: Teleoprators & Virtual Environments, vol. 11, no. 4, pp. 404-424, 2002.
[45] M.M. North et al., "Virtual Reality and Transfer of Learning," Usability Evaluation and Interface Design: Cognitive Engineering, Intelligent Agents and Virtual Reality, J.S. Michael, et al., ed., pp. 634-638, Lawrence Erlbaum Assoc., 2001.
[46] S.E. Mead and A.D. Fisk, "Measuring Skill Acquisition and Retention with an ATM Simulator: The Need for Age-Specific Training," Human Factors, vol. 40, pp. 516-523, 1998.
[47] L. Liu, B. Watson, and M. Miyazaki, "VR for the Elderly: Quantitative and Qualitative Differences in Performance with a Driving Simulator," Cyberpschology & Behavior, vol. 2, no. 6, pp. 567-576, 1999.
[48] X.D. Yang, W.F. Bischof, and P. Boulanger, "Validating the Perofrmance of Haptic Motor Skill Training," Proc. Symp. Haptic Interfaces for Virtual Environments and Teleoprator Systems, pp. 129-135, 2008.
[49] D. Jia, A. Bhatti, and N. Saeid, "The Impact of Self-Efficacy and Perceived System Efficacy on Effectiveness of Virtual Training Systems," J. Behaviour and Information Technology, http://www. 0144929X.2012.681067, 2012.
[50] D. Jia et al., "User-Centred Evaluation of a Virtual Environment Training System: Utility of User Perception Measures," Proc. Third Int'l Conf. Virtual and Mixed Reality, R. Shumaker, ed., pp. 196-205, 2009.
[51] D. Jia, A. Bhatti, and S. Nahavandi, "MUSTe Method for Quantifying Virtual Environment Training System Efficacy," Proc. Ann. Conf. Australian Computer-Human Interaction Special Interest Group (CHISIG) of the Human Factors and Ergonomics Soc. of Australia (HFESA) (OZCHI '09), 2009.
[52] D. Jia, A. Bhatti, and S. Nahavandi, "Design and Evaluation of a Haptically Enable Virtual Environment for Object Assembly Training," Proc. IEEE Int'l Workshop Haptic Audio-Visual Environments and Games (HAVE '09), 2009.
[53] J. Vince, Virtual Reality Systems. Pearson Education, 1995.
[54] V.W. Caracelli and J.C. Greene, "Data Analysis Strategies for Mixed-Method Evaluation Designs," Educational Evaluation and Policy Analysis, vol. 15, no. 2, pp. 195-207, 1993.
[55] A.M. Huberman and M.B. Miles, "Data Management and Analysis Methods," Handbook of Qualitative Research, N.K. Denzin and Y.S. Lincoln, ed., pp. 428-444, Sage, 1994.
[56] M. Sandelowski, C.L. Voils, and G. Knafl, "On Quantitizing Journal of Mixed Methods Research," vol. 3, no. 3, pp. 208-222, 2009.
[57] K. Kelkar et al., "The Added Usefulness of Process Measures over Performance Measures in Interface Design," Int'l J. Human-Computer Interaction, vol. 18, no. 1, pp. 1-18, 2005.
[58] R.M. Furr and V.R. Bacharach, Psychometrics: An Introduction. Sage Publications, 2008.
[59] J. Pallant, "Development and Validation of a Scale to Measrue Perceived Control of Internal States," J. Personality Assessment, vol. 75, no. 2, pp. 308-37, 2000.
[60] R.F. DeVellis, Scale Development: Theory and Applications, second ed. Sage Publications, 2003.
[61] S. Okamoto, M. Konyo, and S. Tadokoro, "Vibrotactile Stimuli Applied to Finger Pads as Biases for Perceived Inertial and Viscous Loads," IEEE Trans. Haptics, vol. 4, no. 4, pp. 307-315, 2011.
[62] G. Borghesan, A. Macchelli, and C. Melchiorri, "Interconnection and Simulation Issues in Haptics," IEEE Trans. Haptics, vol. 3, no. 4, pp. 266-279, Oct.-Dec. 2010.
[63] E.L.M. Su et al., "Effect of Grip Force and Training in Unstable Dynamics on Micromanipulation Accuracy," IEEE Trans. Haptics, vol. 4, no. 3, pp. 167-174, May/June 2011.
[64] G.D. Lecakes et al., "Virtual Reality Environments for Integrated Systems Health Management of Rocket Engine Tests," IEEE Trans. Instrumentation and Measurement, vol. 58, no. 9, pp. 3050-3057, Sept. 2009.
[65] A Guide to Usability: Human Factors in Computing, J. Preece, ed. Addison Wesley, 1993.
[66] C.P. Nemeth, Human Factors Method for Design: Making Systems Human-Centered, B. Raton, ed. CRC Press, 2004.
[67] A. Crystal and B. Ellington, "Task Analysis and Human-Computer Interaction: Approaches, Techniques, and Levels of Analysis," Proc. 10th Am. Conf. Information Systems, 2004.
[68] D. Hix et al., "User-Centered Design and Evalaution of a Real-Time Battlefield Visualization Virtual Environment," Proc. IEEE Virtual Reality '99, 1999.
61 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool