This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Ginger2: An Environment for Computer-Aided Empirical Software Engineering
July/August 1999 (vol. 25 no. 4)
pp. 474-492

Abstract—Empirical software engineering can be viewed as a series of actions to obtain knowledge and a better understanding about some aspects of software development given a set of problem statements in the form of issues, questions or hypotheses. Our experience in conducting empirical software engineering from a variety of viewpoints for the last decade has made us aware of the criticality of integrating the various types of data that are collected and analyzed as well as the criticality of integrating the various types of activities that take place such as experiment design and the experiment itself. This has led us to develop a Computer-Aided Empirical Software Engineering (CAESE) framework as a substrate for supporting the empirical software engineering lifecycle. CAESE supports empirical software engineering in the same manner as a CASE environment serves as a substrate for supporting the software development lifecycle. This paper first presents the CAESE framework that consists of three elements. The first element is a process model for the “lifecycle” of empirical software engineering studies, including needs analysis, experiment design, actual experimentation, and analyzing and packaging results. The second element is a model that helps empirical software engineers decide how to look at the “world” to be studied in a coherent manner. The third element is an architecture based on which CAESE environments can be built, consisting of tool sets for each phase of the process model, a process management mechanism, and the two types of integration mechanism that are vital for handling multiple types of data: data integration and control integration. The second half of this paper describes the Ginger2 environment as an instantiation of our framework. The paper concludes with reports on case studies using Ginger2, which dealt with a variety of empirical data types including mouse and keystrokes, eye traces, three-dimensional movement, skin resistance level, and video-taped data.

[1] V.R. Basili and H.D. Rombach, "The TAME Project: Towards Improvement-Oriented Software Environments," IEEE Trans. Software Eng., Vol. 14, No. 6, 1988, pp. 758-773.
[2] V.R. Basili, “The Experience Factory and its Relationship to Other Improvement Paradigms,” Proc. Fourth European Software Eng. Conf. (ESEC), Sept. 1993.
[3] V.R. Basili, “The Role of Experimentation: Past, Current, and Future,” Proc. 18th Int'l Conf. Software Eng., pp. 442–450, IEEE Computer Society, 1996.
[4] A. Brooks, L. Scott, and S. Takada, “Evaluating the Application of Software Metrics to Data Flow Diagrams and Class Diagrams in Usability Laboratory Experiments (CADPRO Pilot #1),” Nara Inst. of Science and Technology, Information Science Technical Report, NAIST-IS-TR98010, 1998.
[5] J. Daly, A. Brooks, J. Miller, M. Roper, and M. Wood, “Evaluating Inheritance Depth on the Maintainability of Object-Oriented Software,” Empirical Software Eng., vol. 1, no. 2, pp. 109–132, 1996.
[6] D. Day, M. Ahuja, and L. Scott, “Constraints in Design Engineering: A Report on Research in Progress,” Technical Report 97/5, Centre for Advanced Empirical Software Research, The Univ. of New South Wales, 1997.
[7] T. Furuyama, Y. Arai, and K. Iio, “Fault Generation Model and Mental Stress Effect Analysis,” J. Systems and Software, vol. 26, no. 1, pp. 31–42, 1994.
[8] P. Fusaro and F. Lanubile, “A Replicated Experiment to Assess Requirements Inspection Techniques,” Empirical Software Eng., vol. 2, no. 1, pp. 39–57, 1997.
[9] W. Harrison and V.R. Basili, “Editorial,” Empirical Software Eng. vol. 1, no. 1, pp. 5–10, 1966.
[10] A. Ito, S. Morisaki, A. Monden, K. Matsumoto, and K. Torii, “An Experimental Evaluation of 2-person Debugging,” Technical Report of IEICE, KBSE, 1999. in Japanese
[11] E. Kamsties and C. Lott, “An Empirical Evaluation of Three Defect Detection Techniques,” Proc. Fifth European Software Eng. Conf., 1995.
[12] S. Kusumoto, K. Matsumoto, T. Kikuno, and K. Torii, “Experimental Evaluation of the Cost Effectiveness of Software Reviews,” Proc. 15th Int'l Computer Software and Applications Conf., pp. 424–429, 1991.
[13] C. Lott and H.D. Rombach, “Repeatable Software Engineering Experiments for Comparing Defect Detection Techniques,” Empirical Software Eng., vol. 1, no. 3, pp. 241–277, 1996.
[14] K. Matsumoto, K. Inoue, H. Kudo, Y. Sugiyama, and K. Torii, “Error Life Span and Programmer Performance,” Proc. 11th Int'l Computer Software and Applications Conf., pp. 259–265, 1987.
[15] K. Matsumoto, K. Inoue, T. Kikuno, and K. Torii, “Experimental Evaluation of Software Reliability Growth Models,” Proc. 18th Int'l Symp. Fault Tolerant Computing, pp. 148–153, 1988.
[16] K. Matsumoto, S. Kusumoto, T. Kikuno, and K. Torii, “An Experimental Evaluation of Team Performance in Program Development Based on Model—Extension of Programmer Performance Model,” J. Information Processing, vol. 15, no. 3, pp. 466–473, 1992.
[17] A. Monden, Y. Takada, and K. Torii, “An Experiment to Observe Debugging Process by Using an Eye Tracking Device,” Technical Report of IEICE, SS96-5, 1996. in Japanese
[18] S. Morisaki, K. Nakakoji, S. Takada, K. Matsumoto, and K. Torii, “AID-SIM: Architecture for Integrating Distributed Software Development Simulators,” Proc. Int'l Symp. Future Software Technology, pp. 95–100, 1998.
[19] T. Muragishi, “Research on the Assessment of Software Developers' Loads Utilizing Skin Resistance Levels,” masters thesis, Nara Inst. of Science and Tech nology, 1998. in Japanese
[20] Y. Nakashimada, S. Morisaki, A. Monden, K. Matsumoto, and K. Torii, “A Video Recording Retrieval System Supporting Analysis of Software Development Process,” Technical Report of IEICE, SS96-61 1997. in Japanese
[21] F. Osawa, K. Shima, K. Matsumoto, and K. Torii, “A Method for Searching Characteristic Behavior Based on Software Developers' Movement,” Technical Report of IEICE, SS96-62, 1997. in Japanese
[22] A.A. Porter, L.G. Votta, and V.R. Basili, "An Experiment to Assess Different Defect Detection Methods for Software Requirements Inspections," Proc. 16th Int'l Conf. Software Eng., 1994, pp. 103-112.
[23] P. Robillard, P. d'Astous, F. Detienne, and W. Visser, “Measuring Cognitive Activities in Software Engineering,” Proc. 20th Int'l Conf. Software Eng., pp. 292–300, 1998,
[24] S.E. Sim and R.C. Holt, “The Ramp-Up Problem in Software Projects: A Case Study of How Software Immigrants Naturalize,” Proc. 20th Int'l Conf. Software Eng., pp. 361–370, 1998.
[25] C. Sugahara, “Making and Evaluation of a Guideline of Experimental Usability Evaluation,” masters thesis, Nara Inst. of Science and Tech nology, 1998. in Japanese
[26] Y. Takada, K. Matsumoto, and K. Torii, “A Programmer Performance Measure Based on Programmer State Transitions in Testing and Debugging Process,” Proc. 16th Int'l Conf. Software Eng., pp. 123–132, 1994.
[27] S. Uchida, H. Kudo, and A. Monden, “An Experiment and an Analysis of Debugging Process with Periodic Interviews,” Proc. Software Symp., pp. 53–58, 1998.
[28] I. Thomas, "PCTE Interfaces: Supporting Tools in Software-Engineering Environments," IEEE Software, vol. 6, no. 6, pp. 15-23, Nov. 1989.
[29] K. Torii and K. Matsumoto, “Quantitative Analytic Approaches in Software Engineering,” Information and Software Technology, no. 38, pp. 15–3, 1996.
[30] G. Visaggio, “Assessing the Maintenance Process through Replicated, Controlled Experiment,” ISERN Technical Report, ISERN-97-27, 1996.
[31] L. Votta and A. Porter, “Experimental Software Engineering: A Report on the State of the Art,” Proc. 17th Int'l Conf. Software Eng., pp. 277–279, 1995.
[32] M. Yanagi, A. Monden, Y. Takada, and K. Torii, “A Tool Detecting Patterns of Programmers' Behavior when They Likely Inject Bugs,” Proc. 17th Int'l Conf. Software Eng., pp. 277–279, 1995.

Index Terms:
Computer-aided empirical software engineering (CAESE), empirical software engineering lifecycle, in vitro studies, CAESE environments, monitoring, analysis, measurement, human behavior.
Citation:
Koji Torii, Ken-ichi Matsumoto, Kumiyo Nakakoji, Yoshihiro Takada, Shingo Takada, Kazuyuki Shima, "Ginger2: An Environment for Computer-Aided Empirical Software Engineering," IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 474-492, July-Aug. 1999, doi:10.1109/32.799942
Usage of this product signifies your acceptance of the Terms of Use.