This Article 
 Bibliographic References 
 Add to: 
Evaluating Emerging Software Development Technologies: Lessons Learned from Assessing Aspect-Oriented Programming
July/August 1999 (vol. 25 no. 4)
pp. 438-455

Abstract—Determining whether a new software development technique is useful and usable is a challenging task. Various flavors of empirical study may be used to help with this task, including surveys, case studies, and experiments. Little guidance is available within the software engineering community to help choose among these alternatives when assessing a new and evolving software development technique within some cost bounds. We faced this challenge when assessing a new programming technique called aspect-oriented programming. To assess the technique, we chose to apply both a case study approach and a series of four experiments because we wanted to understand and characterize the kinds of information that each approach might provide. In this paper, we describe and critique the evaluation methods we employed, and discuss the lessons we have learned. These lessons are applicable to other researchers attempting to assess new programming techniques that are in an early stage of development.

[1] S. Lawrence Pfleeger, “Design and Analysis in Software Engineering, Part 1: The Language of Case Studies and Formal Experiments,” ACM SIGSOFT Software Engineering Notes, vol. 19, no. 4, pp. 16–20, Oct. 1994.
[2] M. Zelkowitz and D. Wallace, “Experimental Models for Validating Technology,” Computer, vol. 31, no. 5, pp. 23–31, May 1998.
[3] S.A. Lee, “A Scientific Methodology for MIS Case Studies,” MIS Quarterly, vol. 13, no. 1, pp. 33–50, 1989.
[4] E.A. Trahan and L.J. Gitman, “Bridging the Theory-Practice Gap in Corporate Finance: A Survey of Chief Financial Officers,” Quarterly Review of Economics and Finance, vol. 35, no. 1, pp. 73–88, 1995.
[5] S. Ravden and G. Johnson, Evaluating Usability of Human-Computer Interfaces: A Practical Method. Chichester, England: Ellis Hornwood Ltd., 1989.
[6] J. Preece, “Human-Computer Interaction,” Interaction Design: Evaluation, ch. Part VI. Wokingham, England: Addison-Wesley, 1994.
[7] B. Curtis, S.B. Sheppard, E. Kruesi-Bailey, J. Bailey, and D.A. Boehm-Davis, “Experimental Evaluation of Software Documentation Formats,” J. Systems and Software, vol. 9, no. 2, pp. 167–207, Feb. 1989.
[8] A. Porter, H. Siy, C. Toman, and L. Votta, “An Experiment to Assess the Cost-Benefits of Code Inspections in Large Scale Software Development,” IEEE Trans. Software Eng., vol. 23, no. 6, pp. 329–346, June 1997.
[9] M.D. Storey, K. Wong, P. Fong, D. Hooper, K. Hopkins, and H.A. Müller, “On Designing an Experiment to Evaluate a Reverse Engineering Tool,” Proc. Third Working Conf. Reverse Eng., pp. 31–40, IEEE CS Press, 1996.
[10] G. Kiczales, J. Lamping, A. Mendhekar, C. Maeda, C. Lopes, J.-M Loingtier, and J. Irwin, “Aspect-Oriented Programming,” Proc. 11th European Conf., Object-Oriented Programming, ECOOP'97, pp. 220–242, Lecture Notes in Computer Science, vol. 1,241, Springer-Verlag, June 1997.
[11] V.R. Basili, “The Role of Experimentation: Past, Current, and Future,” Proc. 18th Int'l Conf. Software Eng., pp. 442–450, IEEE Computer Society, 1996.
[12] R.J. Walker, E.L.A. Baniassad, and G.C. Murphy, “An Initial Assessment of Aspect-Oriented Programming,” Proc. 21st Int'l Conf. Software Eng., ACM, pp. 120–130, May 1999.
[13] R.K. Yin, Case Study Research: Design and Methods, second edition. Thousand Oaks, Calif.: Sage Publications, 1994
[14] B. Schneiderman, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Iterative Design, Testing, and Evaluation, ch. 10. Reading, Mass.: Addison-Wesley, 1987.
[15] J.E. McGrath, “Methodology Matters: Doing Research in the Behavioral and Social Sciences,” Readings in Human-Computer Interaction: Toward the Year 2000, second edition, R.M. Baecker, J. Grudin, and W.A.S. Buxton, eds., pp. 152–169. San Francisco: Morgan Kaufmann, 1995.
[16] V.R. Basili, R.W. Selby, and D.H. Hutchens, "Experimentation in Software Engineering," IEEE Trans. Software Eng., vol. 12, pp. 733-743, 1986.
[17] S. Lawrence Pfleeger, “Experimental Design and Analysis in Software Engineering, Part 2: How to Set Up an Experiment,” ACM SIGSOFT Software Eng. Notes, vol. 20, no. 1, pp. 22–26, Jan. 1995.
[18] S. Lawrence Pfleeger, “Experimental Design and Analysis in Software Engineering, Part 3: Types of Experimental Design,” ACM SIGSOFT Software Eng. Notes, vol. 20, no. 2, pp. 14–16, Apr. 1995.
[19] S. Lawrence Pfleeger, “Experimental Design and Analysis in Software Engineering, Part 4: Choosing an Experimental Design,” ACM SIGSOFT Software Eng. Notes, vol. 20, no. 3, pp. 13–15, July 1995.
[20] J. Irwin, J.M. Loingtier, J.R. Gilbert, G. Kiczales, J. Lamping, A. Mendhekar, and T. Shpeisman, “Aspect-Oriented Programming OS Sparse Matrix Code,” Proc. Scientific Computing in Object-Oriented Parallel Environments First Int'l Conf., ISCOPE'97, pp. 249–256, Springer-Verlag, Dec. 1997.
[21] A. Mendhekar, G. Kiczales, and J. Lamping, “RG: A Case-Study for Aspect-Oriented Programming,” Technical Report SPL97-009 P9710044, Xerox PARC, Feb. 1997.
[22] Xerox PARC, “AspectJ Home Page,” http://www. parc. xerox. com/spl/projects/ aop/aspectj, 1998.
[23] J. Gosling, B. Joy, and G. Steele, The Java Language Specification, Addison-Wesley, Reading, Mass., 1996.
[24] G.C. Murphy and E.L.A. Baniassad, “Qualitative Case Study Results,” UBC-CS-SE-AOP-1, Oct. 1997.
[25] N.E. Fenton, S. Lawrence Pfleeger, and R. Glass, “Science and Substance: A Challenge to Software Engineers,” IEEE Software, vol. 11, no. 4, pp. 86–95, July 1994.
[26] A. Black, N. Hutchinson, E. Jul, and H. Levy, “Object Structure in the Emerald System,” ACM SIGPLAN Notices, vol. 21, no. 11 pp. 78–86, Nov. 1986.
[27] R.K. Raj, E. Tempero, H.M. Levy, A.P. Black, N.C. Hutchinson, and E. Jul, “Emerald: A General-Purpose Programming Language,” Software—Practice and Experience, vol. 21, no. 1, pp. 91–118, Jan. 1991.
[28] R. Guindon, H. Krasner, and B. Curtis, “Breakdowns and Processes During the Early Activities of Software Design by Professionals,” Empirical Studies of Programmers: Proc. Second Workshop G.M. Olson, S. Sheppard, and E. Soloway, eds., pp. 65–82, 1987.
[29] C. O'Malley, S. Draper, and M. Riley, “Constructive Interaction: A Method for Studying User-Computer-User Interaction,” Proc. First IFIP Conf. Human-Computer Interaction, INTERACT'84, vol. 2, pp. 1–5, Elsevier Science, 1984.
[30] D. Wildman, “Getting the Most from Paired-User Testing,” ACM Interactions, vol. 2, no. 3, pp. 21–27, 1995.
[31] K.E. Weick, Systematic Observational Methods, vol. 2, Research Methods, ch. 13, pp. 357–451, Addison-Wesley, 1968.
[32] R.W. Bowdidge and W.G. Griswold, “How Software Tools Organize Programmer Behavior During the Task of Data Encapsulation,” Empirical Software Eng., vol. 2, no. 3, pp. 221–267, 1997.
[33] B. Boehm, "A Spiral Model of Software Development and Enhancement," Computer, May 1988, pp. 61-72.
[34] A. von Mayrhauser and A.M. Mans, “Identification of Dynamic Comprehension Processes During Large Scale Maintenance,” IEEE Trans. Software Eng., vol. 22, no. 6, pp. 424–437, 1996.
[35] D.E. Perry, N.A. Staudenmayer, and L.G. Votta, "People, Organizations, and Process Improvement." IEEE Software, vol. 11, no. 4, pp. 36-45, 1994.
[36] M. Rosson and J.M. Carroll, “The Reuse of Uses in Smalltalk Programming,” ACM Trans. Computer-Human Interaction, vol. 3, no. 3, pp. 219–253, Sept. 1996.
[37] B. Curtis, H. Krasner, and N. Iscoe, "A Field Study of the Software Design Process for Large Systems," Comm. ACM, vol. 31, no. 11, pp. 1,268-1,287, 1988.

Index Terms:
Empirical study, software development technique, qualitative assessment, case study, experiment.
Gail C. Murphy, Robert J. Walker, Elisa L.A. Baniassad, "Evaluating Emerging Software Development Technologies: Lessons Learned from Assessing Aspect-Oriented Programming," IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 438-455, July-Aug. 1999, doi:10.1109/32.799936
Usage of this product signifies your acceptance of the Terms of Use.