Issue No. 02 - Feb. (2016 vol. 42)
Sira Vegas , Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Boadilla del Monte, Madrid, Spain
Cecilia Apa , Instituto de Computación, Facultad de Ingeniería, Montevideo, Uruguay
Natalia Juristo , Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Boadilla del Monte, Madrid, Spain
In experiments with crossover design subjects apply more than one treatment. Crossover designs are widespread in software engineering experimentation: they require fewer subjects and control the variability among subjects. However, some researchers disapprove of crossover designs. The main criticisms are: the carryover threat and its troublesome analysis. Carryover is the persistence of the effect of one treatment when another treatment is applied later. It may invalidate the results of an experiment. Additionally, crossover designs are often not properly designed and/or analysed, limiting the validity of the results. In this paper, we aim to make SE researchers aware of the perils of crossover experiments and provide risk avoidance good practices. We study how another discipline (medicine) runs crossover experiments. We review the SE literature and discuss which good practices tend not to be adhered to, giving advice on how they should be applied in SE experiments. We illustrate the concepts discussed analysing a crossover experiment that we have run. We conclude that crossover experiments can yield valid results, provided they are properly designed and analysed, and that, if correctly addressed, carryover is no worse than other validity threats.
Software engineering, Animals, Atmospheric measurements, Particle measurements, Psychology, US Government agencies, Information processing,carryover, Data analysis, crossover designs,carryover, Experimental software engineering, controlled experiment, data analysis, crossover design
Sira Vegas, Cecilia Apa, Natalia Juristo, "Crossover Designs in Software Engineering Experiments: Benefits and Perils", IEEE Transactions on Software Engineering, vol. 42, no. , pp. 120-135, Feb. 2016, doi:10.1109/TSE.2015.2467378