This Article 
 Bibliographic References 
 Add to: 
How Reliable Are Systematic Reviews in Empirical Software Engineering?
September/October 2010 (vol. 36 no. 5)
pp. 676-687
Stephen MacDonell, Auckland University of Technology, Auckland
Martin Shepperd, Brunel University, West London
Barbara Kitchenham, Keele University, Keele
Emilia Mendes, The University of Auckland, Auckland
BACKGROUND—The systematic review is becoming a more commonly employed research instrument in empirical software engineering. Before undue reliance is placed on the outcomes of such reviews it would seem useful to consider the robustness of the approach in this particular research context. OBJECTIVE—The aim of this study is to assess the reliability of systematic reviews as a research instrument. In particular, we wish to investigate the consistency of process and the stability of outcomes. METHOD—We compare the results of two independent reviews undertaken with a common research question. RESULTS—The two reviews find similar answers to the research question, although the means of arriving at those answers vary. CONCLUSIONS—In addressing a well-bounded research question, groups of researchers with similar domain experience can arrive at the same review outcomes, even though they may do so in different ways. This provides evidence that, in this context at least, the systematic review is a robust research method.

[1] V. Basili and K. Freburger, "Programming Measurement and Estimation in the Software Engineering Laboratory," J. Systems and Software, vol. 2, pp. 47-57, 1981.
[2] Experimental Software Engineering Issues: A Critical Assessment and Future Directions, H. Rombach, V. Basili, and R. Selby, eds., Springer, 1993.
[3] B. Kitchenham, T. Dybå, and M. Jørgensen, "Evidence-Based Software Engineering," Proc. 26th IEEE Int'l Conf. Software Eng., 2004.
[4] T. Dybå, B. Kitchenham, and M. Jørgensen, "Evidence-Based Software Engineering for Practitioners," IEEE Software, vol. 22, no. 1, pp. 58-65, Jan./Feb. 2005.
[5] B. Kitchenham, E. Mendes, and G. Travassos, "A Systematic Review of Cross- vs. Within-Company Cost Estimation Studies," Proc. 10th Int'l Conf. Empirical Assessment in Software Eng., 2006.
[6] S. MacDonell and M. Shepperd, "Comparing Local and Global Software Effort Estimation Models Reflections on a Systematic Review," Proc. First Int'l Symp. Empirical Software Eng. and Measurement, 2007.
[7] A. Oakley, D. Gough, S. Oliver, and J. Thomas, "The Policy of Evidence and Methodology: Lessons from the EPPI-Centre," Evidence and Policy, vol. 1, no. 1, pp. 5-31, 2005.
[8] S. Oliver, G. Peersman, A. Harden, and A. Oakley, "Discrepancies in Findings from Effectiveness Reviews: The Case of Health Promotion for Older People in Accident and Injury Prevention," Health Education J., vol. 58, pp. 66-77, 1999.
[9] A. Oakley, "Social Science and Evidence-Based Everything: The Case of Education," Educational Rev., vol. 54, pp. 277-286, 2002.
[10] A. Oakley and D. Fullerton, A Systematic Review of Smoking Prevention Programmes for Young People, EPPI Centre, Inst. for Education, 1995.
[11] L. Pauling, How to Live Longer and Feel Better. Atlantic Books, 1986.
[12] P. Knipschild, "Some Examples of Systematic Reviews," British Medical J., vol. 309, pp. 719-721, 1994.
[13] W. Shadish, "Author Judgements about Work They Cite: Three Studies from Psychological Journals," Social Studies of Science, vol. 25, pp. 477-498, 1995.
[14] A. Davis, O. Dieste, A. Hickey, N. Juristo, and A. Moreno, "Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review," Proc. 14th IEEE Int'l Requirements Eng. Conf., pp. 179-188, 2006.
[15] M. Blettner, W. Sauerbrei, B. Schlehofer, T. Scheuchenpflug, and C. Friedenreich, "Traditional Reviews, Meta-Analyses and Pooled Analyses in Epidemiology," Int'l J. Epidemiology, vol. 28, no. 1, pp. 1-9, 1999.
[16] V.R. Basili, F. Shull, and F. Lanubile, "Building Knowledge through Families of Experiments," IEEE Trans. Software Eng., vol. 25, no. 4, pp. 456-473, July/Aug. 1999.
[17] W. Hayes, "Research Synthesis in Software Engineering: A Case for Meta-Analysis," Proc. Sixth IEEE Int'l Software Metrics Symp., pp. 143-151, 1999.
[18] L. Pickard, B. Kitchenham, and P. Jones, "Combining Empirical Results in Software Engineering," Information and Software Technology, vol. 40, no. 14, pp. 811-821, 1998.
[19] J. Miller, "Can Results from Software Engineering Experiments be Safely Combined?" Proc. IEEE Sixth Int'l Software Metrics Symp., L. Briand, ed., 1999.
[20] B. Kitchenham, P. Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, "Systematic Literature Reviews in Software Engineering—A Systematic Literature Review," Information and Software Technology, vol. 51, no. 1, pp. 7-15, 2009.
[21] DARE, "What are the Criteria for the Inclusion of Reviews on DARE?" technical report, Centre for Rev. and Dissemination, 2007.
[22] M. Petticrew, "Why Certain Systematic Reviews Reach Uncertain Conclusions," British Medical J., vol. 326, no. 7392, pp. 756-758, 2003.
[23] J. Higgins, S. Thompson, J. Deeks, and D. Altman, "Measuring Inconsistency in Meta-Analysis," British Medical J., vol. 327, pp. 557-560, 2003.
[24] P. Katrak, A.E. Bialocerkowski, N. Massy-Westropp, S. Kumar, and K.A. Grimmer, "A Systematic Review of the Content of Critical Appraisal Tools," BMC Medical Research Methodology (electronic resource), vol. 4, no. 1, p. 22, 2004.
[25] B. Kitchenham, E. Mendes, and G. Travassos, "Cross versus Within-Company Cost Estimation Studies: A Systematic Review," IEEE Trans. Software Eng., vol. 33, no. 5, pp. 316-329, May 2007.
[26] B. Kitchenham and S. Charters, "Guidelines for Performing Systematic Literature Reviews in Software Engineering," Version 2.3, EBSE Technical Report EBSE-2007-01, Keele Univ., 2007.

Index Terms:
Empirical software engineering, meta-analysis, systematic review, cost estimation.
Stephen MacDonell, Martin Shepperd, Barbara Kitchenham, Emilia Mendes, "How Reliable Are Systematic Reviews in Empirical Software Engineering?," IEEE Transactions on Software Engineering, vol. 36, no. 5, pp. 676-687, Sept.-Oct. 2010, doi:10.1109/TSE.2010.28
Usage of this product signifies your acceptance of the Terms of Use.