The Community for Technology Leaders
Green Image
Issue No. 02 - February (2007 vol. 33)
ISSN: 0098-5589
pp: 124-141
Cemal Yilmaz , IBM T.J. Watson Research Center, Hawthorne, NY
Adam Porter , Department of Computer Science, University of Maryland, College Park, MD
Arvind S. Krishna , Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN
Atif M. Memon , Department of Computer Science, University of Maryland, College Park, MD
Douglas C. Schmidt , Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN
Aniruddha S. Gokhale , Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN
Balachandran Natarajan , Symantec, Bhau Patil Marg, Pune 411 020, India
ABSTRACT
Developers of highly configurable performance-intensive software systems often use in-house performance-oriented "regression testing" to ensure that their modifications do not adversely affect their software's performance across its large configuration space. Unfortunately, time and resource constraints can limit in-house testing to a relatively small number of possible configurations, followed by unreliable extrapolation from these results to the entire configuration space. As a result, many performance bottlenecks escape detection until systems are fielded. In our earlier work, we improved the situation outlined above by developing an initial quality assurance process called "main effects screening". This process 1) executes formally designed experiments to identify an appropriate subset of configurations on which to base the performance-oriented regression testing, 2) executes benchmarks on this subset whenever the software changes, and 3) provides tool support for executing these actions on in-the-field and in-house computing resources. Our initial process had several limitations, however, since it was manually configured (which was tedious and error-prone) and relied on strong and untested assumptions for its accuracy (which made its use unacceptably risky in practice). This paper presents a new quality assurance process called "reliable effects screening" that provides three significant improvements to our earlier work. First, it allows developers to economically verify key assumptions during process execution. Second, it integrates several model-driven engineering tools to make process configuration and execution much easier and less error prone. Third, we evaluate this process via several feasibility studies of three large, widely used performance-intensive software frameworks. Our results indicate that reliable effects screening can detect performance degradation in large-scale systems more reliably and with significantly less resources than conventional techniques
INDEX TERMS
Quality assurance, Monitoring, Degradation, Software systems, Performance evaluation, Software performance, Software testing, System testing, Time factors, Extrapolation
CITATION

C. Yilmaz et al., "Reliable Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems," in IEEE Transactions on Software Engineering, vol. 33, no. 2, pp. 124-141, 2007.
doi:10.1109/TSE.2007.20
182 ms
(Ver 3.3 (11022016))