During development, how can you determine with confidence whether your system will exceed its allowable downtime when delivered? When customers change their minds about functionality, how can you be confident that you've made the right trade-offs to deliver the system within your original fixed budget? During the early weeks of a sprint, how can you be confident that you can deliver all the functionality described in the stories committed in the sprint? How can you determine when a system's technical challenge has exceeded your team's capability and will undermine your commitments for quality, functionality, cost, or schedule? During a project, how can you detect that the discipline required to perform some development practices has become lax? If you're delivering life-critical software, how do you know that you're on track to have removed all the safety- and security-related defects before delivery?
Getting answers to these and other questions is often difficult because hard data is just not available. Even if it is, you still need to identify trends and isolate the root causes of problems. These types of questions drive organizations to embed statistical methods into their software engineering practices. They adopt these quantitative measures so that they can manage products and the processes used to generate them.
This special section of IEEE Software explores the use of statistical methods embedded into software engineering practice to support quantitative process management (QPM). These methods complement others used for product management whose adoption is equally important. These process-based techniques aim to monitor development results in real time or to simulate development activities to support in situ decisions and actions regarding progress or quality. What distinguishes these methods is that we collect data at the process event or task level rather than at the milestone or phase-end level. We can use these data to
• evaluate the performance of process tasks and their results,
• gain insight into the causes of variation in process performance,
• analyze trends and pinpoint root causes of problems, and
• predict future process or project outcomes.
Statistical methods have been used on software projects since the 1970s. However, most uses weren't embedded in the process because the measures were aggregated at the end of a development phase to estimate future results. Typical examples included the use of cost-estimating tools at the beginning of a project and the use of software reliability models at the beginning of testing. Only recently have we begun to see sustained deployment of statistical methods embedded in development processes.
The two articles in this section come from companies in two industry segments, aerospace and IT outsourcing, where the use of statistical methods for QPM has grown fastest. These two segments have been particularly motivated to achieve CMMI's higher maturity levels, in which quantitative management of the development process is essential. CMMI embraces statistical methods in its quest to promote rigorous process improvement practices across an entire development organization.
You're probably asking, what's the value of embedding statistical methods into software engineering practices? Even though such methods revolutionized the improvement of quality and cycle time in manufacturing, can these methods yield similar results when used in creating intellectual rather than physical artifacts? The benefits reported from high-maturity organizations suggest that they can, 1
but they don't provide definitive proof because isolating the impact of statistical methods from other improvements is difficult. However, the two articles in this section provide insight into how software organizations can gain value from embedded statistical methods.
In the first article, Uma Sudhakar Rao, Srikanth Kestur, and Chinmay Pradhan at Unisys Global Services India demonstrate how stochastic optimization modeling (SOM) can be embedded into the project management process. They used SOM to select among alternative methods for implementing project processes, to identify processes to be statistically controlled, and to periodically recalibrate expectations about project results to guide corrective actions. Because this technique is sensitive to underlying assumptions about process performance, it highlights the importance of understanding distributions of task performance results and grounding model assumptions on historical process capability data. When the proper QPM foundation has been laid for SOM, it can become a powerful management tool guiding critical decisions and exploring what-if scenarios.
In the second article, David Card, Kevin Domzalski, and Glyn Davies, using data from BAE Systems NS, demonstrate how embedded statistical methods can reduce delivered defect rates over time. Their measurement system combines techniques such as control charts and profile matching. They show how to use a combination of control charts to identify process problems during inspections. They also show how to quantify defect detection rates across the life cycle to develop and test predictions about quality results. There are several challenges in using control charts with software data, and they provide grist for the debate presented in the Point/Counterpoint discussion in this issue.
More than helping attain a high-maturity CMMI level, these QPM methods' real value is in providing what W.E. Deming called "a system of profound knowledge" to support in-process decisions about development. 2
Embedded process measures have been used for decades to control and improve manufacturing processes. This practice has expanded into business process management, where measures are now being embedded into end-to-end business processes to manage the workflow's effectiveness, the quality of its work products, and the delivery of value to the customer. 3
The challenge in applying these techniques to software development is to adapt them from their original use with predominately routine tasks for application to the knowledge-intense, nonroutine development of computer logic.
In developing the field of statistical process control, Walter A. Shewhart's objective was "through the use of past experience" to "predict, at least within limits, how the phenomena may be expected to vary in the future." 4
Donald J. Wheeler reinforced this sentiment, maintaining that "prediction is the essence of doing business." 5
At the beginning of this introduction, we listed some outcomes that executives, project managers, and developers would like to be able to predict and control throughout a project, especially when safety, business, or reputation are at risk. Statistical methods can help these professionals achieve such results, especially when the added expense of achieving them is justified.
Consider figure 1
's example of one form of QPM and the predictive use of process measures. Throughout a project, software managers and executives want accurate predictions of whether the project will meet its commitments for schedule, cost, features, quality, or other critical outcomes. There are many methods for developing these predictions; this example presents only one.
Figure 1. One example of using quantitative process management techniques predictively.
The first step in providing these predictions for each critical outcome is to develop a model of the project's subprocesses or tasks that most affect each outcome, because the tasks having the greatest impact on each outcome might differ. Next, for each process task in the model, we define the measures of process performance and work product attributes that we believe are most predictive of final project outcomes, along with methods for analyzing them. We enable these two steps by having a standard software development process with associated measures. Consequently, QPM using in-process measures is beyond the capability of immature software organizations.
The next step is to formulate an analysis strategy supporting the predictions of the critical project-performance parameters. In many cases, the predictive model will incorporate additional predictors as the project progresses. The predictive model is represented here as a general linear equation characteristic of regression-based approaches, but other statistical techniques might be more appropriate depending on the predictive objective and nature of the data. Throughout the project, project managers recompute outcome predictions with data emerging from the performance of recent process tasks to determine whether the intermediate results indicate that the project will achieve its desired outcomes or whether corrective action is warranted.
The validity of any predictive model depends partly on the quality of the data entered into it. So, we must ensure that the measurements taken from the performance of process tasks are valid indicators of actual process or product attributes. For instance, as Card, Domzalski, and Davies demonstrate in their article, the number of defects found in a hastily performed peer review probably doesn't accurately indicate the actual number of defects in the work product under review.
Accordingly, to have faith that the resulting measures are accurate predictors of project outcomes, developers must evaluate whether they performed their process tasks with appropriate discipline. Card and his colleagues evaluated the time spent in review preparation as an indicator that the process was under control and was being performed in a disciplined way. When review time was inadequate, inspection results were unreliable indicators of work product quality. Thus, process discipline is important because it not only improves task results but also produces more accurate predictors of project outcomes.
Developers performing process tasks use their process performance data to evaluate and control the performance of their tasks. Measures of their task outcomes become leading indicators or intermediate predictors of final project outcomes. The ability of developers to use their own performance data to manage and control their work processes provides a strong foundation for empowering the development staff. Empowering developers frees managers to focus more time on customers or strategic issues. Consequently, a concomitant benefit of QPM is its contribution to an empowered culture.
Rao, Kestur, and Pradhan demonstrate a different approach to predicting project outcomes—by simulating the effects of project processes on a combination of project outcomes. Initially, project managers can use such simulations to select the best composition of project subprocesses for optimizing several outcomes simultaneously. During the project, project managers can use the data emerging from the performance of process tasks to drive predictive simulations by assessing the probability that the project will achieve targeted outcomes on the basis of current performance.
Regardless of the techniques selected, the analytics behind these predictive models rest on the statistical relationships developed from historical project data. When the historical data are insufficient for rigorous statistical characterization, simulations can work from inferences about distributions and relationships among factors in the model. Even so, it's critical that the organization constantly evaluate these models to ensure their predictive validity is sustained. Updating the models' parameters when necessary provides profound knowledge concerning changes in the underlying conditions affecting project performance.
Statistical methods and development paradigms
Although this issue has no article on the Personal Software Process (PSP) 6
or Team Software Process (TSP), 7
these methods present uniquely powerful opportunities for embedding statistical methods into software development practices. PSP and TSP apply statistics in a way that removes from the data the greatest source of performance variation in software development: differences among individual developers and development teams. When this source of variation doesn't complicate the data, individuals and teams who study their results over time can gain profound knowledge of the causes of variation in their performance.
As individual developers and their teams better understand their capabilities and the factors that affect them, they can
• make more accurate performance predictions,
• control their performance in real time,
• quickly identify the need for corrective action, and
• provide continually updated predictions about task and project outcomes.
Consequently, these methods provide a strong foundation for high-maturity practices. Such techniques could also make substantial contributions to the disciplined performance of agile methods.
The growing base of data from PSP and TSP implementations offers an excellent research platform for investigating how individuals and teams can predict, control, and improve their performance. With sources of variation due to individuals and teams controlled or removed from the data, it's far easier to study other factors that cause variation in software development and develop statistical methods for predicting and managing them. Lessons learned in PSP and TSP research have implications for the performance management and improvement of knowledge-intense work in domains beyond software development.
Neither do we have an article on agile methods. 8
Some of the more advanced users of agile processes have embraced statistical methods in a quest to improve their processes. By looking for root causes, they map trend data to their workflows in novel ways. Agile projects therefore provide yet another research platform for determining ways to predict performance as we try various software development methods. By investigating sources of variation in the data, we can identify best processes and practices.
What's needed to expand deployment?
To accelerate QPM's adoption and use in software development, we must take steps in five communities: executives and project managers, developers, system acquirers, process improvement professionals, and software engineering researchers.
Executives and project managers
Too few executives and project managers have experience or training in using in-process data to manage work processes and predict project outcomes. Without this preparation, executives and managers are averse to risking their performance on seemingly esoteric methods. Six Sigma programs were more widely and successfully deployed than Total Quality Management programs because they insisted on extensive training of executives and managers and on establishing management alignment behind the program.
Successful deployment of QPM methods and other high-maturity techniques in software will require this same concentrated effort to gain understanding and commitment. Executives' interest will rise if they believe that these methods will provide predictive leading indicators early in a project and that these indicators can guide them in taking timely corrective actions or renegotiating commitments with the customer. Project managers will appreciate that these methods provide more powerful management tools than the standard techniques they learn while studying for a project-manager certification. Agile project managers will benefit because their short time scales require the earliest possible detection of problems.
Developers are rarely trained to measure their own performance and then use the data to control and improve their work. PSP and TSP have been rare exceptions in providing such training, and a growing number of schools are adopting these methods in their curricula. Developers also resist adopting such practices when the overhead of collecting and using measures is high. Automated assistance is critical to sustained QPM.
In most areas of human endeavor, the shorter the period in which a task or process must be performed, the more important the role of measurement and statistics in its performance and improvement. From this perspective, agile methods should be fertile ground for using statistical methods to estimate, predict, control, and improve performance at the individual and team levels. In the more advanced agile shops, we frequently see graphs of test run results and story completion rates on the walls. Such data beg for statistical treatment to understand and improve agile teams' capabilities.
Those who acquire software-intensive systems often have difficulty determining whether a system is on track to meet its cost, schedule, functionality, and quality targets. Data summarized at significant milestones often fail as leading indicators of problems. Acquisition managers whose suppliers can provide in-process statistics have better leading indicators and more insight into the results that their suppliers will ultimately deliver. Acquisition managers need greater understanding of the visibility they can gain from in-process statistics and how to embed such statistics in their acquisition methods.
Process improvement professionals
The deployment of QPM in software projects requires support from one or more statistically savvy members of an improvement or measurement team. People with the kind of training that Six Sigma Black Belts typically receive can support the statistical methods for evaluating performance on individual tasks. However, the development, validation, and recalibration of statistical prediction models typically require more sophisticated statistical knowledge. For organizations to better understand what controls their development performance, and for software engineering to become a more predictable discipline, people with such skills must become as indispensable as they are in business intelligence. 3
Software engineering researchers
The bar to adoption is high if software organizations must develop their own QPM techniques and models. The transition to these methods will occur much more rapidly if the research or entrepreneurial community can develop the underlying models and techniques and then help companies calibrate them to local conditions. This was the path that led to broad use of software-cost-estimating models. However, researchers must still address several critical issues. Here are five such issues:
• Process measurement. The available measures are often the wrong measures for analyzing process performance at the task or event level. Measures defined for monthly reporting are often poor measures for analyzing process tasks. Also, different predictive models might require different operational definitions from the same area of measurement. For instance, system availability might be predicted more accurately by mean time to failure than by defect density so long as the test data fairly represent the operational environment. Identifying the most effective measurement definitions for different QPM objectives would save industry much trial-and-error frustration.
• Statistical analysis of design work. Software development is nonroutine, knowledge-intense design work, from the architecture phase through coding. Process events differ in terms of actors, component difficulties, and conditions and are therefore typically affected by multiple sources of variation. Are statistical techniques borrowed from manufacturing appropriate for characterizing this behavior, or do we need different statistical techniques for characterizing the capability and sources of variation in design work? The Point/Counterpoint essays associated with this special section (pp. 48–51) debate the appropriateness of statistical process control methods drawn from manufacturing. These answers are critical for industrial practice.
• Individual and team capability. Individual and team performance is affected by multiple sources of variation, some differing among successive tasks and others differing across projects. We need ways to represent how these sources affect capability and how to develop expectations for evaluating process performance as these sources of variation change. For instance, how should our expectations of process performance change on the basis of a developer's experience or skill? How should we characterize a team's capability on the basis of its mix of experience and skill levels? How do we adjust and apply QPM methods to get the best control and prediction in a mixed capability environment?
• Aggregating measures for system prediction. The example in figure 1 appears simple until we realize that we're trying to predict a system characteristic using bundles of measures taken at the component level. For instance, we're using defect data from inspections of components to predict system quality. When and how should we aggregate these predictors from the component level to predict a system attribute?
• Validating benefits. QPM methods need stronger validation than the results offered in case studies, given that negative results are rarely published. We must test these techniques' marginal value by comparing their additional contributions to prediction and control versus the value of statistical methods that are computed only at major milestones during development. Validations of benefits in multiple industries help develop confidence in managers who have little previous experience with such methods.
QPM in software engineering is still in its infancy. Many high maturity organizations have used these methods to gain profound knowledge about their development capability and the factors that create variation in their performance. Predicting project outcomes using in-process data is a nascent capability. However, with advances in statistical methods for software phenomena, early-phase prediction of effort, duration, functionality, quality, and other important project outcomes could become commonplace.
A common characteristic of engineering disciplines is their use of measures integrated into engineering work for analysis and decision-making. Software engineering work hasn't historically relied on embedded measurement and analysis to the same extent as other engineering disciplines. If software engineering is to be truly an engineering discipline, techniques such as those demonstrated in this special section should become much more widespread on software projects.
is senior vice president and chief scientist of CAST Software, where he focuses on predicting long-term system quality and costs from statistical characterizations of system attributes and architecture. He's also a coauthor of the CMM, People CMM, and Business Process Maturity Model. He holds a PhD from Texas Christian University and is an IEEE fellow. Contact him at firstname.lastname@example.org.
Girish V. Seshagiri
is CEO of Advanced Information Services. He's a cofounder of the Watts Humphrey Software Quality Institute in Chennai, India, and of three software process improvement networks. His interests are in business-goals-driven software process improvement and defect-free software delivery on time. He received his MSc in physics from the University of Madras and his MBA in marketing from Michigan State University. Contact him at email@example.com.
is president of RCI, a software firm specializing in metrics and management. His professional interests focus on developing business cases for justifying changes in software organizations. He's a senior member of the ACM and IEEE. He has an MS in operations research from the University of Southern California. Contact him at firstname.lastname@example.org.
is principal of AMS, a software process improvement firm. His research interest is in model-based process improvement based on CMM, CMMI, TSP, and PSP. He holds a PhD in computer science from Florida Atlantic University. Contact him at email@example.com.
is a vice president of Tata Consultancy Services. Her research interests include process improvement, quality management systems, and business excellence. She's a senior member of the IEEE, a member of the National Association of Software and Services Companies, and a member of the IEEE Software
Advisory Board. She received her PhD in physics from Tohoku University, Japan. Contact her at firstname.lastname@example.org.