, University of Southern California
, Oregon Graduate Institute
Pages: pp. 22-26
How much is 68 + 73? Engineer: "It's 141." Short and sweet. Mathematician: "68 + 73 = 73 + 68 by the commutative law of addition." True, but not very helpful. Accountant: "Normally it's 141, but what are you going to use it for?"
Often, the accountant's answer is the most helpful of all. You may have been looking for the answer to the following question: "The Inspections people estimate that they can remove 68% of my software defects; the Test people estimate that they can remove 73%. How many will they remove together?" Clearly, 141% is not a good answer; there must be some defects that are being counted twice.
This story brings out two important points about software estimation:
Grant Rule's essay on " Bees and the Art of Estimating" (see the sidebar) illustrates point 1 well. Jumping into an estimate of the number of bees in a hive without understanding the counting rules can produce a pretty useless estimate. The same is true for software: Does an estimate of 100 person-months for "software development" include analysis and design, integration and test, deployment, management, or uncompensated overtime? If you use the estimate without knowing the answers, you can get yourself into serious trouble.
Point 2 highlights the fact that estimates have a number of uses, and you can often get both better and simpler estimates if you keep the use of your estimate in mind. For example, suppose you are doing estimates for a make-or-buy analysis. The vendors' quotes for the "buy" option are clustering around $100K for satisfactory-looking products, and it is looking like the "make" option will be a good deal more expensive. In this case, you can make a simpler and even stronger estimate for the "make" option by using optimistic assumptions about its size, complexity, and staff capability. If even the resulting optimistic "make" estimate comes out at $130,000, you have both saved yourself a good deal of estimating effort and produced a stronger conclusion that the "buy" option is better than even the best-case "make" option.
Besides setting budgets and schedules and supporting make-or-buy analyses, software estimation techniques can have several additional decision support uses:
For the latter, for example, the Cocomo II productivity ranges shown in Figure 7 of Bradford Clark's article provide the basis for analyzing various mixed strategies of investments in personnel capability improvement, process maturity, software reuse, software tools, and multisite software development support.
Another observation in Grant Rule's essay about estimation perspectives involves the dynamic nature of the software field. One perspective is that software projects will necessarily evolve during development and that up-front estimates cannot be precise. Several software cost and schedule estimation models now provide (optimistic, most likely, and pessimistic) estimation ranges rather than point estimates.
These ranges support entirely new process models better attuned to the dynamic nature of modern software, such as cost- or schedule-as-independent variable (CAIV or SAIV). At USC, we have evolved a highly successful SAIV approach for developing Web-based digital library systems on a necessarily-fixed schedule of 24 weeks. It works as follows:
Even with a considerable dynamism and uncertainty in the nature of the desired product, this approach almost always produces a satisfactory result in a short, fixed development time.
As a final perspective, the dynamism of the software field means that the software estimation discipline needs to be continually reinventing itself. The articles in this special focus are good examples of this. Traditional software estimation models did not have to deal with graphical user interface builders, objects, process maturity, and Web-based systems. Traditional software estimation methods were either expert-based or model-based, and did not try to mix the two. The articles here show healthy new approaches to these phenomena and indications that the estimation field is rising to the challenge of continually reinventing itself.
Consistent with the theme of "Recent Developments in Software Estimation," this issue of IEEE Software presents six articles that report on promising estimation techniques, each of which can potentially improve the estimation process, the accuracy of the resulting estimates, and the productivity and quality achieved by software developers. The techniques reported are not "tried and true" because it is in the nature of recent developments in science and engineering that others must subject them to trial use before they become accepted practices. However, these authors present approaches that are worthy of consideration by estimators and by those who affect and are affected by estimates for software projects.
In "Improving Size Estimates Using Historical Data," James Bielak presents an analysis of a completed C++ project and shows that the number of GUI elements in a component and the number of GUI events handled by the component provide a rough estimate of the component's size in source lines of code. The number of methods in a component's interface and the number of components reused from the architecture can also be used to estimate size, but the estimate depends on a component's position within the overall architecture.
Analysis of defect data obtained from software inspections is often used to identify problem areas in software projects. Stefan Biffl, in "Using Inspection Data for Defect Estimation" presents the design and results of a large-scale experiment in which he investigated the accuracy of defect estimation models based on inspection data. He shows that the accuracy of subjective defect estimation models based on weighted averages of estimates by individual team members is superior to the accuracy of objective DEMs.
In "Enhancing the Cocomo Estimation Models," Joanne Hale, Allen Parrish, Randy Smith, and Brandon Dixon propose estimation adjustment factors based on the task assignments of project team members that can be used to improve the accuracy of existing cost estimation models. They show improvements in the predictive abilities of Cocomo I and Cocomo II when these factors are included.
"Empirically Guided Software Guesstimation" by Philip Johnson, Joseph Dane, Carleton Moore, and Robert Brewer reports on an experiment in which developer-generated "guesstimates" of software effort were more accurate than analytical estimates. However, they also found that access to a range of analytical estimation methods appeared to be useful to developers in generating their guesstimates and improving them over time.
In the article "Web Development: Estimating Quick-to-Market Software," Donald Reifer proposes a cost model for Web development projects that combines a size measure based on Halstead's Volume measure (using number of operators and operands) and a function-point-like table of complexity weights. Size estimates are used in Cocomo-like equations to produce estimates of effort and duration. He proposes eight cost drivers for the effort adjustment factor. As Reifer points out, there are still a large number of open issues to be resolved; however, his article shows an approach to developing estimation models for this important and growing domain of software engineering.
The final article, "Effects of Process Maturity on Software Development" by Brad Clark, presents the results of his analysis of 161 software projects (the USC Cocomo II database). His results indicate that, for the projects analyzed, a one-level change in process maturity resulted in a 4% to 11% reduction in project effort. Larger projects realize the larger gains. Clark applied the analysis across the five maturity levels of the Software CMM. He speculates that the percentage reduction in effort is not uniform across all levels. He could not determine this because his data did not contain a sufficient level of detail to permit analysis of improvement between levels. This work does, however, summarize a solid analytical analysis of what till now has been largely anecdotal evidence that process maturity results in decreased project effort.
Software engineering is concerned with building software-intensive systems and products within the constraints of time, resources, technology, quality, and business considerations. The ability to scope projects accurately is an essential element of an engineering discipline. Estimation models, procedures, and techniques are essential components of the software engineering discipline. As the field changes, the techniques of estimation must, of necessity, change. The articles in this issue present new developments in software estimation that show the way to accommodating the needs of the always-changing world of software engineering.
Have you heard the one about the bees? Are there any apiarists amongst IEEE Software's readership? Well never mind. Even if you're not a beekeeper and absolutely hate bread and honey, you can still participate in this simple experiment.
I've run this trial on numerous courses and at several conferences in the US, the Netherlands, Germany, Scandinavia, and the UK—and each time, I get pretty similar results.
Try it now. Take paper and pencil (or use your handheld, if you're a gadget freak) and write your estimate for the number of insects in an average hive of English honeybees. No cheating now!
Are you done? Oh, you don't know anything about bees? Well, this is an estimating task. Estimating is about predicting in the face of uncertainty and incomplete knowledge, so that's no bar. We estimate for unfamiliar software projects every day. You don't bother? Oh! How ever do you prioritize your set of possible tasks (that's a rhetorical question)? What do I mean by "average"? Now, that's a good question. Do I mean the "mean population of bees in a hive over time, taking a typical annual cycle into account"? Or do I mean "the mean, calculated from the total population of bees in England at a specific time, divided by the number of hives in England"? And anyway, what do I mean by "insect"? Should you include critters such as mites, pillbugs, and wasps that live on, in, or among the bees? Hang on! Bees are hive animals; arguably the queen and her workers and drones constitute a "single genetic individual," so should you just count the number of queens? And if the hive is on the point of swarming (May or June maybe), should you count the queens that leave to find a new hive?
The point, of course, has nothing to do with bees. It is about the habits and practice of estimating.
When I've run this trial with a "live" audience, very few people have questioned the implicit assumptions (actually 10 from a total of 1,100 ± 10%). Subsequent discussion, and daily observation, leads me to suppose that the same is true when those people estimate for software projects. By the way, I forgot to tell you that all the "live" trials were with groups of software measurement or improvement enthusiasts. The absence of documented assumptions, or even any consideration of the assumptions, pretty much makes an estimate unrepeatable and probably worthless. Especially if you believe, as I do, that each software project is a "complex system with sensitive dependence on initial conditions" (but that's a topic for another time).
To return to the bees. It seems that the population of worker honeybees in a hive fluctuates during the course of the seasons and with the weather and depending on the food supply. Taking any populated hive at random, there is a 95% probability that it will contain a population in the range of 20,000 to 60,000 workers and drones. Add one for the queen if you like.
The surprising fact, to me, is that during the course of my trials, only about eight people have ever given a range estimate (that is, less than 1% of participants). Most go for precise answers, with a high probability of being incorrect. I would forecast that 80-90% of those of you who made an attempt at an answer wrote down a single number (95% confidence). I'm assuming that Software's readership, with an engineering background, is 10 to 20 times more likely to give a range estimate than a point estimate.
It seems to be a built-in human characteristic to prefer precision over accuracy. By this I mean that humans prefer a specific, single value that pretends to certainty (and which the estimator "knows" is probably wrong) over a range value that almost certainly includes the most probable value and hence is correct, but which retains some level of uncertainty. As there are very few certainties in this life, this proclivity is—just maybe—a contributing factor to the well-reported high percentage of failed software projects.
The most common basis used for estimating is so-called "expert opinion." That is, project managers make estimates by referring to their past project experiences. But few can produce any historic data when challenged, and typically there is very little rigor in the approach. No assumptions are documented, no baseline measurements made, and often no provision exists for feedback. Few organizations consistently hold post-implementation reviews, and the project manager who makes personal records of the accuracy of estimates seems a rarity.
Estimating is more craft than science. Good craft work requires lots of practice. As a degree of uncertainty can be guaranteed, the method used to deal with this uncertainty is crucial to ensure repeatability and accuracy and to build confidence in one's ability as an estimator. The specific calibrations, and even the details of the algorithms and tools used, are secondary. After all, once you can repeatably estimate the range in which an ultimate result will fall, even though the range is wide, it is only a matter of tweaking the calibrations to narrow the range and improve the precision of your estimating. That's a matter of comparing the estimate with the result, adjusting the calibrations, then doing it all again. Over and over. Just as athletes do when they practice.
The key is repetition and practice. It's very difficult to produce reliable estimates if you rarely exercise the craft, say once at the beginning of each project (maybe once in nine months?), or if the feedback loop between estimate and result is too long to facilitate recalibration. If you can't estimate in the small, then don't estimate in the large. It's too expensive.
Good estimators practice on inconsequential topics, so they develop the habits of estimating. They do so all the time, on various kinds of estimate. For example, How long will this journey take? What is the percentage of home-built versus imported cars in town? How many emails will I receive tomorrow? What effort is needed to produce each use case? How much effort will I "waste" on unplanned work tomorrow? Even when it is not their responsibility, good estimators challenge themselves to produce an estimate, for their own benefit if not for others. And then they improve over time.
Or, on the other hand, you might just think we are very sad people.