Software Engineering: Two Directions at Once
David Alan Grier is a writer and scholar on computing technologies and was President of the IEEE Computer Society in 2013. He writes for Computer magazine. You can find videos of his writings at video.dagrier.net. He has served as editor in chief of IEEE Annals of the History of Computing, as chair of the Magazine Operations Committee and as an editorial board member of Computer. Grier formerly wrote the monthly column "The Known World." He is an associate professor of science and technology policy at George Washington University in Washington, DC, with a particular interest in policy regarding digital technology and professional societies. He can be reached at email@example.com.
You've got to read the audience. It's one of hardest things you need to do when you are giving a technical talk. Are they looking for details and guidance? Are they looking for inspiration and vision of the future? Again, if that is what they want then you had better be prepared to give it to them. How about a fair and honest assessment of the state of technical careers? That is not as common among audiences. Therefore, you had better be ready for comments if you are giving that kind of talk.
Last month, I was trying to read the audience as I was preparing to give a talk on the development and evolution of software engineering. The subject is relatively new to me. I've been developing the ideas behind the talk since I have been leading the Computer Society and have been promoting the Guide to the Software Engineering Body of Knowledge (SWEBOK) and the various software engineering certifications offered by the Computer Society. (I wrote about SWEBOK in the December 2013 issue of the Communications of the Chinese Computing Federation.)
In my talk, I was making the argument that software engineering is a very adaptive field and that it drew most of its original ideas from manufacturing engineering and not from electrical engineering. Indeed, one of the key concepts of software engineering, the idea of the software lifecycle, came directly from ideas of manufacturing engineering. Specifically, it was based on the cycle of continuous improvement in quality control and quality. In the software lifecycle, we are always working to improve our software. We start the cycle by specifying the goals of the software. We then design it, code it, test it, and deploy it. Once we have deployed or released the software, we assess what we have done and start the cycle all over again.
The last time I gave this talk, I was facing an audience of senior software engineers. They had worked for IBM, Oracle, and a half dozen other firms. By the end of the talk, I began to realize that they had more to teach me than I had to teach them. They generally liked the idea that software engineering had grown not come from electrical engineering and agreed that my argument about how it developed helped them understand their careers and their collective experience. However, they felt that I had missed a key point. The software lifecycle, they argued, fully described software engineering in only one setting, the setting in which your software didn't interact with other pieces of software and in which the users of the software don't need to be retrained for a new release of the system. In any other setting, the traditional software lifecycle is too slow and too complicated, and thoroughly unable to respond to the needs of the market.
Many members of the audience proved to be practitioners of agile methods or lean software development. Both of these approaches appeared in the late 1990s, shortly after the Software Lifecycle was codified in 1995 with the international standard ISO 12207. This standard reflected an approach that was common before the Internet emerged in the 1990s. The Internet, as much as anything, forced software developers to work with a shorter software lifecycle. It allowed developers to get new researchers to customers more quickly and it also allowed customers to respond with their feedback with equal speed.
Agile is the older of the two approaches. It was codified in 2001, when a group met at a ski resort in the United States to discuss a more flexible was of developing software. As the name "agile" suggests, this approach allowed software developers to respond more quickly and with greater sophistications to changes and opportunities. It weakened the classic software lifecycle and allow development teams to work on different parts of the system. They needn't start with a full specification of the system but might create a small description that would outline the basic ideas behind the software. From that point, they would work to create specifications, and code and tests as circumstances demanded. They might code one day, write specifications the next, and debug on the third.
At the start, agile software development was less a set of fixed process and more a set of principles. The original creators of agile methods identified two fundamental principles for their approach. They began with the goal of providing customers with quality software through the quick delivery of high-quality software and ending with the ability to adapt to changing circumstances. Since then it has developed a literature of best practices and methods but these are not the fixed techniques of classical software engineering. If anything, they suggest that agile software engineering is an attitude and an style of management rather than rigid rules.
Lean development is a slightly newer approach and it built upon agile. It is based on a book, published in 2003, which was entitled "Lean Software Development: An Agile Toolkit." As the name suggests, this approach suggested simplifying software and software development, using smaller teams, making decisions about the software as late in the development process as possible, and producing software as fast as possible.
Both agile and lean methods have become quite popular in the development communities. Many of the large developers have been able to deploy the methods with great success. Whenever a company is able to release new versions of software every few days or every few weeks, they are generally utilizing some combination of agile and lean methods. At the same time, a number of firms have had a few public failures and many private failures using both approaches. It will not take you long to search the Internet and find websites that rail against agile and lean methods. "Why are most Agile Adoptions Failing?" asks one blog and "Why Lean really isn't lean" explains another.
In listening to the audience at the end of my talk, I got a clearer understanding of the forces that produced classical software engineering and then drove developers to the agile and lean methods. These methods are efforts to discipline a process that fundamentally wants to embrace anarchy. We can all too easily approach software development thinking that we can create a product with little discipline. We can conceive the idea of the system and then convince ourselves that the system will be easy to code without any bugs at all. Software development involves discipline, a discipline that pulls us in two directions at once. It demands both rigor and flexibility from our development teams. It asks that we work together in a careful systematic way and yet be responsible to our customers, our users, and the market. As my audience told me that night, you can't understand the force of that discipline until you have managed a real software development.
Deng Xiaopeng addressed some of the problems of managing the kinds of intellectual labor we must organize for software engineering. In an 1978 talk at a Conference on Science. In organizing science to build the Chinese economy, he argued that there "must be a division of labor" and then added that the country with a system of specific individual responsibility for each post." The challenge of software engineering is to divide the labor well and then decide how we allocate authority to those that make the specifications, do the coding test the results, and release the product, as well as those that pay for it and those that use it.