Issue No. 06 - November/December (2008 vol. 25)
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/MS.2008.153
Anatol W. Kark , National Research Council Canada
Cornelius Ncube , Bournemouth University
Patricia Oberndorf , Software Engineering Institute, Carnegie Mellon University
The face of software engineering is changing. We're pressured to deliver systems more and more quickly, and those systems must provide increasingly ambitious functionality. On the other hand, we live in an era in which we can no longer afford—in terms of either time or money—to custom-develop every system. This situation is driving the software development industry toward composing systems from readily available software components, including other whole systems. Opportunistic software systems development (OSSD) is an approach in which developers meld together software pieces that they have found. Most often they find unrelated software components and systems that weren't designed to work together but that provide functionality they want to include in a new system. Typically, in opportunistic development, developers spend less effort developing software functionality to meet particular requirements and more time developing "glue code" and using other techniques for integrating the various software pieces.
The OSSD approach is akin to recent television shows in the UK ( Scrapheap Challenge) and the US ( Junkyard Wars). In these settings, competing teams are given a capability they must implement using only what's available at a junkyard or scrap heap, a set of appropriate tools for integrating the pieces, and their own wits and innovation. OSSD is in some ways the new Junkyard Wars: making new capabilities out of useful software that's at hand, applying innovative but effective engineering and management techniques, and often using an item creatively in a way that its creator never envisioned. The key objective of the OSSD approach is to advance the state of the practice by building on what exists rather than recreating everything every time. And herein also lies a key distinction from one premise of the junkyard/scrap heap analogy: there is nothing "junky" about the software and systems used in OSSD. Rather, using these existing items acknowledges their continued utility and value. Their incorporation into a new, larger system is a logical evolution from their original stand-alone existence to one in which they now contribute to a greater capability.
Recent publications such as Barry Boehm's "A View of 20th and 21st Century Software Engineering," 1 James Noble and Robert Biddle's Notes on Postmodern Programming, 2 and the SEI's Ultra-Large-Scale Systems study 3 have highlighted the situation that has led us to OSSD. These all discuss the paradigm changes needed to supply the demand for software. In fact, we think that the biggest (demand for) changes come from traditionally nonsoftware data-processing industries—healthcare, automotive, mobile communications, energy conservation, and others. We software engineers will need a variety of new approaches, including OSSD.
Characteristics of OSSD
Although from a distance, the OSSD approach might look like the traditional software reuse paradigm, there are key differences. Software and systems that are used in systematic reuse paradigms are designed to be reused in a specific context. The development processes are systematic and planned. Methods and guidelines are clear about how to perform and manage each development task, therefore yielding a degree of predictability for the result. Even in some COTS-based (commercial-off-the-shelf-based) system developments—those centered on a single COTS product or product suite—most of the work involves tailoring the system with mechanisms that have been built into the product for just that purpose. Also, both in systematic reuse and COTS-based development, processes have been developed to methodically design reuse components, evaluate components for inclusion in software systems, produce reusable architectures, and maintain resulting systems. This trend continues today with the service-oriented architecture (SOA) approach, in which reuse has moved into the services domain. However, the developer community hasn't yet completely resolved the SOA ideals of the runtime discovery of service components or fully established methods for evaluation of their services and patterns for use.
In the OSSD paradigm, the focus is on combining systems that were never meant to work together or even to be reused. It's about taking existing software systems and performing minimal coding with the emphasis on gluing them together—usually without interaction with the original developers or access to source code—to achieve new desired functionality with a desired look. The selection of candidate software is often experimental. OSSD relies on a great deal of prototyping and requires new kinds of skills, collaboration, and engineering. It's a major shift that challenges the current software systems development paradigms and processes. It requires a major change of mind-set: from designing and writing original program code that meets a particular requirement, to a world in which few rules for design, construction, or process management exist. The development process is ad hoc and opportunistic, with the team continuously scavenging and trying out candidate solutions while constantly paying attention to architectural mismatches, differences in programming languages, uniform ways of describing data, and so on. The major development emphasis is on smart engineering, creativity, innovation, and the most imaginative ways of gluing together seemingly unrelated software pieces to provide interoperable and maintainable systems that meet users' needs. It's about fabricating hybrid systems by scavenging for functionality wherever we can find it to "weld" together pieces into a composite system.
Challenges in the OSSD Approach
OSSD poses unique challenges to developers, development processes, requirements engineering, system architecture, maintenance, and evolution. The OSSD paradigm requires a new set of skills, strategies, and practices; definition of new frameworks and infrastructures to handle the complexities; codification of concepts, techniques, and tools; and flexible development methods and processes. Requirements engineers, developers, and system integrators are challenged to come up with new techniques for success in this environment. In opportunistic development, we have few theories or recipes to follow and not many prescribed ways of doing things. To cope with these challenges, we need creative, innovative, and imaginative engineering and management practices. Yet our universities, practitioner books, and industrial training organizations continue to focus on teaching students how to develop software systems from first principles and largely ignore the very different skills required to integrate and test software they didn't create and don't control.
This special issue aims to gather together insights into the viability, or perhaps the inevitability, of OSSD and to bring forward the most effective practices known today. Another goal is to point the way to what needs to be done to make OSSD more accessible to all practitioners. We couldn't include all the excellent articles that were submitted for the issue, but the articles we chose reflect the current state of the practice and point to the future. Lessons and experiences from our past work in COTS-based and SOA-based systems can help guide us in approaching OSSD. We offer some of them here.
First, requirements cannot and will not be known completely in advance. The watchword for these new systems is flexibility and adaptability. Demands are high and ever-changing. Requirements will be more codeveloped along with the system, as developers discover and select what's possible and negotiate with users regarding the gaps that remain. "Pragmatic and Opportunistic Reuse in Innovative Start-Up Companies" by Slinger Jansen, Sjaak Brinkkemper, Ivo Hunink, and Cetin Demir illustrates this. Also, "Situated Software: Concepts, Motivation, Technology, and the Future" by Sriram Balasubramaniam, Grace Lewis, Soumya Simanta, and Dennis Smith addresses requirements discovery mechanisms.
Second, architectural decisions aren't left solely to the architect-developer. Although architecture will remain important, it too will evolve as the developers discover and add new pieces to make up the new capability. Some pieces might be discarded because they aren't compatible with an emerging architecture. But it's just as likely that architectural adjustments will be made as one approach to bringing more pieces into the whole. For example, Jansen and his colleagues address architecture's impact on "the scale on which the software architecture and software's interaction model needed adjustment to accommodate new functionality." Also, the framework discussed in "Balancing Opportunities and Risks in Component-Based Software Development," by Barry Boehm and Jesal Bhuta, "defines 62 interoperability analysis rules to identify architectural mismatches for a given architecture."
Third, developers (and system "owners") won't own or even be able to view all the source code. OSSD will often depend on the integration of black boxes—pieces of software and systems that someone else owns and in which they have strong intellectual-property interests. With COTS products, we often find that if the price demanded by a component's originator for privileged information about the design or source code is too expensive to be worth it, developers must decide either to move on to discover another usable piece or to make do with the black box. "Stimulating Creativity through Opportunistic Software Development," by Zeljko Obrenović, Dragan Gašević, and Anton Eliëns, addresses how to creatively deal with problems caused by working with black-box components without detailed internal (API) descriptions.
Also, new integration paradigms and techniques will be required and are one key to OSSD's success. Methods have been discovered over the years for figuring out what black boxes do and planning approaches for their integration. Manufacturer-provided specifications and manuals will continue to be important, but chances are no one will ever think to put in a manual all the little pieces of insight that will prove to be the key to successful integration. This circumstance is taken to the extreme in a world in which users end up doing their own integration, as "Monoliths to Mashups: Increasing Opportunistic Assets," by M. Todd Gamble and Rose Gamble, extensively addresses.
Testing will be a challenge. Testing these systems can't rely on code instrumentation and traditional white-box approaches. There will be more reliance on platform-level tools that can "sniff out" what's going on between pieces at the bits-and-bytes level. However, Jansen and his colleagues identify testing "as one main disadvantage to opportunistically reusing third-party functionality that wasn't originally intended for intensive reuse."
Finally, multidisciplinary teaming and collaboration will be essential. Developers increasingly will find that they're not in a world of their own—they share it with both component providers and those who would use an aggregated system as a component in a still-larger system. Collaboration and negotiation with users will be key skills that will need to be taught, nurtured, and rewarded. Obrenović and his colleagues address this point and emphasize the importance of a new type of education. Boehm and Bhuta also emphasize the importance of collaboration.
The State of the Art
This special issue addresses the gap between modern demand and current development approaches by articulating the current and future situation and describing paradigms and practices that point the way forward. Questions have been raised about the viability of the approach and the circumstances under which it's best undertaken, so this special issue also addresses some of those concerns.
Throughout the effort to bring this issue together, we tried to expose what's being done today—what works and what needs to be improved—and to approach OSSD from various perspectives. The set of articles we chose helps to achieve that objective, often in surprising ways, such as when similar conclusions are drawn by authors who started from different perspectives. Notice, for example, the match in fitness criteria in the articles by Balasubramaniam and his colleagues and by Gamble and Gamble. Note also the similarity in mechanism suitability conclusions in the articles by Gamble and Gamble and by Jansen and his colleagues.
At the same time, a few other interesting ideas and themes have emerged.
First, high priority is given to early or fast release. The emphasis is placed on the discovery of suitable components, which "can do the job" at some satisfactory level. Negotiations with the stakeholders then take place on either accepting lesser or different functionality or continuing a search for better components. The articles by Jansen and his colleagues and by Obrenović and his colleagues illustrate this point and discuss the feasibility of the OSSD approach.
Second, we suspect that Release 1 of the software developed in this opportunistic way has a limited lifetime, either because there will be no Release 2 or because a different methodology will be used. The proponent in our Point-Counterpoint debate, Michiel van Genuchten, states that "opportunistic software development isn't about forgetting all the good engineering practices put into place over the last decades. Calling it opportunistic won't make bugs disappear."
Finally, we need a different type of curriculum to prepare future software developers for a different way of "doing software." Hardware engineers have no problems looking for a best chip or a part that would fit whatever design they have in mind. We need to instill similar behavioral patterns in software engineers.
Despite all the difficulties encountered in each incarnation of reuse, we persist along the path of trying to figure out how we're going to create systems that meet the ever-increasing demand for capability and, with it, complexity and sheer size. OSSD is a reality today and for the foreseeable future, as the five articles we chose demonstrate. With this special issue, we hope to help the software community realize the importance of this new trend and the many aspects of it that we have yet to conquer in order to be successful. However, the OSSD approach will pose challenges for requirements engineers, developers, and system integrators to be creative, imaginative, and innovative in coming up with new ways of putting together pieces of software solutions that were never designed to work together. We hope that new tools and techniques will be developed to handle these challenges. Above all, we hope that universities, practitioner books, and industrial training organizations will start to emphasize different developer skills for integrating and testing software.
Cornelius Ncube is a senior lecturer in software systems engineering at the Software Systems Research Centre, Bournemouth University. His core research area is in software systems engineering, with a particular focus on systems-of-systems engineering, composition-based software systems, opportunistic software systems development, COTS-based systems development, and requirements engineering. Ncube received his PhD in computer science from City University London. Contact him at firstname.lastname@example.org; http://dec.bournemouth.ac.uk/staff/cncube/index.html.
Patricia Oberndorf is a senior member of the technical staff at Carnegie Mellon University's Software Engineering Institute. Until recently, she directed the SEI's Dynamic Systems Program, researching interoperability, service-oriented architecture, and model-based engineering. Oberndorf received her master's of science degree in computer science from the University of California at San Diego. She coauthored, with Craig Meyers, Managing Software Acquisition: Open Systems and COTS Products.
Anatol W. Kark is a leader of the Software Engineering Group at the National Research Council, Canada, where his research interests include integration and interoperability of software systems and software life cycles. He is also research director of the Canadian Consortium for Software Engineering Research; a convener for Work Group 7 of the ISO/IEC JTC1 SC7, dealing with the software and systems life-cycle processes; and coeditor of ISO/IEC 12207, the Systems and Software Engineering—Software Life Cycle Processes standard. Kark received his master's degree in math from the University of Waterloo. Contact him at email@example.com.