This Article 
 Bibliographic References 
 Add to: 
Components in the Pipeline
May/June 2011 (vol. 28 no. 3)
pp. 34-40
Ian Gorton, Pacific Northwest National Laboratory
Adam Wynne, Pacific Northwest National Laboratory
Yan Liu, Pacific Northwest National Laboratory
Jian Yin, Pacific Northwest National Laboratory
Scientists commonly describe their data-processing systems metaphorically as software pipelines. These pipelines input one or more data sources and apply steps to transform the data and create useful results. Although conceptually simple, pipelines often adopt complex topologies and must meet stringent quality-of-service requirements that stress the software infrastructure used to construct the pipeline. The Middleware for Data-Intensive Computing (MeDICi) Integration Framework (MIF) is a component-based framework for constructing complex software pipelines. It supports composing pipelines from distributed heterogeneous software components and provides mechanisms for controlling the quality of service to meet demanding performance, reliability, and communication requirements.

1. M. Shaw and D. Garlan, Software Architecture: Perspectives on an Emerging Discipline, Prentice-Hall, 1996.
2. M. Broy and R. Reussner, "Architectural Concepts in Programming Languages," Computer, vol. 43, no. 10, 2010, pp. 88–91.
3. P.M.K. Gordon and C.W. Sensen, A Pilot Study into the Usability of a Scientific Workflow Construction Tool, tech. report 2007-874-26, Sun Center of Excellence for Visual Genomics, 2007.
4. W. Tan, J. Zhang, and I. Foster, "Network Analysis of Scientific Workflows: A Gateway to Reuse," Computer, vol. 43, no. 9, 2010, pp. 54–61.
5. P. Clements et al., Documenting Software Architectures: Views and Beyond, 2nd ed., Addison-Wesley, 2010.
6. I. Crnkovic et al., "Specification, Implementation, and Deployment of Components," Comm. ACM, vol. 45, no. 10, 2002, pp. 35–40.
7. G.T. Heineman and W.T. Councill, Component-Based Software Engineering: Putting the Pieces Together, Addison-Wesley, 2001
8. X. Wu and M. Woodside, "Performance Modeling from Software Components," Proc. 4th Int'l Workshop Software and Performance (WOSP 04)," ACM Press, 2004, pp. 290–301.
1. T. McPhillips et al., "Scientific Workflow Design for Mere Mortals," Future Generation Computer Systems, vol. 25, no. 5, pp. 541–551, 2009.
2. B Ludscher et al., "Scientific Workflows: Business as Usual?" Proc. 7th Int'l Conf. Business Process Management (BPM 09), Springer, 2009, pp. 31–47.
3. M. Wilde et al., "Parallel Scripting for Applications at the Petascale and Beyond," Computer, vol. 42, no. 11, 2009, pp. 50–60.
4. A. Barker and J. van Hemert, "Scientific Workflow: A Survey and Research Directions," Parallel Processing and Applied Mathematics, LNCS 4967, Springer, 2008, pp. 746–753.
5. X. Yang, R.P. Bruin, and M.T. Dove, "Developing an End-to-End Scientific Workflow," Computing in Science and Eng., vol. 12, no. 3, 2010, pp. 52–61.
1. A.R. Shah et al., "An Architecture for Real Time Data Acquisition and Online Signal Processing for High Throughput Tandem Mass Spectrometry," Proc. 5th Int'l Conf. e-Science and Grid Computing, IEEE CS Press, 2009, pp. 88–93.
2. I. Gorton, A.S. Wynne, and Y. Liu, "Engineering High Performance Service-Oriented Pipeline Applications with MeDICi," to be published in Service-Oriented Computing, LNCS 6568, April 2010.
3. P. Tran, J. Gosper, and I. Gorton, "Evaluating the Sustained Performance of COTS-Based Messaging Systems," Software Testing, Verification and Reliability, vol. 13, no. 4, 2003, pp. 229–240.

Index Terms:
scientific software, pipelines, components, software engineering
Ian Gorton, Adam Wynne, Yan Liu, Jian Yin, "Components in the Pipeline," IEEE Software, vol. 28, no. 3, pp. 34-40, May-June 2011, doi:10.1109/MS.2011.23
Usage of this product signifies your acceptance of the Terms of Use.