The Clear Cloud - Home


Orchestrating Scientific Workflows on the Cloud
Junaid Arshad
FEB 19, 2015 15:33 PM
A+ A A-

By Junaid Arshad

Cloud Computing

One of the most compelling benefits of Cloud computing has been its ability for on-demand provision of extraordinarily scalable resources. This has led to a widespread adoption of cloud computing for diverse application domains including both industry and academia. By virtue of its commercial viability, it has become an extremely attractive prospect especially for Small and Medium Enterprises (SMEs). An excellent example of cloud adoption within this industry segment is the EU funded CloudSME project which is aimed at this specific industry segment and has successfully demonstrated the feasibility of cloud computing technology for SMEs in general and for SMEs in simulation and manufacturing in particular. For academia, the ability to call upon gigantic compute and storage resources at tiny or no cost at all is a godsend and has transformed research across diverse scientific disciplines. The EU funded ER-Flow project is a very good example and is aimed at facilitating scientists from disciplines such as Astrophysics, Heliophysics and Computational Chemistry to be able to run their computations on the community supported clouds at no cost.

Cloud Orchestration

Cloud orchestration services have recently emerged to improve the overall usability of the cloud infrastructures i.e. to simplify user interaction, deployment of instances and so on. These services allow configuring, managing, maintaining, deploying and scaling on the cloud. They provide flexibility in instance, service and infrastructure deployments with a few clicks/commands. A cloud orchestration service will facilitate deploying an entire infrastructure in accordance with the requirements of the workflow. Having said that, the automation of processes underpins success of cloud computing which becomes a major challenge when managing heterogeneous environments with disparate systems as emphasized here. Cloud orchestration attempts to address the issue of messy automation by:

  1. Integrating cloud resources across heterogeneous platforms to simplify service construction, deployment and management
  2. Dynamic creation and reconfiguration of cloud services
  3. Provision of applications as ready-to-deploy services agnostic of underlying infrastructure  
  4. Dynamic management of relationships among cloud services
  5. Real-time monitoring and management of underlying infrastructure

Evolution of Scientific Workflows

Grid technologies (Globus, gLite and UNICORE) have been being used extensively for scientific workflow execution. However, a number of outstanding issues have been identified with grid technologies spanning across functionality provided, deployment and maintenance which hamper usage and advancements within the scientific workflows. Cloud computing has been identified as the next generation technology to address the challenges posed by contemporary grid technologies by providing flexibility in deployment, maintenance and use of the infrastructure required for workflow execution. Within the ER-Flow project, we have chosen cloud orchestration technologies to support cloud based workflow execution mainly due to the flexibility and on the fly deployment benefits



Cloud Orchestration Services

Recently, there has been growing number of available solutions for orchestrating services on the Cloud. These include Juju, Heat, Puppet, Chef, BOSH and OCCO. However, each of these solutions has different focus and therefore address different problems and can be adopted as complimentary solutions. The figure below presents one such comparison of these solutions.


Due to their focus on hassle-free service orchestration, we shortlisted JuJu, Heat and OCCO for our analysis which is presented in the figure below.

Comparison of Cloud Orchestration Solutions

Based on our analysis, we have selected JuJu as the orchestration solution to facilitate cloud based workflow execution. Our selection of JuJu is based on the following factors which we believe will prove detrimental in near future.

·         Multi-cloud support - supports Azure, Openstack, EC2 and HP clouds compared to just Openstack by Heat

·         TOSCA compliance - JuJu is progressing rapidly with TOSCA in collaboration with IBM

·         Greater organizational support - Canonical  has been recently joined by IBM to support JuJu

·         Focus on the service layer - JuJu facilitates provision of dedicated deployment environments without the hassle of managing underlying infrastructure

·         Programmatic interface to managing service deployment lifecycle - API for creation, deletion, maintenance of services

We are thrilled by the flexibility and ease of use offered by JuJu Charms and look to continue our work with it to support workflows from diverse scientific disciplines to be executed on the cloud.

[%= name %]
[%= createDate %]
[%= comment %]
Share this:
Please login to enter a comment:

Computing Now Blogs
Business Intelligence
by Keith Peterson
Cloud Computing
A Cloud Blog: by Irena Bojanova
The Clear Cloud: by STC Cloud Computing
Computing Careers: by Lori Cameron
Display Technologies
Enterprise Solutions
Enterprise Thinking: by Josh Greenbaum
Healthcare Technologies
The Doctor Is In: Dr. Keith W. Vrbicky
Heterogeneous Systems
Hot Topics
NealNotes: by Neal Leavitt
Industry Trends
Internet Of Things
Sensing IoT: by Irena Bojanova