The Community for Technology Leaders
2014 9th Workshop on Workflows in Support of Large-Scale Science (WORKS) (2014)
New Orleans, LA, USA
Nov. 16, 2014 to Nov. 16, 2014
ISBN: 978-1-4799-7067-4
pp: 31-40
ABSTRACT
The Advanced lightsource (ALS) is a X-ray synchrotron facility at Lawrence Berkeley National Laboratory. The ALS generates terabytes of raw and derived data each day and serves 1,000's of researchers each year. Only a subset of the data is analyzed due to barriers in terms of processing that small science teams are ill-equipped to surmount. In this paper, we discuss the development and application of a computational framework, termed SPOT, fed with synchrotron data, powered by storage, networking and compute resources at NERSC and ESnet. We describe issues and recommendations for an end-to-end analysis workflow for ALS data. After one year of operation, the collection contains over 90,000 datasets (550 TB) from 85 users across three beamlines. For 16 months, beamline data taken has been promptly and automatically analyzed and annotated with metadata, allowing users to focus on analysis, conclusions and experiments.
INDEX TERMS
Detectors, Real-time systems, Databases, Data analysis, Pipelines, Data transfer, Image reconstruction
CITATION
Jack Deslippe, Abdelilah Essiari, Simon J. Patton, Taghrid Samak, Craig E. Tull, Alexander Hexemer, Dinesh Kumar, Dilworth Parkinson, Polite Stewart, "Workflow Management for Real-Time Analysis of Lightsource Experiments", 2014 9th Workshop on Workflows in Support of Large-Scale Science (WORKS), vol. 00, no. , pp. 31-40, 2014, doi:10.1109/WORKS.2014.9
95 ms
(Ver 3.3 (11022016))