This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Fifth IEEE/ACM International Workshop on Grid Computing (GRID'04)
Parallel and Distributed Astronomical Data Analysis on Grid Datafarm
Pittsburgh, PA
November 08-November 08
ISBN: 0-7695-2256-4
Naotaka Yamamoto, Grid Technology Research Center, AIST, Japan
Osamu Tatebe, Grid Technology Research Center, AIST, Japan
Satoshi Sekiguchi, Grid Technology Research Center, AIST, Japan
A comprehensive study of the whole petabyte-scale archival data of astronomical observatories has a possibility of new science and new knowledge in the field, while it was not feasible so far due to lack of enough data analysis environment. The Grid Datafarm architecture is designed for global petabyte-scale data-intensive computing, which provides a Grid file system with file replica management for fault tolerance and load balancing, and parallel and distributed data computing support for a set of files, to meet with the requirements of the comprehensive study of the whole archival data.
In the paper, we discuss about worldwide parallel and distributed data analysis in the observational astronomical field. The archival data is stored, replicated and dispersed in a Gfarm file system. All the astronomical data analysis tools successfully access files in Gfarm file system without any code modification, using a syscall hooking library regardless of file replica locations. Performance evaluation of the parallel data analysis in several ways shows file-affinity process scheduling plays an essential role for scalable and efficient parallel file I/O performance. A data calibration tools shows scalable file I/O performance, and achieved the file I/O performance of 5.9 GB/sec and 4.0 GB/sec for reading and writing FITS files, respectively, using 30 cluster nodes (60 CPUs). On-demand file replica creation mitigates the overhead of access concentration. Another tool shows the performance improvement at a factor of six for reading a shared file by creating file replicas.
Citation:
Naotaka Yamamoto, Osamu Tatebe, Satoshi Sekiguchi, "Parallel and Distributed Astronomical Data Analysis on Grid Datafarm," grid, pp.461-466, Fifth IEEE/ACM International Workshop on Grid Computing (GRID'04), 2004
Usage of this product signifies your acceptance of the Terms of Use.