Pegasus (workflow management) explained
Pegasus is an open-source workflow management system.[1] [2] [3] It provides the necessary abstractions for scientists to create scientific workflows[4] and allows for transparent execution of these workflows on a range of computing platforms including high performance computing clusters, clouds, and national cyberinfrastructure.[5] In Pegasus, workflows are described abstractly as directed acyclic graphs (DAGs) using a provided API for Jupyter Notebooks, Python, R, or Java.[6] During execution, Pegasus translates the constructed abstract workflow into an executable workflow[7] [8] which is executed and managed by HTCondor.[9] [10]
Pegasus is being used in a number of different disciplines including astronomy, gravitational-wave physics, bioinformatics, earthquake engineering, and helioseismology.[11] Notably, the LIGO Scientific Collaboration has used it to directly detect a gravitational wave for the first time.[7] [12] [13]
Area of applications
Application examples:[14] [5]
- Gravitational-Wave Physics
- Earthquake Science
- Bioinformatics
- Workflows for Volcanic Mass Flows
- Diffusion Image Processing and Analysis
- Spallation Neutron Source (SNS)
History
The development of Pegasus started in 2001.
See also
Notes and References
- [Ewa Deelman|E. Deelman]
- E.A. Huerta, R. Haas, E. Fajardo, D.S. Katz, S. Anderson, P. Couvares,J. Willis, T. Bouvet, J. Enos, W.T.C. Kramer, H.W. Leong, and D. Wheeler, "BOSS-LDG: A Novel Computational Framework That Brings Together Blue Waters, Open Science Grid, Shifter and the LIGO Data Grid to Accelerate Gravitational Wave Discovery", 2017 IEEE 13th International Conference on e-Science (e-Science); pp. 335-344 (2017)
- B. Riedel, B. Bauermeister, L. Bryant, J. Conrad, P. de Perio, R. W. Gardner,L. Grandi, F. Lombardi, A. Rizzo, G. Sartorelli, M. Selvi, E. Shockley, J. Stephen, S. Thapa, and C. Tunnell "Distributed Data and Job Management for the XENON1T Experiment", PEARC '18: Proceedings of the Practice and Experience on Advanced Research Computing;9, pp. 1-8 (2018)
- G. Amalarethinam, T. Lucia, A. Beena, “Scheduling Framework for Regular Scientific Workflows in Cloud”, International Journal of Applied Engineering Research; 10, no. 82 (2015)
- https://cacr.iu.edu/projects/swip/index.html/ The Scientific Workflow Integrity with Pegasus (SWIP)
- D. Weitzel, B. Bockelman, D. Brown, P. Couvares, F. Würthwein, and E.F. Hernandez, “Data Access for LIGO on the OSG”, Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact - PEARC17; 24, no. 1-6 (2017)
- Web site: Testing LIGO's Sensitivity. September 1, 2007. Research.gov. April 30, 2020.
- Duncan Brown and Ewa Deelman, "Looking for gravitational waves: A computing perspective", at Science Node; published June 8, 2011; retrieved April 30, 2020
- https://itnews.iu.edu/articles/2016/1m-nsf-award-goes-to-iu-led-data-integrity-project.php/ $1M NSF award goes to IU-led data integrity project
- Brian Mattmiller, "High Throughput Computing helps LIGO confirm Einstein's last unproven theory", at Morgridge Institute for Research; published March 7, 2016; retrieved May 1, 2020
- Sanden Totten, "Caltech Wasn't the Only SoCal School Helping Discover Gravitational Waves", at KPCC; published 11 February 2016; retrieved May 1, 2020
- D.A. Brown, P.R. Brady, A. Dietz, J. Cao, B. Johnson, J. McNabb, “A Case Study on the Use of Workflow Technologies for Scientific Analysis: Gravitational Wave Data Analysis. In: I.J Taylor, E. Deelman, D.B. Gannon, M. Shields (eds) Workflows for e-Science”, Springer, London; 13, pp. 39-59 (2007)
- D. Davis, T. Massinger, A. Lundgren, J.C. Driggers, A.L. Urban, and L. Nuttall, “Improving the sensitivity of Advanced LIGO using noise subtraction”, Classical and Quantum Gravity; 36, no. 5 (2019)
- [Ewa Deelman|E. Deelman]
