Developing Large-Scale Parallel Programs in Python with Parsl
DescriptionAddressing grand science challenges increasingly relies on extreme-scale computing capabilities to solve big data problems, scale machine learning algorithms, and integrate multi-scale techniques. Such extreme-scale needs arise, for example, when using machine learning to screen billions of small molecules in the search for SARS-CoV-2 therapeutics, to understand the origins of the universe, and to to explore the brain’s neuroanatomic connectome at extreme-scale resolution.
This workshop will introduce Parsl, a Python library that allows the automated and reproducible execution of many tasks (Python functions and external executables) concurrently using parallel and distributed computing systems ranging from multi-core laptops to clouds and supercomputers with thousands of nodes. It will also include hands-on activities to develop a real-world molecular science example using machine learning to demonstrate how Parsl enables large-scale execution of simulation and model training.
Researchers can use Parsl to decrease the iteration time between asking questions and calculating answers, allowing qualitative differences in how the next set of questions are developed, and developers can use it to build scalable platforms. Parsl is an open source project with many contributors, aiming to expand and diversify its user and developer communities.
TimeThursday, September 14th11:00am - 12:15pm CDT