Fall 2022 Workflow Automation for Many Task Computing
views
comments
Historically, high-performance computing (HPC), has primarily involved tightly coupled simulations executed in a synchronous fashion across nodes. More recently, other paradigms within research computing have become common where the workload is not a single large coupled task, but instead very many modest tasks with little to no dependency on each other. These could be data processing and analysis tasks, machine-learning experiments, bioinformatics tasks, or parameter searches in a calculation.
In these many-task, high-throughput computing (HTC) workloads, researchers often attempt some form of parallelization within their application language (e.g., Python, R, or MATLAB ®). This is tedious, fraught with difficulty for most users, lacking flexibility, and is a distraction from the research concerns of the project. It would be better to use a distinct workflow automation tool to manage the many individual tasks.
This workshop outlines this paradigm on a traditional SLURM cluster along with several solutions. Different tools and techniques are discussed in an escalating fashion covering various features and pitfalls. Finally, a demonstration of the hyper-shell utility will showcase the level of sophistication possible both for individuals and research groups in managing task execution. (hyper-shell.readthedocs.io)
Purdue University, 610 Purdue Mall, West Lafayette, IN 47907, (765) 494-4600
© 2015 Purdue University | An equal access/equal opportunity university | Copyright Complaints | Maintained by ITaP
Trouble with this page? Disability-related accessibility issue? Please contact ITaP at [email protected].