Ergonomie a efektivita workflow na HPC klastrech

Abstract

This thesis deals with the execution of task graphs on High-performance Computing (HPC) clusters (supercomputers), with a focus on efficient usage of hardware resources and ergonomic interfaces for task graph submission. Task-based programming is a popular approach for defining scientific workflows that can be computed on distributed clusters. However, executing task graphs on su- percomputers introduces unique challenges, such as performance issues caused by the large scale of HPC workflows or cumbersome interactions with HPC allocation managers like PBS (Portable Batch System) or Slurm. This work examines what are the main challenges in this area and how do they affect task graph execution, and it proposes various approaches for alleviating these challenges, both in terms of efficiency and developer ergonomics. This thesis provides three main contributions. Firstly, it provides a task graph simulation en- vironment that enables prototyping and benchmarking of various task scheduling algorithms, and performs a comprehensive study of the performance of various task schedulers using this envi- ronment. Secondly, it analyzes the bottlenecks and overall performance of a state-of-the-art task runtime Dask and provides an implementation of an alternative Dask server which significantly improves its performance in HPC use-cases. And primarily, it introduces a unified meta-scheduling and resource management design for effortless execution of task graphs on heterogeneous HPC clusters that facilitates efficient usage of hardware resources. It also provides a reference im- plementation of this design within an HPC-tailored task runtime called HyperQueue, which is available as open-source software under the MIT (Massachusetts Institute of Technology) license at https://github.com/it4innovations/hyperqueue.