Events

Presentation: Bridging gaps: The Maestro Data-Aware Middleware
Description: Description: We will introduce the Maestro middleware framework, a data- and memory-aware abstraction for workflow coupling, inter- and intra-application data exchange and redistribution. A central design goal was to enable modelling of memory and storage hierarchies to allow for reasoning about data movement and placement based on costs of moving data objects. At the same time, data objects can carry user-defined metadata as it is typical in meteorological production workflows, allowing workflow management software to reason about the data without inspecting it, at scale. We will also present an outlook on future requirements for such frameworks in the context of federated data handling and cross-site workflow coupling.
Speaker: Utz-Uwe Haus (HPE HPC EMEA Research Lab)
Location: Virtual Event
More information: http://optimizationworkshop2020.zib.de/

Presentation: Bridging gaps: The Maestro Data-Aware Middleware
Speaker: Utz-Uwe Haus (HPE HPC EMEA Research Lab)
Location: Virtual Event
More information: https://events.ecmwf.int/event/169/overview
Slides: https://events.ecmwf.int/event/169/contributions/2736/attachments/1430/2571/HPC_WS-Haus.pdf
Video recording: https://vimeo.com/610643511

Title: Middleware for Memory and Data-Awareness in Workflows (Maestro) Presenter Dirk Pleiter
Session Time: Tuesday, 6 July 2021 16:30 – 19:00 CEST
Location: Virtual Event

More information: https://pasc21.pasc-conference.org/program/schedule/presentation/?id=post132&sess=sess182 

Session: Data Movement Orchestration on HPC Systems
Time: Tuesday, 6 July 202114:00 – 16:00 CEST
Location: Mère Royaume
Description: The next HPC systems are expected to feature millions of cores leading to more parallelism for large-scale simulations and workflows in varied scientific domains. However, there is a price to pay: the gap between computing power and data movement performance will keep increasing more and more and will inevitably intensify the bottleneck caused by data movements. Thus, it becomes necessary to consider data movement within those modern architectures at all the stages of the data lifetime and to come up with software components for orchestrating this movement. In particular, we can identify several key aspects among which: data semantics for expressing applications needs and memory abstraction for portability and extendibility given the multiplicity of memory and storage tiers. During this Minisymposium, we will focus on those two facets and present very promising ongoing projects coming from academia and industry proposing to address the data orchestration challenge and facilitate efficient exploitation of future exascale systems. An open discussion with the speakers will close the session.

More information: https://pasc21.pasc-conference.org/program/schedule/session/?sess=sess137

• Title: Dynamic Provisioning of Storage Resources: A Case Study with Burst Buffers
• Type: Presentation
• Speaker: Maxime Martinasso
• Description: Complex applications and workflows needs are often exclusively expressed in terms of computational resources on HPC systems. In many cases, other resources like storage or network are not allocatable and are shared across the entire HPC system. By looking at the storage resources in particular, any workflow or application should be able to select both its preferred data manager and its required storage capability or capacity. To achieve such a goal, new mechanisms should be introduced. In this work, we present such a tool that dynamically provision a data management system on top of storage devices. We propose a proof-of-concept that is able to deploy, on-demand, a parallel filesystem across intermediate storage nodes on a Cray XC50 system. We show how this mechanism can be easily extended to support more data managers and any type of intermediate storage. Finally, we evaluate the performance of the provisioned storage system with a set of benchmarks.
• More info: https://indico.psi.ch/event/11205/overview
• PDF presentation: https://indico.psi.ch/event/11205/contributions/28778/attachments/18949/29919/HPC-CH0521_DynPro-3.pdf

• Presentation title: ” Open Source Object Storage for All: Join the Mass-Capacity Storage Revolution” by Utz Uwe Haus.
• Description:In this session, you’ll learn about the CORTX community, as well as how you can participate in influencing the roadmap of the world’s only 100% open source object storage that’s optimized for data-intensive workloads.
• More info: https://www.seagate.com/es/es/datasphere-2021/sessions/cortx/

• Presentation title: “Challenges and Solutions for Data Abstraction and Movement in Heterogeneous High Performance Computing Systems” by Christopher Haine.
• Description: High Performance Computing architectures are becoming increasingly heterogeneous, introducing complex memory hierarchies that present significant challenges for the performance, portability, and productivity, of scientific applications and developers. We present efforts to address these challenges from the perspectives of programmability and data movement optimisation, resulting from activities of the HPE EMEA Research Lab in two ongoing research projects, EPiGRAM-HS and Maestro, funded by the EU Horizon 2020 programme.
• More info: https://www.hpe.com/be/en/compute/hpc/emea-research-lab.html

• Title: Orchestrating Data at System Level and Beyond
• Type: Talk
• Speaker: Dirk Pleiter
• Description: Data orchestration in the presence of more complex memory and storage hierarchies, as well as distributed e-infrastructures, is an ongoing challenge. In this talk, we consider different strategies for addressing this challenge in different context with focus on generic data transport functionality. We discuss different options for how to exploit the middleware both from an application perspective as well as the perspective of adding scheduling and placement capabilities.
• More info: https://sos24.cscs.ch/session/orchestrating-data-at-system-level-and-beyond/

– Virtual Conference on Trends in Mathematical Modelling, Simulation and Optimisation: Theory and Applications
– Presentation title: “The brave new world of Exascale computing: Computation is free, Data Movement is not” by Utz-Uwe Haus.
– More info: https://www.trr154.fau.de/trends-in-mathematical-modelling-simulation-and-optimisation/ 

– Virtual Workshop on AI and HPC convergence, virtual event organised by EXDCI2
– Presentation title: “Maestro: Orchestrating Data for HPC (and AI) Applications” by Dirk Pleiter.
– Slides are available here: https://exdci.eu/sites/default/files/public/files/maestro-pleiter-20201126.pdf

Maestro will participate at the EXDCI2 booth at the exhibitor dedicated space Europa Village at the Forum Teratec 2020.
The TERATEC Forum is a major event in France and Europe that brings together the best international experts in HPC, Simulation and Big Data.