Objective
Maestro will build a data and memory-aware middleware framework that addresses the ubiquitous problems of data movement in complex memory hierarchies that exist at multiple levels of the HPC software stack.
Though High Performance Computing (HPC) and High Performance Data Analytics (HPDA) applications pose a broad variety of efficiency challenges, the performance of both has become dominated by data movement through the memory and storage systems, rather than the floating point computational capability. Despite this shift, current software technologies remain severely limited in their ability to optimise data movement. MAESTRO, the ‘Middleware for memory and data-awareness in workflows’ project, will address the two following major impediments of the modern HPC software:
- DATA AWARENESS:Moving data through memory has not always been a performance bottleneck. The software stack that HPC relies upon was built during the decades where the cost of performing floating point operations (FLOPS) was paramount. This has resulted in a software stack and set of programming models that are optimised for floating point operations but lacking in basic data handling functionality. We characterise the set of technical issues as ‘missing data-awareness’.
- MEMORY AWARENESS: Software rightfully insulates users from hardware details, particularly as we move up the software stack. But HPC applications, programming environments and system software cannot make key data movement decisions without some understanding of the hardware, particularly of the increasingly complex memory hierarchy. With the exception of runtimes, which treat memory in a domain-specific manner, software typically makes hardware-neutral decisions which can often result in poor performance We characterise this issue as ‘missing memory-awareness’.
Maestro proposes a middleware framework that enables memory and data-awareness. Maestro has developed new data-aware abstractions that can be used in any level of software, e.g. compiler, runtime or application. A core element are the Core Data Objects (CDO), which are provided to the middleware through a give/take semantics. The middleware is being designed such that it enables modelling of memory and storage hierarchies to allow for reasoning about data movement and placement based on costs of moving data objects. The middleware will support automatic movement and promotion of data in memories and storage and allow for data transformations and optimisation.
Maestro follows a co-design methodology using a set of applications and workflows from diverse areas including numerical weather forecasting, earth-system modelling, materials sciences and in-situ data analysis pipelines for computational fluid dynamics simulations.
Societal impact
Maestro will develop a framework to improve performance of data movement in applications. By improving the ease-of-use of complex memory hierarchies Maestro will help by:
- improving the performance of software, and therefore the energy consumption and CPU hours used by software;
- encouraging the uptake of parallel computing and HPC systems by new communities by lowering the memory performance barrier.
Maestro has the potential to influence a broad range of human discovery and knowledge, as every computational application relies on data movement.
Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far
The project adopted a co-design process for designing the new Maestro middleware. For a relevant set of applications, which all require high-performance computing (HPC) capabilities, requirements have been identified and documented. On this basis the core middleware as well as different system software components designed and initial versions implemented. The system software components include tools for code analysis, a data-aware run-time, a prototype for an Maestro-aware workflow framework, a framework for guided I/O, telemetry tools as well as tools for dynamics provisioning of storage.
The main technical achievements of the initial part of the Maestro projects are the following:
- Establishment of detailed requirements, justified through relevant HPC use cases, to influence the design decisions for the Maestro middleware
- Specification of the core middleware and its first implementations
- Design of execution framework architecture as well as development of the access semantics
- Realisation of the first prototype working implementation of the MIO interface
- Specification of demonstrators for the Maestro technology as well as some early prototypes
Progress beyond the state of the art and expected potential impact
Including the socio-economic impact and the wider societal implications of the project so far
- Data models: A new approach has been specified and implemented that provide a much higher abstraction level than existing approaches
- Workflow management: A concept for introducing data- and memory-awareness in existing workflow frameworks have been formulated
- Dynamic Provisioning: A new solution for dynamic provisioning of storage has been implemented and demonstrated
Latest updated: December 2020
