About the Project

PNRR M4C2 I1.4 Campione Nazionale ICSC – Cascade call SPOKE 1 – “Future HPC and Big Data” (CN00000013 - CUP J33C22001170001)
Thematic Area: 3 – Development and testing of methodologies, middleware and tools for distributed computing for cloud-HPC, AI-HPC, AI-cloud-HPC convergence and their use in interdisciplinary applications.

scope

General objectives

The DAGonCAPIO project proposal “Directed Acyclic Graph on Cross-Application Programmable I/O” has general objectives:

1) formalize, design and implement an engine for scientific workflows capable of maximizing the overlap between I/O and computation;

2) integrate the management of the scratch directory into the workflow engine itself through an ad hoc file system;

3) develop, thanks to the synergy with the companies forming the partnership, an industrial tool that leverages the HPC-Cloud-AI convergence;

4) demonstrate the functionality of DAGonCAPIO through an application that leverages DNSH objectives.

Tools

Dagon & capio

Tools for orchestrating computational resources are one of the fundamental enabling technologies for computational sciences (e-science). The representation of producer/consumer data relationship-dependent tasks finds theoretical solidity in the Directed Acyclic Graph (DAG) and its technological implementation in engines managing scientific workflows. In a workflow consisting of two tasks (A and B, with A producing a[0..n[ file) such that B uses this data, B will be executed when A is completed. This leads to inefficiency due to a lack of overlap between computation and input/output: when A has already produced a[0..m[, B could start processing a[0..m[ while A produces a [m,..,n[.

CAPIO is a middleware capable of inserting I/O streaming functionality into workflows between tasks with file-based communication, improving the overlap between I/O and computing without the need to change the application code.

activities

Work packages

WP1 focuses on several objectives, including ensuring timely execution of the work plan, quality control, knowledge management, and effective communication between stakeholders. This work package includes procurement activities for the necessary hardware and software infrastructure. Furthermore, WP1 manages the project as a whole, including reporting activities, supervision of data management, and deals with financial, legal, administrative, and technical issues. It also includes technical project management, such as coordinated activities between various WPs, regular communication between project members via video conferences, organizing and conducting meetings, monitoring project performance, risk analysis, and contingency management, as well as planning, monitoring, and justifying resource allocation.

WP2 carries out the analysis of the requirements aimed at defining DAGonCAPIO, following the collection of specifications. The analysis activity allows setting up the scientific methodological approach to ensure the results expected from the project.
The necessary design effort to achieve the project objectives will be assessed, in particular, what the use case requires to determine how much is fundamental for subsequent development. This WP will also evaluate the new CAPIO features to be designed and implemented. The analysis activity is synergically related to the design activity (WP3), which will define the fundamental elements of DAGonCAPIO.

WP3 will focus on the DAGonCAPIO design activity and, in particular, on the following:
1) the architectural component intended for the formal representation of the workflow and the related technical implementation through the appropriate data structures with particular reference to the methodologies for the representation of data producing/consuming relationships in the presence of streaming I/O;
2) the primary core of the workflow engine;
3) the system quality monitoring and assurance system.
This WP will also design any new necessary CAPIO features for the project.

WP4 includes activities related to the development and release of the DAGonCAPIO prototype. These activities partially overlap in time with those envisaged in WP2 and WP3 to have a first working prototype approximately halfway through the project. This WP will also implement new necessary CAPIO features for the project.

The activities planned as part of WP5 will see the use of DAGonCAPIO for an actual use case consisting of a multidisciplinary application contextualized in computational marine environmental sciences to assess pollution in the sea. The workflow will allow the coupling of a meteorological model (WRF) to a coastal dynamics model (ROMS). The data produced by the ROMS model will feed the pollutant transport and dispersion model WaComM++, coupled with the AIQUAM water quality model. The use case requires at least the coupling between WaComM++ and AIQUAM to be implemented through the I/O streaming typical of DAGonCAPIO.

As part of WP6, the use case will be tested on on-premises, cloud, and hybrid computing resources thanks to the experience of the industrial partners.

WP7 concerns the dissemination of research results, the promotion of DAGonCAPIO releases, and the public dissemination of the project. A web portal dedicated to the project will be created. Workshops and technical presentations will be held to encourage the exchange of knowledge with other research groups, companies, and associations, as well as training events, discussion forums, and mailing lists. Promotional material will be prepared to present the project results and encourage their use.