Compute4PUNCH & Storage4PUNCH: Federated Infrastructure for Particle, Astro-, and Nuclear Physics

1. Introduction

Particles, Universe, NuClei and Hadrons for the National Research Data Infrastructure (PUNCH4NFDI) is a German consortium funded by the DFG (Deutsche Forschungsgemeinschaft). It represents approximately 9,000 scientists from particle, astro-, astroparticle, hadron, and nuclear physics communities. The consortium's prime goal is to establish a federated and FAIR (Findable, Accessible, Interoperable, Reusable) science data platform. This platform aims to provide unified access to the diverse and heterogeneous computing and storage resources contributed by its member institutions across Germany, addressing the common challenge of analyzing exponentially growing data volumes with complex algorithms.

2. Federated Heterogeneous Compute Infrastructure – Compute4PUNCH

The Compute4PUNCH concept addresses the challenge of providing seamless access to a wide array of in-kind contributed High-Throughput Compute (HTC), High-Performance Compute (HPC), and Cloud resources. These resources vary in architecture, OS, software, and authentication, and are already operational and shared, necessitating a non-intrusive integration approach.

3. Federated Storage Infrastructure – Storage4PUNCH

Parallel to compute, the Storage4PUNCH concept federates community-supplied storage systems, primarily based on dCache and XRootD technologies, which are well-established in High-Energy Physics (HEP).

4. Technical Details & Mathematical Framework

The core scheduling challenge can be modeled as a resource optimization problem. Let $R = \{r_1, r_2, ..., r_n\}$ represent the set of heterogeneous resources, each with attributes like architecture, available cores $c_i$, memory $m_i$, and queue wait time $w_i$. Let $J = \{j_1, j_2, ..., j_m\}$ represent jobs with requirements $\hat{c}_j, \hat{m}_j$.

The meta-scheduler (COBalD/TARDIS) aims to maximize overall utility or throughput. A simplified objective function for job placement could be to minimize the makespan or maximize resource utilization, considering constraints:

$\text{Minimize } \max_{r \in R} (\text{completionTime}(r))$

subject to: $\sum_{j \in J_r} \hat{c}_j \leq c_r \quad \text{and} \quad \sum_{j \in J_r} \hat{m}_j \leq m_r \quad \forall r \in R$

where $J_r$ is the set of jobs assigned to resource $r$. The dynamic nature is handled by TARDIS, which "tricks" HTCondor into seeing remote resources as part of its local pool.

5. Experimental Results & Prototype Status

The paper reports on the current status and first experiences with scientific applications on available prototypes. While specific benchmark numbers are not detailed in the provided excerpt, the successful execution of real scientific workloads is implied. The integration of HTCondor with COBalD/TARDIS has been demonstrated to dynamically integrate resources from different administrative domains. Initial user access via JupyterHub and token-based AAI has been tested, providing a proof-of-concept for the unified entry point. The use of CVMFS has been validated for delivering necessary software environments across the federated infrastructure.

6. Analysis Framework: A Conceptual Case Study

Consider a multi-messenger astrophysics analysis searching for neutrino counterparts to gamma-ray bursts. The workflow involves:

1. Data Discovery: A researcher uses the federated metadata catalog (under evaluation in Storage4PUNCH) to locate relevant neutrino event data from IceCube and gamma-ray data from Fermi-LAT, stored in dCache instances at DESY and Bielefeld.
2. Workflow Submission: Via the JupyterHub interface, the researcher defines a parameter sweep analysis. The job requirements (software: Python, IceCube software suite via CVMFS; compute: 1000 CPU-hours) are specified.
3. Orchestration: The HTCondor overlay, guided by COBalD/TARDIS, dynamically matches and dispatches hundreds of jobs to available slots across KIT's HPC, Bonn's HTC, and cloud resources. The token AAI handles authentication seamlessly.
4. Execution & Data Access: Jobs pull software from CVMFS, read input data directly from the federated storage via XRootD doors, and write intermediate results to a temporary storage space.
5. Result Aggregation: Final results are aggregated and written back to a persistent, FAIR-compliant repository within the Storage4PUNCH federation.

This case demonstrates the value proposition: a scientist interacts with a single, coherent system to leverage nationally dispersed, heterogeneous resources without managing the underlying complexity.

7. Application Outlook & Future Directions

The combined Compute4PUNCH and Storage4PUNCH infrastructure has significant potential beyond the initial PUNCH communities:

• Cross-Domain Federation: The model could be extended to other NFDI consortia or European Open Science Cloud (EOSC) initiatives, creating a true pan-European federated infrastructure.
• Integration of Edge Computing: For fields like radio astronomy or detector monitoring, integrating edge compute resources near sensors could be a logical next step.
• AI/ML Workload Support: Enhancing the scheduler to natively support GPU/accelerator resources and frameworks like Kubernetes for large-scale ML training jobs.
• Advanced Data Management: Deeper integration of intelligent data placement, lifecycle management, and active metadata catalogs to optimize data-intensive workflows.
• Quantum Computing Hybrid: As quantum computing matures, the federation could incorporate quantum processors as specialized resources for specific algorithm steps.

The success of this federation will depend on sustainable funding, operational robustness, and continued community buy-in to the federated model over local optimization.

8. References

1. PUNCH4NFDI Consortium. "PUNCH4NFDI – Particles, Universe, NuClei and Hadrons for the NFDI." White Paper, 2021.
2. Thain, D., Tannenbaum, T., & Livny, M. "Distributed computing in practice: the Condor experience." Concurrency and Computation: Practice and Experience, 17(2-4), 323-356, 2005.
3. Blomer, J., et al. "CernVM-FS: delivering scientific software to globally distributed computing resources." Journal of Physics: Conference Series, 396(5), 052018, 2012.
4. Fuhrmann, P., & Gulzow, V. "dCache, storage system for the future." In European Conference on Parallel Processing (pp. 1106-1113). Springer, Berlin, Heidelberg, 2006.
5. XRootD Collaboration. "XRootD – A highly scalable architecture for data access." WSEAS Transactions on Computers, 10(11), 2011.
6. Isard, M., et al. "Quincy: fair scheduling for distributed computing clusters." In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles (pp. 261-276), 2009. (For scheduling theory context).
7. Wilkinson, M. D., et al. "The FAIR Guiding Principles for scientific data management and stewardship." Scientific data, 3(1), 1-9, 2016.

9. Original Analysis: Core Insight, Logical Flow, Strengths & Flaws, Actionable Insights

Core Insight: PUNCH4NFDI isn't building a new supercomputer; it's engineering a federation layer of minimal viable intrusion. This is a pragmatic, politically astute response to the real-world constraint of Germany's fragmented, community-owned research computing landscape. The true innovation lies not in the individual technologies—HTCondor, dCache, CVMFS are battle-tested—but in their orchestration into a coherent national system with a token-based AAI as the glue. It's a classic "overlay network" strategy applied to cyberinfrastructure, reminiscent of how the internet itself was built atop diverse physical networks. As the European Open Science Cloud (EOSC) grapples with similar federation challenges, PUNCH's approach offers a concrete, operational blueprint.

Logical Flow: The logic is compellingly simple: 1) Accept heterogeneity as a permanent state, not a problem to be eliminated. 2) Use lightweight meta-scheduling (COBalD/TARDIS) to create a virtual pool, avoiding the need to modify entrenched local schedulers (SLURM, PBS, etc.). 3) Decouple identity and access management via tokens, sidestepping the nightmare of reconciling institutional accounts. 4) Decouple software from infrastructure via CVMFS/containers. 5) Apply the same federation logic to storage. The flow is from user-facing simplicity (JupyterHub) down through abstraction layers to underlying complexity.

Strengths & Flaws: The overwhelming strength is practical deployability. By demanding minimal changes from resource providers, it lowers the barrier to participation, which is crucial for bootstrapping a consortium. Leveraging mature HEP tools ensures reliability and reduces development risk. However, the flaws are in the trade-offs. The overlay model can introduce performance overheads in job dispatch and data access compared to a tightly integrated system. The "lowest common denominator" abstraction might limit access to unique features of specific HPC systems. Most critically, the long-term sustainability model is unproven—who pays for the central coordination, the meta-scheduler maintenance, and the user support? The project risks building a brilliant prototype that withers after the initial 5-year DFG funding.

Actionable Insights: For other consortia, the key takeaway is to start with governance and lightweight integration, not a grand technical redesign. 1) Immediately adopt a token-based AAI; it's the foundational enabler. 2) Prioritize the user experience (JupyterHub) to drive adoption; scientists won't use a cumbersome system. 3) Instrument everything from day one. To secure future funding, they must generate compelling metrics on increased resource utilization, cross-institutional collaboration, and scientific throughput. 4) Plan for the "second federation"—how to interconnect with other NFDI consortia or EOSC. The technical architecture should be explicitly designed for nested federation. Finally, they must develop a clear cost-sharing model for the central services, moving beyond project grants to a cooperative operational funding model akin to WLCG (Worldwide LHC Computing Grid). The technology is ready; the enduring challenge is socio-technical.