1. Gabatarwa & Bayyani
PUNCH4NFDI (Barbashi, Sararin Samaniya, Ƙwayoyin Nukiliya da Hadrons don Tsarin Bayanan Bincike na Ƙasa) babbar ƙungiya ce ta Jamus da DFG (Deutsche Forschungsgemeinschaft) ta ba da kuɗi. Tana wakiltar kimanin masana kimiyya 9,000 daga fagen ilimin barbashi, taurari, barbashin taurari, hadron, da ilimin nukiliya. Babban manufar ƙungiyar ita ce kafa dandalin bayanan kimiyya na haɗin kai, mai bin ka'idodin FAIR (Ana iya Samu, Ana iya Kaiwa, Ana iya Haɗawa, Ana iya Sake Amfani). Babban kalubalen da aka magance a cikin wannan takarda shine haɗa manyan na'urorin lissafi (HPC, HTC, Girgije) da albarkatun ajiya waɗanda cibiyoyin membobin suka ba da gudummawa "a cikin nau'i" a duk faɗin Jamus, kowannensu yana da tsarukan gine-gine, tarin software, da manufofin shiga daban-daban.
Don magance wannan, an ƙirƙira ra'ayoyin Compute4PUNCH da Storage4PUNCH. Waɗannan suna nufin samar da cikakkiyar damar shiga wannan tsarin da aka rarraba, ba tare da tsangwama ga tsarin aiki na yanzu ba.
Ƙungiya a Tsinkaye
- Masana Kimiyya da Aka Wakilta: ~9,000 PhD
- Manyan Cibiyoyi: Thüringer Landessternwarte, Jami'ar Bonn, KIT, Jami'ar Bielefeld, DESY
- Kudade: DFG, lokacin farko na shekaru 5
- Babban Manufa: Gina dandalin bayanai & lissafi na haɗin kai mai bin ka'idodin FAIR
2. Tsarin Kwamfuta na Haɗin Kai (Compute4PUNCH)
Ra'ayin Compute4PUNCH yana magance kalubalen yin amfani da yadda ya kamata da guntun albarkatun HTC, HPC, da Girgije masu tsarukan gine-gine, tsarin aiki (OS), da tsarin tabbatar da ainihi daban-daban.
2.1 Tsarin Tsakiya & Kalubalen Haɗawa
Babban ƙuntatawa shine ka'idar "ƙaramin tsangwama". Albarkatun sun riga suna aiki kuma ana raba su, don haka dole ne gyare-gyaren tsarin masu samarwa su zama ƙanƙanta. Maganin shine tsarin da ya mamaye wanda ke haɗa albarkatun ba tare da buƙatar canje-canje masu zurfi ga tsarin gudanarwa na asali ba.
2.2 Manyan Fasahohi & Abubuwan Haɗawa
- Tsarin Ayyuka na Mamaye: Ana amfani da HTCondor a matsayin babban mai tsara ayyukan haɗin kai, yana ƙirƙirar tafki ɗaya daga albarkatun daban-daban.
- Mai Tsara Albarkatun Meta: COBalD/TARDIS yana haɗa albarkatun daban-daban (misali, jerin HPC, VMs na girgije) cikin tafkin HTCondor cikin sauri da kuma a bayyane. Yana aiki azaman tsarin "matukin jirgi", yana ɗaukar albarkatun a madadin masu amfani.
- Yanayin Software: Fasahar Kwantena (misali, Docker, Singularity) da Tsarin Fayil na Na'ura ta CERN (CVMFS) suna tabbatar da samar da software mai iya faɗaɗawa, mai maimaitawa, da na musamman ga al'umma a duk faɗin nodes.
- Tabbatar da Ainihi & Izinin: Tsarin Tabbatar da Ainihi da Izinin (AAI) na tushen alama yana ba da damar shiga daidaitacce, wanda ke da mahimmanci ga yanayin cibiyoyi da yawa.
2.3 Shiga & Fuskar Mai Amfani
Masu amfani suna hulɗa da tsarin haɗin kai ta hanyoyin shiga da suka saba:
- Nodes na Shiga na Al'ada: Samun damar SSH zuwa tsarin ƙofar shiga.
- JupyterHub: Fuskar yanar gizo da ke ba da damar hulɗa da albarkatun lissafi da bayanai, yana rage matakin shiga don nazari mai sarƙaƙiya.
3. Tsarin Ajiya na Haɗin Kai (Storage4PUNCH)
Dangane da lissafi, Storage4PUNCH yana mai da hankali kan haɗa tsarin ajiya da al'umma ke samarwa.
3.1 Dabarun Haɗin Ajiya
Dabarar tana amfani da fasahohin da aka kafa daga al'ummar Kimiyyar Makamashi Mai Girma (HEP) don ƙirƙirar suna ɗaya da matakin shiga na rarraba ajiya.
3.2 Fasahar Aiki & Haɗawa
- Manyan Fasahohin Ajiya: Da farko dCache da XRootD. Waɗannan ana amfani da su sosai a cikin HEP don sarrafa bayanai masu girman petabyte a wurare daban-daban.
- Matakin Haɗin Kai: Ana haɗa tsarin zuwa tsarin gama gari, yana gabatar da tsarin fayil na ma'ana guda ɗaya ga masu amfani.
- Fasalolin Ci Gaba da Ake Bincike: Haɗa fasahar ajiyar ƙwaƙwalwa (don rage jinkiri/faɗin bandwidth) da tsarin sarrafa metadata don haɓaka gano bayanai da gudanarwa, da nufin zurfin haɗawa fiye da sauƙin samun fayil.
4. Aiwatar da Fasaha & Cikakkun Bayanai
4.1 Tsarin Lissafi don Tsara Albarkatun
Babban kalubalen tsarawa ana iya zayyana shi azaman matsalar ingantawa. Bari $R = \{r_1, r_2, ..., r_n\}$ ya zama tarin albarkatun daban-daban, kowannensu yana da halaye kamar tsarin gine-gine $(arch_i)$, cores masu samuwa $(c_i)$, ƙwaƙwalwa $(m_i)$, da farashi/fifiko $(p_i)$. Bari $J = \{j_1, j_2, ..., j_m\}$ ya zama tarin ayyuka tare da buƙatu $(c^{req}_j, m^{req}_j, arch^{req}_j)$.
Mai tsara meta (COBalD/TARDIS) yana nufin haɓaka amfani ko kayan aiki gabaɗaya, wanda za'a iya ƙirƙira shi kamar haka: $$\text{Ƙara } U = \sum_{j \in J} \sum_{r \in R} x_{jr} \cdot u(j, r)$$ bisa ga ƙuntatawa: $$\sum_{j} x_{jr} \cdot c^{req}_j \leq c_r, \quad \sum_{j} x_{jr} \cdot m^{req}_j \leq m_r, \quad x_{jr} \in \{0,1\}$$ inda $x_{jr}=1$ idan an tsara aikin $j$ akan albarkatu $r$, kuma $u(j,r)$ aikin amfani ne da ke dacewa da buƙatun aiki zuwa iyawar albarkatu da manufofi. Wannan yayi kama da bin packing ko multi-dimensional knapsack problem, wanda aka saba da shi a cikin wallafe-wallafen tsarawa na grid/girgije.
4.2 Sakamakon Ƙirar Farko & Aiki
Takardar ta ambaci "kwarewar farko tare da aikace-aikacen kimiyya da ake aiwatarwa akan ƙirar farko da ake da su." Duk da yake ba a ba da takamaiman ma'auni na ƙididdiga a cikin taƙaitaccen bayani ba, aiwatar da nasara tana nufin:
- Haɗin Aiki: Tsarin mamaye HTCondor/COBalD/TARDIS ya yi nasarar gabatar da ayyuka da sarrafa su a tsakanin tsarin baya daban-daban (jerin ayyukan HPC, gonakin HTC).
- Isar da Software: CVMFS da kwantena sun isar da yanayin software da ake buƙata ga nodes masu aiki daban-daban cikin aminci.
- Tsarin Aiki na AAI: AAI na tushen alama ya ba masu amfani daga wata cibiya damar shiga albarkatun wata cibiya ba tare da matsala ba.
Ra'ayin Zane (Tsarin Gine-gine da aka nuna): Zane na ma'ana zai nuna Wuraren Shiga na Mai Amfani (JupyterHub, Nodes na Shiga) suna haɗawa zuwa "Matakin Haɗin Kai" na tsakiya. Wannan matakin ya ƙunshi HTCondor (Mai Tsara Lissafi) da Haɗin Ajiya (dCache/XRootD). Ƙarƙashin wannan matakin, an nuna "Wuraren Albarkatu" da yawa (Site A, B, C...), kowannensu yana da HPC na gida, HTC, ko albarkatun girgije da ajiya, an haɗa su ta hanyar wakilan COBalD/TARDIS da ƙa'idodin ajiya. Bayanai suna gudana daga ajiyar haɗin kai zuwa albarkatun lissafi ta hanyar hanyoyin sadarwa masu sauri.
5. Tsarin Nazari & Binciken Misali
Binciken Misali: Nazarin Taurari Mai Saƙo Da Yawa
Labari: Mai bincike yana buƙatar haɗa babban bayanai daga na'urar hangen nesa ta neutrino (wanda aka adana akan misalin dCache a DESY) tare da bayanan binciken sararin samaniya na gani (wanda aka adana akan tsarin XRootD a Tautenburg) kuma ya gudanar da ƙirar koyon inji don rarrabuwa.
Tsarin Aiki a cikin Tsarin PUNCH4NFDI:
- Shiga & Gano: Mai bincike ya shiga cikin sabis na JupyterHub ta amfani da alamar AAI na ƙungiyar.
- Samun Bayanai: Ta hanyar sunan ajiya ɗaya na Storage4PUNCH, suna samun damar bayanan nesa biyu kamar su fayiloli na gida ne (
/punche/data/neutrino/event_set.h5,/punche/data/optical/survey_catalog.fits). - Gabatar da Aikin Lissafi: Sun rubuta rubutun nazari (Python tare da PyTorch). Rubutun ya ƙayyade buƙatun albarkatu: GPUs 4, RAM 64 GB, da takamaiman hoton kwantena (
registry.punch4nfdi.de/astroml:v2.1). - Haɗawa: Gabatarwa ta hanyar HTCondor, COBalD/TARDIS ya gano albarkatun da suka dace da buƙatun—watakila nodes na GPU a tarin HPC na KIT da girgije na Bielefeld.
- Aiwatarwa: An ja kwantena ta hanyar CVMFS, an kwarara bayanan daga ajiyar haɗin kai, kuma aikin ya gudana. Mai bincike yana lura da ci gaba ta hanyar fuskar JupyterHub ko umarnin HTCondor.
Wannan yana nuna "ba tare da tsangwama ba" ra'ayi: mai amfani yana ma'amala da tsarin guda ɗaya, ba sarƙaƙiyar tsarin cibiyoyi daban-daban 5+ ba.
6. Nazari Mai Mahimmanci & Hikimar Ƙwararru
Babban Hikima: PUNCH4NFDI ba ta gina sabon babban kwamfuta ba; tana injiniyan matakin haɗin kai na ban mamaki na siyasa da fasaha. Haƙiƙanin ƙirƙira yana cikin umarnin "ƙaramin tsangwama", wanda ya yarda da ikon mallakar masu samar da albarkatun da ake da su—gaskiyar da ba za a iya sasantawa ba a cikin kimiyyar Jamus da aka ba da kuɗaɗen jama'a. Wannan ya sa ya zama ƙasa da wani aikin "fasahar wata" kuma ya zama aikin "manne" mai sarƙaƙiya, yana nufin cire ƙima daga rarrabuwa.
Tsarin Ma'ana & Dalilin Dabarun: Ma'ana tana da kyau: 1) Karɓi bambancin a matsayin yanayi na dindindin, 2) Yi amfani da tsaka-tsakin tsaka-tsaki, waɗanda al'umma suka karɓa (HTCondor, dCache, XRootD) don rage haɗari da haɓaka karɓuwa, 3) Gabatar da tsarin haɗawa na zamani (COBalD/TARDIS, kwantena) don sarrafa bambancin, 4) Samar da ƙofofin shiga masu dacewa da mai amfani (JupyterHub) don ɓoye sarƙaƙiyar da ke ƙasa. Wannan yana bin nasarar zanen Tsarin Lissafi na Duniya na LHC (WLCG) amma ya faɗaɗa shi zuwa ƙarin nau'ikan kimiyya da nau'ikan albarkatu.
Ƙarfi & Kurakurai:
Ƙarfi: Zaɓin fasaha yana da kyau sosai. HTCondor yana da shekarun da aka tabbatar da iya faɗaɗawa. COBalD/TARDIS, wanda ya fito daga HEP, an gina shi da gangan don wannan samarwa mai ƙarfi, daban-daban. CVMFS shine ma'auni na zinare don rarraba software a cikin kimiyya mai yawan bayanai. Mayar da hankali kan AAI yana da mahimmanci ga amincewa tsakanin cibiyoyi.
Kurakurai/Hatsarori Masu Muhimmanci: Giwa a cikin daki shine ƙarin aiki da sarƙaƙiya. Tsarin mamaye dole ne ya ƙara jinkiri da wuraren gazawa. Tsarawa a tsakanin manufofin HPC daban-daban (lokacin jira na jerin gwano, iyakokin agogon bango) mafarki ne COBalD/TARDIS dole ne ya yi amfani da shi. Rashin bayanan aiki mai ƙarfi na takardar yana nuni—hujja za ta kasance a cikin ci gaba mai dorewa akan ayyukan samarwa, ba gudanar da ƙira ba. Bugu da ƙari, yayin da ake "binciken" ajiyar ƙwaƙwalwa, harkokin bayanai tsakanin ajiyar haɗin kai da wuraren lissafi na iya zama babban cikas, mai tunawa da kalubalen da aka fuskanta a farkon lissafi na grid. Rahoton 2023 daga Girgije na Bude Kimiyya na Turai (EOSC) akan "Haɗin kai na Tsarin Bincike" ya nuna cewa motsin bayanai ba tare da tsangwama ba ya kasance cikin manyan kalubale 5 ga irin waɗannan ƙungiyoyin haɗin kai.
Hikima Mai Aiki: Ga sauran ƙungiyoyin haɗin kai da ke ƙoƙarin irin wannan haɗin kai: 1. Ba da fifiko ga Siyasa fiye da Fasaha: Sami takardar yarjejeniya ta ɗaure kan raba albarkatu da manufofin AAI da farko. Fasaha shine ɓangaren da ya fi sauƙi. 2. Kafa Kowane Abu daga Rana ɗaya: Tura cikakken saka idanu (kamar Grafana/Prometheus) don auna ƙarin aiki na ainihi, ƙimar nasarar aiki, da ingancin canja wurin bayanai. Waɗannan bayanan suna da mahimmanci don tabbatar da ƙimar haɗin kai. 3. Yi Shirin "Dogon Wutsiya": Masu amfani na farko za su zama ƙwararrun lissafi. Don gaskiya ya bauta wa masana kimiyya 9,000, saka kuɗi mai yawa a cikin tsarin JupyterHub, tsarin aiki na samfuri, da horar da al'umma don shigar da dogon wutsiya na masu binciken waɗanda ke buƙatar ƙarfin amma ba su da ƙwarewar sys-admin. Nasarar ayyuka kamar shirin "Ƙofofin Kimiyya" na NSF na Amurka yana jaddada wannan buƙatar. 4. Duba Bayan Fasahar HEP: Duk da yake dCache/XRootD suna da ƙarfi, kuma a kimanta ƙarin hanyoyin shiga na ajiyar abubuwa na asalin girgije (S3) don haɗin kai mai faɗi tare da girgije na kasuwanci da sauran fannonin kimiyya, kamar yadda takarda farar "Ajiya ta Asalin Girgije" na CNCF ta ba da shawarar.
7. Ayyukan Gaba & Taswirar Ci Gaba
Tsarin PUNCH4NFDI ya kafa tushe don wasu manyan hanyoyin gaba:
- Haɗuwa tare da Girgije & Gefe: Tsarin zai iya haɗa fashewar girgije na kasuwanci (AWS, GCP, Azure) ba tare da tsangwama ba ta hanyar COBalD/TARDIS don buƙatu mai girma, kuma mai yuwuwa wuraren lissafi na gefe kusa da kayan aiki (misali, wuraren na'urar hangen nesa) don tacewa na ainihi.
- Haɗa Ayyukan AI/ML: Ƙarfafa goyon baya don tarin gungu masu ƙarfi, masu sassauƙa don horar da ML da aka rarraba (misali, haɗa masu aiki na Kubernetes tare da mamaye HTCondor) da wuraren adana da aka tsara na ƙirar da aka riga aka horar a cikin ajiyar haɗin kai.
- Haɗin Bayanai Mai Aiki & Lissafi: Ƙetare sauƙin samun fayil. Aikin gaba zai iya ba da damar ƙirar "aiki-azaman-sabisi" inda ake aika lambar nazari zuwa wurin bayanai (wanda aka yi wahayi daga tsarin "Bayanai Kusa da Ayyuka" a cikin ayyuka kamar Girgije na Bude Kimiyya na Turai).
- Haɗin Kai tsakanin Ƙungiyoyin Haɗin Kai: PUNCH4NFDI na iya zama abin koyi don haɗawa tare da sauran ƙungiyoyin haɗin kai na NFDI (misali, don kimiyyar rayuwa ko binciken yanayi), ƙirƙirar ainihin masana'antar bincike ta ƙasa. Wannan yana buƙatar aiki akan AAI na fage daban-daban da tsarin metadata.
- Tsarin Dorewa: Haɓaka takamaiman tsari don aiki na dogon lokaci da ba da kuɗi bayan kuɗin DFG na farko na shekaru 5, mai yuwuwa ya haɗa da raba farashi ko hanyoyin lissafi tsakanin cibiyoyi bisa ga amfani da albarkatu.
8. Nassoshi
- Ƙungiyar Haɗin Kai ta PUNCH4NFDI. (2024). Takarda Fari ta PUNCH4NFDI: Zuwa Dandalin Bayanan Kimiyya na FAIR. (Takarda ta Ciki ta Ƙungiyar Haɗin Kai).
- Thain, D., Tannenbaum, T., & Livny, M. (2005). Lissafi da aka rarraba a aikace: gwanintar Condor. Haɗin kai da Lissafi: Aiki da Gwaninta, 17(2-4), 323-356. https://doi.org/10.1002/cpe.938
- Ƙungiyar Girgije na Bude Kimiyya na Turai (EOSC). (2023). Rahoton Tsarin Haɗin Kai na EOSC. https://www.eosc.eu/advisory-groups/interoperability
- Blomer, J., et al. (2011). Rarraba software tare da CernVM-FS. Jaridar Physics: Taron, 331(4), 042003. https://doi.org/10.1088/1742-6596/331/4/042003
- Gidauniyar Lissafin Asalin Girgije (CNCF). (2022). Ajiya ta Asalin Girgije: Takarda Fari. https://www.cncf.io/reports/cloud-native-storage-whitepaper/
- Gidauniyar Kimiyya ta Ƙasa. (2021). Cibiyar Al'ummar Ƙofofin Kimiyya: Rahoton Shekara. https://sciencegateways.org
- Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Fassarar Hotuna-zuwa-Hoto mara Haɗin gwiwa ta amfani da Cibiyoyin Adawa na Ma'ana na Zagayowar. A cikin Proceedings na taron kwamfuta na IEEE (shafi na 2223-2232). (An ambata a matsayin misali na algorithm mai sarƙaƙiya, mai cike da albarkatu wanda zai amfana daga irin wannan tsarin haɗin kai).