Scalability Evaluation of Data Transfer Framework for Multi-Component Applications

• Item type: SIG Technical Reports(1)
• 公開日: 2020-12-14
• タイトル: Scalability Evaluation of Data Transfer Framework for Multi-Component Applications
• タイトル / 言語: en
• タイトル: Scalability Evaluation of Data Transfer Framework for Multi-Component Applications
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: 通信
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_18gh
• 資源タイプ: technical report
• 著者所属: RIKEN Center for Computational Science
• 著者所属: RIKEN Center for Computational Science
• 著者所属: RIKEN Center for Computational Science
• 著者所属: RIKEN Center for Computational Science
• 著者所属(英): en
• 著者所属(英): RIKEN Center for Computational Science
• 著者所属(英): en
• 著者所属(英): RIKEN Center for Computational Science
• 著者所属(英): en
• 著者所属(英): RIKEN Center for Computational Science
• 著者所属(英): en
• 著者所属(英): RIKEN Center for Computational Science
• 著者名: Jie, Yin; Balazs, Gerofi; Atsushi, Hori; Yutaka, Ishikawa
• 著者名(英): Jie, Yin; Balazs, Gerofi; Atsushi, Hori; Yutaka, Ishikawa
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Multi-component workflows play a significant role in High-Performance Computing and Big Data applications. They usually contain multiple, independently developed components that execute side-by-side to perform sophisticated computation and exchange data through file I/O over the parallel file system. However, file I/O can become an impediment in such systems and cause undesirable performance degradation due to its relatively low speed (compared to the interconnect fabric), which is unacceptable especially for applications with strict time constraints. The Data Transfer Framework (DTF) is an I/O arbitration layer working with the PnetCDF I/O library to eliminate the bottleneck by transparently redirecting file I/O operations through the parallel file system, to message passing via the high-speed interconnect fabrics between coupled components. Scalable and high-speed data transfer between components can be thus easily achieved with minimal development effort by using DTF. However, previous work provides insufficient scalability evaluation of DTF. In order to comprehensively evaluate the scalability of an I/O middleware like DTF and highlight its major advantages, we have designed an ensemble-based I/O benchmark that adopts the I/O model of the real-time weather forecasting application called SCALE-LETKF and present the scalability evaluation results of DTF against file I/O on two supercomputers, Fugaku and Oakforest-PACS, respectively. We provide insights into DTF's scalability and performance enhancements with the intention to impact future I/O middleware design.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Multi-component workflows play a significant role in High-Performance Computing and Big Data applications. They usually contain multiple, independently developed components that execute side-by-side to perform sophisticated computation and exchange data through file I/O over the parallel file system. However, file I/O can become an impediment in such systems and cause undesirable performance degradation due to its relatively low speed (compared to the interconnect fabric), which is unacceptable especially for applications with strict time constraints. The Data Transfer Framework (DTF) is an I/O arbitration layer working with the PnetCDF I/O library to eliminate the bottleneck by transparently redirecting file I/O operations through the parallel file system, to message passing via the high-speed interconnect fabrics between coupled components. Scalable and high-speed data transfer between components can be thus easily achieved with minimal development effort by using DTF. However, previous work provides insufficient scalability evaluation of DTF. In order to comprehensively evaluate the scalability of an I/O middleware like DTF and highlight its major advantages, we have designed an ensemble-based I/O benchmark that adopts the I/O model of the real-time weather forecasting application called SCALE-LETKF and present the scalability evaluation results of DTF against file I/O on two supercomputers, Fugaku and Oakforest-PACS, respectively. We provide insights into DTF's scalability and performance enhancements with the intention to impact future I/O middleware design.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AN10463942
• 書誌情報: 研究報告ハイパフォーマンスコンピューティング（HPC）; 巻 2020-HPC-177, 号 24, p. 1-7, 発行日 2020-12-14
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 2188-8841
• Notice: SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc.
• 出版者 / 言語: ja
• 出版者: 情報処理学会