2024-03-28T23:44:28Zhttps://ipsj.ixsq.nii.ac.jp/ej/?action=repository_oaipmhoai:ipsj.ixsq.nii.ac.jp:001744692023-04-27T10:00:04Z01164:02240:08543:08896
Spark as Data Supplier for MPI Deep Learning ProcessesSpark as Data Supplier for MPI Deep Learning Processesengストレージhttp://id.nii.ac.jp/1001/00174435/Technical Reporthttps://ipsj.ixsq.nii.ac.jp/ej/?action=repository_action_common_download&item_id=174469&item_no=1&attribute_id=1&file_no=1Copyright (c) 2016 by the Information Processing Society of JapanESIEE PARIS/FUJITSU LABORATORIES LTD.FUJITSU LABORATORIES LTD.FUJITSU LABORATORIES LTD.FUJITSU LABORATORIES LTD.FUJITSU LABORATORIES LTD.Amir, HaderbacheMasahiro, MiwaMasafumi, YamazakiTsuguchika, TabaruKohta, NakashimaRecent works in deep learning show that training large models can improve accuracy. Many distributed deep learning frameworks have been so far developed to scale up machine learning algorithms. For the sake of performance, we believe these intensive computations must be combined with a clever data parallelism strategy. This paper brings one possible answer to the issue of supplying data to deep learning worker nodes on HPC systems. We design a two sides system where independent MPI Process executions match Spark tasks whose job is to provide data partition. We test and evaluate different Spark configurations and show that this system provides a flexible and scalable data supply mechanism which leverage MPI high performance and Spark high level data management.Recent works in deep learning show that training large models can improve accuracy. Many distributed deep learning frameworks have been so far developed to scale up machine learning algorithms. For the sake of performance, we believe these intensive computations must be combined with a clever data parallelism strategy. This paper brings one possible answer to the issue of supplying data to deep learning worker nodes on HPC systems. We design a two sides system where independent MPI Process executions match Spark tasks whose job is to provide data partition. We test and evaluate different Spark configurations and show that this system provides a flexible and scalable data supply mechanism which leverage MPI high performance and Spark high level data management.AN10463942研究報告ハイパフォーマンスコンピューティング(HPC)2016-HPC-15611192016-09-082188-88412016-08-31