Asynchronous Deep Learning Test-bed to Analyze Gradient Staleness Effect

• Item type: SIG Technical Reports(1)
• 公開日: 2018-07-23
• タイトル: Asynchronous Deep Learning Test-bed to Analyze Gradient Staleness Effect
• タイトル / 言語: en
• タイトル: Asynchronous Deep Learning Test-bed to Analyze Gradient Staleness Effect
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: 機械学習・ニューラルネットワーク
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_18gh
• 資源タイプ: technical report
• 著者所属: University of Tsukuba／National Institute of Advanced Industrial Science and Technology
• 著者所属: National Institute of Advanced Industrial Science and Technology／University of Tsukuba
• 著者所属: National Institute of Advanced Industrial Science and Technology／University of Tsukuba
• 著者所属(英): en
• 著者所属(英): University of Tsukuba / National Institute of Advanced Industrial Science and Technology
• 著者所属(英): en
• 著者所属(英): National Institute of Advanced Industrial Science and Technology / University of Tsukuba
• 著者所属(英): en
• 著者所属(英): National Institute of Advanced Industrial Science and Technology / University of Tsukuba
• 著者名: Duo, Zhang; Yusuke, Tanimura; Hidemoto, Nakada
• 著者名(英): Duo, Zhang; Yusuke, Tanimura; Hidemoto, Nakada
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: For modern machine learning systems, including deep learning systems, parallelization is inevitable since they are required to process massive amount of training data. One of the hot topic of this area is the data parallel learning where multiple nodes cooperate each other exchanging parameter / gradient periodically. In order to efficiently implement data-parallel machine learning in a collection of computers with a relatively sparse network, it is indispensable to asynchronously update model parameters through gradients, but the effect of the learning model through asynchronous analysis has not yet been fully understood. In this paper, we propose a software test-bed for analyzing gradient staleness effect on prediction performance, using deep learning framework TensorFlow and distributed computing framework Ray. We report the architecture of the test-bed and initial evaluation results.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: For modern machine learning systems, including deep learning systems, parallelization is inevitable since they are required to process massive amount of training data. One of the hot topic of this area is the data parallel learning where multiple nodes cooperate each other exchanging parameter / gradient periodically. In order to efficiently implement data-parallel machine learning in a collection of computers with a relatively sparse network, it is indispensable to asynchronously update model parameters through gradients, but the effect of the learning model through asynchronous analysis has not yet been fully understood. In this paper, we propose a software test-bed for analyzing gradient staleness effect on prediction performance, using deep learning framework TensorFlow and distributed computing framework Ray. We report the architecture of the test-bed and initial evaluation results.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AN10096105
• 書誌情報: 研究報告システム・アーキテクチャ（ARC）; 巻 2018-ARC-232, 号 28, p. 1-6, 発行日 2018-07-23
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 2188-8574
• Notice: SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc.
• 出版者 / 言語: ja
• 出版者: 情報処理学会