| Item type |
SIG Technical Reports(1) |
| 公開日 |
2019-02-21 |
| タイトル |
|
|
タイトル |
A Hybrid Simulator to Analyze Gradient Staleness Effect |
| タイトル |
|
|
言語 |
en |
|
タイトル |
A Hybrid Simulator to Analyze Gradient Staleness Effect |
| 言語 |
|
|
言語 |
eng |
| キーワード |
|
|
主題Scheme |
Other |
|
主題 |
ディープ・ラーニング |
| 資源タイプ |
|
|
資源タイプ識別子 |
http://purl.org/coar/resource_type/c_18gh |
|
資源タイプ |
technical report |
| 著者所属 |
|
|
|
筑波大学／産業技術総合研究所 |
| 著者所属 |
|
|
|
産業技術総合研究所／筑波大学 |
| 著者所属 |
|
|
|
産業技術総合研究所／筑波大学 |
| 著者所属(英) |
|
|
|
en |
|
|
Uniersity of Tsukuba / National Institute of Advanced Science and Technology |
| 著者所属(英) |
|
|
|
en |
|
|
National Institute of Advanced Science and Technology / Uniersity of Tsukuba |
| 著者所属(英) |
|
|
|
en |
|
|
National Institute of Advanced Science and Technology / Uniersity of Tsukuba |
| 著者名 |
Duo, Zhang
Yusuke, Tanimura
Hidemoto, Nakada
|
| 著者名(英) |
Duo, Zhang
Yusuke, Tanimura
Hidemoto, Nakada
|
| 論文抄録 |
|
|
内容記述タイプ |
Other |
|
内容記述 |
One of the obstacles for parallel execution of Deep Learning is the Gradient exchange overhead, and there are numerous exchange methods are proposed to mitigate the overhead. Investigating these methods with real machines requires a lot of resources. Furthermore, it is impossible to investigate the behavior under other circumstances, such as different network latency. We propose a hybrid simulator that combines gradient computation on real machine and virtual time management using discrete event simulator, that enables to accurately reproduce the behavior under arbitrary gradient exchange methods and arbitrary setup. We implemented this simulator using Python coroutine. We confirmed that we can reproduce the behavior of asynchronous gradient exchange, and it can handle 64 nodes with single node. |
| 論文抄録(英) |
|
|
内容記述タイプ |
Other |
|
内容記述 |
One of the obstacles for parallel execution of Deep Learning is the Gradient exchange overhead, and there are numerous exchange methods are proposed to mitigate the overhead. Investigating these methods with real machines requires a lot of resources. Furthermore, it is impossible to investigate the behavior under other circumstances, such as different network latency. We propose a hybrid simulator that combines gradient computation on real machine and virtual time management using discrete event simulator, that enables to accurately reproduce the behavior under arbitrary gradient exchange methods and arbitrary setup. We implemented this simulator using Python coroutine. We confirmed that we can reproduce the behavior of asynchronous gradient exchange, and it can handle 64 nodes with single node. |
| 書誌レコードID |
|
|
収録物識別子タイプ |
NCID |
|
収録物識別子 |
AN10444176 |
| 書誌情報 |
研究報告システムソフトウェアとオペレーティング・システム（OS）
巻 2019-OS-145,
号 8,
p. 1-6,
発行日 2019-02-21
|
| ISSN |
|
|
収録物識別子タイプ |
ISSN |
|
収録物識別子 |
2188-8795 |
| Notice |
|
|
|
SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc. |
| 出版者 |
|
|
言語 |
ja |
|
出版者 |
情報処理学会 |
IPSJ-OS19145008.pdf (1.3 MB)
