A survey of sparse structures in the multi-layer perceptron of large language models

• Item type: SIG Technical Reports(1)
• 公開日: 2024-12-09
• タイトル: A survey of sparse structures in the multi-layer perceptron of large language models
• タイトル / 言語: en
• タイトル: A survey of sparse structures in the multi-layer perceptron of large language models
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: 省電力
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_18gh
• 資源タイプ: technical report
• 著者所属: Fujitsu Ltd.
• 著者所属: University of Kyushu
• 著者所属: Fujitsu Ltd.
• 著者所属: Fujitsu Ltd.
• 著者所属(英): en
• 著者所属(英): Fujitsu Ltd.
• 著者所属(英): en
• 著者所属(英): University of Kyushu
• 著者所属(英): en
• 著者所属(英): Fujitsu Ltd.
• 著者所属(英): en
• 著者所属(英): Fujitsu Ltd.
• 著者名: Sameer, Deshmukh; Mingchuan, Lyu; Hiroki, Tokura; Takumi, Honda
• 著者名(英): Sameer, Deshmukh; Mingchuan, Lyu; Hiroki, Tokura; Takumi, Honda
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Large language models using the transformer architecture require massive computational resources for training to acceptable levels of accuracy. Recent advances have shown that the MLP layers within such models can be pruned to up to 90% sparsity to reduce the computational requirement of training and inference. However, achieving high performance for the sparse matrix multiplication remains a challenge on GPUs. Several approaches have been suggested for improving the performance of sparse matrix multiplication using structured sparsity. In this paper, we first survey and benchmark some of the sparsity structures that can be applied to dense matrices, and then examine the training loss curves of a 162M Mistral model using various structures of sparsity. Our results show promising future directions for research in improving the training time of transformers using sparsity.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Large language models using the transformer architecture require massive computational resources for training to acceptable levels of accuracy. Recent advances have shown that the MLP layers within such models can be pruned to up to 90% sparsity to reduce the computational requirement of training and inference. However, achieving high performance for the sparse matrix multiplication remains a challenge on GPUs. Several approaches have been suggested for improving the performance of sparse matrix multiplication using structured sparsity. In this paper, we first survey and benchmark some of the sparsity structures that can be applied to dense matrices, and then examine the training loss curves of a 162M Mistral model using various structures of sparsity. Our results show promising future directions for research in improving the training time of transformers using sparsity.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AN10463942
• 書誌情報: 研究報告ハイパフォーマンスコンピューティング（HPC）; 巻 2024-HPC-197, 号 25, p. 1-6, 発行日 2024-12-09
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 2188-8841
• Notice: SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc.
• 出版者 / 言語: ja
• 出版者: 情報処理学会