2026-05-11T11:45:13Z https://ipsj.ixsq.nii.ac.jp/oai

oai:ipsj.ixsq.nii.ac.jp:00241718 2025-01-19T07:34:10Z 1164:2240:11467:11814

A survey of sparse structures in the multi-layer perceptron of large language models A survey of sparse structures in the multi-layer perceptron of large language models Sameer, Deshmukh Mingchuan, Lyu Hiroki, Tokura Takumi, Honda Sameer, Deshmukh Mingchuan, Lyu Hiroki, Tokura Takumi, Honda 省電力 Large language models using the transformer architecture require massive computational resources for training to acceptable levels of accuracy. Recent advances have shown that the MLP layers within such models can be pruned to up to 90% sparsity to reduce the computational requirement of training and inference. However, achieving high performance for the sparse matrix multiplication remains a challenge on GPUs. Several approaches have been suggested for improving the performance of sparse matrix multiplication using structured sparsity. In this paper, we first survey and benchmark some of the sparsity structures that can be applied to dense matrices, and then examine the training loss curves of a 162M Mistral model using various structures of sparsity. Our results show promising future directions for research in improving the training time of transformers using sparsity. Large language models using the transformer architecture require massive computational resources for training to acceptable levels of accuracy. Recent advances have shown that the MLP layers within such models can be pruned to up to 90% sparsity to reduce the computational requirement of training and inference. However, achieving high performance for the sparse matrix multiplication remains a challenge on GPUs. Several approaches have been suggested for improving the performance of sparse matrix multiplication using structured sparsity. In this paper, we first survey and benchmark some of the sparsity structures that can be applied to dense matrices, and then examine the training loss curves of a 162M Mistral model using various structures of sparsity. Our results show promising future directions for research in improving the training time of transformers using sparsity. technical report 情報処理学会 2024-12-09 application/pdf 研究報告ハイパフォーマンスコンピューティング（HPC） 25 2024-HPC-197 1 6 2188-8841 AN10463942 https://ipsj.ixsq.nii.ac.jp/record/241718/files/IPSJ-HPC24197025.pdf eng