Linear Effect of Neuron Activations in Transformer-based Language Models

• Item type: SIG Technical Reports(1)
• 公開日: 2024-12-05
• タイトル: Linear Effect of Neuron Activations in Transformer-based Language Models
• タイトル / 言語: en
• タイトル: Linear Effect of Neuron Activations in Transformer-based Language Models
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: ポスターセッション
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_18gh
• 資源タイプ: technical report
• 著者所属: The University of Tokyo
• 著者所属: The University of Tokyo
• 著者所属: Institute of Industrial Science, The University of Tokyo
• 著者所属(英): en
• 著者所属(英): The University of Tokyo
• 著者所属(英): en
• 著者所属(英): The University of Tokyo
• 著者所属(英): en
• 著者所属(英): Institute of Industrial Science, The University of Tokyo
• 著者名: Xin, Zhao; Zehui, Jiang; Naoki, Yoshinaga
• 著者名(英): Xin, Zhao; Zehui, Jiang; Naoki, Yoshinaga
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Neurons in feed-forward layers of Transformers have shown the ability to store factual knowledge. However, previous analyses mostly focused on qualitative evaluation, leaving the numerical relationship between neuron activations and model outputs less understood. Our study conducts a quantitative analysis through neuron-wise intervention experiments using the knowledge probing dataset. Our findings first reveal that neurons exhibit linearity and polarity in producing output tokens probabilities, quantified by “neuron empirical gradients.” Empirical gradients provide a direct measure of neurons' importance in representing knowledge. However, neuron-wise intervention experiments are costly, making it impractical to obtain empirical gradients in large language models. To address this, we propose NeurGrad, an efficient method for measuring neuron empirical gradients. Our experimental results show that NeurGrad outperforms several baseline methods in both efficiency and accuracy.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Neurons in feed-forward layers of Transformers have shown the ability to store factual knowledge. However, previous analyses mostly focused on qualitative evaluation, leaving the numerical relationship between neuron activations and model outputs less understood. Our study conducts a quantitative analysis through neuron-wise intervention experiments using the knowledge probing dataset. Our findings first reveal that neurons exhibit linearity and polarity in producing output tokens probabilities, quantified by “neuron empirical gradients.” Empirical gradients provide a direct measure of neurons' importance in representing knowledge. However, neuron-wise intervention experiments are costly, making it impractical to obtain empirical gradients in large language models. To address this, we propose NeurGrad, an efficient method for measuring neuron empirical gradients. Our experimental results show that NeurGrad outperforms several baseline methods in both efficiency and accuracy.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AN10115061
• 書誌情報: 研究報告自然言語処理（NL）; 巻 2024-NL-262, 号 8, p. 1-6, 発行日 2024-12-05
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 2188-8779
• Notice: SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc.
• 出版者 / 言語: ja
• 出版者: 情報処理学会