WEKO3
アイテム
Provide Interpretability of Document Classification by Large Language Models Based on Word Masking
https://ipsj.ixsq.nii.ac.jp/records/233823
https://ipsj.ixsq.nii.ac.jp/records/233823e3961e0f-28bd-40f8-a889-b061d0917e8a
| 名前 / ファイル | ライセンス | アクション |
|---|---|---|
IPSJ-TOD1702002.pdf (5.3 MB)
2026年4月23日からダウンロード可能です。
|
Copyright (c) 2024 by the Information Processing Society of Japan
|
|
| 非会員：¥0, IPSJ:学会員：¥0, DBS:会員：¥0, IFAT:会員：¥0, DLIB:会員：¥0 | ||
| Item type | Trans(1) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 公開日 | 2024-04-23 | |||||||||
| タイトル | ||||||||||
| タイトル | Provide Interpretability of Document Classification by Large Language Models Based on Word Masking | |||||||||
| タイトル | ||||||||||
| 言語 | en | |||||||||
| タイトル | Provide Interpretability of Document Classification by Large Language Models Based on Word Masking | |||||||||
| 言語 | ||||||||||
| 言語 | eng | |||||||||
| キーワード | ||||||||||
| 主題Scheme | Other | |||||||||
| 主題 | [テクニカルノート] deep learning, news documents classification, LLM, BERT, Attention, word masking | |||||||||
| 資源タイプ | ||||||||||
| 資源タイプ識別子 | http://purl.org/coar/resource_type/c_6501 | |||||||||
| 資源タイプ | journal article | |||||||||
| 著者所属 | ||||||||||
| Kogakuin University | ||||||||||
| 著者所属 | ||||||||||
| Kogakuin University | ||||||||||
| 著者所属(英) | ||||||||||
| en | ||||||||||
| Kogakuin University | ||||||||||
| 著者所属(英) | ||||||||||
| en | ||||||||||
| Kogakuin University | ||||||||||
| 著者名 |
Atsuki, Tamekuri
× Atsuki, Tamekuri
× Saneyasu, Yamaguchi
|
|||||||||
| 著者名(英) |
Atsuki, Tamekuri
× Atsuki, Tamekuri
× Saneyasu, Yamaguchi
|
|||||||||
| 論文抄録 | ||||||||||
| 内容記述タイプ | Other | |||||||||
| 内容記述 | Deep neural networks have greatly improved natural language processing and text analysis technologies. In particular, pre-trained large language models have achieved significant improvement. However, it has been argued that they are black boxes and that it is important to provide interpretability. In our previous work, we focused on self-attention and proposed methods for providing and evaluating interpretability. However, the work did not use large language models, and the evaluation method used unusual sentences by deleting words. In this paper, we focus on BERT, which is a popular large language model, and its masking function instead of deleting words. We then show a problem of using this masking function to provide interpretability, which is that the mask token is not neutral for decision. We then propose an evaluation method based on this masking function with training to learn that the mask token is neutral. ------------------------------ This is a preprint of an article intended for publication Journal of Information Processing(JIP). This preprint should not be cited. This article should be cited as: Journal of Information Processing Vol.32(2024) (online) ------------------------------ |
|||||||||
| 論文抄録(英) | ||||||||||
| 内容記述タイプ | Other | |||||||||
| 内容記述 | Deep neural networks have greatly improved natural language processing and text analysis technologies. In particular, pre-trained large language models have achieved significant improvement. However, it has been argued that they are black boxes and that it is important to provide interpretability. In our previous work, we focused on self-attention and proposed methods for providing and evaluating interpretability. However, the work did not use large language models, and the evaluation method used unusual sentences by deleting words. In this paper, we focus on BERT, which is a popular large language model, and its masking function instead of deleting words. We then show a problem of using this masking function to provide interpretability, which is that the mask token is not neutral for decision. We then propose an evaluation method based on this masking function with training to learn that the mask token is neutral. ------------------------------ This is a preprint of an article intended for publication Journal of Information Processing(JIP). This preprint should not be cited. This article should be cited as: Journal of Information Processing Vol.32(2024) (online) ------------------------------ |
|||||||||
| 書誌レコードID | ||||||||||
| 収録物識別子タイプ | NCID | |||||||||
| 収録物識別子 | AA11464847 | |||||||||
| 書誌情報 |
情報処理学会論文誌データベース（TOD） 巻 17, 号 2, 発行日 2024-04-23 |
|||||||||
| ISSN | ||||||||||
| 収録物識別子タイプ | ISSN | |||||||||
| 収録物識別子 | 1882-7799 | |||||||||
| 出版者 | ||||||||||
| 言語 | ja | |||||||||
| 出版者 | 情報処理学会 | |||||||||
