A Novel Dataset Development Method for Evaluating Sentiment Recognition Bias of Large Language Models in Conflict Structures

• Item type: SIG Technical Reports(1)
• 公開日: 2024-07-19
• タイトル: A Novel Dataset Development Method for Evaluating Sentiment Recognition Bias of Large Language Models in Conflict Structures
• タイトル / 言語: en
• タイトル: A Novel Dataset Development Method for Evaluating Sentiment Recognition Bias of Large Language Models in Conflict Structures
• 言語: eng
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_18gh
• 資源タイプ: technical report
• 著者所属: Faculty of Data Science, Shiga University
• 著者所属(英): en
• 著者所属(英): Faculty of Data Science, Shiga University
• 著者名: Keito, Inoshita
• 著者名(英): Keito, Inoshita
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: The rapid development of AI technology has enabled Large Language Models (LLMs) to acquire extensive general knowledge from vast amounts of text data, making them useful for various tasks. However, it has become evident that LLMs also acquire biases present in their training data, leading to discriminatory behavior towards attributes such as gender, race, and political ideologies. This is particularly concerning in the field of national security, where sentiment recognition bias towards specific countries by LLMs could cause serious problems. Although previous studies have developed datasets for evaluating these biases, several challenges remain in their development methods. This study proposes a novel dataset development method for evaluating sentiment recognition biases of LLMs, based on tweet data related to the Ukraine-Russia war. Specifically, the method involves automated sentiment labeling and anonymization processes using LLMs, aiming to create efficient and high-accuracy datasets. Experimental results confirm that the proposed method effectively evaluates the sentiment recognition biases of LLMs in various conflict structures. In conclusion, this study provides a new method for evaluating biases in LLMs and demonstrates its effectiveness. Future research should focus on developing larger datasets and improving anonymization techniques.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: The rapid development of AI technology has enabled Large Language Models (LLMs) to acquire extensive general knowledge from vast amounts of text data, making them useful for various tasks. However, it has become evident that LLMs also acquire biases present in their training data, leading to discriminatory behavior towards attributes such as gender, race, and political ideologies. This is particularly concerning in the field of national security, where sentiment recognition bias towards specific countries by LLMs could cause serious problems. Although previous studies have developed datasets for evaluating these biases, several challenges remain in their development methods. This study proposes a novel dataset development method for evaluating sentiment recognition biases of LLMs, based on tweet data related to the Ukraine-Russia war. Specifically, the method involves automated sentiment labeling and anonymization processes using LLMs, aiming to create efficient and high-accuracy datasets. Experimental results confirm that the proposed method effectively evaluates the sentiment recognition biases of LLMs in various conflict structures. In conclusion, this study provides a new method for evaluating biases in LLMs and demonstrates its effectiveness. Future research should focus on developing larger datasets and improving anonymization techniques.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AN1010060X
• 書誌情報: 研究報告人文科学とコンピュータ（CH）; 巻 2024-CH-136, 号 2, p. 1-4, 発行日 2024-07-19
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 2188-8957
• Notice: SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc.
• 出版者 / 言語: ja
• 出版者: 情報処理学会