WEKO3
アイテム
Effects and Mitigation of Out-of-vocabulary in Universal Language Models
https://ipsj.ixsq.nii.ac.jp/records/211922
https://ipsj.ixsq.nii.ac.jp/records/21192249afc794-3ae2-4858-aa5e-3fc3f1a0c951
| 名前 / ファイル | ライセンス | アクション |
|---|---|---|
IPSJ-TOD1403003.pdf (980.4 kB)
|
Copyright (c) 2021 by the Information Processing Society of Japan
|
|
| オープンアクセス | ||
| Item type | Trans(1) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 公開日 | 2021-07-13 | |||||||||
| タイトル | ||||||||||
| タイトル | Effects and Mitigation of Out-of-vocabulary in Universal Language Models | |||||||||
| タイトル | ||||||||||
| 言語 | en | |||||||||
| タイトル | Effects and Mitigation of Out-of-vocabulary in Universal Language Models | |||||||||
| 言語 | ||||||||||
| 言語 | eng | |||||||||
| キーワード | ||||||||||
| 主題Scheme | Other | |||||||||
| 主題 | [研究論文] Natural language processing, Machine learning, Transfer learning, Language models | |||||||||
| 資源タイプ | ||||||||||
| 資源タイプ識別子 | http://purl.org/coar/resource_type/c_6501 | |||||||||
| 資源タイプ | journal article | |||||||||
| 著者所属 | ||||||||||
| Tokyo Institute of Technology／Presently with Odd Concepts Inc. | ||||||||||
| 著者所属 | ||||||||||
| Tokyo Institute of Technology | ||||||||||
| 著者所属(英) | ||||||||||
| en | ||||||||||
| Tokyo Institute of Technology / Presently with Odd Concepts Inc. | ||||||||||
| 著者所属(英) | ||||||||||
| en | ||||||||||
| Tokyo Institute of Technology | ||||||||||
| 著者名 |
Sangwhan, Moon
× Sangwhan, Moon
× Naoaki, Okazaki
|
|||||||||
| 著者名(英) |
Sangwhan, Moon
× Sangwhan, Moon
× Naoaki, Okazaki
|
|||||||||
| 論文抄録 | ||||||||||
| 内容記述タイプ | Other | |||||||||
| 内容記述 | One of the most important recent natural language processing (NLP) trends is transfer learning - using representations from language models implemented through a neural network to perform other tasks. While transfer learning is a promising and robust method, downstream task performance in transfer learning depends on the robustness of the backbone model's vocabulary, which in turn represents both the positive and negative characteristics of the corpus used to train it. With subword tokenization, out-of-vocabulary (OOV) is generally assumed to be a solved problem. Still, in languages with a large alphabet such as Chinese, Japanese, and Korean (CJK), this assumption does not hold. In our work, we demonstrate the adverse effects of OOV in the context of transfer learning in CJK languages, then propose a novel approach to maximize the utility of a pre-trained model suffering from OOV. Additionally, we further investigate the correlation of OOV to task performance and explore if and how mitigation can salvage a model with high OOV. ------------------------------ This is a preprint of an article intended for publication Journal of Information Processing(JIP). This preprint should not be cited. This article should be cited as: Journal of Information Processing Vol.29(2021) (online) ------------------------------ |
|||||||||
| 論文抄録(英) | ||||||||||
| 内容記述タイプ | Other | |||||||||
| 内容記述 | One of the most important recent natural language processing (NLP) trends is transfer learning - using representations from language models implemented through a neural network to perform other tasks. While transfer learning is a promising and robust method, downstream task performance in transfer learning depends on the robustness of the backbone model's vocabulary, which in turn represents both the positive and negative characteristics of the corpus used to train it. With subword tokenization, out-of-vocabulary (OOV) is generally assumed to be a solved problem. Still, in languages with a large alphabet such as Chinese, Japanese, and Korean (CJK), this assumption does not hold. In our work, we demonstrate the adverse effects of OOV in the context of transfer learning in CJK languages, then propose a novel approach to maximize the utility of a pre-trained model suffering from OOV. Additionally, we further investigate the correlation of OOV to task performance and explore if and how mitigation can salvage a model with high OOV. ------------------------------ This is a preprint of an article intended for publication Journal of Information Processing(JIP). This preprint should not be cited. This article should be cited as: Journal of Information Processing Vol.29(2021) (online) ------------------------------ |
|||||||||
| 書誌レコードID | ||||||||||
| 収録物識別子タイプ | NCID | |||||||||
| 収録物識別子 | AA11464847 | |||||||||
| 書誌情報 |
情報処理学会論文誌データベース（TOD） 巻 14, 号 3, 発行日 2021-07-13 |
|||||||||
| ISSN | ||||||||||
| 収録物識別子タイプ | ISSN | |||||||||
| 収録物識別子 | 1882-7799 | |||||||||
| 出版者 | ||||||||||
| 言語 | ja | |||||||||
| 出版者 | 情報処理学会 | |||||||||
