| Item type |
SIG Technical Reports(1) |
| 公開日 |
2021-06-18 |
| タイトル |
|
|
タイトル |
A GAN Based Approach to Lip-Sync 2D Cartoon Animations without Requiring Raw Cartoon Dataset |
| タイトル |
|
|
言語 |
en |
|
タイトル |
A GAN Based Approach to Lip-Sync 2D Cartoon Animations without Requiring Raw Cartoon Dataset |
| 言語 |
|
|
言語 |
eng |
| 資源タイプ |
|
|
資源タイプ識別子 |
http://purl.org/coar/resource_type/c_18gh |
|
資源タイプ |
technical report |
| 著者所属 |
|
|
|
The University of Tokyo |
| 著者所属 |
|
|
|
The University of Tokyo／Currently working in AI Lab at CyberAgent, Inc. |
| 著者所属 |
|
|
|
The University of Tokyo |
| 著者所属(英) |
|
|
|
en |
|
|
The University of Tokyo |
| 著者所属(英) |
|
|
|
en |
|
|
The University of Tokyo / Currently working in AI Lab at CyberAgent, Inc. |
| 著者所属(英) |
|
|
|
en |
|
|
The University of Tokyo |
| 著者名 |
Mitsuhiko, Nakamoto
Xueting, Wang
Toshihiko, Yamasaki
|
| 著者名(英) |
Mitsuhiko, Nakamoto
Xueting, Wang
Toshihiko, Yamasaki
|
| 論文抄録 |
|
|
内容記述タイプ |
Other |
|
内容記述 |
We present a generative adversarial networks (GAN) based approach to lip-sync 2D cartoon animations. Most of the previous works have worked on lip-sync for the real people talking videos. However, lip-sync for 2D cartoon animations was rarely discussed while the traditional workflow of creating 2D cartoon animations is highly time-consuming. The main problem of automatically lip-syncing a 2D cartoon animation, especially using a deep learning approach, is the lack of datasets which consist of well lip-synced cartoon animations. Therefore, In this paper we present a GAN-based approach to achieve 2D cartoon animation lip-sync with no need of collecting raw cartoon animation datasets. Alternatively, we construct a cartoon talking video dataset by applying CartoonGAN to transform real-life speaking videos into cartoon styles. The dataset after the style transfer was used to train a lip-synchronization model, Wav2Lip. Our approach can generate natural lip-synchronized cartoon animations. We also conduct a user study and the results demonstrate the effectiveness of our approach. |
| 論文抄録(英) |
|
|
内容記述タイプ |
Other |
|
内容記述 |
We present a generative adversarial networks (GAN) based approach to lip-sync 2D cartoon animations. Most of the previous works have worked on lip-sync for the real people talking videos. However, lip-sync for 2D cartoon animations was rarely discussed while the traditional workflow of creating 2D cartoon animations is highly time-consuming. The main problem of automatically lip-syncing a 2D cartoon animation, especially using a deep learning approach, is the lack of datasets which consist of well lip-synced cartoon animations. Therefore, In this paper we present a GAN-based approach to achieve 2D cartoon animation lip-sync with no need of collecting raw cartoon animation datasets. Alternatively, we construct a cartoon talking video dataset by applying CartoonGAN to transform real-life speaking videos into cartoon styles. The dataset after the style transfer was used to train a lip-synchronization model, Wav2Lip. Our approach can generate natural lip-synchronized cartoon animations. We also conduct a user study and the results demonstrate the effectiveness of our approach. |
| 書誌レコードID |
|
|
収録物識別子タイプ |
NCID |
|
収録物識別子 |
AN10100541 |
| 書誌情報 |
研究報告コンピュータグラフィックスとビジュアル情報学（CG）
巻 2021-CG-182,
号 1,
p. 1-5,
発行日 2021-06-18
|
| ISSN |
|
|
収録物識別子タイプ |
ISSN |
|
収録物識別子 |
2188-8949 |
| Notice |
|
|
|
SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc. |
| 出版者 |
|
|
言語 |
ja |
|
出版者 |
情報処理学会 |
IPSJ-CG21182001.pdf (4.4 MB)
