Comparing Likert Scale and Pairwise Comparison for Human Evaluation in Rapport-Building Dialogue Systems

• Item type: SIG Technical Reports(1)
• 公開日: 2024-12-05
• タイトル: Comparing Likert Scale and Pairwise Comparison for Human Evaluation in Rapport-Building Dialogue Systems
• タイトル / 言語: en
• タイトル: Comparing Likert Scale and Pairwise Comparison for Human Evaluation in Rapport-Building Dialogue Systems
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: 話者認識・音声分析
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_18gh
• 資源タイプ: technical report
• 著者所属: Nara Institute of Science and Technology／Guardian Robot Project RIKEN
• 著者所属: Guardian Robot Project RIKEN
• 著者所属: Guardian Robot Project RIKEN／Nara Institute of Science and Technology
• 著者所属: Institute of Science Tokyo／Guardian Robot Project RIKEN／Nara Institute of Science and Technology
• 著者所属(英): en
• 著者所属(英): Nara Institute of Science and Technology / Guardian Robot Project RIKEN
• 著者所属(英): en
• 著者所属(英): Guardian Robot Project RIKEN
• 著者所属(英): en
• 著者所属(英): Guardian Robot Project RIKEN / Nara Institute of Science and Technology
• 著者所属(英): en
• 著者所属(英): Institute of Science Tokyo / Guardian Robot Project RIKEN / Nara Institute of Science and Technology
• 著者名: Muhammad, Yeza Baihaqi; Angel, García Contreras; Seiya, Kawano; Koichiro, Yoshino
• 著者名(英): Muhammad, Yeza Baihaqi; Angel, García Contreras; Seiya, Kawano; Koichiro, Yoshino
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Human evaluation plays a critical role in dialogue systems research, especially in non-task-oriented systems such as rapport-building dialogue systems. Current evaluations often rely on Likert scales to assess user experience, but this method introduces challenges such as inconsistent scale perception, inefficiency, and central tendency bias. Moreover, it is difficult to compare the agent's performance across multiple criteria due to the problem of uneven scoring interpretations by participants on the Likert scale. On the other hand, pairwise comparison emphasizes direct item-to-item evaluation based on defined criteria, producing scores that more closely align with participants' preferences and minimizing biases. This paper compares an evaluation framework for rapport-building dialogue systems using pairwise comparison with a conventional Likert scale system. These approaches are tested through dialogue experiments involving six participants and four dialogue systems embedded in a conversational robot: CommA, CommI, CommO, and CommE, to measure human-agent rapport. Our experimental results indicated that the pairwise comparison method better represented systems' overall performance compared to the Likert scale. It also demonstrated lower variability, higher reliability, and a shorter completion time.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Human evaluation plays a critical role in dialogue systems research, especially in non-task-oriented systems such as rapport-building dialogue systems. Current evaluations often rely on Likert scales to assess user experience, but this method introduces challenges such as inconsistent scale perception, inefficiency, and central tendency bias. Moreover, it is difficult to compare the agent's performance across multiple criteria due to the problem of uneven scoring interpretations by participants on the Likert scale. On the other hand, pairwise comparison emphasizes direct item-to-item evaluation based on defined criteria, producing scores that more closely align with participants' preferences and minimizing biases. This paper compares an evaluation framework for rapport-building dialogue systems using pairwise comparison with a conventional Likert scale system. These approaches are tested through dialogue experiments involving six participants and four dialogue systems embedded in a conversational robot: CommA, CommI, CommO, and CommE, to measure human-agent rapport. Our experimental results indicated that the pairwise comparison method better represented systems' overall performance compared to the Likert scale. It also demonstrated lower variability, higher reliability, and a shorter completion time.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AN10442647
• 書誌情報: 研究報告音声言語情報処理（SLP）; 巻 2024-SLP-154, 号 43, p. 1-5, 発行日 2024-12-05
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 2188-8663
• Notice: SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc.
• 出版者 / 言語: ja
• 出版者: 情報処理学会