Revisiting NTCIR ACLIA IR4QA with Additional Relevance Assessments

• Item type: SIG Technical Reports(1)
• 公開日: 2009-07-21
• タイトル: Revisiting NTCIR ACLIA IR4QA with Additional Relevance Assessments
• タイトル / 言語: en
• タイトル: Revisiting NTCIR ACLIA IR4QA with Additional Relevance Assessments
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: 英語セッション
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_18gh
• 資源タイプ: technical report
• 著者所属: Microsoft Research Asia
• 著者所属: National Institute of Informatics
• 著者所属: National Taiwan Ocean University
• 著者所属: Microsoft Research Asia
• 著者所属: Carnegie Mellon University
• 著者所属: Carnegie Mellon University
• 著者所属(英): en
• 著者所属(英): Microsoft Research Asia
• 著者所属(英): en
• 著者所属(英): National Institute of Informatics
• 著者所属(英): en
• 著者所属(英): National Taiwan Ocean University
• 著者所属(英): en
• 著者所属(英): Microsoft Research Asia
• 著者所属(英): en
• 著者所属(英): Carnegie Mellon University
• 著者所属(英): en
• 著者所属(英): Carnegie Mellon University
• 著者名: Tetsuya, Sakai; Noriko, Kando; Chuan-JieLin; Ruihua, Song; Hideki, Shima; Teruko, Mitamura
• 著者名(英): Tetsuya, Sakai; Noriko, Kando; Chuan-Jie, Lin; Ruihua, Song; Hideki, Shima; Teruko, Mitamura
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: At the NTCIR-7 Workshop Meeting held in December 2008, participating systems of the ACLIA IR4QA task were evaluated based on “qrels version 1,” which covered the depth-30 pool for every topic and went further down the pool for a limited number of topics, due to time constraints. This paper reports on revised results based on “qrels version 2” which covers the depth-100 pool for every topic. While the version 1 and version 2 results are generally in agreement, some differences in system rankings and significance test results suggest that the additional effort was worthwhile. This paper also reports on a set of additional experiments with new “pseudo-qrels,” which mimics the qrels without relying on any manual relevance assessments. Our pseudo-qrels experiments are surprisingly successful: the Pearson correlation coefficients between performances based on our “size-100” pseudo-qrels and those based on qrels version 2 are over 0.9, and even the Kendall rank correlations are 0.58-0.86. Hence, for the next round of IR4QA at NTCIR-8, we may be able to predict system rankings with reasonable accuracy using size-100 pseudo-qrels, right after the run submission deadline.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: At the NTCIR-7 Workshop Meeting held in December 2008, participating systems of the ACLIA IR4QA task were evaluated based on “qrels version 1,” which covered the depth-30 pool for every topic and went further down the pool for a limited number of topics, due to time constraints. This paper reports on revised results based on “qrels version 2” which covers the depth-100 pool for every topic. While the version 1 and version 2 results are generally in agreement, some differences in system rankings and significance test results suggest that the additional effort was worthwhile. This paper also reports on a set of additional experiments with new “pseudo-qrels,” which mimics the qrels without relying on any manual relevance assessments. Our pseudo-qrels experiments are surprisingly successful: the Pearson correlation coefficients between performances based on our “size-100” pseudo-qrels and those based on qrels version 2 are over 0.9, and even the Kendall rank correlations are 0.58-0.86. Hence, for the next round of IR4QA at NTCIR-8, we may be able to predict system rankings with reasonable accuracy using size-100 pseudo-qrels, right after the run submission deadline.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AN10112482
• 書誌情報: 研究報告データベースシステム（DBS）; 巻 2009-DBS-148, 号 9, p. 1-8, 発行日 2009-07-21
• Notice: SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc.
• 出版者 / 言語: ja
• 出版者: 情報処理学会