Controlling the Penalty on Late Arrival of Relevant Documents in Information Retrieval Evalution with Graded Relevance

• Item type: SIG Technical Reports(1)
• 公開日: 2006-09-12
• タイトル: Controlling the Penalty on Late Arrival of Relevant Documents in Information Retrieval Evalution with Graded Relevance
• タイトル / 言語: en
• タイトル: Controlling the Penalty on Late Arrival of Relevant Documents in Information Retrieval Evalution with Graded Relevance
• 言語: eng
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_18gh
• 資源タイプ: technical report
• 著者所属: Knowledge Media Laboratory Toshiba Corporate R&D center
• 著者所属(英): en
• 著者所属(英): Knowledge Media Laboratory, Toshiba Corporate R&D center
• 著者名: Tetsuya, Sakai
• 著者名(英): Tetsuya, Sakai
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Large-scale information retrieval evalution efforts such as TREC and NTCIR have always used binary-relevance evalution metrics even when graded relevance data were available. However the NTCIR-6 crosslingual task has finally announced that it will use graded-relevance metrics though only as additional metrics. This paper compares graded-relevance metrics in terms of the ability to control the balance between retrieving highly relevant documents and retrieving any relevant documents early in the ranked list. We argue and demonstrate that Q-measure is more flexible than normalised Discounted Cumulative Gain and generalised Average Precision. We then suggest a brief guideline for conducting a reliable information retrieval evalution with graded relevance.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Large-scale information retrieval evalution efforts such as TREC and NTCIR have always used binary-relevance evalution metrics, even when graded relevance data were available. However, the NTCIR-6 crosslingual task has finally announced that it will use graded-relevance metrics, though only as additional metrics. This paper compares graded-relevance metrics in terms of the ability to control the balance between retrieving highly relevant documents and retrieving any relevant documents early in the ranked list. We argue and demonstrate that Q-measure is more flexible than normalised Discounted Cumulative Gain and generalised Average Precision. We then suggest a brief guideline for conducting a reliable information retrieval evalution with graded relevance.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AN10115061
• 書誌情報: 情報処理学会研究報告自然言語処理（NL）; 巻 2006, 号 94(2006-NL-175), p. 57-64, 発行日 2006-09-12
• Notice: SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc.
• 出版者 / 言語: ja
• 出版者: 情報処理学会