On the Robustness of Information Retrieval Metrics to Biased Relevance Assessments

• Item type: JInfP(1)
• 公開日: 2009-04-08
• タイトル: On the Robustness of Information Retrieval Metrics to Biased Relevance Assessments
• タイトル / 言語: en
• タイトル: On the Robustness of Information Retrieval Metrics to Biased Relevance Assessments
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: Regular Paper
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_6501
• 資源タイプ: journal article
• その他タイトル / その他のタイトル: Information Retrieval
• 著者所属: NewsWatch, Inc.
• 著者所属(英): en
• 著者所属(英): NewsWatch, Inc.
• 著者名: Tetsuya, Sakai
• 著者名(英): Tetsuya, Sakai
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Information Retrieval (IR) test collections are growing larger, and relevance data constructed through <i>pooling</i> are suspected of becoming more and more <i>incomplete</i> and <i>biased</i>. Several studies have used IR evaluation metrics specifically designed to handle this problem, but most of them have only examined the metrics under <i>incomplete but unbiased</i> conditions, using random samples of the original relevance data. This paper examines nine metrics in more realistic settings, by reducing the number of pooled systems and the number of pooled documents. Even though previous studies have shown that metrics based on a <i>condensed list</i>, obtained by removing all unjudged documents from the original ranked list, are effective for handling very incomplete but unbiased relevance data, we show that these results do not hold when the relevance data are biased towards particular systems or towards the top of the pools. More specifically, we show that the condensed-list versions of <i>Average Precision</i>, <i>Qmeasure</i> and <i>normalised Discounted Cumulative Gain</i>, which we denote as AP', Q' and nDCG', are not necessarily superior to the original metrics for handling biases. Nevertheless, AP' and Q' <i>are</i> generally superior to <i>bpref</i>, <i>Rank-Biased Precision</i> and its condensed-list version even in the presence of biases.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Information Retrieval (IR) test collections are growing larger, and relevance data constructed through <i>pooling</i> are suspected of becoming more and more <i>incomplete</i> and <i>biased</i>. Several studies have used IR evaluation metrics specifically designed to handle this problem, but most of them have only examined the metrics under <i>incomplete but unbiased</i> conditions, using random samples of the original relevance data. This paper examines nine metrics in more realistic settings, by reducing the number of pooled systems and the number of pooled documents. Even though previous studies have shown that metrics based on a <i>condensed list</i>, obtained by removing all unjudged documents from the original ranked list, are effective for handling very incomplete but unbiased relevance data, we show that these results do not hold when the relevance data are biased towards particular systems or towards the top of the pools. More specifically, we show that the condensed-list versions of <i>Average Precision</i>, <i>Qmeasure</i> and <i>normalised Discounted Cumulative Gain</i>, which we denote as AP', Q' and nDCG', are not necessarily superior to the original metrics for handling biases. Nevertheless, AP' and Q' <i>are</i> generally superior to <i>bpref</i>, <i>Rank-Biased Precision</i> and its condensed-list version even in the presence of biases.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AA00700121
• 書誌情報: Journal of information processing; 巻 17, p. 156-166, 発行日 2009-04-08
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 1882-6652
• 出版者 / 言語: ja
• 出版者: 情報処理学会