On the Properties of Evaluation Metrics for Finding One Highly Relevant Document

• Item type: Trans(1)
• 公開日: 2007-09-15
• タイトル: On the Properties of Evaluation Metrics for Finding One Highly Relevant Document
• タイトル / 言語: en
• タイトル: On the Properties of Evaluation Metrics for Finding One Highly Relevant Document
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: 研究論文（IPSJ Best Paper Award、論文賞受賞）
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_6501
• 資源タイプ: journal article
• 著者所属: NewsWatch Inc. (This work was done when the author was at Toshiba.)
• 著者所属(英): en
• 著者所属(英): NewsWatch, Inc. (This work was done when the author was at Toshiba.)
• 著者名: Tetsuya, Sakai
• 著者名(英): Tetsuya, Sakai
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Traditional information retrieval evaluation relies on both precision and recall. However modern search environments such as the Web in which recall is either unimportant or immeasurable require precision-oriented evaluation. In particular finding one highly relevant document is very important for practical tasks such as known-item search and suspected-item search. This paper compares the properties of five evaluation metrics that are applicable to the task of finding one highly relevant document in terms of the underlying assumptions how the system rankings produced resemble each other and discriminative power. We employ two existing methods for comparing the discriminative power of these metrics: The Swap Method proposed by Voorhees and Buckley at ACM SIGIR 2002 and the Bootstrap Sensitivity Method proposed by Sakai at SIGIR 2006. We use four data sets from NTCIR to show that while P($^{+}$)-measure O-measure and NWRR (Normalised Weighted Reciprocal Rank) are reasonably highly correlated to one another P($^{+}$)-measure and O-measure are more discriminative than NWRR which in turn is more discriminative than Reciprocal Rank. We therefore conclude that P($^{+}$)-measure and O-measure each modelling a different user behaviour are the most useful evaluation metrics for the task of finding one highly relevant document.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Traditional information retrieval evaluation relies on both precision and recall. However, modern search environments such as the Web, in which recall is either unimportant or immeasurable, require precision-oriented evaluation. In particular, finding one highly relevant document is very important for practical tasks such as known-item search and suspected-item search. This paper compares the properties of five evaluation metrics that are applicable to the task of finding one highly relevant document in terms of the underlying assumptions, how the system rankings produced resemble each other, and discriminative power. We employ two existing methods for comparing the discriminative power of these metrics: The Swap Method proposed by Voorhees and Buckley at ACM SIGIR 2002, and the Bootstrap Sensitivity Method proposed by Sakai at SIGIR 2006. We use four data sets from NTCIR to show that, while P($^{+}$)-measure, O-measure and NWRR (Normalised Weighted Reciprocal Rank) are reasonably highly correlated to one another, P($^{+}$)-measure and O-measure are more discriminative than NWRR, which in turn is more discriminative than Reciprocal Rank. We therefore conclude that P($^{+}$)-measure and O-measure, each modelling a different user behaviour, are the most useful evaluation metrics for the task of finding one highly relevant document.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AA11464847
• 書誌情報: 情報処理学会論文誌データベース（TOD）; 巻 48, 号 SIG14(TOD35), p. 29-46, 発行日 2007-09-15
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 1882-7799
• 出版者 / 言語: ja
• 出版者: 情報処理学会