Web Search Evaluation with Informational and Navigational Intents

• Item type: JInfP(1)
• 公開日: 2013-01-15
• タイトル: Web Search Evaluation with Informational and Navigational Intents
• タイトル / 言語: en
• タイトル: Web Search Evaluation with Informational and Navigational Intents
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: [Regular Papers] diversified search, evaluation, intents, metrics, search result diversification, web search
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_6501
• 資源タイプ: journal article
• 著者所属: Microsoft Research Asia
• 著者所属(英): en
• 著者所属(英): Microsoft Research Asia
• 著者名: Tetsuya, Sakai
• 著者名(英): Tetsuya, Sakai
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Given an ambiguous or underspecified web search query, search result diversification aims at accomodating different user intents within a single “entry-point” result page. However, some intents are informational, for which many relevant pages may help, while others are navigational, for which only one web page is required. We propose new evaluation metrics for search result diversification that considers this distinction, as well as the condordance test for comparing the intuitiveness of a given pair of metrics quantitatively. Our main experimental findings are: (a) In terms of discriminative power which reflects statistical reliability, the proposed metrics, DIN#-nDCG and P+Q#, are comparable to intent recall and D#-nDCG, and possibly superior to α-nDCG; (b) In terms of the concordance test which quantifies the agreement of a diversity metric with a gold standard metric that represents a basic desirable property, DIN#-nDCG is superior to other diversity metrics in its ability to reward both diversity and relevance at the same time. Moreover, both D#-nDCG and DIN#-nDCG significantly outperform α-nDCG in their ability to reward diversity, to reward relevance, and to reward both at the same time. In addition, we demonstrate that the randomised Tukey's Honestly Significant Differences test that takes the entire set of available runs into account is substantially more conservative than the paired bootstrap test that only considers one run pair at a time, and therefore recommend the former approach for significance testing when a set of runs is available for evaluation.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Given an ambiguous or underspecified web search query, search result diversification aims at accomodating different user intents within a single “entry-point” result page. However, some intents are informational, for which many relevant pages may help, while others are navigational, for which only one web page is required. We propose new evaluation metrics for search result diversification that considers this distinction, as well as the condordance test for comparing the intuitiveness of a given pair of metrics quantitatively. Our main experimental findings are: (a) In terms of discriminative power which reflects statistical reliability, the proposed metrics, DIN#-nDCG and P+Q#, are comparable to intent recall and D#-nDCG, and possibly superior to α-nDCG; (b) In terms of the concordance test which quantifies the agreement of a diversity metric with a gold standard metric that represents a basic desirable property, DIN#-nDCG is superior to other diversity metrics in its ability to reward both diversity and relevance at the same time. Moreover, both D#-nDCG and DIN#-nDCG significantly outperform α-nDCG in their ability to reward diversity, to reward relevance, and to reward both at the same time. In addition, we demonstrate that the randomised Tukey's Honestly Significant Differences test that takes the entire set of available runs into account is substantially more conservative than the paired bootstrap test that only considers one run pair at a time, and therefore recommend the former approach for significance testing when a set of runs is available for evaluation.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AA00700121
• 書誌情報: Journal of information processing; 巻 21, 号 1, p. 145-155, 発行日 2013-01-15
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 1882-6652
• 出版者 / 言語: ja
• 出版者: 情報処理学会