WEKO3
アイテム
One Click One Revisited: Enhancing Evaluation based on Information Units
https://ipsj.ixsq.nii.ac.jp/records/83209
https://ipsj.ixsq.nii.ac.jp/records/83209d0a40309-6701-4bd2-aa03-ae059b25c095
名前 / ファイル | ライセンス | アクション |
---|---|---|
![]() |
Copyright (c) 2012 by the Information Processing Society of Japan
|
|
オープンアクセス |
Item type | SIG Technical Reports(1) | |||||||
---|---|---|---|---|---|---|---|---|
公開日 | 2012-07-25 | |||||||
タイトル | ||||||||
タイトル | One Click One Revisited: Enhancing Evaluation based on Information Units | |||||||
タイトル | ||||||||
言語 | en | |||||||
タイトル | One Click One Revisited: Enhancing Evaluation based on Information Units | |||||||
言語 | ||||||||
言語 | eng | |||||||
キーワード | ||||||||
主題Scheme | Other | |||||||
主題 | 情報検索と情報抽出 | |||||||
資源タイプ | ||||||||
資源タイプ識別子 | http://purl.org/coar/resource_type/c_18gh | |||||||
資源タイプ | technical report | |||||||
著者所属 | ||||||||
Microsoft Research Asia, China | ||||||||
著者所属 | ||||||||
Kyoto University, Japan | ||||||||
著者所属(英) | ||||||||
en | ||||||||
Microsoft Research Asia, China | ||||||||
著者所属(英) | ||||||||
en | ||||||||
Kyoto University, Japan | ||||||||
著者名 |
Tetsuya, Sakai
× Tetsuya, Sakai
|
|||||||
著者名(英) |
Tetsuya, Sakai
× Tetsuya, Sakai
|
|||||||
論文抄録 | ||||||||
内容記述タイプ | Other | |||||||
内容記述 | This paper extends the evaluation framework of the NTCIR-9 One Click Access Task (1CLICK-1), which required systems to return a single, concise textual output in response to a query in order to satisfy the user immediately after a click on the SEARCH button. Unlike traditional nugget-based summarisation and question answering evaluation methods, S-measure, the official evaluation measure of 1CLICK-1, discounts the value of each information unit based on its position within the textual output. We first show that the discount parameter L of S-measure affects system ranking and discriminative power, and that using multiple values, e.g. L = 250 (user has only 30 seconds to view the text) and L = 500 (user has one minute), is beneficial. We then complement the recall-like S-measure with a simple, precision-like measure called T-measure as well as a combination of S-measure and T-measure, called S#. We show that S# with a heavy emphasis on S-measure imposes an appropriate length penalty to 1CLICK-1 system outputs and yet achieves discriminative power that is comparable to S-measure. These new measures will be used at NTCIR-10 1CLICK-2. | |||||||
論文抄録(英) | ||||||||
内容記述タイプ | Other | |||||||
内容記述 | This paper extends the evaluation framework of the NTCIR-9 One Click Access Task (1CLICK-1), which required systems to return a single, concise textual output in response to a query in order to satisfy the user immediately after a click on the SEARCH button. Unlike traditional nugget-based summarisation and question answering evaluation methods, S-measure, the official evaluation measure of 1CLICK-1, discounts the value of each information unit based on its position within the textual output. We first show that the discount parameter L of S-measure affects system ranking and discriminative power, and that using multiple values, e.g. L = 250 (user has only 30 seconds to view the text) and L = 500 (user has one minute), is beneficial. We then complement the recall-like S-measure with a simple, precision-like measure called T-measure as well as a combination of S-measure and T-measure, called S#. We show that S# with a heavy emphasis on S-measure imposes an appropriate length penalty to 1CLICK-1 system outputs and yet achieves discriminative power that is comparable to S-measure. These new measures will be used at NTCIR-10 1CLICK-2. | |||||||
書誌レコードID | ||||||||
収録物識別子タイプ | NCID | |||||||
収録物識別子 | AN10112482 | |||||||
書誌情報 |
研究報告データベースシステム(DBS) 巻 2012-DBS-154, 号 6, p. 1-7, 発行日 2012-07-25 |
|||||||
Notice | ||||||||
SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc. | ||||||||
出版者 | ||||||||
言語 | ja | |||||||
出版者 | 情報処理学会 |