Toward Cranfield-inspired reusability assessment in interactive information retrieval evaluation |
| |
| Institution: | 1. Institute of Environmental Science and Technology, Universitat Autònoma de Barcelona, Spain;2. Graduate School of Economics and Management, Ural Federal University, Yekaterinburg, Russian Federation;1. School of Information Management, Nanjing University, Nanjing 210023, China;2. School of Computer Science and Engineering, Southeast University, Nanjing 210096, China;3. School of Economics and Management, Nanjing University of Science and Technology, Nanjing 210094, China |
| |
| Abstract: | Re-using research resources is essential for advancing knowledge and developing repeatable, empirically solid experiments in scientific fields, including interactive information retrieval (IIR). Despite recent efforts on standardizing research re-use and documentation, how to quantitatively measure the reusability of IIR resources still remains an open challenge. Inspired by the reusability evaluations on Cranfield experiments, our work proactively explores the problem of measuring IIR test collection reusability and makes threefold contributions: (1) constructing a novel usefulness-oriented framework with specific analytical methods for evaluating the reusability of IIR test collections consisting of query sets, document/page sets, and sets of task-document usefulness (tuse); (2) explaining the potential impacts of varying IIR-specific factors (e.g. search tasks, sessions, user characteristics) on test collection reusability; (3) proposing actionable methods for building reusable test collections in IIR and thereby amortizing the true cost of user-oriented evaluations. The Cranfield-inspired reusability assessment framework serves as an initial step towards accurately evaluating the reusability of IIR research resources and measuring the reproducibility of IIR evaluation results. It also demonstrates an innovative approach to integrating the insights from individual heterogeneous user studies with the evaluation techniques developed in standardized ad hoc retrieval experiments, which will facilitate the maturation of IIR fields and eventually benefits both sides of research. |
| |
| Keywords: | Interactive information retrieval (IIR) IIR evaluation Usefulness Test collection Reusability assessment Reproducibility studies |
| 本文献已被 ScienceDirect 等数据库收录！ |
|
