Median measure: an approach to IR systems evaluation |
| |
| Authors: | Howard Greisdorf Amanda Spink |
| |
| Institution: | a School of Library and Information Science, University of North Texas, P.O. Box 311068 Denton, TX 76203-1068, USA;b School of Information Sciences and Technology, The Pennsylvania State University, 511 Rider I Building, 120 S. Burrowes St, University Park, PA 16801, USA |
| |
| Abstract: | In this paper results from three studies examining 1295 relevance judgments by 36 information retrieval (IR) system end-users is reported. Both the region of the relevance judgments, from non-relevant to highly relevant, and the motivations or levels for the relevance judgments are examined. Three major findings are studied. First, the frequency distributions of relevance judgments by IR system end-users tend to take on a bi-modal shape with peaks at the extremes (non-relevant/relevant) with a flatter middle range. Second, the different type of scale (interval or ordinal) used in each study did not alter the shape of the relevance frequency distributions. And third, on an interval scale, the median point of relevance judgment distributions correlates with the point where relevant and partially relevant items begin to be retrieved. The median point of a distribution of relevance judgments may provide a measure of user/IR system interaction to supplement precision/recall measures. The implications of investigation for relevance theory and IR systems evaluation are discussed. |
| |
| Keywords: | |
| 本文献已被 ScienceDirect 等数据库收录！ |
|
