共查询到20条相似文献，搜索用时 31 毫秒
1.
Scaling Up the TREC Collection 总被引：3，自引：3，他引：0
Due to the popularity of Web search engines, a large proportion of real text retrieval queries are now processed over collections measured in tens or hundreds of gigabytes. A new Very Large test Collection (VLC) has been created to support qualification, measurement and comparison of systems operating at this level and to permit the study of the properties of very large collections. The VLC is an extension of the well-known TREC collection and has been distributed under the same conditions. A simple set of efficiency and effectiveness measures have been defined to encourage comparability of reporting. The 20 gigabyte first-edition of the VLC and a representative 10% sample have been used in a special interest track of the 1997 Text Retrieval Conference (TREC-6). The unaffordable cost of obtaining complete relevance assessments over collections of this scale is avoided by concentrating on early precision and relying on the core TREC collection to support detailed effectiveness studies. Results obtained by TREC-6 VLC track participants are presented here. All groups observed a significant increase in early precision as collection size increased. Explanatory hypotheses are advanced for future empirical testing. A 100 gigabyte second edition VLC (VLC2) has recently been compiled and distributed for use in TREC-7 in 1998. 相似文献
2.
Tetsuya Sakai 《Information Retrieval》2016,19(3):256-283
Traditional pooling-based information retrieval (IR) test collections typically have \(n= 50\)–100 topics, but it is difficult for an IR researcher to say why the topic set size should really be n. The present study provides details on principled ways to determine the number of topics for a test collection to be built, based on a specific set of statistical requirements. We employ Nagata’s three sample size design techniques, which are based on the paired t test, one-way ANOVA, and confidence intervals, respectively. These topic set size design methods require topic-by-run score matrices from past test collections for the purpose of estimating the within-system population variance for a particular evaluation measure. While the previous work of Sakai incorrectly used estimates of the total variances, here we use the correct estimates of the within-system variances, which yield slightly smaller topic set sizes than those reported previously by Sakai. Moreover, this study provides a comparison across the three methods. Our conclusions nevertheless echo those of Sakai: as different evaluation measures can have vastly different within-system variances, they require substantially different topic set sizes under the same set of statistical requirements; by analysing the tradeoff between the topic set size and the pool depth for a particular evaluation measure in advance, researchers can build statistically reliable yet highly economical test collections. 相似文献
3.
We consider text retrieval applications that assign query-specific relevance scores to documents drawn from particular collections.
Such applications represent a primary focus of the annual Text Retrieval Conference (TREC), where the participants compare
the empirical performance of different approaches. P(K), the proportion of the top K documents that are relevant, is a popular measure of retrieval effectiveness.
Participants in the TREC Very Large Corpus track have observed that when the target is a random sample from a collection,
P(K) is substantially smaller than when the target is the entire collection. Hawking and Robertson (2003) confirmed this finding
in a number of experimental settings. Hawking et al. (1999) posed as an open research question the cause of this phenomenon
and proposed five possible explanatory hypotheses. In this paper, we present a mathematical analysis that sheds some light
on these hypotheses and complements the experimental work of Hawking and Robertson (2003). We will also introduce C(L), contamination at L, the number of irrelevant documents amongst the top L relevant documents, and describe its properties.
Our analysis shows that while P(K) typically will increase with collection size, the phenomenon is not universal. That is, the asymptotic behavior of P(K) and C(L) depends on the score distributions and relative proportions of relevant and irrelevant documents in the collection.
While this article went to press, Yehuda Vardi passed away. We dedicate the paper to his memory. 相似文献
4.
Collection selection is a crucial function, central to the effectiveness and efficiency of a federated information retrieval
system. A variety of solutions have been proposed for collection selection adapting proven techniques used in centralised
retrieval. This paper defines a new approach to collection selection that models the topical distribution in each collection.
We describe an extended version of latent Dirichlet allocation that uses a hierarchical hyperprior to enable the different
topical distributions found in each collection to be modelled. Under the model, resources are ranked based on the topical
relationship between query and collection. By modelling collections in a low dimensional topic space, we can implicitly smooth
their term-based characterisation with appropriate terms from topically related samples, thereby dealing with the problem
of missing vocabulary within the samples. An important advantage of adopting this hierarchical model over current approaches
is that the model generalises well to unseen documents given small samples of each collection. The latent structure of each
collection can therefore be estimated well despite imperfect information for each collection such as sampled documents obtained
through query-based sampling. Experiments demonstrate that this new, fully integrated topical model is more robust than current
state of the art collection selection algorithms. 相似文献
5.
Server selection is an important subproblem in distributed information retrieval (DIR) but has commonly been studied with
collections of more or less uniform size and with more or less homogeneous content. In contrast, realistic DIR applications
may feature much more varied collections. In particular, personal metasearch—a novel application of DIR which includes all
of a user’s online resources—may involve collections which vary in size by several orders of magnitude, and which have highly
varied data. We describe a number of algorithms for server selection, and consider their effectiveness when collections vary
widely in size and are represented by imperfect samples. We compare the algorithms on a personal metasearch testbed comprising
calendar, email, mailing list and web collections, where collection sizes differ by three orders of magnitude. We then explore
the effect of collection size variations using four partitionings of the TREC ad hoc data used in many other DIR experiments.
Kullback-Leibler divergence, previously considered poorly effective, performs better than expected in this application; other
techniques thought to be effective perform poorly and are not appropriate for this problem. A strong correlation with size-based
rankings for many techniques may be responsible. 相似文献
6.
Modern retrieval test collections are built through a process called pooling in which only a sample of the entire document
set is judged for each topic. The idea behind pooling is to find enough relevant documents such that when unjudged documents
are assumed to be nonrelevant the resulting judgment set is sufficiently complete and unbiased. Yet a constant-size pool represents
an increasingly small percentage of the document set as document sets grow larger, and at some point the assumption of approximately
complete judgments must become invalid. This paper shows that the judgment sets produced by traditional pooling when the pools
are too small relative to the total document set size can be biased in that they favor relevant documents that contain topic
title words. This phenomenon is wholly dependent on the collection size and does not depend on the number of relevant documents
for a given topic. We show that the AQUAINT test collection constructed in the recent TREC 2005 workshop exhibits this biased
relevance set; it is likely that the test collections based on the much larger GOV2 document set also exhibit the bias. The
paper concludes with suggested modifications to traditional pooling and evaluation methodology that may allow very large reusable
test collections to be built.
相似文献
| Ellen VoorheesEmail: |
7.
This paper describes a method, using Genetic Programming, to automatically determine term weighting schemes for the vector
space model. Based on a set of queries and their human determined relevant documents, weighting schemes are evolved which
achieve a high average precision. In Information Retrieval (IR) systems, useful information for term weighting schemes is
available from the query, individual documents and the collection as a whole.
We evolve term weighting schemes in both local (within-document) and global (collection-wide) domains which interact with
each other correctly to achieve a high average precision. These weighting schemes are tested on well-known test collections
and are compared to the traditional tf-idf weighting scheme and to the BM25 weighting scheme using standard IR performance metrics.
Furthermore, we show that the global weighting schemes evolved on small collections also increase average precision on larger
TREC data. These global weighting schemes are shown to adhere to Luhn’s resolving power as both high and low frequency terms
are assigned low weights. However, the local weightings evolved on small collections do not perform as well on large collections.
We conclude that in order to evolve improved local (within-document) weighting schemes it is necessary to evolve these on
large collections. 相似文献
8.
《The Journal of Academic Librarianship》2023,49(4):102739
Diversity is commonly invoked as a goal for libraries in a variety of domains, including collections. This study explores how diversity language is used to characterize collection development efforts at a sample of large North American academic libraries, through content analysis of strategic planning documents as well as library webpages providing information about diversity efforts. The topic of collection diversity was found to be not consistently addressed in these documents. While 90 % of the institutions whose strategic plans were examined mentioned diversity in some way, less than half of these did so in relation to collections; about one-third of the libraries with diversity information pages omitted reference to collections in these documents. Strategic plans and webpages that mentioned collection diversity often did so cursorily, and in vague terms that lacked specific details about the nature and purpose of diversification. This indicates that the academic libraries still have far to go in developing a shared conceptualization of what collection diversity involves. Analysis of updated documents, retrieved two years after the initial sample, suggests that there may be increasing attention to social justice concerns in these contexts, which would potentially enable greater precision in discussions of collection diversification. 相似文献
9.
Robin Aly Aiden Doherty Djoerd Hiemstra Franciska de Jong Alan F. Smeaton 《Information Retrieval》2013,16(5):557-583
Concept based video retrieval often relies on imperfect and uncertain concept detectors. We propose a general ranking framework to define effective and robust ranking functions, through explicitly addressing detector uncertainty. It can cope with multiple concept-based representations per video segment and it allows the re-use of effective text retrieval functions which are defined on similar representations. The final ranking status value is a weighted combination of two components: the expected score of the possible scores, which represents the risk-neutral choice, and the scores’ standard deviation, which represents the risk or opportunity that the score for the actual representation is higher. The framework consistently improves the search performance in the shot retrieval task and the segment retrieval task over several baselines in five TRECVid collections and two collections which use simulated detectors of varying performance. 相似文献
10.
11.
Empirical modeling of the score distributions associated with retrieved documents is an essential task for many retrieval
applications. In this work, we propose modeling the relevant documents’ scores by a mixture of Gaussians and the non-relevant
scores by a Gamma distribution. Applying Variational Bayes we automatically trade-off the goodness-of-fit with the complexity
of the model. We test our model on traditional retrieval functions and actual search engines submitted to TREC. We demonstrate
the utility of our model in inferring precision-recall curves. In all experiments our model outperforms the dominant exponential-Gaussian
model. 相似文献
12.
社会信息服务演进中的图书馆 总被引：1，自引：0，他引：1
图书馆本质上是一个“信息资源体系服务系统”，资源建设上致力于信息资源体系的形成和发展，服务上致力于信息的自由平等获取。图书馆、书店、因特网、信息咨询公司等服务系统之间相互依存竞争、不断整合，共同构成了完整而富有生机的社会信息服务体系。图书馆要主动参与现有竞争，又要在资源格局、服务格局和用户格局中不断拓展“竞争蓝海”。 相似文献
13.
Operational multimodal information retrieval systems have to deal with increasingly complex document collections and queries that are composed of a large set of textual and non-textual modalities such as ratings, prices, timestamps, geographical coordinates, etc. The resulting combinatorial explosion of modality combinations makes it intractable to treat each modality individually and to obtain suitable training data. As a consequence, instead of finding and training new models for each individual modality or combination of modalities, it is crucial to establish unified models, and fuse their outputs in a robust way. Since the most popular weighting schemes for textual retrieval have in the past generalized well to many retrieval tasks, we demonstrate how they can be adapted to be used with non-textual modalities, which is a first step towards finding such a unified model. We demonstrate that the popular weighting scheme BM25 is suitable to be used for multimodal IR systems and analyze the underlying assumptions of the BM25 formula with respect to merging modalities under the so-called raw-score merging hypothesis, which requires no training. We establish a multimodal baseline for two multimodal test collections, show how modalities differ with respect to their contribution to relevance and the difficulty of treating modalities with overlapping information. Our experiments demonstrate that our multimodal baseline with no training achieves a significantly higher retrieval effectiveness than using just the textual modality for the social book search 2016 collection and lies in the range of a trained multimodal approach using the optimal linear combination of the modality scores. 相似文献
14.
Jacques Savoy 《Information Retrieval》2004,7(1-2):121-148
This paper describes and evaluates different retrieval strategies that are useful for search operations on document collections written in various European languages, namely French, Italian, Spanish and German. We also suggest and evaluate different query translation schemes based on freely available translation resources. In order to cross language barriers, we propose a combined query translation approach that has resulted in interesting retrieval effectiveness. Finally, we suggest a collection merging strategy based on logistic regression that tends to perform better than other merging approaches. 相似文献
15.
Lorraine Goeuriot Gareth J. F. Jones Liadh Kelly Johannes Leveling Mihai Lupu Joao Palotti Guido Zuccon 《Information Retrieval》2018,21(6):507-540
Since its inception in 2013, one of the key contributions of the CLEF eHealth evaluation campaign has been the organization of an ad-hoc information retrieval (IR) benchmarking task. This IR task evaluates systems intended to support laypeople searching for and understanding health information. Each year the task provides registered participants with standard IR test collections consisting of a document collection and topic set. Participants then return retrieval results obtained by their IR systems for each query, which are assessed using a pooling procedure. In this article we focus on CLEF eHealth 2013 and 2014s retrieval task, which saw topics created based on patients’ information needs associated with their medical discharge summaries. We overview the task and datasets created, and the results obtained by participating teams over these two years. We then provide a detailed comparative analysis of the results, and conduct an evaluation of the datasets in the light of these results. This twofold study of the evaluation campaign teaches us about technical aspects of medical IR, such as the effectiveness of query expansion; the quality and characteristics of CLEF eHealth IR datasets, such as their reliability; and how to run an IR evaluation campaign in the medical domain. 相似文献
16.
Tuomo Korenius Jorma Laurikkala Martti Juhola Kalervo Järvelin 《Information Retrieval》2006,9(1):33-53
Search facilitated with agglomerative hierarchical clustering methods was studied in a collection of Finnish newspaper articles
(N = 53,893). To allow quick experiments, clustering was applied to a sample (N = 5,000) that was reduced with principal components analysis. The dendrograms were heuristically cut to find an optimal partition,
whose clusters were compared with each of the 30 queries to retrieve the best-matching cluster. The four-level relevance assessment
was collapsed into a binary one by (A) considering all the relevant and (B) only the highly relevant documents relevant, respectively.
Single linkage (SL) was the worst method. It created many tiny clusters, and, consequently, searches enabled with it had high
precision and low recall. The complete linkage (CL), average linkage (AL), and Ward's methods (WM) returned reasonably-sized
clusters typically of 18–32 documents. Their recall (A: 27–52%, B: 50–82%) and precision (A: 83–90%, B: 18–21%) was higher
than and comparable to those of the SL clusters, respectively. The AL and WM clustering had 1–8% better effectiveness than
nearest neighbor searching (NN), and SL and CL were 1–9% less efficient that NN. However, the differences were statistically
insignificant. When evaluated with the liberal assessment A, the results suggest that the AL and WM clustering offer better
retrieval ability than NN. Assessment B renders the AL and WM clustering better than NN, when recall is considered more important
than precision. The results imply that collections in the highly inflectional and agglutinative languages, such as Finnish,
may be clustered as the collections in English, provided that documents are appropriately preprocessed. 相似文献
17.
Shiva Imani Moghadasi Sri Devi Ravana Sudharshan N. Raman 《Journal of Informetrics》2013,7(2):301-312
For a system-based information retrieval evaluation, test collection model still remains as a costly task. Producing relevance judgments is an expensive, time consuming task which has to be performed by human assessors. It is not viable to assess the relevancy of every single document in a corpus against each topic for a large collection. In an experimental-based environment, partial judgment on the basis of a pooling method is created to substitute a complete assessment of documents for relevancy. Due to the increasing number of documents, topics, and retrieval systems, the need to perform low-cost evaluations while obtaining reliable results is essential. Researchers are seeking techniques to reduce the costs of experimental IR evaluation process by the means of reducing the number of relevance judgments to be performed or even eliminating them while still obtaining reliable results. In this paper, various state-of-the-art approaches in performing low-cost retrieval evaluation are discussed under each of the following categories; selecting the best sets of documents to be judged; calculating evaluation measures, both, robust to incomplete judgments; statistical inference of evaluation metrics; inference of judgments on relevance, query selection; techniques to test the reliability of the evaluation and reusability of the constructed collections; and other alternative methods to pooling. This paper is intended to link the reader to the corpus of ‘must read’ papers in the area of low-cost evaluation of IR systems. 相似文献
18.
《Cataloging & classification quarterly》2013,51(4):73-85
ABSTRACT This article discusses how samples of records for laboratory IR experiments on OPACs can be constructed so that results obtained from different experiments can be compared. The literature on laboratory IR experiments seems to indicate that the retrieval effectiveness (recall and precision) is affected by the way the samples of records for such experiments are generated. Especially the amount of records and the subject area coverage of the records seems to affect the retrieval effectiveness. This article contains suggestions for the construction of samples for laboratory IR experiments on OPACs and demonstrates that the retrieval effectiveness is affected by different sample size and composition. 相似文献
19.
We first present in this paper an analytical view of heuristic retrieval constraints which yields simple tests to determine
whether a retrieval function satisfies the constraints or not. We then review empirical findings on word frequency distributions
and the central role played by burstiness in this context. This leads us to propose a formal definition of burstiness which
can be used to characterize probability distributions with respect to this phenomenon. We then introduce the family of information-based
IR models which naturally captures heuristic retrieval constraints when the underlying probability distribution is bursty
and propose a new IR model within this family, based on the log-logistic distribution. The experiments we conduct on several
collections illustrate the good behavior of the log-logistic IR model: It significantly outperforms the Jelinek-Mercer and
Dirichlet prior language models on most collections we have used, with both short and long queries and for both the MAP and
the precision at 10 documents. It also compares favorably to BM25 and has similar performance to classical DFR models such
as InL2 and PL2. 相似文献
20.
A Task-Oriented Non-Interactive Evaluation Methodology for Information Retrieval Systems 总被引：2，自引：0，他引：2
Jane Reid 《Information Retrieval》2000,2(1):115-129
Past research has identified many different types of relevance in information retrieval (IR). So far, however, most evaluation of IR systems has been through batch experiments conducted with test collections containing only expert, topical relevance judgements. Recently, there has been some movement away from this traditional approach towards interactive, more user-centred methods of evaluation. However, these are expensive for evaluators in terms both of time and of resources. This paper describes a new evaluation methodology, using a task-oriented test collection, which combines the advantages of traditional non-interactive testing with a more user-centred emphasis. The main features of a task-oriented test collection are the adoption of the task, rather than the query, as the primary unit of evaluation and the naturalistic character of the relevance judgements. 相似文献