共查询到20条相似文献，搜索用时 15 毫秒
1.
The influential Text REtrieval Conference (TREC) retrieval conference has always relied upon specialist assessors or occasionally participating groups to create relevance judgements for the tracks that it runs. Recently however, crowdsourcing has been championed as a cheap, fast and effective alternative to traditional TREC-like assessments. In 2010, TREC tracks experimented with crowdsourcing for the very first time. In this paper, we report our successful experience in creating relevance assessments for the TREC Blog track 2010 top news stories task using crowdsourcing. In particular, we crowdsourced both real-time newsworthiness assessments for news stories as well as traditional relevance assessments for blog posts. We conclude that crowdsourcing not only appears to be a feasible, but also cheap and fast means to generate relevance assessments. Furthermore, we detail our experiences running the crowdsourced evaluation of the TREC Blog track, discuss the lessons learned, and provide best practices. 相似文献
2.
《Microprocessing and Microprogramming》1995,41(5-6):333-337
In this introduction, we will summarize the main contributions of the papers collected in this special issue. Moreover, the topics addressed in these papers will be linked to the major research trends in the domain of parallel algorithms, architectures and compilation. 相似文献
3.
Omar Alonso 《Information Retrieval》2013,16(2):101-120
Crowdsourcing has emerged as a viable platform for conducting different types of relevance evaluation. The main reason behind this trend is that it makes possible to conduct experiments extremely fast, with good results at a low cost. However, like in any experiment, there are several implementation details that would make an experiment work or fail. To gather useful results, clear instructions, user interface guidelines, content quality, inter-rater agreement metrics, work quality, and worker feedback are important characteristics of a successful crowdsourcing experiment. Furthermore, designing and implementing experiments that require thousands or millions of labels is different than conducting small scale research investigations. In this paper we outline a framework for conducting continuous crowdsourcing experiments, emphasizing aspects that should be of importance for all sorts of tasks. We illustrate the value of characteristics that can impact the overall outcome using examples based on TREC, INEX, and Wikipedia data sets. 相似文献
4.
《Journal of Informetrics》2023,17(3):101432
The disruption (D) index is a network-based indicator to quantify the extent to which a focal paper disrupts its predecessors. This study focuses on what disruption means by examining example articles related to “sleeping beauties in science” and frequency-inverse document frequency (TF-IDF). We investigated the structure of the citation network and subsequent papers’ motivations for citing the focal papers. Based on the observation that conceptual work is more likely to disrupt science than technical work, we hypothesize that disruption reflects the mechanism of how paradigms shift in the development of science. We also assume that the disruption identified by the D index indicates more than generating a new direction. Disruptive contributions include revolutionary studies such as Nobel-prize-winning papers, as suggested in previous work. However, disruptive contributions also include scientific dissemination of new terminology created by popular proposals, such as “sleeping beauties in science.” Such contributions redefine and popularize phenomena in science. 相似文献
5.
介绍有目的、有方向,以约稿方式出版增刊的做法.以此种方式刊登的研究论文整体质量较好,且某一学科论文可得到较集中地展示.读者可以通过印刷版阅览,也可以通过期刊网进行浏览和检索.实践证明,只要充分开发、利用好增刊资源,增刊同样能为高校的教学、科研、人才培养发挥作用. 相似文献
6.
为了纪念科技期刊创刊各周年,可编撰一期刊庆纪念特刊.刊庆特刊通常的内容有导语、贺信、题词、寄语、感言、杂志发展史、刊庆主题约稿或征文选登、高质量专业论文、历年载文分析、优秀论文评选结果或重要论文题录等.编撰刊庆纪念特刊有助于系统梳理杂志历史,扩大杂志影响力,对杂志载文进行深入分析,加深作者、读者、编者感情.由于刊庆特刊通常为全彩页设计,编撰时要重视色彩的选择、图片和文字的分布、专业元素的体现,等等;在编撰过程中应尽早启动,构建框架,收集资料,进行约稿. 相似文献
7.
Kezban Dilek Onal Ye Zhang Ismail Sengor Altingovde Md Mustafizur Rahman Pinar Karagoz Alex Braylan Brandon Dang Heng-Lu Chang Henna Kim Quinten McNamara Aaron Angert Edward Banner Vivek Khetan Tyler McDonnell An Thanh Nguyen Dan Xu Byron C. Wallace Maarten de Rijke Matthew Lease 《Information Retrieval》2018,21(2-3):111-182
A recent “third wave” of neural network (NN) approaches now delivers state-of-the-art performance in many machine learning tasks, spanning speech recognition, computer vision, and natural language processing. Because these modern NNs often comprise multiple interconnected layers, work in this area is often referred to as deep learning. Recent years have witnessed an explosive growth of research into NN-based approaches to information retrieval (IR). A significant body of work has now been created. In this paper, we survey the current landscape of Neural IR research, paying special attention to the use of learned distributed representations of textual units. We highlight the successes of neural IR thus far, catalog obstacles to its wider adoption, and suggest potentially promising directions for future research. 相似文献
8.
The evaluation of diversified web search results is a relatively new research topic and is not as well-understood as the time-honoured evaluation methodology of traditional IR based on precision and recall. In diversity evaluation, one topic may have more than one intent, and systems are expected to balance relevance and diversity. The recent NTCIR-9 evaluation workshop launched a new task called INTENT which included a diversified web search subtask that differs from the TREC web diversity task in several aspects: the choice of evaluation metrics, the use of intent popularity and per-intent graded relevance, and the use of topic sets that are twice as large as those of TREC. The objective of this study is to examine whether these differences are useful, using the actual data recently obtained from the NTCIR-9 INTENT task. Our main experimental findings are: (1) The $\hbox{D}\,\sharp$ evaluation framework used at NTCIR provides more “intuitive” and statistically reliable results than Intent-Aware Expected Reciprocal Rank; (2) Utilising both intent popularity and per-intent graded relevance as is done at NTCIR tends to improve discriminative power, particularly for $\hbox{D}\,\sharp$ -nDCG; and (3) Reducing the topic set size, even by just 10 topics, can affect not only significance testing but also the entire system ranking; when 50 topics are used (as in TREC) instead of 100 (as in NTCIR), the system ranking can be substantially different from the original ranking and the discriminative power can be halved. These results suggest that the directions being explored at NTCIR are valuable. 相似文献
9.
Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems 总被引：1，自引：0，他引：1
Guido Zuccon Teerapong Leelanupab Stewart Whiting Emine Yilmaz Joemon M. Jose Leif Azzopardi 《Information Retrieval》2013,16(2):267-305
In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations. 相似文献
10.
A standard approach to Information Retrieval (IR) is to model text as a bag of words. Alternatively, text can be modelled
as a graph, whose vertices represent words, and whose edges represent relations between the words, defined on the basis of
any meaningful statistical or linguistic relation. Given such a text graph, graph theoretic computations can be applied to measure various properties of the graph, and hence of the text. This work
explores the usefulness of such graph-based text representations for IR. Specifically, we propose a principled graph-theoretic
approach of (1) computing term weights and (2) integrating discourse aspects into retrieval. Given a text graph, whose vertices
denote terms linked by co-occurrence and grammatical modification, we use graph ranking computations (e.g. PageRank Page et al.
in The pagerank citation ranking: Bringing order to the Web. Technical report, Stanford Digital Library Technologies Project,
1998) to derive weights for each vertex, i.e. term weights, which we use to rank documents against queries. We reason that our
graph-based term weights do not necessarily need to be normalised by document length (unlike existing term weights) because
they are already scaled by their graph-ranking computation. This is a departure from existing IR ranking functions, and we
experimentally show that it performs comparably to a tuned ranking baseline, such as BM25 (Robertson et al. in NIST Special
Publication 500-236: TREC-4, 1995). In addition, we integrate into ranking graph properties, such as the average path length, or clustering coefficient, which
represent different aspects of the topology of the graph, and by extension of the document represented as a graph. Integrating
such properties into ranking allows us to consider issues such as discourse coherence, flow and density during retrieval.
We experimentally show that this type of ranking performs comparably to BM25, and can even outperform it, across different
TREC (Voorhees and Harman in TREC: Experiment and evaluation in information retrieval, MIT Press, 2005) datasets and evaluation measures. 相似文献
11.
《Government Information Quarterly》2022,39(4):101742
Literature shows there is a growing interest in studies involving Artificial Intelligence (AI) in the public sector; and while there is evidence of many governmental initiatives that have been established to harness the power of AI, empirical research on the topic and evidence-based insights are rather lacking. The aim of this Special Issue on Artificial Intelligence for Data-Driven Decision-Making and Governance in Public Affairs is to extend both the theoretical and practical boundaries of AI research in the public sector in order to improve governmental decision-making and governance, thus enhancing public value creation. The papers in this special issue focus on AI risks and guidelines, AI governance, the risks of governmental implementation of AI to citizens' privacy, increasing citizen satisfaction through AI-enabled government services, the enablers and challenges of AI implementation in specific public sectors, and using AI to study political opinion. These papers not only advance our knowledge and understanding of the use of AI in government and public governance, but they also help to set out a renewed research agenda. Future research should, among other things, focus on inter- and multi-disciplinary empirical studies that call for the collaboration of a variety of stakeholders; on the longitudinal dynamics of creating public value through the breadth and depth of AI assimilation; and on the investigation of the ethical challenges (particularly data privacy) in AI implementation. 相似文献
12.
盛京将军辖区内混居着大量旗人与民人,在日常生活中,旗民的接触必然会产生各种各样的纠纷,进而引发各种司法案件。本文就《黑图档·嘉庆朝》对盛京将军辖区的刑民案件展开研究,进行梳理,以期对盛京将军辖区的司法状况研究作出贡献。首先,本文对盛京将军衙门的设置缘由和司法官员进行梳理;其次,对盛京将军辖区的刑事案件展开分析,通过对不同刑事案件种类的分析,对清代刑事立法展开研究;再次,对清代旗人在法律中所特有的特权进行探讨,旗人在法律中享有优待和特权;最后,对盛京将军辖区的司法状况进行总结。 相似文献
13.
14.
INEX与TREC是检索领域的两大检索系统评价平台,在检索技术发展迅速的今天依然保持强大生命力,在当今检索技术评价领域起着十分重要的作用。本篇文章通过对INEX与TREC的研究目标以及平台的构成要素包括三个方面:测试集、检索问题的构造、相关性评估的比较,找出INEX相对于TREC评测平台的创新及不同点,以便更加深入和全面地了解INEX的评测方法。 相似文献
15.
The Reliable Information Access (RIA) Workshop was held in the summer of 2003, with a goal of improved understanding of information
retrieval systems, in particular with regard to the variability of retrieval performance across topics. The workshop ran massive
cross-system failure analysis on 45 of the TREC topics and also performed cross-system experiments on pseudo-relevance feedback.
This paper presents an overview of that workshop, along with some preliminary conclusions from these experiments. Even if
this workshop was held 6 years ago, the issues of improving system performance across all topics is still critical to the
field and this paper, along with the others in this issue, are the first widely published full papers for the workshop. 相似文献
16.
[目的/意义] 分析科研众包项目的协作模式,为我国在开放环境下开展协作研究提供理论指导和行动指南。[方法/过程] 通过分析Crowd Research项目的协作过程,从公众科研人员招募、研究过程协作、结果提交、成果产出4方面入手,详细说明各部分的关键步骤,总结Crowd Research对我国发展科研众包项目的启示。[结果/结论] 研究发现,Crowd Research基于众包思想,发展了一种新型的科研协作模式,公众科研人员在专业科研人员的带领下可以协作参与研究过程,包括自主选择里程碑、同行评估提交内容、互相分配贡献值、合作发表学术论文等。 相似文献
17.
Farzad Ashrafi Hafez Mohammadhassanzadeh Farhad Shokraneh Ali Valinejadi Karim Johari Nazanin Saemi Alireza Zali Niloofar Mohaghegh Hassan Ashayeri 《Health information and libraries journal》2012,29(4):323-332
Objective: The purpose of this study is to analyse Iranian scientific publications in the neuroscience subfields by librarians and neuroscientists, using Science Citation Index Expanded (SCIE) via Web of Science data over the period, 2002–2008. Methods: Data were retrieved from the SCIE. Data were collected from the ‘subject area’ of the database and classified by neuroscience experts into 14 subfields. To identify the citation patterns, we applied the ‘impact factor’ and the ‘number of publication’. Data were also analysed using HISTCITE, Excel 2007 and SPSS. Results: Seven hundred and thirty‐four papers have been published by Iranian between 2002 and 2008. Findings showed a growing trend of neuroscience papers in the last 3 years with most papers (264) classified in the neuropharmacology subfield. There were fewer papers in neurohistory, psychopharmacology and artificial intelligence. International contributions of authors were mostly in the neurology subfield, and ‘Collaboration Coefficient’ for the neuroscience subfields in Iran was 0.686 which is acceptable. Most international collaboration between Iranians and developed countries was from USA. Eighty‐seven percent of the published papers were in journals with the impact factor between 0 and 4; 25% of papers were published by the researchers affiliated to Tehran University of Medical Sciences. Conclusion: Progress of neuroscience in Iran is mostly seen in the neuropharmacology and the neurology subfields. Other subfields should also be considered as a research priority by health policymakers. As this study was carried out by the collaboration of librarians and neuroscientists, it has been proved valuable for both librarians and policymakers. This study may be encouraging for librarians from other developing countries. 相似文献
18.
从第一作者论文、作者合作论文、省内外作者比例、基金论文、论文被《复印报刊资料》全文收录、论文被引频次等方面对2003-2005年“图书情报专家学者论坛专辑”——《图书馆论坛》第6期与1-5期进行比较分析，总结专辑的特点，还分析了《大学图书馆学报》读者沙龙上所有网友对《图书馆论坛》及专辑的评论意见，探讨专辑的学术影响．剖析存在的问题，以期对该刊的可持续发展有所启示。 相似文献
19.
An information retrieval (IR) system can often fail to retrieve relevant documents due to the incomplete specification of information need in the user’s query. Pseudo-relevance feedback (PRF) aims to improve IR effectiveness by exploiting potentially relevant aspects of the information need present in the documents retrieved in an initial search. Standard PRF approaches utilize the information contained in these top ranked documents from the initial search with the assumption that documents as a whole are relevant to the information need. However, in practice, documents are often multi-topical where only a portion of the documents may be relevant to the query. In this situation, exploitation of the topical composition of the top ranked documents, estimated with statistical topic modeling based approaches, can potentially be a useful cue to improve PRF effectiveness. The key idea behind our PRF method is to use the term-topic and the document-topic distributions obtained from topic modeling over the set of top ranked documents to re-rank the initially retrieved documents. The objective is to improve the ranks of documents that are primarily composed of the relevant topics expressed in the information need of the query. Our RF model can further be improved by making use of non-parametric topic modeling, where the number of topics can grow according to the document contents, thus giving the RF model the capability to adjust the number of topics based on the content of the top ranked documents. We empirically validate our topic model based RF approach on two document collections of diverse length and topical composition characteristics: (1) ad-hoc retrieval using the TREC 6-8 and the TREC Robust ’04 dataset, and (2) tweet retrieval using the TREC Microblog ’11 dataset. Results indicate that our proposed approach increases MAP by up to 9% in comparison to the results obtained with an LDA based language model (for initial retrieval) coupled with the relevance model (for feedback). Moreover, the non-parametric version of our proposed approach is shown to be more effective than its parametric counterpart due to its advantage of adapting the number of topics, improving results by up to 5.6% of MAP compared to the parametric version. 相似文献
20.
We have studied the efficiency of research in the EU by a percentile-based citation approach that analyzes the distribution of country papers among the world papers. Going up in the citation scale, the frequency of papers from efficient countries increases while the frequency from inefficient countries decreases. In the percentile-based approach, this trend, which is uniform at any citation level, is measured by the ep index that equals the Ptop 1%/Ptop 10% ratio. By using the ep index we demonstrate that EU research on fast-evolving technological topics is less efficient than the world average and that the EU is far from being able to compete with the most advanced countries. The ep index also shows that the USA is well ahead of the EU in both fast- and slow-evolving technologies, which suggests that the advantage of the USA over the EU in innovation is due to low research efficiency in the EU. In accord with some previous studies, our results show that the European Commission’s ongoing claims about the excellence of EU research are based on a wrong diagnosis. The EU must focus its research policy on the improvement of its inefficient research. Otherwise, the future of Europeans is at risk. 相似文献