User simulations for evaluating answers to question series |
| |
| Authors: | Jimmy Lin |
| |
| Institution: | College of Information Studies, Department of Computer Science, Institute for Advanced Computer Studies, University of Maryland, College Park, MD 20742, USA |
| |
| Abstract: | Recently, question series have become one focus of research in question answering. These series are comprised of individual factoid, list, and “other” questions organized around a central topic, and represent abstractions of user–system dialogs. Existing evaluation methodologies have yet to catch up with this richer task model, as they fail to take into account contextual dependencies and different user behaviors. This paper presents a novel simulation-based methodology for evaluating answers to question series that addresses some of these shortcomings. Using this methodology, we examine two different behavior models: a “QA-styled” user and an “IR-styled” user. Results suggest that an off-the-shelf document retrieval system is competitive with state-of-the-art QA systems in this task. Advantages and limitations of evaluations based on user simulations are also discussed. |
| |
| Keywords: | Question answering Evaluation User simulations |
| 本文献已被 ScienceDirect 等数据库收录！ |
|
