Change search
ReferencesLink to record
Permanent link

Direct link
A Use Case Framework for Information Access Evaluation
KTH, School of Computer Science and Communication (CSC), Theoretical Computer Science, TCS.ORCID iD: 0000-0003-4042-4919
2014 (English)In: Professional Search in the Modern World: COST Action IC1002 on Multilingual and Multifaceted Interactive Information Access / [ed] Paltoglou, Georgios, Loizides, Fernando, Hansen, Preben, Springer, 2014, 6-22 p.Chapter in book (Refereed)
Abstract [en]

Information access is no longer only a question of retrieving topical text documents in a work-task related context. Information search has become one of the most common uses of the personal computers; a daily task for millions of individual users searching for information motivated by information needs they experience for some reason, momentarily or continuously. Instead of professionally edited text documents, multilingual and multimedia content from a variety of sources of varying quality needs to be accessed. Even the scope of the research efforts in the field must therefore be broadened to better capture the mechanisms for the systems’ impact, take-up and success in the marketplace. Much work has been carried out in this direction: graded relevance, and new evaluation metrics, more varied document collections used in evaluation and different search tasks evaluated. The research in the field is however fragmented. Despite that the need for a common evaluation framework is widely acknowledged, such framework is still not in place. IR system evaluation results are not regularly validated in Interactive IR or field studies; the infrastructure for generalizing Interactive IR results over tasks, users and collections is still missing. This chapter presents a use case-based framework for experimental design in the field of interactive information access. Use cases in general connect system design and evaluation to interaction and user goals, and help identifying test cases for different user groups of a system. We suggest that use cases can provide a useful link even between information access system usage and evaluation mechanisms and thus bring together research from the different related research fields. In this chapter we discuss how use cases can guide the developments of rich models of users, domains, environments, and interaction, and make explicit how the models are connected to benchmarking mechanisms. We give examples of the central features of the different models. The framework is highlighted by examples that sketch out how the framework can be productively used in experimental design and reporting with a minimal threshold for adoption.

Place, publisher, year, edition, pages
Springer, 2014. 6-22 p.
, Lecture Notes in Computer Science, ISSN 0302-9743 ; 8330
Keyword [en]
Benchmarking, Evaluation, Interaction, Use cases
National Category
Human Computer Interaction
URN: urn:nbn:se:kth:diva-157333DOI: 10.1007/978-3-319-12511-4_2ScopusID: 2-s2.0-84921412557ISBN: 978-3-319-12510-7OAI: diva2:769719

QC 20150331

Available from: 2014-12-08 Created: 2014-12-08 Last updated: 2015-03-31Bibliographically approved

Open Access in DiVA

No full text

Other links

Publisher's full textScopus

Search in DiVA

By author/editor
Karlgren, Jussi
By organisation
Theoretical Computer Science, TCS
Human Computer Interaction

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Altmetric score

Total: 68 hits
ReferencesLink to record
Permanent link

Direct link