A Search Log-Based Approach to Evaluation

A Search Log-Based Approach to Evaluation,10.1007/978-3-642-15464-5_26,Junte Zhang,Jaap Kamps

A Search Log-Based Approach to Evaluation   (Citations: 2)
BibTex | RIS | RefWorks Download
Anyone offering content in a digital library is naturally interested in assessing its performance: how well does my system meet the users’ information needs? Standard evaluation benchmarks have been developed in information retrieval that can be used to test retrieval effectiveness. However, these generic benchmarks focus on a single document genre, language, media-type, and searcher stereotype that is radically different from the unique content and user community of a particular digital library. This paper proposes to derive a domain-specific test collection from readily available interaction data in search logs files that captures the domain-specificity of digital libraries. We use as case study an archival institution’s complete search log that spans over multiple years, and derive a large-scale test collection. We manually derive a set of topics judged by human experts—based on a set of e-mail reference questions and responses from archivists—and use this for validation. Our main finding is that we can derive a reliable and domain-specific test collection from search log files.
Conference: European Conference on Digital Libraries - ECDL , pp. 248-260, 2010
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.
    • ...In prior research, it has been shown that search log files can be eectively used to construct domainspecific IR test collections that agree with traditional methods of evaluation [35]...
    • ...In a previous study [35], it is explained how the log files can be used to construct a massive test collection for IR evaluation, which leads to very similar system rankings as a set of humanly judged topics...
    • ...This explanation is unlikely since we validated the resulting log-based evaluation against a human judged topic set and obtained very similar system rankings [35]...

    Junte Zhanget al. Search log analysis of user stereotypes, information seeking behavior,...

Sort by: