Expected Results

From ATIRE

Jump to: navigation, search

This page documents some of the expected results when searching with the ATIRE search engine. The documented commands assume that the default filenames are used and all binaries from the project are in the PATH. The exact output may not match due to parallel indexing, but should match to 3 or 4 decimal points.

Contents

Wall Street Journal

The Wall Street Journal topics and assessments are contained within the evaluation folder of the ATIRE search engine repository. These are not necessarily the best results possible, but are used to verify that changes made to the search engine have not affected the results

atire -a evaluation/WSJ.qrels -q evaluation/51-100.titles

Expected output:

Processed 50 topics:
MAP: 0.221737

The following was generated 2014-02-11 using ATIRE changeset 1847:118b625a87ce on Mac OS X 10.9.1 .

WSJ

Ranking FunctionMAP
-RBM25:0.9:0.40.222400
-Rlmds:25000.224130
-RDPH0.214867
-RDFRee0.195040

Wikipedia 2009

The Wikipedia 2009 queries and assessments are available from the INEX website.

The queries first need to be run through the topic extractor found in the evaluation directory like so:

INEXtopics_to_ANTquery -clean 2009-topics.xml > 2009.topics
INEXtopics_to_ANTquery -clean 2010-topics.xml > 2010.topics

Then after removing topics that have no assessments:

atire -a inex2009.qrels -q 2009.topics
atire -a inex2010-article.qrels -q 2010.topics

Expected output:

Processed 68 topics:
MAP: 0.344935

Processed 52 topics:
MAP: 0.351517

TREC Collections

See TREC Results