This page documents some of the expected results when searching with the ATIRE search engine. The documented commands assume that the default filenames are used and all binaries from the project are in the PATH. The exact output may not match due to parallel indexing, but should match to 3 or 4 decimal points.
Wall Street Journal
The Wall Street Journal topics and assessments are contained within the evaluation folder of the ATIRE search engine repository. These are not necessarily the best results possible, but are used to verify that changes made to the search engine have not affected the results
atire -a evaluation/WSJ.qrels -q evaluation/51-100.titles
Processed 50 topics: MAP: 0.221737
The following was generated 2014-02-11 using ATIRE changeset 1847:118b625a87ce on Mac OS X 10.9.1 .
The Wikipedia 2009 queries and assessments are available from the INEX website.
The queries first need to be run through the topic extractor found in the evaluation directory like so:
INEXtopics_to_ANTquery -clean 2009-topics.xml > 2009.topics INEXtopics_to_ANTquery -clean 2010-topics.xml > 2010.topics
Then after removing topics that have no assessments:
atire -a inex2009.qrels -q 2009.topics atire -a inex2010-article.qrels -q 2010.topics
Processed 68 topics: MAP: 0.344935 Processed 52 topics: MAP: 0.351517
See TREC Results