The evaluation measures of an information retrieval system is the process of assessing how well the search results satisfied the user's query intent. The metrics are often split into multiple categories. Online metrics measure actual users' interactions with the search system. Offline metrics measure the relevance of the search engine by having expert judges measure how likely each result (or the SERP page as a whole) is to meet the information needs of the user.
Contents
- Online metrics
- Session abandonment rate
- Click through rate
- Session success rate
- Zero result rate
- Offline metrics
- Precision
- Recall
- Fall out
- F score F measure
- Average precision
- Precision at K
- R Precision
- Mean average precision
- Discounted cumulative gain
- Other measures
- Visualization
- Top queries list
- Queries per time
- References
The mathematical symbols used in the formulas below mean:
Online metrics
Online metrics are generally created from data mined from search logs. The metrics are often used to determine the success of an A/B test.
Session abandonment rate
Session abandonment rate is a ratio of search session which do not result in a click.
Click-through rate
Click-through rate (CTR) is the ratio of users who click on a specific link to the number of total users who view a page, email, or advertisement. It is commonly used to measure the success of an online advertising campaign for a particular website as well as the effectiveness of email campaigns.
Session success rate
Session success rate measures the ratio of user sessions that lead to a success. Defining "success" is often dependent on context, but for search a successful result is often measured using dwell time as a primary factor along with secondary user interaction, for instance, the user copying the result URL is considered a successful result, as is copy/pasting from the snippet.
Zero result rate
Zero result rate (ZRR) is the ratio of SERPs which returned with zero results. The metric either indicates a recall issue, or that the information being searched for is not in the index.
Offline metrics
Offline metrics are generally created from relevance judgement sessions where the judges score the quality of the search results. The judges often score each result of a query as either binary (good/bad), or on a multi-level scale of satisfying the needs of the searcher. In practice, queries may be ill-posed and there may be different shades of relevancy. For instance there is ambiguity in the query "mars", the judge does not know if the user is search for Mars the planet, Mars the chocolate bar, or Bruno Mars the singer.
Precision
Precision is the fraction of the documents retrieved that are relevant to the user's information need.
In binary classification, precision is analogous to positive predictive value. Precision takes all retrieved documents into account. It can also be evaluated at a given cut-off rank, considering only the topmost results returned by the system. This measure is called precision at n or P@n.
Note that the meaning and usage of "precision" in the field of information retrieval differs from the definition of accuracy and precision within other branches of science and statistics.
Recall
Recall is the fraction of the documents that are relevant to the query that are successfully retrieved.
In binary classification, recall is often called sensitivity. So it can be looked at as the probability that a relevant document is retrieved by the query.
It is trivial to achieve recall of 100% by returning all documents in response to any query. Therefore, recall alone is not enough but one needs to measure the number of non-relevant documents also, for example by computing the precision.
Fall-out
The proportion of non-relevant documents that are retrieved, out of all non-relevant documents available:
In binary classification, fall-out is closely related to specificity and is equal to
It is trivial to achieve fall-out of 0% by returning zero documents in response to any query.
F-score / F-measure
The weighted harmonic mean of precision and recall, the traditional F-measure or balanced F-score is:
This is also known as the
The general formula for non-negative real
Two other commonly used F measures are the
The F-measure was derived by van Rijsbergen (1979) so that
F-measure can be a better single metric when compared to precision and recall; both precision and recall give different information that can complement each other when combined. If one of them excels more than the other, F-measure will reflect it.
Average precision
Precision and recall are single-value metrics based on the whole list of documents returned by the system. For systems that return a ranked sequence of documents, it is desirable to also consider the order in which the returned documents are presented. By computing a precision and recall at every position in the ranked sequence of documents, one can plot a precision-recall curve, plotting precision
That is the area under the precision-recall curve. This integral is in practice replaced with a finite sum over every position in the ranked sequence of documents:
where
This finite sum is equivalent to:
where
Some authors choose to interpolate the
where
An alternative is to derive an analytical
Precision at K
For modern (Web-scale) information retrieval, recall is no longer a meaningful metric, as many queries have thousands of relevant documents, and few users will be interested in reading all of them. Precision at k documents (P@k) is still a useful metric (e.g., P@10 or "Precision at 10" corresponds to the number of relevant results on the first search results page), but fails to take into account the positions of the relevant documents among the top k. Another shortcoming is that on a query with fewer relevant results than k, even a perfect system will have a score less than 1. It is easier to score manually since only the top k results need to be examined to determine if they are relevant or not.
R-Precision
R-precision requires knowing all documents that are relevant to a query. The number of relevant documents,
Precision is equal to recall at the R-th position.
Empirically, this measure is often highly correlated to mean average precision.
Mean average precision
Mean average precision for a set of queries is the mean of the average precision scores for each query.
where Q is the number of queries.
Discounted cumulative gain
DCG uses a graded relevance scale of documents from the result set to evaluate the usefulness, or gain, of a document based on its position in the result list. The premise of DCG is that highly relevant documents appearing lower in a search result list should be penalized as the graded relevance value is reduced logarithmically proportional to the position of the result.
The DCG accumulated at a particular rank position
Since result set may vary in size among different queries or systems, to compare performances the normalised version of DCG uses an ideal DCG. To this end, it sorts documents of a result list by relevance, producing an ideal DCG at position p (
The nDCG values for all queries can be averaged to obtain a measure of the average performance of a ranking algorithm. Note that in a perfect ranking algorithm, the
Other measures
Visualization
Visualizations of information retrieval performance include:
Top queries list
Top queries is noting the most common queries over a fixed amount of time. The top queries list assists in knowing the style of queries entered by users.
Queries per time
Measuring how many queries are performed on the search system per (month/day/hour/minute/sec) tracks the utilization of the search system. It can be used for diagnostics to indicate an unexpected spike in queries, or simply as a baseline when comparing with other metrics, like query latency. For example, a spike in query traffic, may be used to explain a spike in query latency.