#
Metric List

##
Automatic metrics for multiple-choice tasks

These metrics use log-likelihood of the different possible targets.

- `loglikelihood_acc`: Fraction of instances where the choice with the best logprob was correct - we recommend using a normalization by length
- `loglikelihood_f1`: Corpus level F1 score of the multichoice selection
- `mcc`: Matthew’s correlation coefficient (a measure of agreement between statistical distributions).
- `recall_at_k`: Fraction of instances where the choice with the k-st best logprob or better was correct
- `mrr`: Mean reciprocal rank, a measure of the quality of a ranking of choices ordered by correctness/relevance
- `target_perplexity`: Perplexity of the different choices available.
- `acc_golds_likelihood`: A bit different, it actually checks if the average logprob of a single target is above or below 0.5.
- `multi_f1_numeric`: Loglikelihood F1 score for multiple gold targets.

##
Automatic metrics for perplexity and language modeling

These metrics use log-likelihood of prompt.

- `word_perplexity`: Perplexity (log probability of the input) weighted by the number of words of the sequence.
- `byte_perplexity`: Perplexity (log probability of the input) weighted by the number of bytes of the sequence.
- `bits_per_byte`: Average number of bits per byte according to model probabilities.
- `log_prob`: Predicted output’s average log probability (input’s log prob for language modeling).

##
Automatic metrics for generative tasks

These metrics need the model to generate an output. They are therefore slower.

- Base:
  - `exact_match`: Fraction of instances where the prediction matches the gold. Several variations can be made through parametrization:
    - normalization on string pre-comparision on whitespace, articles, capitalization, …
    - comparing the full string, or only subsets (prefix, suffix, …)
  - `maj_at_k`: Model majority vote. Samples k generations from the model and assumes the most frequent is the actual prediction.
  - `f1_score`: Average F1 score in terms of word overlap between the model output and gold (normalisation optional).
  - `f1_score_macro`: Corpus level macro F1 score.
  - `f1_score_macro`: Corpus level micro F1 score.
- Summarization:
  - `rouge`: Average ROUGE score (Lin, 2004).
  - `rouge1`: Average ROUGE score (Lin, 2004) based on 1-gram overlap.
  - `rouge2`: Average ROUGE score (Lin, 2004) based on 2-gram overlap.
  - `rougeL`: Average ROUGE score (Lin, 2004) based on longest common subsequence overlap.
  - `rougeLsum`: Average ROUGE score (Lin, 2004) based on longest common subsequence overlap.
  - `rouge_t5` (BigBench): Corpus level ROUGE score for all available ROUGE metrics.
  - `faithfulness`: Faithfulness scores based on the SummaC method of Laban et al. (2022).
  - `extractiveness`: Reports, based on (Grusky et al., 2018):
    - `summarization_coverage`: Extent to which the model-generated summaries are extractive fragments from the source document,
    - `summarization_density`: Extent to which the model-generated summaries are extractive summaries based on the source document,
    - `summarization_compression`: Extent to which the model-generated summaries are compressed relative to the source document.
  - `bert_score`: Reports the average BERTScore precision, recall, and f1 score (Zhang et al., 2020) between model generation and gold summary.
- Translation:
  - `bleu`: Corpus level BLEU score (Papineni et al., 2002) - uses the sacrebleu implementation.
  - `bleu_1`: Average sample BLEU score (Papineni et al., 2002) based on 1-gram overlap - uses the nltk implementation.
  - `bleu_4`: Average sample BLEU score (Papineni et al., 2002) based on 4-gram overlap - uses the nltk implementation.
  - `chrf`: Character n-gram matches f-score.
  - `ter`: Translation edit/error rate.
- Copyright:
  - `copyright`: Reports:
    - `longest_common_prefix_length`: Average length of longest common prefix between model generation and reference,
    - `edit_distance`: Average Levenshtein edit distance between model generation and reference,
    - `edit_similarity`: Average Levenshtein edit similarity (normalized by the length of longer sequence) between model generation and reference.
- Math:
  - Both `exact_match` and `maj_at_k` can be used to evaluate mathematics tasks with math specific normalization to remove and filter latex.

##
LLM-as-Judge

- `llm_judge_gpt3p5`: Can be used for any generative task, the model will be scored by a GPT3.5 model using the OpenAI API.
- `llm_judge_llama_3_405b`: Can be used for any generative task, the model will be scored by a Llama 3.405B model using the HuggingFace API.
- `llm_judge_multi_turn_gpt3p5`: Can be used for any generative task, the model will be scored by a GPT3.5 model using the OpenAI API. It is used for multiturn tasks like mt-bench.
- `llm_judge_multi_turn_llama_3_405b`: Can be used for any generative task, the model will be scored by a Llama 3.405B model using the HuggingFace API. It is used for multiturn tasks like mt-bench.
- `tvd_mi`: Corpus-level LLM-as-a-judge metric that estimates a lower bound on total variation mutual information using paired responses. It assumes each example has two responses and a binary label indicating whether they are from the same underlying item (`1`) or from different items (`0`), and computes `TPR + TNR - 1` from the judge’s binary decisions.

Update on GitHub