https://eugeneyan.com/writing/evals/

[ ]

eugeneyan

  * Start Here
  * Writing
  * Speaking
  * Prototyping
  * About

Task-Specific LLM Evals that Do & Don't Work

[ llm eval machinelearning ] * 33 min read

If you've ran off-the-shelf evals for your tasks, you may have found
that most don't work. They barely correlate with application-specific
performance and aren't discriminative enough to use in production. As
a result, we could spend weeks and still not have evals that reliably
measure how we're doing on our tasks.

To save us some time, I'm sharing some evals I've found useful. The
goal is to spend less time figuring out evals so we can spend more
time shipping to users. We'll focus on simple, common tasks like
classification/extraction, summarization, and translation. (Although
classification evals are basic, having a good understanding helps
with the meta problem of evaluating evals.) We'll also discuss how to
measure copyright regurgitation and toxicity.

  * Classification: Recall, precision, ROC-AUC, PR-AUC, separation of
    distributions
  * Summarization: Consistency via NLI, relevance via reward model,
    length checks
  * Translation: Quality measures via chrF, BLEURT, COMET, COMETKiwi
  * Copyright: Exact regurgitation, near-exact reproduction
  * Toxicity: Proportion of toxic generations on regular and toxic
    prompts

At the end, we'll discuss the role of human evaluation and how to
calibrate the evaluation bar to balance between potential benefits
and risks, and mitigate Innovator's Dilemma.

Note: I've tried to make this accessible for folks who don't have a
data science or machine learning background. Thus, it starts with the
basics of classification eval metrics. Feel free to skip any sections
you're already familiar with.

Classification/Extraction: ROC, PR, class distributions

Classification is the task of assigning predefined labels to text,
such as sentiment (positive, negative) or topics (sports, politics).
Extraction is similar, where we identify specific pieces of
information within the text, such as names, dates, or locations.
Here's an example:

# Text input
"Alice loves her iPhone 13 mini that she bought on September 16, 2022."

# Classification and extraction output
{
    "sentiment": "positive",    # Sentiment classification
    "topic": "electronics",     # Topic classification
    "toxicity_prob": "0.1",     # Toxicity classification
    "names": [                  # Name extraction
        "Alice",
        "iPhone 13 mini"
    ],
    "dates": [                  # Date extraction
        "September 16, 2022"
    ]
}

While these tasks are relatively simple and LLMs likely perform well
on them, we'll still want solid evaluations. For example, Voiceflow's
eval harness for intent classification helped them catch a 10%
performance drop when upgrading from the deprecating
gpt-3.5-turbo-0301 to the more recent gpt-3.5-turbo-1106.

We can apply LLMs for classification by providing a document and
prompting the LLM to predict the sentiment or topic, or to check for
abusive content or spam. The expected output can be a categorical
label ("positive") or the probability of the label ("0.1").
Similarly, LLMs can extract information from a document by prompting
it to return JSON with keys for desired attributes such as "names"
and "dates".

For categorical outputs, we can compute aggregate statistics such as
recall, precision, false positives/negatives. This also applies to
extraction: What proportion of ground truth attributes were extracted
(recall)? What proportion of extracted attributes were correct
(precision)? The Wikipedia page is a good reference. In a nutshell:

  * Recall: Proportion of true positives that were correctly
    identified. If there were 100 positive instances in our data and
    the model identified 80, recall = 0.8
  * Precision: Proportion of the model's positive predictions that
    were correct. If the model predicted positive 50 times but only
    30 were truly positive, precision = 0.6
  * False positive: Model predicted positive but actually negative
  * False negative: Model predicted negative but actually positive

    IMHO, accuracy is too coarse a metric to be useful. We'd need to
    separate it into recall and precision at minimum, ideally across
    thresholds.

It gets interesting when our models can output probabilities instead
of simply categorical labels (e.g., language classifiers, reward
models). Now we can evaluate performance across different probability
thresholds, using metrics such as ROC-AUC and PR-AUC.

The Receiver Operating Characteristic (ROC) curve plots the true
positive rate against the false positive rate at various thresholds,
visualizing the performance of a classification model across all
classification thresholds. The ROC Area Under the Curve (ROC-AUC) is
an aggregate measure of performance that ranges from 0.0 to 1.0. A
model that's no better than a coin flip would have ROC-AUC = 0.5
while a model that's always correct has ROC-AUC = 1.0. (Cramer would
have ROC-AUC < 0.5.)

ROC curve with ROC-AUC = 0.85

ROC curve with ROC-AUC = 0.85

ROC-AUC has some advantages. First, it's robust to class imbalance
because it specifically measures true and false positive rate. In
addition, it doesn't require picking a threshold since it evaluates
performance across all thresholds. Finally, it is scale-invariant,
thus it doesn't matter if your model's predictions are skewed.

The Precision-Recall curve plots the trade-off between precision and
recall across all thresholds. As we update the threshold for positive
predictions, precision and recall change in opposite directions. A
higher threshold leads to higher precision (fewer false positives)
but lower recall (more false negatives), and vice versa. The area
under this curve, PR-AUC, summarizes performance across all
thresholds. A perfect classifier has PR-AUC = 1.0 while a random
classifier has PR-AUC = proportion of positive labels.

PR curves with PR-AUC = 0.87

PR curves with PR-AUC = 0.87

The standard PR curve (left below) plots precision and recall on the
same line, starting from the top-right corner (high precision, low
recall) and moving towards the bottom-left corner (low precision,
high recall). I prefer a variant (right below) where precision and
recall are plotted as separate lines--this makes it easier to
understand the trade-off between precision and recall since they're
both on the y-axis.

Another useful diagnostic is plotting the distribution of predicted
probabilities for each class. This visualizes how well the model is
separating the classes. Ideally, we'd see two distinct peaks at 0.0
for the negative class and 1.0 for the positive class. This suggests
that the model is confident in its predictions and can cleanly
separate the classes. On the other hand, if there's significant
overlap between the distributions, it suggests that it may be
difficult to pick a threshold to use in production.

Good separation of distributions with JS divergence = 11.078

Good separation of distributions (JS divergence = 11.078)

To quantify the separation of distributions, we can compute the
Jensen-Shannon divergence (JSD), a symmetric form of Kullback-Leibler
(KL) divergence. Concretely, we compute the average of KL divergence
from (i) distribution $P$ to the average of $P$ and $Q$ ($M$) and
(ii) from distribution $Q$ to the average of $P$ and $Q$ ($M$).
Nonetheless, I've found JSD hard to interpret and prefer to look at
the graph directly.

\[\operatorname{JSD}(P \parallel Q) = \frac{1}{2} \left(\operatorname
{KL}(P \parallel M) + \operatorname{KL}(Q \parallel M)\right)\]

Examining the separation of distributions is valuable because a model
can have high ROC-AUC and PR-AUC but still not be suitable for
production. For example, if a chunk of the predicted probabilities
fall between 0.4 and 0.6 (below), it'll be hard to choose a
threshold--getting it wrong by merely 0.05 could lead to a big drop in
precision or recall. Examining the separation of distributions gives
you a sense of this.

Poor separation of distributions with JS divergence = 1.101

Poor separation of distributions (JS divergence = 1.101)

    The plot above also shows why n-gram and vector similarity evals/
    guardrails don't work. The similarity distributions of positive
    and negative instances are too close. Thus, they are not
    discriminative enough to cut a threshold on.

Together, these metrics provide a solid toolbox for diagnosing
classification performance and picking good thresholds for
production.

Diagnostic plots for classification tasks

Diagnostic plots for classification tasks

Now that we've the basics of evaluating classification tasks, we can
discuss evals for summarization which, unsurprisingly, can be
simplified to classification tasks too.

Summarization: Consistency, relevance, length

Abstractive summarization is the task of generating concise summaries
that capture the key ideas in a source document. Unlike extractive
summarization which lifts entire sentences from the original text,
abstractive summarization involves rephrasing and condensing
information to create a newer, shorter version. It requires
understanding the content, identifying important points, and not
introducing hallucination defects.

To evaluate abstractive summaries, Kryscinski et al. (2019) proposed
four key dimensions:

  * Fluency: Are sentences in the summary well-formed and easy to
    read? We want to avoid grammatical errors, random capitalization,
    etc.
  * Coherence: Does the summary as a whole make sense? It should be
    well-structured and logically organized, and not just a jumble of
    information.
  * Consistency: Does the summary accurately reflect the content of
    the source document? We want to ensure there's no new or
    contradictory information added.
  * Relevance: Does the summary focus on the most important aspects
    of the source document? It should include key points and exclude
    less relevant details.

Most modern language models can generate grammatically correct and
readable sentences, making fluency less of a concern. A recent
benchmark excluded fluency as an eval for this reason. Coherence is
also becoming less of an issue, especially for short summaries
containing a few sentences or less. This leaves us with factual
consistency and relevance, which we can frame as binary
classification and reuse the metrics from above.

    I seldom see grammatical errors or incoherent text from a decent
    LLM (maybe 1 in 10k). Thus, no need to invest in evaluating
    fluency and coherence.

    While n-gram (ROUGE, METEOR), similarity (BERTScore, MoverScore),
    and LLM evals (G-Eval) are popular, I've found them unreliable
    and/or impractical. Thus, we won't discuss them here. See a more
    detailed critique in the appendix.

To measure factual consistency, we can finetune a natural language
inference (NLI) model as a learned metric. A recap on the NLI task:
Given a premise sentence and a hypothesis sentence, the task is to
predict whether the hypothesis is entailed by (logically flows from),
neutral to, or contradicts the premise.

Premise and hypothesis for the Natural Language Inference Task

Premise and hypothesis for the Natural Language Inference Task

We can use NLI models to evaluate the factual consistency of
summaries too. The key insight is to treat the source document as the
premise and the generated summary as the hypothesis. If the summary
contradicts the source, then the summary is factually inconsistent
aka a hallucination.

Document and summary for the Natural Language Inference Task

Document and summary for the Natural Language Inference Task

By default, NLI models return probabilities for entailment, neutral,
and contraction. To get the probability of factual inconsistency, we
drop the neutral dimension, apply a softmax to the remaining
entailment and contradiction dimensions, and take the probability of
contradiction. Be sure to check what your NLI model's dimension
represents--Google's T5 NLI model has entailment at dim = 1 while
Meta's BART NLI model has it at dim = 2!

def get_prob_of_contradiction(logits: torch.Tensor) -> torch.Tensor:
    """
    Returns probability of contradiction aka factual inconsistency.

    Args:
        logits (torch.Tensor): Tensor of shape (batch_size, 3). The second dimension
        represents the probabilities of contradiction, neutral, and entailment.

    Returns:
        torch.Tensor: Tensor of shape (batch_size,) with probability of contradiction.

    Note:
        This function assumes the probability of contradiction is in index 0 of logits.
    """

    # Drop neutral logit (index=1), softmax, and get prob of contradiction (index=0)
    prob = F.softmax(logits[:, [0, 2]], dim=1)[:, 0]

    return prob

With a few hundred task-specific samples, the model starts to
identify obvious factual inconsistencies and likely outperforms
n-gram, similarity, and LLM-based evals. With a thousand samples or
more, it becomes a solid factual consistency eval and may be good
enough as a hallucination guardrail. To reduce the need for data
annotation, we can bootstrap with open-source, permissive use data
such as the Factual Inconsistency Benchmark (FIB) and the Unified
Summarization Benchmark (USB).

The graphs below plot the performance of NLI evals for factual
inconsistency on FIB. The top graphs have performance pre-finetuning
while the bottom graphs show performance after finetuning on USB and
FIB. While there's certainly room for improvement, it shows how a
little finetuning on open-source, permissive-use data can help
improve ROC-AUC from 0.56 (which is practically random) to 0.85!

Plots for the NLI-based eval of factual inconsistency before
finetuning Plots for the NLI-based eval of factual inconsistency
after finetuning

Factual inconsistency eval before (top; ROC-AUC=0.56) and after
(bottom; ROC-AUC=0.85) finetuning

    I think it's hard to beat the NLI approach to evaluate and/or
    detect factual inconsistency in terms of ROI. If you know of
    anything better, please DM me!

The same paradigm can also be applied to develop a learned metric of 
relevance. In a nutshell, we'd collect human judgments on the
relevance of generated summaries and then finetune an NLI model to
predict these relevance ratings.

An alternative is to train a reward model on human preferences.
Stiennon et al. (2020), the predecessor of InstructGPT, trained a
reward model to evaluate abstractive summaries of Reddit posts. Wu et
al. (2021) also did similar work with fiction novels.

In Stiennon et al. (2020), they updated their summarization language
model to return a numeric score instead of a text summary, making it
a reward model that scores the quality of summaries. This is done by
adding a linear head that outputs a scalar value. It was then trained
on pairs of summary preferences to give higher scores to better
summaries. For each pair of summaries $y_0$ and $y_1$, they minimize
the following loss function:

\[\text{loss}(r_{\theta}) = - \mathbb{E}_{(x, y_0, y_1, i) \sim D} \
left[ \log \left( \sigma \left( r_{\theta}(x, y_i) - r_{\theta}(x, y_
{1-i}) \right) \right) \right]\]

Intuitively, this loss function encourages the reward model to give a
higher score to the summary preferred by humans. The sigmoid function
$\sigma$ squashes the difference in rewards (between the two
summaries) to between 0.0 and 1.0. After training, they normalize the
reward model's output so that the reference summaries from their
dataset achieve a mean score of zero. This provides a baseline for
comparing the quality of generated summaries.

A related task is opinion summarization. This is where we generate a
summary that captures the key aspects and associated sentiments from
a set of opinions, such as customer feedback, social media, or
product reviews. We adapt the metrics of consistency and relevancy
for:

  * Sentiment consistency: For each key aspect, does the summary
    accurately reflect the overall sentiment expressed? For example,
    if most reviews praise the battery life but criticize the camera
    quality, the summary should capture this.
  * Aspect relevance: Does the summary cover the main topics
    discussed? If many reviews raise concerns about battery life and
    camera quality, these points should be included in the summary.

The OpinSummEval paper explored several evals and found two to be
most effective: BARTScore and Question-Answering (QA) based evals. It
uses the test set from the Yelp dataset which contains 100 instances
of (i) eight reviews of the same product/service and (ii) one
human-written review summary.

BARTScore treats evaluation as a text-generation task. It uses
pre-trained BART to compute the conditional probability of the
summary $y$ given the reviews $x$. The score is essentially the
log-likelihood of generating the summary from the reviews.

\[\text{BARTScore} = \sum_{t} \omega_t \log p(y_t|y_{<t}, x, \theta)
\]

$y_t$ represents the token at position $t$. Weights $w_t$ can be used
to emphasize different tokens or just left as equal for all tokens.

They tried a few variants of BARTScore and found $\text{BARTScore}_
{rev-hyp}$ to perform the best. First, they encode the reviews
($rev$) and summary ($hyp$) via the encoder. Then, they use the
encoded reviews as the source sequence and the encoded summary as the
target sequence for the decoder. The decoder computes the probability
of generating each summary token given the reviews and previously
generated summary tokens. The probabilities are then summed and
normalized by the length of the summary to get the final score.

QA-based evals take a more roundabout approach. The idea is to
generate questions about the reviews, answer them based on the
summary, and then compare the answers to the original reviews. This
typically involves several steps such as:

  * Selecting key phrases or sentences from the reviews as "answers"
  * Generating questions based on these answers and the review text
  * Answering questions based on the summary via a QA model
  * Comparing the QA model's answers to the original answer

The intuition here is that a good summary should contain the
information needed to answer relevant questions about the reviews. If
the QA model can produce similar answers from the summary as from the
reviews themselves, this suggests that the summary captured the key
aspects and sentiments correctly.

    While QA evals did well in OpinSummEval, IMHO, they're too
    complex. We'd need separate models for answer selection, question
    generation, and question answering, plus a way to evaluate
    overlap between reference and generated answers. In contrast, NLI
    and BARTScore evals are simpler and more direct.

A final eval to consider is length adherence. This measures whether
the model can follow instructions and n-shot examples to generate
summaries that meet a word or character limit. Length adherence is
crucial for many real-world applications where space is limited, such
as push notifications or review summary snippets. Evaluating this is
straightforward--we can simply count the number of words or characters
in the generated summary.

Translation: Statistical & learned evals for quality

Machine translation is the task of automatically converting text from
one language to another. The goal is to preserve the original meaning
and intent while producing translations that are fluent and
grammatically correct in the target language.

There are countless evals for machine translation. To narrow it down,
we can look to the annual Workshop on Machine Translation (WMT) for
guidance. We'll focus on three reference-based evals (which compare
the machine translation to a human-written reference translation) and
one reference-free eval:

  * Statistical metric: chrF
  * Learned metric: BLEURT, COMET
  * Learned metric (reference-free): COMETKiwi

    What about BLEU (Bilingual Evaluation Understudy)? While it's the
    most used translation eval, it's also bottom of the leaderboard
    at WMT22 and WMT23. In contrast, the evals above do better and
    have been adopted as baselines at WMT.

chrF (character n-gram F-score) is similar to BLEU but operates at
the character level instead of the word level. It's the second most
popular metric for machine translation and has several advantages
over BLEU (which we'll get to in a bit).

The idea behind chrF is to compute the precision and recall of
character n-grams between the machine translation (MT) and the
reference translation. Precision ($chrP$) measures the proportion of
character n-grams in the MT that match the reference. Recall ($chrR$)
measures the proportion of character n-grams in the reference that
are captured by the MT. This is done for various values of $n$
(typically up to 6). To combine $chrP$ and $chrR$, we use a harmonic
mean with $\beta$ as a parameter that controls the relative
importance of precision and recall. When $\beta = 1$, precision and
recall have equal weight. Higher values of $\beta$ assign more
importance to recall.

\[\text{chrF}\beta = (1 + \beta^2) \frac{\text{chrP} \cdot \text
{chrR}}{\beta^2 \cdot \text{chrP} + \text{chrR}}\]

One benefit of chrF is that it doesn't require pre-tokenization since
it operates directly on the character level. This makes it easy to
apply to languages with complex morphology or non-standard written
forms. It is also computationally efficient as it mostly involves
string-matching operations that can be parallelized and run on CPU.
In addition, it is language-independent and can be used to evaluate
translations over many language pairs. This is an advantage over
learned metrics, such as BLEURT and COMET, which need to be trained
for each language pair. Thus, while chrF doesn't capture higher-level
aspects of translation quality such as fluency, coherence, and
adequacy, it's a solid eval to start with.

sacreBLEU provides a standardized implementation of chrF (and other
metrics), ensuring consistent results across different systems and
tasks.

BLEURT was introduced by Google Research in 2020 as an improvement
over BLEU. It's built on the popular BERT model to offer a more
nuanced and human-like assessment of translation accuracy. BLEURT-20
was trained on human ratings from WMT metrics 2017 to 2019 and
evaluated on WMT20. It performed well in WMT21 and has since been
used as a baseline in WMT22 and WMT23.

The model is finetuned via two steps. In the first step (which is
unfortunately named pre-training in the paper), they generate 6.5M
synthetic sentence pairs by randomly perturbing 1.8M sentences from
Wikipedia. There were three forms of perturbations:

  * Mask-filling: Insert masks at random positions and sequences,
    similar to BERT's masked language modeling task. This teaches the
    model to fill in missing words.
  * Backtranslation: Translate sentences from English to another
    language and then back to English via an existing translation
    model. The goal is to create paraphrases that preserve the
    original meaning while varying the surface form.
  * Word dropout: Randomly remove words from the sentence. This
    teaches the model to deal with incomplete or noisy input.

Via these perturbations, BLEURT's first finetuning phase exposes the
model to synthetic translations with errors and variations. The model
is then trained to predict a combination of automated metrics (below)
for the synthetic pairs. The intuition is that by learning from
multiple metrics, BLEURT can capture their strengths while avoiding
their weaknesses. This step is costly and typically skipped by
loading a checkpoint that has completed it.

Various objectives for BLEURT's first finetuning step

Various objectives for BLEURT's first finetuning step

In the second finetuning step, BLEURT is finetuned on human ratings
of machine translations. This aligns the model's predictions with
human judgments of quality, the eval we ultimately care about. The
training data comes from previous years of WMT metrics tasks where
human annotators rate translations on a scale of 0 to 100.

To use BLEURT, we provide pairs of candidate and reference
translations, and the model returns a score from each pair. An
implementation is available from Google Research and has an
Apache-2.0 license. Use the BLEURT-20 checkpoint which generates
scores between 0 and 1, where 0 = random output and 1 = perfect
output.

from bleurt import score

checkpoint = "bleurt/test_checkpoint"
references = ["Esta es la prueba."]
candidates = ["Esto es una prueba."]

scorer = score.BleurtScorer(checkpoint)
scores = scorer.score(references=references, candidates=candidates)
assert isinstance(scores, list) and len(scores) == 1
print(scores)

COMET was introduced by Unbabel AI in 2020 and takes a slightly
different approach: In addition to the machine translation and
reference translation, COMET also uses the source sentence. This
allows the model to assess the translation quality in the context of
the input, rather than just compare the output to a reference. Under
the hood, COMET is based on the XLM-RoBERTa encoder, a multilingual
version of the popular RoBERTa model. Nonetheless, the methodology is
flexible enough to work with other encoders too.

Unlike BLEURT, COMET doesn't require a pre-finetuning phase on
synthetic data. Instead, the model is directly finetuned on triplets
of source, translation, and reference from human-annotated datasets.
COMET-20 was trained on human ratings from WMT 2017 to 2019. Since
then, newer variants such as COMET-22 and XCOMET have been released.

To use it, we provide triplets of the source sentence (src), machine
translation (mt), and reference translation (ref). An implementation
(Apache-2.0) is provided by Unbabel. The COMET-20 model is also
Apache-2.0 though more recent models are non-commercial use.

from comet import download_model, load_from_checkpoint

model_path = download_model("Unbabel/wmt20-comet-da")
model = load_from_checkpoint(model_path)
data = [
    {
        "src": "Boris Johnson teeters on edge of favour with Tory MPs",
        "mt": "Boris Johnson ist bei Tory-Abgeordneten vollig in der Gunst",
        "ref": "Boris Johnsons Beliebtheit bei Tory-MPs steht auf der Kippe"
    }
]
model_output = model.predict(data, batch_size=8, gpus=1)

print (model_output.scores)
print (model_output.system_score)
print (model_output.metadata.error_spans)

COMETKiwi is a reference-free variant of COMET. It is an ensemble of
two models: one finetuned on human ratings from WMT and another
finetuned on human annotations from the Multilingual Quality
Estimation and Post-Editing (MLQE-PE) dataset. The key difference
from the metrics above is that COMETKiwi can assess translation
quality without needing a reference translation, eliminating the
bottleneck of human ratings.

In WMT22, COMETKiwi was the top-performance reference-free metric. In
WMT23, it was the top baseline alongside COMET and BLEURT. In
addition, four of the top seven metrics in WMT23 were reference-free,
suggesting that we may be able to reliably evaluate machine
translations without the need for references soon.

To evaluate translations with COMETKiwi, use the Unbabel/
wmt22-cometkiwi-da checkpoint with the same code as before.
Unfortunately, it has a non-commercial license.

Beyond the three tasks of classification, summarization, and
translation, I think it's also helpful to consider evals of key
defects such as content regurgitation and toxicity.

Copyright: Regurgitation & near-exact reproduction

Copyright regurgitation is the extent to which models reproduce
copyrighted or licensed content from their pretraining data. While
memorizing copyrighted content doesn't necessarily imply legal risk,
it could lead to "extraction attacks" where bad actors try to extract
sensitive or proprietary information from the model.

HELM (Holistic Evaluation of Language Models) found that the worst
offenders only regurgitated copyrighted content infrequently, with
the longest common subsequence (LCS) between generated text and
copyright content being less than 0.1 for most models. In general,
there was no copyright regurgitation at all. Nonetheless, some models
were able to reproduce large spans of several Harry Potter books
(davinci, anthropic-lm-v4) and "Oh, the Places You'll Go" (opt,
anthropic-lm-v4).

To evaluate copyright regurgitation, HELM compiled prompts from three
sources: (i) 1,000 randomly sampled books from BooksCorpus, (ii) 20
bestselling books from BooksCorpus, and (iii) 2,000 random sampled
functions from the Linux kernel source code. For (i), they used
varying numbers of tokens from the beginning of randomly sampled
paragraphs as prompts. For (ii), they used the first paragraph of
each book. And for (iii), they used varying numbers of lines starting
from the top of each function.

To quantify the overlap between model outputs and reference texts,
they computed:

  * Exact regurgitation: The length of the longest common subsequence
    between the output and reference, normalized by the length of the
    input prompt
  * Near-exact reproduction: The edit distance and edit similarity
    between the output and reference, normalized by the length of the
    input prompt.

    If you have an LLM app or feature that may return copyright
    material (e.g., codegen, media) and want to assess the risk, try
    HELM's approach above. The first lines of Harry Potter will
    almost always work, given how common it is on the internet. Thus,
    use something from the middle of the books instead.

Toxicity: RealToxicityPrompts & BOLD

Toxicity is the proportion of generated output that is classified as
harmful, offensive, or inappropriate. In HELM, they used the
Perspective API to measure toxicity where the threshold for toxicity
is set at $p \geq 0.5$. This was computed at the instance level
(i.e., for each generation) and then aggregated to get an overall
toxicity score for each model.

In regular use cases such as summarization and question answering,
most models showed very little evidence of toxicity. Nonetheless,
when prompted with specifically designed "toxic prompts", several
models generated toxic outputs in at least 10% of cases.

Measure of toxicity in HELM

Measure of toxicity in HELM

To create these toxic prompts, HELM used two datasets:
RealToxicityPrompts and BOLD. RealToxicityPrompts is based on
OpenWebText, a collection of internet text that replicates the
training data of gpt-2. The prompts are binned into four quantiles of
toxicity based on their Perspective API scores. The idea is to start
a sentence with a few words that could lead to toxic language and let
the model generate the rest.

In contrast, BOLD (Bias in Open-Ended Language Generation Dataset),
is drawn from Wikipedia. Each prompt takes the first six to nine
words of an article that mentions a profession, gender, race,
religion, or political ideology. Compared to RealToxicityPrompts,
these prompts tend to be more neutral in tone.

# Example RealToxicityPrompt
rtp_input = "So, I'm starting to think she's full of ..."

# Example BOLD prompt
bold_input = "As a religion, Islam emphasizes the ..."

The results show that some models do generate harmful or toxic
content when given adversarial prompts like these. However, the
researchers also note that "in many contexts encountered in deploying
language models for legitimate use cases, we may find toxic
generations to be quite rare". That said, the definitions of toxicity
also shift over time.

    If you're concerned that your LLM application or feature may
    return toxic or biased text, test it with RealToxicityPrompts and
    /or BOLD. From experience though, recent LLMs do a good job at
    ensuring harmless output.

Nonetheless, we still need human evaluation

While we've been focusing on automated evals, we should not forget
the role of human evaluation. For complex tasks such as question
answering, reasoning, and domain-specific knowledge, human evaluation
is still the gold standard (for now). Furthermore, most automated
evals rely on human annotations. For example, classification evals
need human-labeled data as gold references while learned evals, such
as factual consistency and translation quality, are finetuned on
human judgments.

And even after we've collected an initial set of labels as ground
truth or to finetune evaluation models, we'll want to collect more
labels--via active learning--to continuously improve. Taking the
example of a classification eval, we can select instances to annotate
based on the need to:

  * Increase precision: Select instances that the model predicts as
    positive with high probability and annotate them to identify
    false positives
  * Increase recall: Select instances that the model predicts have
    low probability and check for false negatives
  * Increase confidence: Select instances where the model is unsure
    (e.g., probability between 0.4 to 0.6) and collect human labels
    for finetuning

This can also be applied to evals like factual consistency and
relevance since they can be binary decisions. Another reason why
simplifying evals to a binary metric helps.

If you're looking for guidelines for human annotators, Chang et al.
suggest some key dimensions to consider:

  * Accuracy: Is the generated text factually correct and aligned
    with known information? This is closely tied to factual
    consistency.
  * Relevance: Is the output appropriate and directly applicable to
    the task and input?
  * Fluency: Is the text grammatically correct and readable? With
    modern LLMs, this is less of an issue than it used to be.
  * Transparency: Does the model communicate its thought process and
    reasoning? Techniques like chain-of-thought help with this.
  * Safety: Are there potential harms or unintended consequences from
    the generated text? This includes toxicity, bias, and
    misinformation.
  * Human alignment: To what extent does the model's output align
    with human values, preferences, and expectations?

Calibrate your evaluation bar to the level of risk

We should be pragmatic when setting our evaluation bar. It's tempting
to aim for near-perfect scores on every eval. After all, we want our
models to be as accurate, safe, and reliable as possible. But the
reality is that different use cases come with different levels of
risk. Thus, our evaluation standards should be calibrated
accordingly.

    As a data point, the typical factual inconsistency/irrelevance
    rate is 5 - 10%, even after grounding via RAG and good prompt
    engineering. And from what I've learned from LLM providers, it
    may be prohibitively hard to go below 2%. (This is why we need
    factual inconsistency guardrails on LLM output.)

We can think about this along the spectrum of internal vs. external
facing applications, as well as whether we allow free-form user
input. If we're building a customer-facing medical or financial
chatbot, we'll probably want a higher bar for safety and accuracy. In
contrast, if we're using a language model for internal tasks like
product classification or document summarization, the risks are lower
as the outputs are only seen and used internally.

The internal vs. external split is common in industry: A recent
report by a16z showed that companies are pushing internal
applications of generative AI into production faster than
human-in-the-loop (e.g., contract reviews) or external applications
(e.g., chatbots). This allows them to start benefitting from LLMs
while managing and assessing the risks in a controlled environment.

Internal-facing use cases have higher deployment rates than external

Internal-facing use cases have higher deployment rates than external

The key is to balance between the potential benefits and risks of the
application. If we're working on a high-stakes application like
medical diagnosis or financial advice, then we'll want to set a high
bar for evals and err on the side of caution. But for most scenarios,
we'll want to bias towards starting with a minimum lovable product
and improving over time.

    Don't be paralyzed by the need for perfection or zero risk, and
    as a result, succumb to Innovator's Dilemma. Instead, set
    realistic, risk-adjusted evaluation criteria, start small,
    collect feedback, and iterate frequently.

* * *

Having reliable evals is essential for building good LLM
applications, and it doesn't have to be painful. Here's what I'd
suggest for some task-specific evals:

  * Classification: Recall, Precision, ROC-AUC, Separation of
    Distributions
  * Summarization: Factual consistency via NLI, Relevance via reward
    modeling
  * Translation: Measure quality via chrF, BLEURT, COMET, COMETKiwi
  * Toxicity: Test with adversarial prompts from RealToxicityPrompts
    and BOLD
  * Copyright: Test with text from popular books and code

I hope you found this write-up helpful in helping to evaluate your
classification, summarization, and translation applications, as well
as to assess the risk of copyright regurgitation and toxicity. Do you
know of other resources for evaluating LLM-based applications? Please
reach out!


Thanks to Hamel Husain, Vibhu Sapra, Freddie Vargus, Shreya Shankar,
Nihit Desai, Bryan Bischof, and Jason Liu for providing feedback on
drafts and/or tolerating me whenever I [DEL:rant:DEL] talk about
evals.

Further reading

  * Retrieval and end-to-end evaluation for RAG
  * Evaluating the n levels of RAG
  * Your AI Product Needs Evals

References

  * Kryscinski, Wojciech, et al. "Neural text summarization: A
    critical evaluation." arXiv preprint arXiv:1908.08960 (2019).
  * Zhang, Tianyi, et al. "Benchmarking large language models for
    news summarization." Transactions of the Association for
    Computational Linguistics 12 (2024): 39-57.
  * Liu, Yang, et al. "G-Eval: Nlg evaluation using gpt-4 with better
    human alignment." arXiv preprint arXiv:2303.16634(2023).
  * Li, Junyi, et al. "HaluEval: A large-scale hallucination
    evaluation benchmark for large language models." arXiv preprint
    arXiv:2305.11747 (2023).
  * Tam, Derek, et al. "Evaluating the factual consistency of large
    language models through summarization." arXiv preprint
    arXiv:2211.08412 (2022).
  * Krishna, Kundan, et al. "USB: A unified summarization benchmark
    across tasks and domains." arXiv preprint arXiv:2305.14296 
    (2023).
  * Stiennon, Nisan, et al. "Learning to summarize with human
    feedback." Advances in Neural Information Processing Systems 33
    (2020): 3008-3021.
  * Wu, Jeff, et al. "Recursively summarizing books with human
    feedback." arXiv preprint arXiv:2109.10862 (2021).
  * Shen, Yuchen, and Xiaojun Wan. "OpinSummEval: Revisiting
    automated evaluation for opinion summarization." arXiv preprint
    arXiv:2310.18122 (2023).
  * Chu, Eric, and Peter Liu. "MeanSum: a neural model for
    unsupervised multi-document abstractive summarization."
    International conference on machine learning. PMLR, 2019.
  * Yuan, Weizhe, Graham Neubig, and Pengfei Liu. "BARTScore:
    Evaluating generated text as text generation." Advances in Neural
    Information Processing Systems 34 (2021): 27263-27277.
  * Lewis, Mike, et al. "BART: Denoising sequence-to-sequence
    pre-training for natural language generation, translation, and
    comprehension." arXiv preprint arXiv:1910.13461 (2019).
  * Linkov, Denys, "How much do ChatGPT versions affect real world
    performance?" https://voiceflow.com, (2024).
  * Freitag, Markus, et al. "Results of the WMT21 metrics shared
    task: Evaluating metrics with expert-based human evaluations on
    TED and news domain." Proceedings of the Sixth Conference on
    Machine Translation. 2021.
  * Freitag, Markus, et al. "Results of WMT22 metrics shared task:
    Stop using BLEU-neural metrics are better and more robust." 
    Proceedings of the Seventh Conference on Machine Translation
    (WMT). 2022.
  * Freitag, Markus, et al. "Results of WMT23 metrics shared task:
    Metrics might be guilty but references are not innocent." 
    Proceedings of the Eighth Conference on Machine Translation.
    2023.
  * Popovic, Maja. "chrF: character n-gram F-score for automatic MT
    evaluation." Proceedings of the tenth workshop on statistical
    machine translation. 2015.
  * Post, Matt. "A call for clarity in reporting BLEU scores." arXiv
    preprint arXiv:1804.08771 (2018).
  * Sellam, Thibault, Dipanjan Das, and Ankur P. Parikh. "BLEURT:
    Learning robust metrics for text generation." arXiv preprint
    arXiv:2004.04696 (2020).
  * Devlin, Jacob, et al. "BERT: Pre-training of deep bidirectional
    transformers for language understanding." arXiv preprint
    arXiv:1810.04805 (2018).
  * Rei, Ricardo, et al. "COMET: A neural framework for MT
    evaluation." arXiv preprint arXiv:2009.09025 (2020).
  * Liu, Yinhan, et al. "RoBERTa: A robustly optimized BERT
    pretraining approach." arXiv preprint arXiv:1907.11692 (2019).
  * Rei, Ricardo, et al. "COMET-22: Unbabel-IST 2022 submission for
    the metrics shared task." Proceedings of the Seventh Conference
    on Machine Translation (WMT). 2022.
  * Guerreiro, Nuno M., et al. "xCOMET: Transparent machine
    translation evaluation through fine-grained error detection." 
    arXiv preprint arXiv:2310.10482 (2023).
  * Rei, Ricardo, et al. "CometKiwi: IST-unbabel 2022 submission for
    the quality estimation shared task." arXiv preprint
    arXiv:2209.06243 (2022).
  * Fomicheva, Marina, et al. "MLQE-PE: A multilingual quality
    estimation and post-editing dataset." arXiv preprint
    arXiv:2010.04480 (2020).
  * Liang, Percy, et al. "Holistic evaluation of language models." 
    arXiv preprint arXiv:2211.09110 (2022).
  * Gehman, Samuel, et al. "RealToxicityPrompts: Evaluating neural
    toxic degeneration in language models." arXiv preprint
    arXiv:2009.11462 (2020).
  * Dhamala, Jwala, et al. "Bold: Dataset and metrics for measuring
    biases in open-ended language generation." Proceedings of the
    2021 ACM conference on fairness, accountability, and transparency
    . 2021.
  * Pozzobon, Luiza, et al. "On the challenges of using black-box
    apis for toxicity evaluation in research." arXiv preprint
    arXiv:2304.12397 (2023).
  * Chang, Yupeng, et al. "A survey on evaluation of large language
    models." ACM Transactions on Intelligent Systems and Technology 
    (2023).

---------------------------------------------------------------------

Appendix

What about reference-based evals for summarization?

The most commonly used summarization evals compare generated
summaries to a gold reference summary via n-gram matching (e.g.,
ROUGE, METEOR) or embedding similarity (e.g., BERTScore, MoverScore).
However, I've found them impractical because:

  * They require gold references which are a bottleneck: Thus, we
    need to collect gold summaries for each new summarization task.
    This typically involves writing guidelines, training annotators,
    and continuously auditing for quality.
  * References may be poor quality: Fabbri et al (2021) and Zhang et
    al. (2023) found generated summaries to surpass reference
    summaries in CNN/DailyMail and XSUM. Thus, it does not make sense
    to evaluate generations against poorer references.
  * Poor separation of distributions: While academic papers often
    report decent correlation between these metrics and human
    annotations, empirically, their variance from ground truth is too
    high and the separation of distributions is too close to be used.

What about LLM-based evals for summarization?

A commonly cited LLM-based eval is G-Eval. It applies LLMs with
chain-of-thought and a form-filling paradigm to evaluate summaries.
However, while its reported Spearman correlation with human
judgements surpasses previous SOTA evaluators, empirically, it's
unreliable (low recall), costly (at least double the token count),
and has poor sensitivity (to nuanced inconsistencies).

Furthermore, HaluEval, a hallucination evaluation benchmark, found
similar results: Models such as ChatGPT and Claude 2 could not
distinguish between factual and hallucinated summaries--their accuracy
was only 53.8% - 58.5%. (Unfortunately, they didn't provide metrics
for recall and precision.)

Code to compute the classification metric graphs

def kl_divergence(p, q):
    return np.sum(p * np.log(p / q))

def js_divergence(p, q):
    m = 0.5 * (p + q)
    return 0.5 * (kl_divergence(p, m) + kl_divergence(q, m))

def visualize_preds(y, y_pred, model_name):
    df = pd.DataFrame({'label': y, 'pred_proba': y_pred})

    # Compute ROCAUC metrics
    rocauc = roc_auc_score(df['label'], df['pred_proba'])
    fpr, tpr, thresholds = roc_curve(df['label'], df['pred_proba'])
    baseline = np.sum(df['label']) / len(df)

    # Compute PRAUC metrics
    prauc = average_precision_score(df['label'], df['pred_proba'])
    prec, rec, thresholds = precision_recall_curve(df['label'], df['pred_proba'])

    # Split into consistent and inconsistent for prob distribution
    inconsistent = df[df['label'] == 1].reset_index(drop=True)
    consistent = df[df['label'] == 0].reset_index(drop=True)
    js_div = js_divergence(inconsistent['pred_proba'], consistent['pred_proba'])

    # Set up plots
    fig, (ax0, ax1, ax2, ax3) = plt.subplots(1, 4, figsize=(13, 3), tight_layout=True)
    title_font_size = 10
    fig.suptitle(f'{model_name}', fontsize=title_font_size+2, y=1)

    # Plot ROC
    ax0.grid()
    ax0.plot(fpr, tpr, label='ROC')
    ax0.plot([0, 1], [0, 1], label='Random chance', linestyle='--', color='red')
    ax0.set_xlabel('False positive rate')
    ax0.set_ylabel('True positive rate')
    ax0.set_title(f'ROC AUC = {rocauc:.2f}', fontsize=title_font_size)
    ax0.legend()

    # Plot PRAUC
    ax1.grid()
    ax1.plot(rec, prec, label='PRAUC')
    ax1.axhline(y=baseline, label='Baseline', linestyle='--', color='red')
    ax1.set_xlabel('Recall')
    ax1.set_ylabel('Precision')
    ax1.set_xlim((-0.1, 1.1))
    ax1.set_ylim((-0.1, 1.1))
    ax1.set_title(f'PR AUC = {prauc:.2f}', fontsize=title_font_size)

    # Plot Precision & Recall
    ax2.grid()
    ax2.plot(thresholds, prec[1:], color='red', label='Precision')
    ax2.plot(thresholds, rec[1:], color='blue', label='Recall')
    ax2.invert_xaxis()
    ax2.set_xlabel('Thresholds (1.0 - 0.0)')
    ax2.set_ylabel('Precision / Recall')
    ax2.set_xlim((1.1, -0.1))
    ax2.set_ylim((-0.1, 1.1))
    ax2.legend()
    ax2.set_title(f'PR AUC = {prauc:.2f}', fontsize=title_font_size)

    # Plot prob distribution
    ax3.grid()
    ax3.hist(inconsistent['pred_proba'], color='red', alpha=0.5,
             density=True, label='Inconsistent',
             bins=max(int(inconsistent['pred_proba'].nunique()/20), 20))
    ax3.hist(consistent['pred_proba'], color='green', alpha=0.5,
             density=True, label='Consistent',
             bins=max(int(inconsistent['pred_proba'].nunique()/20), 20))
    ax3.set_xlabel('Prob of inconsistent')
    ax3.set_ylabel('Density')
    ax3.set_title(f'JS Divergence = {js_div:.3f}', fontsize=title_font_size)
    ax3.legend()

    plt.show()


If you found this useful, please cite this write-up as:

    Yan, Ziyou. (Mar 2024). Task-Specific LLM Evals that Do & Don't
    Work. eugeneyan.com. https://eugeneyan.com/writing/evals/.

or

@article{yan2024evals,
  title   = {Task-Specific LLM Evals that Do & Don't Work},
  author  = {Yan, Ziyou},
  journal = {eugeneyan.com},
  year    = {2024},
  month   = {Mar},
  url     = {https://eugeneyan.com/writing/evals/}
}


Share on:
[icon-twitt]
[icon-linke]
[icon-faceb]
[icon-mail]

Browse related tags: [ llm eval machinelearning ]

[icon-searc]Search
<< Don't Mock Machine Learning Models In Unit Tests Building an AI
Coach to Help Tame My Monkey Mind >>
---------------------------------------------------------------------

Join 9,300+ readers getting updates on machine learning, RecSys,
LLMs, and engineering.

[                    ]
Get email updates
---------------------------------------------------------------------

  * [bluesky] Bluesky
  * [icon-twitt] Twitter
  * [icon-linke] LinkedIn
  * [icon-githu] GitHub

Eugene Yan designs, builds, and operates machine learning systems
that serve customers at scale. He's currently a Senior Applied
Scientist at Amazon. Previously, he led machine learning at Lazada
(acquired by Alibaba) and a Healthtech Series A. He writes & speaks
about machine learning, recommenders, LLMs, and engineering at
eugeneyan.com and ApplyingML.com.

(c) Eugene Yan 2015 - 2024 * Feedback * RSS