LLM Evaluation Guide

A skill for evaluating and benchmarking large language models (LLMs) in research settings. Covers automatic metrics, human evaluation protocols, benchmark suites, evaluation pitfalls, and best practices for reporting LLM performance.

Evaluation Taxonomy

Types of Evaluation

1. Intrinsic evaluation:
   Measures model quality on its own terms
   - Perplexity, likelihood, calibration
   - Useful for comparing architectures and training procedures

2. Extrinsic evaluation:
   Measures model quality on downstream tasks
   - Task-specific benchmarks (QA, summarization, classification)
   - Closer to real-world usefulness

3. Human evaluation:
   Human judges rate model outputs
   - Fluency, correctness, helpfulness, safety
   - Gold standard but expensive and slow

Automatic Metrics

Common Metrics by Task

Task	Metric	Description
Language modeling	Perplexity	Lower is better; measures prediction quality
Machine translation	BLEU, COMET	N-gram overlap; learned quality estimation
Summarization	ROUGE-1/2/L	Recall of n-grams against reference
Question answering	Exact Match, F1	Token-level match against reference answer
Classification	Accuracy, F1	Standard classification metrics
Generation quality	BERTScore	Semantic similarity via embeddings
Factuality	FActScore	Proportion of atomic facts supported by evidence

Computing Key Metrics

from collections import Counter
import math


def compute_bleu(reference: list[str], hypothesis: list[str],
                 max_n: int = 4) -> float:
    """
    Compute corpus-level BLEU score (simplified).

    Args:
        reference: List of reference token sequences
        hypothesis: List of hypothesis token sequences
        max_n: Maximum n-gram order
    """
    precisions = []

    for n in range(1, max_n + 1):
        num = 0
        den = 0
        for ref_tokens, hyp_tokens in zip(reference, hypothesis):
            ref_ngrams = Counter(
                tuple(ref_tokens[i:i+n]) for i in range(len(ref_tokens) - n + 1)
            )
            hyp_ngrams = Counter(
                tuple(hyp_tokens[i:i+n]) for i in range(len(hyp_tokens) - n + 1)
            )
            clipped = {ng: min(c, ref_ngrams.get(ng, 0))
                       for ng, c in hyp_ngrams.items()}
            num += sum(clipped.values())
            den += max(sum(hyp_ngrams.values()), 1)

        precisions.append(num / max(den, 1))

    # Brevity penalty
    ref_len = sum(len(r) for r in reference)
    hyp_len = sum(len(h) for h in hypothesis)
    bp = math.exp(1 - ref_len / max(hyp_len, 1)) if hyp_len < ref_len else 1.0

    # Geometric mean of precisions
    log_avg = sum(math.log(max(p, 1e-10)) for p in precisions) / max_n
    return bp * math.exp(log_avg)