An n-gram is a contiguous sequence of n tokens, often words. Unigrams contain one token, bigrams contain two, and trigrams contain three. N-gram overlap compares these sequences between a model-generated candidate and one or more reference answers. Unlike semantic evaluation, it rewards matching wording rather than directly assessing whether two texts communicate the same meaning.
For example, consider the reference “the server is down” and the candidate “the server is unavailable.” With lowercase, whitespace-based tokenization, three of the four candidate unigrams match the reference. Two of the three candidate bigrams also match: “the server” and “server is.” The texts communicate similar information, but the different final word reduces overlap. Larger n-grams generally impose stricter requirements on local wording and word order.
BLEU, or Bilingual Evaluation Understudy, is commonly used for machine translation. It combines modified n-gram precisions, typically across orders one through four, using a geometric mean and a brevity penalty. Modified precision clips matching counts against reference counts so repeated words cannot inflate the score without limit. The brevity penalty discourages candidates that obtain high precision by being too short. Standard BLEU is primarily a corpus-level metric; sentence-level results can be unstable, particularly for short answers, and depend on smoothing choices.
ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is commonly used for summarization. ROUGE-N compares n-grams, with ROUGE-1 and ROUGE-2 evaluating unigrams and bigrams respectively. Recall measures how much reference content appears in the candidate, while precision measures how much candidate content matches the reference; implementations can also report F1. ROUGE-L instead uses the longest common subsequence, so it should not be described as a fixed-length n-gram overlap metric.
In a cloud evaluation workflow, engineers store a versioned test dataset containing prompts and reference answers, generate responses from candidate foundation models, and run scoring code in a managed processing job or evaluation pipeline. For a support-ticket summarizer, the team might compare ROUGE-1 and ROUGE-2 across model versions while separately reviewing whether summaries preserve outage causes and recovery actions. Keep prompts, generation settings, reference data, tokenizer configuration, metric implementation, and aggregation rules consistent so score changes remain interpretable.
The main advantages are low computational cost, reproducibility, and straightforward comparison across many outputs. The main limitations are sensitivity to paraphrases, reference quality, tokenization, punctuation, and language-specific segmentation. Multiple references can accommodate alternative valid phrasings, although their effect depends on the metric. A high overlap score does not establish factual correctness: changing “access is allowed” to “access is not allowed” preserves much of the wording while reversing the meaning. Longer outputs can also increase recall without improving quality.
For foundation-model evaluation, distinguish lexical metrics from semantic and judgment-based methods. BERTScore compares contextual token embeddings and can recognize similarities that exact matching misses, but it does not guarantee factual correctness. LLM-as-a-judge evaluates outputs against a rubric and can assess qualities such as relevance or instruction following, but introduces judge-model cost, variability, and potential bias. Use overlap metrics as task-specific regression signals alongside semantic evaluation, grounded factual checks, and calibrated human review rather than treating any single score as a universal measure of model quality.
Amazon SageMaker