BLEU, short for Bilingual Evaluation Understudy, is an automated metric for comparing generated text with one or more reference texts. Its original and strongest application is machine translation, where engineers need to compare translation systems across a representative test corpus. In foundation-model evaluation, BLEU can help assess translation or other constrained text-generation tasks, but it is less suitable for open-ended conversations that admit many equally valid answers.
BLEU measures modified n-gram precision. An n-gram is a consecutive sequence of tokens: a unigram contains one token, while a bigram contains two. The commonly used BLEU-4 configuration combines precision for unigrams through four-grams using a geometric mean. Matched counts are clipped to the maximum count found in any individual reference, preventing a model from gaining unlimited credit by repeating words that appear in the references.
Because precision alone could reward very short outputs, BLEU applies a brevity penalty when the candidate text is shorter than its effective reference length. With multiple references, standard BLEU chooses the reference length closest to the candidate length, preferring the shorter reference in a tie. The resulting score is commonly reported either between 0 and 1 or on a scale from 0 to 100. Always state the scale; a score of 30 on the latter scale does not mean that 30 percent of translations are correct.
For example, suppose a reference translation says, "The server is temporarily unavailable." A candidate saying, "The server is currently unavailable," preserves much of the wording and receives overlap credit. "The service cannot be reached right now" may convey a similar meaning while receiving less credit. Conversely, a fluent sentence that changes a critical number or negates an instruction may retain substantial overlap despite being dangerously wrong.
In a cloud evaluation pipeline, engineers store a held-out dataset containing inputs and approved references, generate outputs from each candidate model, and run a consistent BLEU implementation in a batch processing job. A multilingual support team could compare a baseline model with a fine-tuned model using the same translation examples and generation settings. Store model identifiers, dataset versions, outputs, tokenizer settings, library versions, and scoring configuration alongside each result so that comparisons can be reproduced.
Corpus BLEU aggregates matched and total n-gram counts across examples before calculating the score; it is not the arithmetic average of sentence-level BLEU scores. Sentence-level scores are unstable for short responses, and a missing higher-order match can make an unsmoothed score zero. Tokenization, case handling, smoothing, and reference selection can materially change results. Standardized tooling such as SacreBLEU helps make these choices explicit and comparisons more reliable.
For exam questions, distinguish BLEU's precision-oriented lexical matching from ROUGE's commonly used recall-oriented overlap measures, particularly in summarization. BERTScore uses contextual embeddings to assess token similarity and can better accommodate some paraphrases. LLM-as-a-judge evaluates outputs against a rubric but introduces judge-model bias, variability, and additional inference cost. BLEU is inexpensive and reproducible, but it should be combined with task-specific correctness checks, human review, or semantic evaluation rather than treated as proof of overall model quality.
Amazon S3