BERTScore evaluates generated text by comparing it with human-written or otherwise trusted reference text. It uses a pretrained contextual language encoder to represent tokens as vectors, then compares those vectors using cosine similarity. Because the representations incorporate surrounding context, BERTScore can recognize some semantic similarities even when the generated response uses different wording from the reference. It is commonly used for translation, summarization, and reference-based answer evaluation.
The calculation starts with a similarity matrix between candidate and reference tokens. For precision, each candidate token is matched to its most similar reference token, and those similarities are averaged. For recall, each reference token is matched to its most similar candidate token. F1 combines precision and recall through their harmonic mean. These are soft similarity measures, not counts of exactly correct words or classifications. Matching is greedy rather than a strict one-to-one alignment, and optional inverse document frequency weighting can give less weight to common tokens.
For example, a reference answer might say, “The application automatically increases capacity when demand rises,” while a generated answer says, “The system scales out as traffic grows.” Word-overlap metrics may give limited credit because several words differ. BERTScore can give credit for related meanings. However, an answer that changes “increases” to “decreases” may still share substantial contextual similarity. Engineers must therefore check whether the metric detects the particular errors that matter for their application.
In a cloud evaluation workflow, teams store versioned prompts and reference answers, generate responses from candidate foundation models, and run BERTScore in a batch processing job. For example, a team comparing two customer-support models can score both against the same held-out reference set, report average F1, and inspect results separately for billing, troubleshooting, and account-access questions. The generation model and the encoder used for BERTScore are separate components. The scorer need not use the same model family as the application.
Reproducibility requires recording the encoder checkpoint, tokenizer, selected encoder layer, package version, weighting settings, and baseline-rescaling configuration. BERTScore is not restricted to the original BERT model; implementations can use other supported contextual encoders. Raw and baseline-rescaled scores are not interchangeable, and scores should not be interpreted as percentages of correctness. Avoid assuming a universal zero-to-one range across configurations, particularly when baseline rescaling is enabled. Long inputs also require an explicit truncation or chunking policy because encoders have input-length limits.
The main trade-offs are compute cost, dependence on reference quality, and sensitivity to the chosen encoder and domain. Running an encoder is generally more expensive than calculating lexical overlap, although batching and appropriate hardware can improve throughput. A valid answer may score poorly if the reference captures only one acceptable response. Conversely, a fluent answer containing an incorrect date, number, negation, or unsupported claim can still score well. BERTScore does not verify claims against retrieved documents or external evidence.
For evaluation questions, distinguish BERTScore from BLEU, ROUGE, and LLM-as-a-judge. BLEU emphasizes modified n-gram precision with a brevity penalty, while common ROUGE variants measure word-sequence overlap and often emphasize reference coverage. BERTScore instead uses contextual token similarity. LLM-as-a-judge asks a generative model to apply an evaluation rubric and can assess broader criteria, but introduces its own biases and variability. A practical evaluation suite combines complementary metrics with factuality checks, task-specific validation, and human review rather than treating any single score as a complete quality assessment.
Amazon S3