Semantic Similarity

Semantic similarity describes the degree to which texts convey similar meanings. For example, “Restart the server” and “Reboot the machine” express similar instructions despite using different words. This matters when evaluating natural language processing systems because a valid translation, summary, or generated answer may not reproduce the wording of a reference answer. Semantic similarity is a general evaluation concept, not one standardized metric with a universally meaningful score.

A common implementation converts each text into an embedding: a numerical vector produced by a language model. Cosine similarity compares the directions of two vectors, with larger values generally indicating greater similarity for models designed for that comparison. Mathematical cosine similarity ranges from -1 to 1, although observed scores depend on the model and data. A score of 0.85 is not an 85 percent probability that two statements mean the same thing. Use a compatible embedding model and configuration for both sides; vectors from unrelated models do not share a meaningful coordinate system.

NLP evaluation requires distinguishing semantic measures from lexical measures. BLEU measures modified n-gram precision and applies a brevity penalty; it is commonly used for machine translation. ROUGE is a family of overlap metrics commonly used for summarization, including n-gram and longest-common-subsequence variants. ROUGE is often associated with recall, although implementations can report precision, recall, and F1. Both BLEU and ROUGE can undervalue acceptable paraphrases because their matching depends heavily on surface wording.

BERTScore instead compares contextual token embeddings between candidate and reference texts. It matches tokens by embedding similarity and aggregates those matches into precision, recall, and F1 scores, optionally using importance weighting. Unlike a single cosine comparison between sentence embeddings, BERTScore evaluates token-level correspondences. It can recognize meaning-preserving wording changes, but its results still depend on the encoder, language, configuration, and reference quality.

In a cloud evaluation pipeline, store representative prompts and reference answers, generate candidate responses, and calculate semantic and lexical metrics in batch processing jobs. For a support assistant, compare “You can return the item within 30 days” with a differently worded reference. Also include difficult negative examples such as “You cannot return the item within 30 days.” Negation, numbers, dates, and entity substitutions can change correctness while leaving embeddings highly similar. Combine similarity with explicit checks for these details, groundedness evaluation, and human review.

The same idea supports semantic search and retrieval-augmented generation. Embed document chunks and queries, retrieve nearby vectors, and optionally rerank the results. Approximate nearest-neighbor indexes reduce retrieval latency at scale but can miss relevant candidates. Retrieval similarity indicates a potentially useful context passage; it does not prove that a generated answer faithfully uses that passage.

Calibrate thresholds against human-labeled examples from the target domain rather than adopting an arbitrary cutoff. Track performance across languages, document lengths, and specialized terminology. Embedding models add inference cost, and cross-encoders that jointly score text pairs usually cost more per comparison than comparing precomputed vectors. Version models, preprocessing, reference datasets, and metric settings so results remain reproducible. When choosing an evaluation approach, match the metric to the requirement: word overlap, preservation of meaning, factual accuracy, and task success are related but distinct objectives.