BLEU score – machine translation evaluation
20 questions and answers on the BLEU metric, explaining n-gram precision, brevity penalty, multiple references and known limitations when evaluating machine translation systems.
What is the BLEU score used for in NLP?
Answer: BLEU (Bilingual Evaluation Understudy) is an automatic metric for evaluating machine translation quality by measuring n-gram overlap between system outputs and one or more human reference translations.
Does BLEU measure precision, recall or both?
Answer: BLEU primarily measures modified n-gram precision, adjusted by a brevity penalty that discourages overly short translations, but it does not directly compute recall over reference n-grams.
What are modified n-gram precisions in BLEU?
Answer: Modified precision counts each candidate n-gram up to the maximum number of times it appears in any reference, preventing systems from inflating scores by repeating the same high-frequency n-grams excessively.
What is the brevity penalty in BLEU and why is it needed?
Answer: The brevity penalty reduces the BLEU score when the candidate translation is too short compared to references, discouraging trivial translations that maximize precision by omitting content.
Why does BLEU combine multiple n-gram orders?
Answer: BLEU multiplies geometric means of 1-gram to 4-gram precisions (typically) to reward both local word choice and short phrase correctness, penalizing translations that only match single words but not phrases or word order.
How does BLEU handle multiple reference translations?
Answer: For each n-gram and sentence length, BLEU considers all references and uses the best match when computing clipped counts and brevity penalties, making it more tolerant of paraphrasing and lexical variation than a single reference.
What is the typical BLEU score range and interpretation?
Answer: BLEU ranges from 0 to 1 (often reported as 0–100); higher scores indicate closer n-gram overlap with references, but absolute values depend on language pair, dataset and reference quality, so relative comparisons are more meaningful.
Why is BLEU more reliable at corpus level than at sentence level?
Answer: At sentence level, BLEU is unstable and often zero due to sparse n-gram matches, while aggregating over many sentences smooths variability and yields scores that correlate better with human judgments.
What are some limitations of BLEU?
Answer: BLEU is insensitive to meaning and adequacy beyond n-gram overlap, may penalize legitimate paraphrases, is influenced by tokenization and reference choice, and can be gamed by certain degenerate strategies if used uncritically.
How does BLEU compare to human evaluation?
Answer: BLEU moderately correlates with human judgments at system level, making it useful for rapid experiments, but human evaluation remains the gold standard for fine-grained quality assessment and nuanced linguistic issues.
Why is tokenization important when computing BLEU?
Answer: BLEU operates on token sequences; inconsistent or poor tokenization can distort n-gram counts, so standardized tokenization (e.g. sacreBLEU) is recommended for fair comparison between systems and papers.
What is sacreBLEU and why is it used?
Answer: sacreBLEU is a tool that standardizes BLEU computation (tokenization, reference sets, case handling) and logs settings, improving reproducibility and comparability of BLEU scores across MT research papers.
Can BLEU be used for evaluation beyond machine translation?
Answer: BLEU has been applied to tasks like summarization and captioning, but its limitations become more pronounced there; specialized metrics such as ROUGE or CIDEr are often preferred for those generation tasks.
How does BLEU treat word order?
Answer: Higher-order n-grams capture local word order; incorrect ordering tends to reduce bigram, trigram and 4-gram matches, lowering BLEU compared to outputs that maintain phrase-level structure in line with references.
Why is smoothing sometimes applied to BLEU?
Answer: Smoothing adjusts n-gram precision when counts are zero (especially for higher n-grams in short sentences), preventing the overall BLEU from collapsing to zero and making sentence-level analysis more informative.
What does a BLEU score of zero indicate?
Answer: A zero BLEU score means there were no matching n-grams (often higher-order) after clipping, typically indicating very poor overlap with references or very short/unnatural candidate outputs for the evaluated set.
How should BLEU be used responsibly in research and industry?
Answer: Use BLEU as one signal alongside human evaluation and other metrics, report standard settings (e.g. sacreBLEU), avoid over-optimizing solely for BLEU and interpret improvements in the context of qualitative examples and user needs.
What effect does adding more reference translations have on BLEU?
Answer: More references increase the chance that a candidate’s n-grams appear in at least one reference, usually raising BLEU scores and making the metric more robust to paraphrasing and lexical variation.
Why might BLEU under-reward high-quality translations?
Answer: If a system produces a correct but significantly paraphrased translation that differs in n-gram choices from references, BLEU may assign a modest score despite good adequacy and fluency, due to limited overlap in surface forms.
Why is BLEU still widely used despite newer metrics?
Answer: BLEU has a long history, many benchmarks and codebases built around it, making it convenient for comparisons over time; newer metrics complement rather than completely replace BLEU in many MT evaluation workflows.
🔍 BLEU concepts covered
This page covers BLEU score: modified n-gram precision, brevity penalty, multiple references, smoothing, limitations and best practices for using BLEU as one component of machine translation evaluation.