Does document-level context improve translation quality?
We built a measurement harness to find out on our own production stack — and the answer is yes, but not in the way surface metrics suggest.
Headline
Translating each sentence with the whole source document available as context produces a small but statistically significant improvement in COMET, a semantic quality metric aligned with human judgment — even when surface metrics like BLEU barely move. The improvement concentrates where it matters most: on the difficult segments that sentence-in-isolation translation gets wrong.
wmt22-comet-da)Why the small number is the interesting number
The mean COMET delta is small in aggregate because most sentences translate well in isolation — Gemini is a strong translator. But the mean hides the tail. On the worst 10% of sentence-level translations, adding document context lifts COMET by +0.037 (0.6314 → 0.6682) — nearly ten times the average effect. Cross-sentence information about pronouns, terminology, and register resolves ambiguities that sentence-in-isolation translation cannot. Context wins on 155 segments and loses on 126 (at ±0.005 COMET); the largest wins reach +0.33.
| Metric | Sentence | Context | Δ |
|---|---|---|---|
| COMET (semantic) | 0.8547 | 0.8595 | +0.0048 |
| BLEU (surface) | 38.13 | 38.63 | +0.50 |
| chrF (surface) | 63.15 | 62.94 | -0.21 |
| Cost, this experiment | $2.28 | $0.71 | −$1.57 (3.2x cheaper) |
Note that context mode is also substantially cheaper — Google Gemini's explicit context caching bills the shared document once instead of once per chunk. The RFI-style question is not just “does it work better” but “does it cost more to work better”. Here the answer is: it costs about a third as much, on the same corpus, on the same model.
What actually changed — three real segments
Every quote below is a real segment from the 500-run. The English text and reference French are unchanged; only how the model saw the segment (in isolation vs with the surrounding document) differs.
EN — I do not believe this is what my constituents want.
Sentence mode picked commettants, dated legalese. Context saw the surrounding parliamentary register and chose électeurs, which is what the reference used.
EN — We are concerned with a statute.
“Concerned with” is idiomatic for “this is about”. Sentence mode calqued it word-for-word. Context produced the idiomatic French, matching the reference exactly.
Honest counterexample — context isn't uniformly better
EN — Mr. Speaker, I ask that the remaining questions be allowed to stand.
Not every context-mode change is an improvement. Here, en souffrance implies unpaid mail; the correct parliamentary term is en instance (still deferred) — which sentence mode produced. Twenty-four of 500 segments regressed like this. We report both the wins and losses.
Methodology
- Corpus: Canadian Hansard EN↔FR (parliamentary proceedings), deterministic sample of 500 segments (seed 42); formal government register, close to real-world Canadian public-sector translation.
- Model:
gemini-3.6-flash (adaptive thinking, same model Synzo runs). Same model runs Synzo's public translation feature. - Modes compared: Sentence translates each segment independently, replicating a traditional segment-by-segment CAT-tool pipeline. Context translates the same segments in chunks with the whole source document supplied as cached context.
- Metrics: BLEU and chrF via
sacreBLEU; COMET via Unbabelwmt22-comet-da— the semantic metric aligned with human judgment recommended in the machine-translation research literature. - Alignment: both modes emit exactly one output segment per input, so all three metrics compare like with like.
- Cost: tracked per-request from each API response's
usage_metadata; priced at Google's published rates. - Reproducibility: harness, corpus loader, and per-segment score files are open source under Apache 2.0.