What is word error rate?
Word error rate is the proportion of words a transcription gets wrong, measured against a human reference transcript. Insertions and deletions are counted alongside substitutions, then divided by the number of words actually spoken. It is the standard figure for transcription quality, and it treats every word as equally important, which is where its limits begin.
Also called: WER
Why a low word error rate can still mean a failed call
A deployment accepted on an overall word error rate will fail on the words that carry the transaction. The figure weights every word identically, so a transcript that misses an account number and gets the courtesies right scores well while being useless. Errors also cluster rather than spreading evenly, which means a respectable average is consistent with a minority of calls being unusable. The figure is worth reporting, and it belongs alongside accuracy on the entities the task depends on.
- Every word counts equally, including words that carry no information.
- Errors cluster on particular speakers and conditions rather than spreading evenly.
- Entity level accuracy predicts task success where the overall figure does not.
- A reference transcript carries its own disagreement between human transcribers.
How word error rate works
- A reference transcript is produced by people against an agreed convention.
- The hypothesis transcript is aligned to the reference word by word.
- Insertions and deletions are counted alongside substitutions at each alignment point.
- The total is divided by the word count of the reference transcript.
- The figure is reported with the corpus and the conditions it was measured on.
- Entity level accuracy is reported beside it for the fields the task needs.
How word error rate is evaluated
Word error rate is only interpretable with its corpus and its segmentation stated. Report the following together with every figure:
- word error rate with the corpus and audio conditions named
- distribution across calls rather than the corpus average alone
- entity level accuracy for numbers and proper nouns
- error rate by language and by speaker accent
- disagreement between human transcribers on the same reference
Limitations, risks and common failure modes
- Normalisation choices in the reference change the figure by several points.
- A corpus assembled from clean audio flatters a model that will meet noise.
- An average across calls hides a subset that is entirely unusable.
- Improvements in the figure fail to appear in task completion rate.