What is lip sync?
Lip sync is the degree to which a synthetic voice and the mouth movement of the presenter rendering it stay aligned. Where the two separate by more than a few frames, a viewer notices before they can say why. It is one of the few quality markers in avatar video that an untrained audience judges instantly.
Also called: Lip synchronisation
Why a few frames of drift costs a module its authority
Learners rarely complain about lip sync in those terms. They report that a module felt cheap, or that they could not take a compliance message seriously. Attendance on the next one then falls away. For training generated at volume from controlled documents, the authority of the delivery is doing real work, since the material is usually an obligation somebody has to follow. A quarter of a second of drift is enough to undo that.
- Production quality is read by learners as a proxy for how seriously the organisation takes the material.
- Drift is most visible on the sibilants and plosives a viewer watches for.
- A module regenerated after a policy revision can lose alignment the original version had.
- Alignment is measurable per segment, so it need never be judged by impression alone.
How lip sync works
- The approved script is segmented into utterances with their intended timings.
- Synthesis produces audio with phoneme boundaries retained for each utterance.
- Viseme targets are derived from those boundaries and mapped onto the presenter model.
- The rendered video is compared against the audio to measure offset per segment.
- Segments outside the tolerance are regenerated rather than corrected by hand.
- A sample of each language version is reviewed by a person before release.
How lip sync is evaluated
Lip sync is assessed segment by segment rather than on an impression of the finished module. Measure the following on every language version before release:
- mean and worst case audio to viseme offset in milliseconds
- proportion of segments outside the agreed tolerance
- offset drift from the opening of a module to its close
- degradation after a regeneration from a revised script
- reviewer detection rate on blinded samples
Limitations, risks and common failure modes
- Offset accumulates across a long module, so an opening that looks correct ends badly.
- Tolerance settled for English carries over to languages with denser consonant clusters.
- Reviewers habituate to small drift after an hour of checking and stop noticing it.
- Compression on the way into a learning platform can shift audio relative to video.