KAIST's DLLM-VSR reads lip movements and converts them to text with a word error rate as low as 19.4%, opening a new path for assistive devices for people who are deaf or hard of hearing.
- KAIST developed DLLM-VSR, an AI system that reads lip movements and converts them to text without any audio.
- The system reaches a 19.4% word error rate on the LRS3 benchmark, using 433 hours of labeled video.
- Instead of reading left to right, the AI locks in the words it is most confident about first, then infers the rest.
- Potential uses include noisy industrial environments and assistive devices for people who are deaf or hard of hearing.
- The technology is not ready for widespread use, and it also raises privacy concerns.
Table of Contents
Why lip reading is hard
A research team at the Korea Advanced Institute of Science and Technology (KAIST) has unveiled DLLM-VSR, an AI system that recognizes lip movements to turn speech into text. According to the official announcement, the system achieves a word error rate of 19.4% on a popular lip-reading benchmark. That is the best result among methods trained on the same labeled dataset.
Lip reading is inherently difficult, even for AI. Sounds like "p," "b," and "m" produce almost identical mouth shapes, so it is hard to tell what a person is saying without hearing it. Traditional speech recognition systems typically process words one at a time, from left to right. The problem is that a single wrong prediction at the start of a sentence throws off everything after it.
A diffusion LLM approach
The team took a different route, using a diffusion large language model (DLLM). The AI is not forced to interpret words one by one. It first identifies the parts of a sentence it is most confident about, then uses those words as context to fill in the gaps step by step. The filling process draws on information from both before and after each gap in the sentence.

The system also estimates several plausible sentence lengths based on the video's duration. For each length, the AI generates a separate transcript and then picks the most plausible result. In the paper, the team reports a 19.4% word error rate on the LRS3 benchmark with 433 hours of labeled video. This figure counts misrecognized, missing, and extra words, not the share of conversations understood perfectly.
Applications and limits
The potential uses go beyond following a conversation in a noisy restaurant. The researchers envision the technology working in industrial settings, where machinery noise makes spoken communication difficult. It could also support devices that convert lip movements into text for people who are deaf or hard of hearing.

Still, DLLM-VSR is not ready for everyday use. Performance still varies by dataset, and the team is working on reducing its compute requirements. The goal is for the system to run in real time on wearables and robots without an Internet connection. Meanwhile, technology that can interpret conversations from silent video also raises legitimate privacy concerns.
A 19.4% error rate sounds impressive, but in practice it still means roughly 1 in 5 words is misrecognized, so it is not reliable enough for situations that demand high accuracy. The more notable part is the approach itself: locking in the confident parts first and then reasoning out the rest could carry over to many other recognition problems. Readers should watch how quickly the compute requirements come down, since that is the precondition for running this on wearables. At the same time, manufacturers need clear privacy safeguards in place before putting it into products.
Source: Digital Trends
Reviewed by: Hero Association








