Microsoft Patent Application Describes AI That Rewrites Live Speech and Lip Movements

According to reports on the filing, AI agents would monitor a live video feed and its audio, pick out words or phrases that match preset rules or a user's profile, and remove or replace them almost instantly. The system would then edit the surrounding audio and video so the change looks natural, including redrawing the speaker's face and mouth to fit the new words. Computerworld reported that the profile could take into account factors such as a viewer's age, the languages they speak or their location.

The filing lists several uses. They include live translation, in which viewers hear an event in their own language with matching lip movements, and removing filler sounds such as "um" and "ah". The system could also correct stutters and drop words that are mumbled, mispronounced, misspoken or considered offensive, and it might replace them with suggestions drawn from internet searches, Digit reported.

The same capabilities raise obvious concerns about deepfakes. Because edits would be tailored to individual profiles, different viewers could hear different versions of the same speech without knowing what was actually said, and regional expressions or accents could be stripped out. The filing reportedly suggests that the original audio and video could be recorded and stored for reference, but the reports do not say who would have access to it.

A patent application does not mean a product is coming. Microsoft has not said whether it plans to build the technology, and none of the reports included a comment from the company. Technology groups routinely file patents on ideas they never ship.

The filing arrives as Microsoft and its rivals push real-time AI translation into video calls and live events. If a patent is granted and the idea reaches products such as Teams, the main questions will be whether viewers are told that speech has been changed and whether the unedited version stays available.