AI to Make Lip Sync Dubbing a More Natural Conversation


Deep Video Portrait to modify the appearance of an actor by transferring head pose and facial expressions with a high level of realism

Researchers had developed a system using artificial intelligence (AI) that can edit the facial expressions of an actor to accurately match dubbed voices. They predicted that the system will save time and reduce costs for the film industry.

Unlike previous methods, that focus on movements of the face interior only, Deep Video Portraits can animate the whole face including eyes, eyebrows and head position in videos. It animates the same by using controls known from computer graphics face animation. It can even synthesise a plausible static video background if the head is moved around.

Can correct gaze and head pose

This technique can be used for post-production in the film industry where computer graphics editing of faces is already used in today’s feature films. With this approach, even the positioning of an actor’s head and their facial expression could be edited to change camera angles or subtly change the framing of a scene to tell the story better.

Apart from positioning an actor’s head, it can also be used in other applications like video and VR teleconferencing. Where it is used to correct gaze and head pose such that a more natural conversation setting is achieved.

Record the detailed movements

It works by using model-based 3D face performance capture to record the detailed movements of the eyebrows, mouth, nose and head position of the dubbing actor in a video. It then transposes these movements onto the target actor in the film to accurately sync the lips and facial movements with the new audio.

Despite extensive post-production manipulation, dubbing films into foreign languages always present a mismatch between the actor on screen and the dubbed voice. The research team is using the same technology to develop in tandem neural networks trained to detect synthetically generated or edited video at high precision to make it easier to spot forgeries.

Well, they have no plans to make the software publicly available but state that any software implementing such creative use cases should include watermarking schemes to clearly mark modifications. They are currently at the proof-of-concept stage and is yet to work at the real time. However, the researchers anticipate the approach that it could make a real difference to the visual entertainment industry.