AI system could make lip sync dubbing accurate

Scientists have developed an artificial intelligence (AI) system that can edit the facial expressions of actors to accurately match dubbed voices, saving time and reducing costs for the film industry.

Deep Video Portraits, developed by researchers including those from Max Planck Institute for Informatics in Germany and Stanford University in the US can also be used to correct gaze and head pose in video conferencing and enables new possibilities for video postproduction and visual effects.

Unlike previous methods that are focused on movements of the face interior only, Deep Video Portraits can also animate the whole face including eyes, eyebrows, and head position in videos, using controls known from computer graphics face animation.

It can even synthesise a plausible static video background if the head is moved around.

"It works by using model-based 3D face performance capture to record the detailed movements of the eyebrows, mouth, nose, and head position of the dubbing actor in a video," said Hyeongwoo Kim from the University of Bath in the UK.

"It then transposes these movements onto the 'target' actor in the film to accurately sync the lips and facial movements with the new audio," said Kim.

The research is currently at the proof-of-concept stage and is yet to work in real-time, however, the researchers anticipate the approach could make a real difference to the visual entertainment industry.

"Despite extensive post-production manipulation, dubbing films into foreign languages always presents a mismatch between the actor on screen and the dubbed voice," said Christian Theobalt, from the Max Planck Institute for Informatics.

"Our new Deep Video Portrait approach enables us to modify the appearance of a target actor by transferring head pose, facial expressions, and eye motion with a high level of realism," said Theobalt.

"This technique could also be used for post-production in the film industry where computer graphics editing of faces is already widely used in today's feature films," said Christian Richardt, from the University of Bath.0

A great example is 'The Curious Case of Benjamin Button' where the face of Brad Pitt was replaced with a modified computer graphics version in nearly every frame of the movie.

This work remains a very time-consuming process, often requiring many weeks of work by trained artists.

The approach can also be used in other applications, which the authors show on their project website, for instance in video and VR teleconferencing, where it can be used to correct gaze and head pose such that a more natural conversation setting is achieved.

The software enables many new creative applications in visual media production, but the authors are also aware of the potential of misuse of modern video editing technology.