New AI-based technology allows characters in movies to perform their lines flawlessly in another language through deepfake technology, eliminating the lip sync problems in traditional dubbing. The AI dubbing tech was developed by UK-based company Flawless, whose co-founder and director Scott Mann had become tired of seeing poor foreign dubbing in his films. After watching a foreign-dubbed version of his movie, Heist, which stars Robert De Niro, Mann says he was appalled by how the dubbing ruined carefully crafted scenes. Often dubbing involves changing the dialogue significantly to match the actor’s lip movements closely.
Mann then began researching academic works related to deepfakes, which led him to an AI dubbing project led by Christian Theobalt, a professor at the Max Planck Institute for Informatics in Germany. Inspired by Theobalt’s project, Flawless’s tech acts as a more sophisticated version of a deepfake. It captures an actor’s facial expressions and movements in a scene as well as the dubbed lines of a foreign voice actor. This data is combined to create a 3D model, which merges the face and the head of the actor with the lip movements of the dubber. The result is then digitally stitched onto the actor in the related scene.
The company is in discussions with various studios to create foreign versions of several movies. Demo clips showcase Jack Nicholson and Tom Cruise from A Few Good Men, Tom Hanks from Forrest Gump, and Robert De Niro from Taxi Driver as they deliver their famous lines flawlessly and fluently in various languages.
Zoom Out: Deepfake technology can reshape the movie industry and foreign dubbing both positively and negatively. It can help studios avoid costly reshoots and deliver more plausible storytelling, but it can also be used to spread misinformation, and as Mann underlines, some actors are a little uncomfortable seeing themselves manipulated through AI.
