At the 2023 I / O developer conference, Google announced that it is developing an experimental technology called Universal Translator.
As the name implies, the tool is designed to translate video from one language to another while retaining the overall tone and atmosphere. This means that the technology not only translates audio from one language to another but also mimics the speaker’s voice, tone and facial expressions, with videos of characters speaking changing their mouth patterns in sync with the pronunciation of the target language.
As shown in the image above, Universal Translator first detects words and translates them. Then, it checks the speaker’s tone of voice and what they emphasize. After combining these two aspects, it generates the speech of the target language. Finally, it synchronizes the speaker’s diction in the video with the pronunciation of the AI-generated speech.
Considering that this tool could be misused to create fake videos, Google said it will restrict access to Universal Translator. As a result, only Google-authorized partners will be able to use it for construction projects, while ordinary users will not be able to use it. i