Although tech giants, including Google, Amazon and Meta, have each put their own powerful speech recognition systems at the heart of their software and services. But speech recognition remains a challenging topic in artificial intelligence and machine learning. The good news is that today OpenAI is solemnly announcing the open-source of Whisper – known as an automatic speech recognition system that officially claims that it can perform powerful transcriptions in multiple languages and translate them into English.
(Source: OpenAI Blog)
What makes Whisper different, OpenAI says, is that it took 680,000 hours of multilingual and “multitasking” training data collected from the network, improving the scheme’s ability to recognize unique accents, background noise and technical terms.
The overview on the official GitHub repository says:
"The primary target users of Whisper models are AI researchers who study the robustness, generalization, capability, bias, and constraints of current models. At the same time, it is also suitable as an automatic speech recognition solution for developers, especially English speech recognition. Interested friends can download multiple versions of the Whisper system from the hosting platform, and its models show strong ASR results in about 10 languages. In addition, if fine-tuned on certain tasks, they are also expected to show additional capabilities in application scenarios such as voice activity detection and narrator classification."
Architecture
Unfortunately, Whisper has not been robustly evaluated in related fields, and the model has its limitations – in the field of text prediction.
Since the system was trained on a lot of “noisy” data, OpenAI decided to give everyone a shot in advance, warning that Whisper could include words in the transcription that weren’t actually spoken.
The reason may be that Whisper is both trying to predict the next word in the audio and trying to transcribe the audio itself.
Process example
In addition, Whisper’s performance in different language scenarios is also inconsistent, especially when it comes to narrators of languages that are not well represented in the training data, the error rate is also higher.
But the latter is nothing new in the field of speech recognition, and even the industry’s premier systems have been plagued by such biases.
Referring to the results of a study shared by Stanford University in 2020 – systems from Amazon, Apple, Google, IBM and Microsoft have a much lower error rate (about 35%) for white users compared to blacks.
About 1/3 of Whisper’s audio dataset is non-English
Even so, OpenAI believes that Whisper’s transcription capabilities can be used to improve existing accessibility tools. It wrote on GitHub:
"While the Whisper model is not suitable for real-time transcription out of the box, its speed and size suggest that others can build on it for near real-time speech recognition and translation applications. Beneficial applications built on top of Whisper's models, whose value is a tangible indication of the different capabilities of these models, are expected to have real economic impact. We hope that everyone will actively use this technology for beneficial purposes, making improvements to automatic speech recognition technology easier and enabling more participants to create more responsible projects. With the dual benefits of speed and accuracy, Whisper will allow for an affordable automated transcription and translation experience for large volumes of communications."
