Whisper (speech recognition system) explained

Whisper (speech recognition system)
Author:OpenAI[1]
Released:September 21, 2022
Repo:https://github.com/openai/whisper
Programming Language:Python
License:MIT License

Whisper is a machine learning model for speech recognition and transcription, created by OpenAI and first released as open-source software in September 2022.[2]

It is capable of transcribing speech in English and several other languages,[3] and is also capable of translating several non-English languages into English. OpenAI claims that the combination of different training data used in its development has led to improved recognition of accents, background noise and jargon compared to previous approaches.[4]

Whisper is a weakly-supervised deep learning acoustic model, made using an encoder-decoder transformer architecture.[5]

Whisper V2 was released on December 8, 2022.[6] Whisper V3 was released in November 2023, on the OpenAI Dev Day.

Background

Speech recognition has had a long history in research; the first approaches made use of statistical methods, such as dynamic time warping, and later hidden Markov models. At around the 2010s, deep neural network approaches became more common for speech recognition models, which were enabled by the availability of large datasets ("big data") and increased computational performance.[7] Early approaches to deep learning in speech recognition included convolutional neural networks, which were limited due to their inability to capture sequential data, which later led to developments of Seq2seq approaches, which include recurrent neural networks which made use of long short-term memory.[8]

Transformers, introduced in 2017 by Google, displaced many prior state-of-the art approaches to many problems in machine learning, and started becoming the core neural architecture in fields such as language modeling and computer vision;[9] weakly-supervised approaches to training acoustic models were recognized in the early 2020s as promising for speech recognition approaches using deep neural networks.[10]

According to a NYT report, in 2021 OpenAI believed they exhausted sources of higher-quality data to train their large language models and decided to complement scraped web text with transcriptions of YouTube videos and podcasts, and developed Whisper to solve this task.[11]

Training and capabilities

Whisper has been trained using semi-supervised learning on 680,000 hours of multilingual and multitask data, of which about one-fifth (117,000 hours) were non-English audio data. Whisper does not outperform models which specialize in the LibriSpeech dataset, although when tested across many datasets, it is more robust and makes 50% fewer errors than other models.

Whisper has a differing error rate with respect to transcribing different languages, with a higher word error rate in languages not well-represented in the training data.[12]

The model has been used as the base for a unified model for speech recognition and more general sound recognition.[13]

Architecture

The Whisper architecture is based on an encoder-decoder transformer. Input audio is split into 30-second chunks converted into a Mel-frequency cepstrum, which is passed to an encoder. A decoder is trained to predict later text captions. Special tokens are used to perform several tasks, such as phrase-level timestamps.[14]

See also

Notes and References

  1. Radford . Alec . Kim . Jong Wook . Xu . Tao . Brockman . Greg . McLeavey . Christine . Sutskever . Ilya . Robust Speech Recognition via Large-Scale Weak Supervision . 2022-12-06 . eess.AS . 2212.04356 .
  2. Web site: Golla . Ramsri Goutham . 2023-03-06 . Here Are Six Practical Use Cases for the New Whisper API . live . https://web.archive.org/web/20230325214704/https://slator.com/six-practical-use-cases-for-new-whisper-api/ . 2023-03-25 . 2023-08-12 . Slator . en-US.
  3. Web site: Dickson . Ben . 2022-10-03 . How will OpenAI's Whisper model impact AI applications? . live . https://web.archive.org/web/20230315105914/https://venturebeat.com/ai/how-will-openais-whisper-model-impact-ai-applications/ . 2023-03-15 . 2023-08-12 . VentureBeat . en-US.
  4. Web site: Wiggers . Kyle . September 21, 2022 . OpenAI open-sources Whisper, a multilingual speech recognition system . February 12, 2023 . TechCrunch . en-US . February 12, 2023 . https://web.archive.org/web/20230212154543/https://techcrunch.com/2022/09/21/openai-open-sources-whisper-a-multilingual-speech-recognition-system/ . live .
  5. 2212.04356 . eess.AS . Alec . Radford . Jong Wook . Kim . Robust Speech Recognition via Large-Scale Weak Supervision . 2022-12-06 . Xu . Tao . Brockman . Greg . McLeavey . Christine . Sutskever . Ilya . 3.
  6. Web site: Announcing the large-v2 model · openai/whisper · Discussion #661 . 2024-01-08 . GitHub . en.
  7. Book: Yu . Dong . Automatic speech recognition: a deep learning approach . Deng . Li . Springer . 2014 . 978-1-4471-5778-6 . 2015th . Signals and communication technology . London Heidelberg . 9 . en.
  8. 2303.11607v1 . Latif . Siddique . Aun . Zaidi . Cuayahuitl . Heriberto . Shamshad . Fahad . Shoukat . Moazzam . Qadir . Junaid . Transformers in Speech Processing: A Survey. 2023 . cs.CL .
  9. Book: Kamath . Uday . Transformers for machine learning: a deep dive . Graham . Kenneth L. . Emara . Wael . CRC Press, Taylor & Francis Group . 2022 . 978-0-367-76734-1 . First . Chapman & Hall/CRC machine learning & pattern recognition . Boca Raton London New York . xix . en.
  10. Book: Paaß . Gerhard . Giesselbach . Sven . Foundation Models for Natural Language Processing . Foundation Models for Speech, Images, Videos, and Control . Artificial Intelligence: Foundations, Theory, and Algorithms . 2023-02-16 . 313–382 . en . 2302.08575 . 10.1007/978-3-031-23190-2_7. 978-3-031-23189-6 . 257019816 .
  11. Web site: Davis . Wes . 2024-04-06 . OpenAI transcribed over a million hours of YouTube videos to train GPT-4 . 2024-04-20 . The Verge . en.
  12. Web site: Wiggers . Kyle . 2023-03-01 . OpenAI debuts Whisper API for speech-to-text transcription and translation . live . https://web.archive.org/web/20230718040023/https://techcrunch.com/2023/03/01/openai-debuts-whisper-api-for-text-to-speech-transcription-and-translation/ . 2023-07-18 . 2023-08-21 . TechCrunch . en-US.
  13. Book: 2307.03183 . Gong . Yuan . Sameer . Khurana . Interspeech 2023 . Leonid . Karlinsky . James . Glass. Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers . 2023 . 2798–2802 . 10.21437/Interspeech.2023-2193 .
  14. Web site: 2022-09-21 . Introducing Whisper . live . https://web.archive.org/web/20230820005801/https://openai.com/research/whisper . 2023-08-20 . 2023-08-21 . openai.com . en-US.

This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia article "Whisper (speech recognition system)".

Except where otherwise indicated, Everything.Explained.Today is © Copyright 2009-2024, A B Cryer, All Rights Reserved. Cookie policy.