1. Introduction

For decades, automated voice response systems meant keypad menus and frustration. That changed when machine learning research achieved three convergent breakthroughs: robust speech recognition, speech synthesis nearly indistinguishable from human voice, and language models capable of real-time dialogue. Today, a voice agent can answer a call, resolve an inquiry, and hand control back to a human when needed, without the user noticing the difference. This article reviews the research that made this shift possible, citing exclusively verified sources.

2. State of the art: three advances that made it possible

The inflection point in speech recognition was Whisper, OpenAI's system trained on 680,000 hours of multilingual audio, which achieved robust transcription without strict supervision (Radford et al., 2022). Building on that foundation, speech synthesis took a qualitative leap with Microsoft's VALL-E, a language model over neural codecs able to clone a voice from just three seconds of reference audio (Wang et al., 2023).

In 2024, NaturalSpeech 3 showed that professional-quality speech synthesis can reach error rates comparable to human speech, using factorized codecs and diffusion models (Ju et al., 2024). In parallel, Google's AudioPaLM unified listening and speaking in a single language model, removing the traditional separation between transcription and synthesis systems (Rubenstein et al., 2023).

The most recent leap is Moshi, from French startup Kyutai: an end-to-end speech model that holds full-duplex conversations, meaning it can speak and listen simultaneously with a latency of about 200 milliseconds, the natural behavior of a phone call (Défossez et al., 2024).

3. Applications in telephony

These advances translate into concrete applications already running in production:

  1. Automated customer service: agents resolve common queries with no wait times and escalate to a human only when necessary.
  2. Personal virtual assistants: they schedule appointments, send messages, and manage reminders through natural conversation.
  3. Contextual outbound calls: appointment reminders, collections, and surveys are handled with fluid conversation instead of rigid scripts.
  4. Multilingual support: thanks to systems such as Whisper, agents transcribe and understand languages without market-specific training.

4. Risks and safeguards

Generated voice also poses serious risks. OpenAI itself documents in its GPT-4o system card the dangers of speech synthesis: identity spoofing, unauthorized use of a voice, and audio deepfakes (OpenAI, 2024). The industry responds with emerging practices such as labeling synthetic audio, explicit consent for voice cloning, and API-level safeguards that restrict permitted voices.

5. Methodology note

This article cites exclusively verified online sources, published between 2021 and 2024. Authors, titles, and years were confirmed against the official arXiv and OpenAI pages before inclusion. Only SpeechBrain (Ravanelli et al., 2021) has formal peer review, having been published at Interspeech 2021; VALL-E, AudioPaLM, NaturalSpeech 3, Whisper, and Moshi are technical preprints or system reports, the standard publication format in the machine learning field.

References

  1. Défossez, A., Mazaré, N., Orsini, M., et al. (2024). Moshi: A speech-text foundation model for real-time dialogue. arXiv. https://arxiv.org/abs/2410.00037
  2. Ju, Z., Wang, Y., Shen, K., et al. (2024). NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv. https://arxiv.org/abs/2403.03100
  3. OpenAI. (2024). GPT-4o system card. OpenAI. https://openai.com/index/gpt-4o-system-card/
  4. Radford, A., Kim, J. W., Xu, T., et al. (2022). Robust speech recognition via large-scale weak supervision. arXiv. https://arxiv.org/abs/2212.04356
  5. Ravanelli, M., Parcollet, T., Plantinga, P., et al. (2021). SpeechBrain: A general-purpose speech toolkit. Proceedings of Interspeech 2021. https://arxiv.org/abs/2106.04624
  6. Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., et al. (2023). AudioPaLM: A large language model that speaks and listens. arXiv. https://arxiv.org/abs/2306.12925
  7. Wang, C., Chen, S., Wu, Y., et al. (2023). Neural codec language models are zero-shot text to speech synthesizers. arXiv. https://arxiv.org/abs/2301.02111