Learning and controlling the source-filter representation of speech with a variational autoencoder - Institut d'Electronique et de Télécommunications de Rennes Accéder directement au contenu
Article Dans Une Revue Speech Communication Année : 2023

Learning and controlling the source-filter representation of speech with a variational autoencoder

Résumé

Understanding and controlling latent representations in deep generative models is a challenging yet important problem for analyzing, transforming and generating various types of data. In speech processing, inspiring from the anatomical mechanisms of phonation, the source-filter model considers that speech signals are produced from a few independent and physically meaningful continuous latent factors, among which the fundamental frequency $f_0$ and the formants are of primary importance. In this work, we start from a variational autoencoder (VAE) trained in an unsupervised manner on a large dataset of unlabeled natural speech signals, and we show that the source-filter model of speech production naturally arises as orthogonal subspaces of the VAE latent space. Using only a few seconds of labeled speech signals generated with an artificial speech synthesizer, we propose a method to identify the latent subspaces encoding $f_0$ and the first three formant frequencies, we show that these subspaces are orthogonal, and based on this orthogonality, we develop a method to accurately and independently control the source-filter speech factors within the latent subspaces. Without requiring additional information such as text or human-labeled data, this results in a deep generative model of speech spectrograms that is conditioned on $f_0$ and the formant frequencies, and which is applied to the transformation speech signals. Finally, we also propose a robust $f_0$ estimation method that exploits the projection of a speech signal onto the learned latent subspace associated with $f_0$.
Fichier principal
Vignette du fichier
sadok2023learning.pdf (11.05 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03650569 , version 1 (25-04-2022)
hal-03650569 , version 2 (04-05-2022)
hal-03650569 , version 3 (21-03-2023)

Identifiants

Citer

Samir Sadok, Simon Leglaive, Laurent Girin, Xavier Alameda-Pineda, Renaud Séguier. Learning and controlling the source-filter representation of speech with a variational autoencoder. Speech Communication, 2023, 148, pp.53-65. ⟨10.1016/j.specom.2023.02.005⟩. ⟨hal-03650569v3⟩
389 Consultations
60 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More