projects

Robust and efficient audio-visual speech enhancement

Robust and Efficient Deep Learning based Audio-VIsual Speech Enhancement

Audio-visual speech enhancement
REAVISE (2023-2026, grant ANR-22-CE23-0026) is a project funded by the Agence Nationale de la Recherche (ANR) via the JCJC program. The project aims to develop a unified audio-visual speech enhancement (AVSE) framework that robustly integrates acoustic data (noisy speech signal) with accompanying visual information (speaker’s lip movements) in order to recover an intelligible, high-quality estimate of the clean speech signal with low computational power and independently of the acoustic and visual noise environments.

Project publications

Publications associated with REAVISE are also available in the HAL–ANR collection.

  1. Colombe Mboungou, Mostafa Sadeghi, Jean-Eudes Ayilo and Romain Serizel, Audio-visual Contrastive Alignment for Diffusion-based Visual-conditioned Speech Enhancement, INTERSPEECH, 2026. [Paper]

  2. Jean-Eudes Ayilo, Mostafa Sadeghi, Romain Serizel and Xavier Alameda-Pineda, Diffusion-based Frameworks for Unsupervised Speech Enhancement, IEEE Transactions on Audio, Speech and Language Processing, 2026. [Paper] [DOI]

  3. Mostafa Sadeghi, Jean-Eudes Ayilo, Romain Serizel and Xavier Alameda-Pineda, Posterior Transition Modeling for Unsupervised Diffusion-Based Speech Enhancement, IEEE Signal Processing Letters, 2025. [Paper] [DOI]

  4. Jean-Eudes Ayilo, Mostafa Sadeghi, Romain Serizel and Xavier Alameda-Pineda, Diffusion-based Unsupervised Audio-visual Speech Enhancement, ICASSP, 2025. [Paper]

  5. Berné Nortier, Mostafa Sadeghi and Romain Serizel, Unsupervised speech enhancement with diffusion-based generative models, ICASSP, 2024. [Paper]

  6. Mostafa Sadeghi and Romain Serizel, Posterior sampling algorithms for unsupervised speech enhancement with recurrent variational autoencoder, ICASSP, 2024. [Paper]

  7. Jean-Eudes Ayilo, Mostafa Sadeghi and Romain Serizel, Diffusion-based speech enhancement with a weighted generative-supervised learning loss, ICASSP, 2024. [Paper]

[1] Z. Kang, M. Sadeghi, R. Horaud, and X. Alameda-Pineda, Expression-preserving face frontalization improves visually assisted speech processing, International Journal of Computer Vision (IJCV), 2022. [Paper]

[2] M. Sadeghi, S. Leglaive, X. Alameda-Pineda, L. Girin, and R. Horaud, Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoders, IEEE Transactions on Audio, Speech and Language Processing, vol. 28, pp. 1788- 1800, May 2020. [Code]

[3] M. Sadeghi and X. Alameda-Pineda, Mixture of Inference Networks for VAE-based Audio-visual Speech Enhancement, IEEE Transactions on Signal Processing, vol. 69, pp. 1899-1909, March 2021. [Code]

[4] M. Sadeghi and X. Alameda-Pineda, Switching Variational Auto-Encoders for Noise-Agnostic Audio-visual Speech Enhancement, in IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), Toronto, Ontario, Canada, June 2021. [Slides, Poster]