This paper proposes a neural end-to-end text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, it is capable of consistently synthesizing high-quality clean speech regardless of the quality of the training data for the target speaker.
Building a TTS model with Gaussian Mixture VAEs enables fine-grained control of speaking style, noise condition, and more. Describes the conditioned GAN model to generate speaker conditioned Mel spectra by augmenting the z-space corresponding to the identification This paper proposes a two layer latent variable model to obtain disentangled latent representation, thus facilitating fine-grained control over various attributes This paper proposes a model that can control non-annotated attributes such as speaking style, accent, background noise, etc.