MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis

0. Contents

1. Abstract
2. Demos -- Emotional speech synthesis by transferring the emotion from reference audio
3. Demos -- Emotional speech synthesis by predicting the emotion from input text
4. Demos -- Emotional speech synthesis by manual control
5. Demos -- Global-level emotion presenting module (GM)
6. Demos -- Utterance-level emotion presenting module (UM)

1. Abstract

Expressive synthetic speech is essential for many human-computer interaction and audio broadcast scenarios, and thus synthesizing expressive speech has attracted much attention in recent years. Previous methods performed the expressive speech synthesis either with explicit labels or with a fixed-length style embedding extracted from reference audio, both of which can only learn an average style and thus ignores the multi-scale nature of speech prosody. In this paper, we propose MsEmoTTS, a multi-scale emotional speech synthesis framework, to model the emotion from different levels. Specifically, the proposed method is a typical attention-based sequence-to-sequence model but with proposed three modules, including global-level emotion presenting module (GM), utterance-level emotion presenting module (UM), and local-level emotion presenting module (LM), to model the global emotion category, utterance-level emotion variation, and syllable-level emotion strength, respectively. In addition to modeling the emotion from different levels, the proposed method also allows us to synthesize emotional speech in different ways, i.e., transferring the emotion from reference audio, predicting the emotion from input text, and controlling the emotion strength manually. Extensive experiments conducted on a Chinese emotional speech corpus demonstrate that the proposed method outperforms the compared reference audio-based and text-based emotional speech synthesis methods on the emotion transfer speech synthesis and text-based emotion prediction speech synthesis respectively. Besides, the experiments also show that the proposed method can control the emotion expressions flexibly. Detail analysis is conducted to show the effectiveness of each module and the good design of the proposed method.

2. Demos -- Emotional speech synthesis by transferring the emotion from reference audio

Corresponding to Section 5.1 in our paper, below lists the samples that are synthesized for evaluations on the emotion transfer task. We compared MsEmoTTS (proposed) with the GST model on both parallel and non-parallel emotion transfer.

Parallel emotion transfer: the input text is the same as the transcription of reference audio.

Non-parallel emotion transfer: the input text is different from the transcription of reference audio.

Short summary: Experiment results for both parallel and non-parallel emotion transfer demonstrate that the proposed method can convey emotion information from the reference audio effectively, and achieves superiority on the reference audiobased emotional speech synthesis compared with the GST model.

3. Demos -- Emotional speech synthesis by predicting the emotion from input text

Corresponding to Section 5.2 in our paper, below lists the samples that are synthesized for evaluations on emotion prediction from input text only. We compared MsEmoTTS (proposed) with the TPSE-GST model.

Short summary: The higher emotion naturalness MOS scores of MsEmoTTS than TPSE-GST demonstrate that MsEmoTTS is suitable for emotional speech synthesis by predicting the emotion from input text only.

4. Demos -- Emotional speech synthesis by manual control

Corresponding to Section 5.3 in our paper, below lists the samples that are synthesized for presenting the ability of emotional speech synthesis by manual control. For the same input text, different global emotion categories and local emotion strengths are utilized to synthesize emotional speech with different expressions. ''All strengths 0'' and ''All strengths 1'' below represent all syllable strengths in the utterance are appointed to 0 or 1 for the manual labels in emotional TTS. ''Strength increasing'' and ''Strength decreasing'' below represent local emotion strengths gradually increase or decrease within an utterance.

Text: 舒家姐妹现在的人气很高啊。(English: The sisters of Shu are very popular now.)

Emotion	All strengths 0	All strengths 1	F0 curves
Happiness
Anger
Sadness
Surprise
Fear
Disgust

Emotion	Strength increasing	Strength decreasing	F0 curves
Happiness
Anger
Sadness
Surprise
Fear
Disgust

Short summary: The results for all emotions indicate that the local emotion strength and global emotion category of synthetic speech can be manually controlled successfully.

5. Demos -- Component analysis - Global-level emotion presenting module (GM)

Corresponding to Section 6.1 in our paper, below lists the samples that are synthesized for evaluations on GM. We compared the consistency of the generated emotion expressions with input text of our proposed ''soft'' emotion embeddings and the conventional ''hard'' emotion embeddings in emotion prediction. ''P-h'' represents the model as same as our proposed model except the global emotion renders are conventional ''hard'' emotion embeddings.

Short summary: The emotion naturalness MOS scores indicate the effectiveness of the proposed soft emotion embedding method.

6. Demos -- Component analysis - Utterance-level emotion presenting module (UM)

Corresponding to Section 6.2 in our paper, below lists the samples that are synthesized for evaluating the effectiveness of UM. ''w/o'' means without and ''w'' mean with. We compare the similarity of generated emotion expressions from reference audio on the emotion transfer task.

Parallel emotion transfer

Non-parallel emotion transfer

Short summary: The results of CMOS and A/B preference test indicate the effectiveness of modeling the utterance-level variation for the emotional speech synthesis, and also demonstrates the good design of the proposed UM.