Accent-Emotion Entanglement in LM-Based TTS Synthesis

To both quantify accent-emotion entanglement and validate its existence, a listening test was conducted. 29 participants were asked to rate the similarity between a synthesised utterance and the ground truth (taken from the MEAD dataset (Wang et al, 2020)) exclusively considering accent. The reference speech provided to each TTS system matched the ground truth in speaker and emotion. CosyVoice 2 accepts fine-tuned instructions where MaskGCT does not. For the happy/angry condition, the instruction given was 'Make this sound angry/happy'. For neutral utterances no instruction was given. The rating system used a 1-5 scale, with lower scores representing lesser similarity. The results of this test are shown in the table below and are visualised above

These results confirmed that CosyVoice 2 often fails to accurately represent the accent of the reference speaker. Happy utterances seemed particularly prone to accent hallucination, with an average SMOS of just 1.84. All 3 emotions though saw low scoring utterances suggesting that accent hallucination was not only caused by certain emotions. Instead, listening to the utterances revealed that nature of the accent hallucination did seem to be guided by the emotion of the input. A selection of the utterances used in the listening test can be heard at the bottom of the page while a preview of the listening test can be seen here (redacted too keep anonymous).

To further investigate the nature of the issues in CosyVoice 2 and to establish the existence of accent-emotion entanglement over random accent hallucination, sythesised utterances were analysed using a selection of objective metrics. 100 utterances were synthesised for each emotion (angry/happy/neutral), target text (5 total), speaker (2 male, 2 female) and system (CosyVoice 2 and MaskGCT). Each utterance was then passed through an accent recognition system, GenAID (Zhong et al, 2025), to produce accent embeddings. These embeddings can be seen in the graphs above, reduced to 2 dimensions through UMAP projection. Ground truth accents were taken from the VCTK (Yamagishi et al, 2020) and CommonAccent (Zuluaga-Gomez et al, 2023) databases to provide context. These visualisations support the listening test resutls and show how the emotion of the input provided to CosyVoice 2 impacts the accent of the output. While the majority of utterances have a General American accent, regardless of the accent of the reference speaker, clusters of outliers can be seen for each emotion. Many angry utterances, for example, are given African accents. This does not occur for neutral input which instead sees a cluster of British-accented utterances. Where the accents of utterances produced by MaskGCT are generally close to their ground truth equivalent, CosyVoice 2 utterances follow a distinct, highly dispersed pattern regardless of reference speaker. This suggests that it is not exclusively some element of the reference speech that is leading to the accent hallucination.

The greater spread of CosyVoice 2 utterance accents is quantified in the table above. For every speaker and emotion, CosyVoice 2 utterances show the greatest average cosine distance from their respective centroid. CosyVoice 2 produces a wide variety of accents for each speaker and emotion. This aligns with the visualisations and the listening test results. A key motivator for this project was to use a variety of evaluation metrics which come together to produce a clear picture of accent-emotion entanglement. The goal was for it to be clear what each metric captures and how they are grounded in human perception. Many systems now present results for metrics such as speaker similarity, handily captured in a single number for each condition. However, upon deeper analysis it is often not obvious what exactly these measures tell us. Can we truly capture speaker similarity in a single number? Evaluation through metrics which are either poorly described, under-utilised or overburdened does not paint the whole picture, leading to unexpected phenomena such as accent-emotion entanglement slipping through the cracks.

In addition to studying the accent of each utterance, their emotion was also evaluated. This knowledge was key in determining the exact nature of the accent hallucination. Was accent in some way replacing the expected emotion or did they conincide? Analysis was conducted in much the same way as with accent. Each of the 100 utterances per condition were passed through emotion2vec (Ma et al. 2023) to produce an emotion embedding. These embeddings were then reduced in dimension through UMAP and plotted for each system. These visualisations can be seen above. Additionally, during the listening test, participants were asked to rate the similarity of a synthesised utterance's emotion to its ground truth equivalent, just as with accent. A summary of these results are shown in the table below.

The visualisations align with listening test findings, showing that although performance is largely similar, CosyVoice 2 does slightly improve over MaskGCT in terms of percieved emotion similarity, particularly for neutral and angry utterances. This suggests that hallucinated accents aren't in some way replacing emotions during synthesis, instead the 2 features develop separately. No significant correlation was found between accent and emotion SMOS scores for CosyVoice 2 utterances with a Pearson correlation of r = 0.073. MaskGCT does however see a weak positive correlation with a Pearson correlation of r = 0.393. This is likely due to deliberately poor utterances being chosen. Utterances were chosen for the listening test based on cosine similarity between the accent embeddings of the synthesised utterance and the ground truth, with the lowest scoring utterances picked. Because there was little to no accent hallucination using MaskGCT low scoring sentences seem to be those which are more generally poor formed, e.g. they sounded robotic. Such utterances may also be likely to perform poorly on emotion similarity. The key takeaway from these results is that the emotion of a CosyVoice 2 utterance is not impacted by accent hallucination. Accent-emotion hallucination only works one way.

Below are a selection of the utterances, ground truth and synthesised, used in the listening test.