ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models

2.4 Task selection rationales and consistency with motivation

Finally, we would like to comment on your general concerns about our evaluation task selection.

First, although we did not elaborate on this in the paper due to space limitations, we do have clear rationales behind our evaluation tasks. Specifically, there are three considerations for our task design:

1. We identified two important aspects of prosody, emotion and emphasis, which both serve key communicative purposes in human speech interactions, and which are covered in both speech analysis and generation benchmarks.
2. These two aspects, paired with the 'content->prosody', 'prosody->content', and 'prosody->prosody' settings, can be projected to the following canonical speech processing tasks.

3. We then need to adapt the tasks to audiobook settings that involve sophisticated semantics understanding and inform practical scenarios. Hence -- Expressive speech synthesis is adapted by blending the style specification in an audiobook context, hence deriving direct style specification and indirect style inference. -- Emphasis synthesis is adapted by embedding the direction of what to emphasize into a contrastive focus setting. -- Emotion recognition and emphasis detection are adapted into an audiobook completion task. -- Style cloning is adapted into an audiobook conversation between two characters with opposite styles, with reference styles provided as the previous round of conversation.

Second, the prosody->prosody task in the paper actually involves non-trivial semantic understanding capabilities, because it requires the model to switch among different styles based on the character identity. A common mistake of existing methods is that they tend to maintain the same style across different speakers, ignoring the character change. Other speech language models would sometimes mismatch the character styles (i.e., the style of Tom in the next paragraph is different from that in the previous one). This is our motivation for introducing the dialogue setting, instead of the monologue one.

Third, to improve the consistency between the motivation and the actual experiments, we plan to make two changes:

1. Add more motivating examples to the introduction. We will add a rising tone vs falling tone example to 'Prosody->Content' (as requested by Reviewer HdRD, question 9), and the aforementioned character consistency challenges to 'Prosody->Prosody'.
2. Add a new 'Prosody->Prosody' experiment closer to the intro motivation. Specifically, we use the emotion classification dataset (the JL corpus). For each utterance-emotion combination, we construct the following template

“The lord swims in the sea.” [Prosody (Ref Speech)] “I know.” Tom said. [___]
“The lord swims in the sea.” [Prosody (Ref Speech)] “I am sorry.” Tom said. [___]

We let ProsodyLM complete the prosody of the second quote. Our interesting observation is that, if the response quote is 'I know', its prosody varies greatly when the emotion of the reference quote changes. If the reference quote is angry, the reponse quote would sound angry and arrogant (example audio 1 and example audio 2 of 'I know' in response to angry); if the reference quote is sad, the response quote would sound soft (example audio 1 and example audio 2 of 'I know' in response to sad).

On the other hand, if the response quote is 'I am sorry', it would sound soft both when the reference quote is angry (example audio 1 and example audio 2 of 'I am sorry' in response to angry) and when the reference quote is sad (example audio 1 and example audio 2 of 'I am sorry' in response to sad).

2.1 Human verification of character descriptions in 'Indirect Style Inference'.

To test whether the character descriptions are truly indicative of the speaking styles, we add a human study where we present each one of the character descriptions to each subject and ask them to choose one from six speaking styles: high-voiced, low-voiced, fast, slow, loud, quiet. We invited 11 AI researchers (who are not authors nor familiar with this project) for the test. The confusion matrix is listed below, where each row corresponds to a description, and each column represents a perceived style.

As can be observed, subjects were able to choose the matching styles with almost 100% accuracy, except that one subject was confused between 'slow' and 'low'. This verifies that the descriptions are indicative of speaking styles perceived by human readers.

Meanwhile, Appendix D.3.2 lists all the character descriptions. We welcome readers to judge their alignment with the intended style.

2.2 Inter-model comparison in the 'contrastive focus' experiment.

Thank you for pointing this out, and you are absolutely right that different models may produce different ranges of prosody contours.

That is why we did not directly compare the absolute F0/energy contours among different models, but rather the relative prosody difference among on-focus, pre-focus, and post-focus words within the same model. More specifically, in Figure 2, what we are comparing are the relative height differences in the blue, red, and grey bars within the same group. As can be observed, only ProsodyLM and GPT-4o have significant height differences, and these differences are all greater than the confidence intervals marked by the black lines, which indicates statistical significance. The other models have little difference. On the other hand, different models do have different height ranges, but that is not our focus.

2.3 Metrics in emphasis detection and emotion recognition

Thank you for bringing this up! As also noted by Reviewer HdRD, one major complication of both emphasis detection and emotion recognition tasks is that the answers depend not only on the prosody of the quote, but also the content of the quote. Since the datasets are designed to be parallel (same content paired with different emotions/emphasis words), the prosody cues and content cues are often in conflict. Consider the following example (question 3 from Reviewer HdRD)

'And that to me was so exhilarating.' He/She emphasized ‘___’

The content of the quote would strongly suggest 'exhilearting' should be emphasized, which creates a significant bias towards certain words even if these words were not prosodically emphasized.

As a result, we need to introduce metrics other than simple F1 scores to remove such biases. That is why we applied the relative log-probability differences as our metrics. Specifically, take emphasis detection as an example, instead of measuring whether ProsodyLM outputs the emphasized word with the highest probability, we measure the probability difference of outputting a word when the word is emphasized v.s. when it is not. This would separate out the effect brought by prosody changes, controlling for the content influence (because the content is the same when calculating the relative probability difference). We did not include this rationale due to space limitations, but we will add it back to the paper.

I have read the authors' responses and other reviewers and discussions, the authors' clarifications make sense to me. I have raised my score accordingly.

6. Would prosody tokens affect performance on semantically-rich tasks?

Applying ProsodyLM to the intent classification (IC) is a non-trivial task, because recall that ProsodyLM was continuously pre-trained on audiobooks only, and thus does not have any instruction-following capabilities. This also makes comparison with a text-based instruction model quite an unfair comparison.

While we are investigating how to adapt IC to an audiobook setting, we would like to present some alternative results. Specifically, we evaluate the text-section perplexity on some validation audiobooks (unseen during continuous pre-training), and compare it with two text-only LLM baselines: 1) Llama3.1-8b-instruct, and 2) Llama3.1-8b-instruct continuously fine-tuned on the same set of audiobooks. The results are as follows:

As can be observed, ProsodyLM only increases perplexity by 2 compared to pure text LLM pre-trained on the same data, which indicates that introducing prosody tokens has only minor effects on text modeling capabilities.

7. Speaker-dependent information

Yes, the log-F0 for the speaker should be provided for both training and inference. The primary purpose of the log-F0 information is to distinguish male speakers from female speakers (males typically speak in lower F0s and females in higher). As a result, ProsodyLM is highly robust to fluctuations in the log-F0 values, as long as it still falls in a reasonable range for the speaker.

In one-shot generation settings where we do not have a global statistics of the speaker, we will specify speaker log-F0 as the average log-F0 of one example utterance of the speaker (we need at least one utterance of a speaker to replicate the voice). In very rare cases where no example utterance is available, we will just specify it as bin 150 if the speaker is male and bin 300 if the speaker is female.

Since log-F0 is a pretty generic information (and it is the only speaker-dependent information accessible to ProsodyLLM), ProsodyLLM would not 'memorize' certain speakers or certain speaker log-F0s. It works well for arbitrary unseen speakers.

4. Consistent Speakers vs. Inconsistence Speakers

This is possibly related to the confusion in question 1. Recall that our utterances were generated in different speaker voices, so to rule out the interference of speaker voices, it is most desirable if the utterances generated by the baselines are also in the same set of speaker voices. Unfortunately, not all baselines support generating in our specified speaker voices -- many baselines, especially commercial ones, only support their proprietary, high-quality voices. We refer to baselines that can support generating in the same speaker voices as ours as 'consistent speakers', and those that cannot as 'inconsistent speakers'. This was introduced in lines 290-291.

The reason we separated out two sets of experiments for consistent speakers and inconsistent speakers is that subjective evaluation is very sensitive to speaker differences. Many listeners may like an utterance simply because they like the speaker's voice. To minimize the interference, we separate out the inconsistent speaker settings from the consistent ones.

To make this more clear, we will change lines 290-291 into 'We conducted two sets of listening tests: 'Consistent speakers', which includes all the baselines that could synthesize the same speakers as ours, and 'Inconsistent speakers', including baselines that cannot. This could minimize the influence of speaker differences on perceived quality.'

5. 'Neutral' class excluded from emotion recognition experiment.

We excluded the 'neutral' class because it is a special, 'emotionless' emotion. Recall that we need to turn the emotion classification task into an audiobook setting, and it is a bit odd to say 'He/She was neutral'. Nevertheless, we follow your advice and add the neutral sentiment back to the data. The revised results are as follows.

As can be observed, the results are largely consistent with Table 3 in the main paper.

We would like to follow up regarding the intent classification (IC) experiment (your question 6). We evaluate our method on the test set of fluent_speech_commands_dataset. Recall that since ProsodyLM is not instruction-tuned, we need to modify the task as a next token prediction task (like Sections 4.3.1 and 4.3.2). Specifically, we adopt the following template for the three aspects of IC, action, object, and location, respectively:

'Turn off the lights.' [Prosody] Based on what the user said, the intention is to ___  
'Turn off the lights.' [Prosody] Based on what the user said, the object is ___  
'Turn off the lights.' [Prosody] Based on what the user said, the location is ___  

We then probe the ProsodyLM's output probabilities for a list of candidate continuations and see if the correct answer has the highest probability.

We include two baselines. 1) Llama3.1-8b-instruct, and 2) Llama3.1-8b-instruct continuously fine-tuned on the same set of audiobooks as ProsodyLM. The second baseline is almost the same as ProsodyLM except that the [Prosody] section is not present during fine-tuning, so it can best demonstrate the effect of the [Prosody] section on this task.

The following table shows the results.

As can be observed, ProsodyLM does not significantly underperform text-based LLMs, which implies introducing the [Prosody] section does not significantly impact the performance even on this task. Note that the results are not directly comparable with the reported results in existing literature because the former are adapted to the audiobook completion setting.

Meanwhile, since it is approaching the deadline of the discussion period, but no discussion has been initiated so far, we would like to follow up on whether your concerns can be adequately addressed. We are happy to engage in discussion with you and answer any further questions you might have. Thank you for your time!

Dear authors, thank you for the answers.

Regarding (4): The use of the word "speakers" suggests that the samples were spoken by human speakers. To avoid this misunderstanding, I would suggest replacing "consistent/inconsistent speakers" with "consistent/inconsistent voices".

Regarding (6): The results on perplexity are interesting. However, it is worth noting that perplexity does not always correlate with downstream task performance, see e.g. Levy et al., 2024; Liu et al., 2023:

Hong Liu, Sang Michael Xie, Zhiyuan Li, Tengyu Ma. "Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models", Proceedings of the 40th International Conference on Machine Learning, PMLR 202:22188-22214, 2023.

Levy, Mosh, Alon Jacoby, and Yoav Goldberg. “Same Task, More Tokens: The Impact of Input Length on the Reasoning Performance of Large Language Models.” In _Proceedings of the ACL (Volume 1: Long Papers), 2024.

Dear authors, thank you for the detailed answers. I have changed my rating accordingly.