Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

Thank you for your detailed comment and constructive suggestions. We appreciate the time and effort you have invested in reviewing our submission.

Weaknesses:

The association between generated text and speech can be further evaluated.

When obtaining predicted transcription and continuation y_hat, we cannot split it to the prompt transcription and the continuation transcription. For the predicted text sequence, we do not have a way to know where the switch between transcription and continuation happens. Thus, we cannot compute WER against prompt only or against continuation only. However we measured the ASR performance of the proposed model trained exclusively for ASR without speech continuation, the results were a WER of 3.1 / 4.9 on Librispeech test-clean / test-other.

In addition, replacing the text with a sequence of null tokens of the same length can help showcase the impact of text on semantic quality.

We have performed this analysis, we replaced the text with a sequence of null tokens, with a sequence of characters (for example “a a a a”) and with some other arbitrary token. We see consistent results, that when doing this the speech continuation of the model contains mumbling or long human voices. We can thus conclude that indeed the text has an impact on the semantic quality of the generated speech.

Questions:

Could you please provide additional information about the training and inference in the Spoken QA task? Does Spectron undergo training for the spoken QA task, whereas the baselines, such as AudioLM and GSLM, does not?

No specific training has been done on spoken question answering. Spectron was only trained for speech continuation as described in Section 3.2. AudioLM, GSLM and TWIST also haven't been explicitly trained for spoken question answering. SpeechGPT, on the other hand, was trained using a carefully curated multimodal question answering dataset. We have added a column to Table 4.

In the provided Spoken QA demos, SpeechGPT tends to generate lengthier and more detailed answers, while SPECTRON prefers more concise responses. Could the comparable performance of both models be attributed to the nature of the QA task, such as the prevalence of shorter answers in the test set?

Spectron hasn't been trained using spoken QA datasets, we only used those datasets for evaluation. SpeechGPT was explicitly trained for question answering using carefully curated spoken QA dataset, therefore its answers are more detailed. Also, note that the evaluation strategy of the spoken QA datasets does prefer short answers. We define that an answer is correct by checking if the correct answer is contained in the model output. For example, if the model output is “The capital of France is Paris” and the correct answer is “Paris” we say that the model answered the question correctly. Thus, long and detailed answers have a higher chance of containing the correct answer inside of their response.

Thank you for your detailed comment and constructive suggestions. We appreciate the time and effort you have invested in reviewing our submission.

Weaknesses:

I don't see major weaknesses. It would be great if the authors can also compare their system with OpenAI voice-version ChatGPT (https://openai.com/blog/chatgpt-can-now-see-hear-and-speak).

Thank you for the suggestion. It's important to note that the voice-version of ChatGPT was released post the abstract submission deadline and in close proximity to the final paper submission. Additionally, the detailed model architecture, performance metrics, and training methodology of the voice-version ChatGPT have not been officially published. Hence, directly comparing to the voice-enabled version of ChatGPT is not straightforward, despite its impressive capabilities. Nonetheless, we believe that the comparison between these two systems holds significant interest and should be addressed in future work.

Questions:

Will Spectron still work when using a LLM that has 100B parameters? Currently, Spectron uses 1B language model (PaLM 2), and that is much smaller than state-of-the-art LLMs.

We experimented with a larger LLM, however due to computational constraints we were not able to complete the training process. We think that Spectron will also work well with a larger LLM, though more data or parameter freezing may be needed to avoid overfitting.

In addition, what happens if you use Whisper (https://cdn.openai.com/papers/whisper.pdf) to transcribe Libri-Light and obtain training transcripts y?

Whisper obtains a 2.7 / 5.2 error rate on Librispeech test-clean/test-other. The model used for the transcription of the Librilight dataset is the NST model. It achieves a WER of 1.7/3.4 on Librispeech test-clean/test-other datasets. We thus believe that using the Whisper model to transcribe that dataset shouldn’t have a significant effect on the performance of the system.

Thank you for your detailed comment and constructive suggestions. We appreciate the time and effort you have invested in reviewing our submission.

Weaknesses:

The MOS evaluation of synthesis is bereft of details

In the MOS experiments 30 raters rated 20 randomly sampled examples. We employed MOS experiments to gauge the naturalness of the synthesized continuous speech. raters were not exposed to the initial prompt. To further clarify this issue, we changed the name of this metric in the revisioned paper to N-MOS (N for naturalness). We report Naturalness MOS as only one of many axes of evaluation; e.g. we transcribed the resulting speech and measured both perplexity and accuracy using our methods.

The CoT idea is interesting as I mentioned before

We conducted experiments involving the introduction of errors into the ASR transcription and text continuation to examine the interconnectedness of the decoding process. Our observations confirm a clear interdependence among the outputs. Specifically, introducing errors into both the text continuation and ASR transcription resulted in notably inferior speech generation outcomes. Consequently, we conclude that the cohesion of the entire process significantly influences the model's performance. We believe that this issue warrants further attention in future research.

Questions:

No details are provided on MOS tests - could the authors share some details ?

In the MOS experiments 30 raters rated 20 randomly sampled examples. We use the MOS experiments to measure the naturalness of the synthesized continuation speech without the prompt. We have added these details to the revised version.

What was the WER of the Librispeech model used to obtain pseudo-labels on LibriLight ? Do the authors have any idea about label quality ?

The Librispeech model that was used to obtain the pseudo-labels is the NST model. It achieves a WER of 1.7/3.4 on Librispeech test-clean/test-other datasets. From manually sampling some of the training data, it seems that the quality of the pseudo labels is high.

The paper says "we use TTS to synthesize questions that fit within this duration" - what does this mean? Do you truncate long questions or modify the durations of phones/words to fit within 3s? Or do you drop questions that are longer than 3s?

We drop questions that are longer than 3s. Since previous methods (GSLM and AudioLM) use only 3s prompts, we only considered questions whose synthesized spoken version was shorter than 3s.

I thank the authors for their response, and for addressing the questions and weaknesses.

1. I still believe that the manner in which MOS was used here is not ideal -> raters should have been exposed to the initial prompt and asked to rate how natural the continuation was as opposed to just the audio.

2. 20 samples are likely small for such tests.

3. The response to CoT is appreciated, but some analysis and numbers in the paper would substantiate the author's motivation behind choosing this CoT style of input.

Questions:

Can you evaluate the ASR performance of this method?

For the ASR task, we have computed the WER between the generated text and the original labels. As you state, it is important to note that the text also contains the continuation, and thus, the WER is not ideal. A large portion of the error comes from the continuation since there is a large variance in text continuation. When comparing a text continuation to a transcript, even a deviation of a single token will mean that the whole sequence that comes after the error is wrong with respect to the ground truth transcription. However, we measured the ASR performance of the proposed model trained exclusively for ASR without speech continuation, the results were a WER of 3.1 / 4.9 on Librispeech test-clean / test-other. For TTS, we are currently exploring the transformation of Spectron into TTS and aim to provide an update shortly.

Why is it 3 seconds? Would it robustly work even if we throw more than (or less than) 3-second spoken prompts?

In order to be consistent with the previous work (GSLM and AudioLM), we use prompts of 3 seconds or less during testing.

Similarly, how did you deal with sentences that are less than 3 seconds during training? Can you discard them or concatenate neighboring sentences to make it longer than 3 seconds?

We discard sentences shorter than 3 seconds during training. Note that in the LibriLight and Librispeech the amount of examples that are less than 3 seconds are only 0.04% and 2.73% respectively. Thus, discarding these examples is a minor change to the datasets.

How about combining multiple sentences and using the original first sentence as a chunk instead of a 3-second chunk? In this case, we would also have access to the transcription corresponding to the spoken prompt part easily. We can have more precise control of the text output to perform ASR inside the framework explicitly.

Thanks for the suggestion. We focused on continuations of 3-second prefixes to match past work, but we can try this in future work regarding the use of the Spectron framework as an ASR or TTS system.

Around equation (3), $\mathcal{P}_s$: why does $\mathcal{P}_s$ depend on position $s$?

Thanks for finding this typo, we removed the $s$ from the projection layer $\mathcal{P}$.

In section 4.1, the last sentence, "For semantic and acoustic quality, Spectron was trained with a model of 350 million parameters, while for the question answering task, Spectron was trained with a model of 1 billion parameters," I could not understand it. Do you mean you use 350M models for Sections 4.2.1 and 4.2.2 and 1B models for Section 4.2.3? How are they different?

Thank you for the comment. Yes, we mean that we use the 350M LM for Sections 4.2.1 and 4.2.2 and 1B LM model for Section 4.2.3. All other configurations of the models are identical in both methods. We did this to better compare Spectron in the audio continuation task since other audio continuation methods use smaller models. We wrote it explicitly in the revision of the paper.

Is the question-answering task performed with zero-shot without fine-tuning it? Please clarify it.

The question answering task is performed without fine-tuning; it is performed zero-shot. Spectron was only trained for speech continuation as described in Section 3.2. The world knowledge encapsulated in the pre-trained LLM allows Spectron to correctly answer spoken questions.

Thank you for your detailed comment and constructive suggestions. We appreciate the time and effort you have invested in reviewing our submission.

Weaknesses

The paper lacks the reproducibility and accessibility of the results:

The release of these upstream pre-trained models and code built atop them are beyond our control. We tried to mitigate this issue in other aspects; beyond training and (to-be released) test sets from public data, we also provide a full description of the model architectures, novel objectives and hyperparameters used in the training (Appendix A.1). With these, we believe our key contribution is reproducible with our test suite and the many public LLMs, ASR encoders, and vocoders (e.g., LLaMA, wav2vec2 CTC, HiFiGAN), especially as our method’s semantic improvements are a step-function over the other methods (except for the concurrent SpeechGPT work), and our acoustic improvements are a step-function over SpeechGPT. We welcome suggestions on how to further improve reproducibility for this discussion period and camera review.

Due to the above issue, it is not clear whether the superiority of the proposed method:

Thank you for the comment. In order to evaluate the origin of our models improvement we compare the pre-trained models that each method use:

LLM pretrained model - Our method is based on the 1B parameters LLM whereas SpeechGPT and TWIST utilize a 7B parameters model. LLM performance is well known to be strongly influenced by the model size. Therefore, we do not believe that the language model used in our method is more powerful than the language models used in SpeechGPT or TWIST methods. In comparison with AudioLM, we use a comparable model size. We acknowledge that the GSLM model is smaller than our model. Moreover, SpeechGPT is fine-tuned with cross-modal instruction objectives derived from carefully curated speech-text datasets. TWIST must learn its speech capability from scratch, and so they use approximately 150k hours of speech-text datasets. Our method exhibits competitive results for multiple tasks using only the public Libri-Light dataset (60k hours) for adaptation.

ASR / Tokenization - TWIST, SpeechGPT and GSLM all use the HuBERT model for tokenization. On the ASR task, from the SUPERB paper, which uses the same type of representation, the Hubert-Large model achieves a WER of 2.94 on the LibriSpeech test-clean test set. In comparison, the USM model we used has a performance of 3.1 on the test-clean test sets. Thus, we do not think that the speech-encoder we have used is more powerful than the open-source methods or any of the encoders used in the methods we have compared against.

Clarity: Section 2 requires more improvements.

Thank you for the comment. We re-wrote Section 2 based on your suggestions (e.g., model architecture, training method, pre-training data, etc.). The re-written section can be viewed in the updated version of the paper. Additionally, extra information about the related work can be found in Appendix A.2.

We have successfully completed the TTS experiment for Spectron. In this experiment, we utilized the prompt for acoustic conditioning and manipulated the text in the CoT to match the requested text. The N-MOS results for this experiment are 3.40±0.31, which is a fair outcome considering that the system was not specifically trained for TTS.