We sincerely thanks for your positive recognition of our work. Our point-by-point responses to all your comments are provided below.

Q1: In Table 3, it might have been helpful to include comparisons with a wider variety of voice interaction models.

Thank you for your suggestion. We have consulted the results of models such as LLaMA-Omni [1] and MiniCPM-o [2] on the SQA task, and we will incorporate their results into the revised version of the paper.

Q2.1: Does the Sensivoice encoder in the Plus model provide continuous embeddings or discretized tokens as input?:

In the VITA-Audio-plus, the SenseVoice [3] encoder provides continuous embeddings as input.

Q2.2: What kind of information do glm-4-voice tokens encode?:

The GLM4-Voice [4] tokenizer is capable of encoding acoustic features such as intonation, stress, and rhythm.

Q3: Does it support multi-round interactions?

VITA-Audio supports multi-turn conversations. In Stage 4, we trained using the VoiceAssistant-400K [5], which includes multi-turn audio dialogues. We will update the data format in the appendix in the revised version to avoid any misunderstandings.

Q4: How many audio tokens are typically accumulated before audio is reconstructed?

We found that GLM4-Voice struggles to encode very short audio tokens (e.g., 2 audio tokens). Based on our experience, typically 8 audio tokens are a suitable choice. This is also the reason we choose the interleaving ratio for the Boost and Balance inference modes. The Balance mode requires two forward passes to generate enough audio tokens for decoding, while the Boost mode only requires one forward pass.

Q5: Is there any evidence showing how much the original text understanding capabilities of the LLM are affected by training with interleaving or parallel generation schemes?

We have tested VITA-Audio after aligning with ASR and TTS tasks on several text modality benchmarks. After the ASR and TTS alignment, the model’s original text understanding capabilities were indeed affected. However, this might be due to the lack of large-scale, high-quality text data used during training. Interestingly, we observed a slight improvement in performance on GSM8K, which may be due to the fact that our training dataset included a significant amount of math-related data. We will include this result in the revised version.

Q6: Was there a particular reason for not using an Instruct-tuned version?

We used Qwen-7B-Instruct as the pre-trained text LLM. We will clarify this point in the revised version.

[1] Fang Q, Guo S, Zhou Y, et al. Llama-omni: Seamless speech interaction with large language models.

[2] Yao Y, Yu T, Zhang A, et al. Minicpm-v: A GPT-4v level mllm on your phone.

[3] An K, Chen Q, Deng C, et al. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and LLMs.

[4] Zeng A, Du Z, Liu M, et al. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot.

[5] Xie Z, Wu C. Mini-omni: Language models can hear, talk while thinking in streaming.

Sincerely thanks for your efforts in reviewing this work. We hope the detailed responses help clarify your concerns. We would greatly appreciate it if you could kindly re-evaluate our work in light of the new explanations and additional results.

Q1: Choosing the Number of MCTP Modules:

The number of MCTP modules is positively correlated with inference acceleration; however, it may also come with an increased risk of error rate. After a comprehensive evaluation, we selected the configuration with 10 modules to achieve the optimal balance between efficiency and accuracy.

Q2: VITA-Audio Data Format.

We will take the inference stage as an example, and denote $Y_t$ as the $t$-th token, $h_n$ and $h^i_n$ as the $t$-th hidden state from LLM and MCTP-$i$, respectively.

Without considering KV cache, LLM takes all previous $t$ tokens $( Y_0, Y_1, ..., Y_{t-1} )$ as input and predicts the next token $Y_{t}$. Note that LLM also generates hidden state $( h_0, h_1, ..., h_{t-1} )$.

Then the first MCTP-1 futher takes $n$ tokens $( Y_1, Y_2, ..., Y_{t} )$ with the hidden states $( h_0, h_1, ..., h_{t-1} )$ as input and predict token $Y_{t+1}$. And MCTP-1 also generates hidden state $h^1_{t}$ for token $Y_{t}$.

The second MCTP-2 takes the all $n$ tokens $( Y_2, Y_3, ..., Y_{t+1} )$ with the hidden states $( h_1, h_2, ..., h_{t-1}, h^1_{t} )$ as input and predict token $Y_{t+2}$.

After one forward pass of LLM and all MCTPs, the model outputs 11 tokens $( Y_{t}, Y_{t+1}, ..., Y_{t+10} )$. Figure 2 may cause some misunderstandings, and we will correct this in the revised version.

During training, the loss function is formatted as:

$\mathcal{L} = - \sum_{t=1}^{T} \log P$ Y_t | {Y_0, Y_1, ..., Y_{t-1} } $+ \log P_1$ Y_{t+1} | {Y_1, Y_2, ..., Y_{t}, h_0, h_1, ..., h_{t-1} } $+ \log P_2$ Y_{t+2} | {Y_2, Y_3, ..., Y_{t+1}, h_1, h_2, ..., h^1_{t} } $+ ... + \log P_{10}$ Y_{t+10} | {Y_10, Y_11, ..., Y_{t+9}, h_9, h_{10}, ..., h^9_{t+9} } $$.

We set the ratio of text tokens to be roughly $1:2$ following recent TTS and SLM literature (GLM4Voice [1] and CosyVoice2 [2]), which causes the audio stream to lag behind the text stream. During both inference and training, we use $n$ text tokens and $2n+2$ audio tokens as an interleaved group where the additional two tokens correspond to special audio tokens: <|begin_of_audio|> and <|end_of_audio|>.

Q3: The variants of VITA-Audio.

The variants of VITA-Audio come from two aspects:

1. Two Models with Different Architectures: VITA-Audio and VITA-Audio-Plus.

VITA-Audio is designed as a universal paradigm to benefit various interleaved Speech LLMs. To validate the broad applicability of this framework, we conducted experiments on different architectures of interleaved speech LLMs. The results show that both VITA-Audio-Plus, based on SenseVoice, and VITA-Audio, based on GLM4-Voice, significantly improve the model's accuracy and inference speed by introducing the MCTP module, demonstrating the compatibility of this approach with different model architectures.

2. Four Different Inference Modes: Turbo, Boost, Balance, and Vanilla.

We have designed four different inference modes: Turbo, Boost, Balance, and Vanilla, to be applied in different scenarios. The core distinction between these modes lies in the configuration of inference parameters (the number of activated MCTP modules and the ratio of interleaved text and audio token generation). As a result, we do not need to train a separate model for each mode. Of course, targeted optimization can still be carried out for the desired inference mode if the application scenario is specified.

Q4: Differences Between MCTP and Speculative Decoding:

Existing speculative decoding methods (such as Medusa[3]) typically require verifying all candidate tokens to ensure the correctness of the generated results. And the verification step significantly slows down the decode speed. If the verification step is omitted, the accuracy rate of the generated results tends to be low. In high-concurrency or resource-limited environments, the acceleration benefits of speculative decoding diminish significantly.

In contrast, the MCTP module aims to reduce the latency for generating the first audio chunk in streaming scenarios, and does not require verification. Our instantiated MCTP generates up to ten audio tokens at once. More importantly, this module has a very low computational cost, ensuring stable and significant interaction acceleration even in high-concurrency or resource-constrained settings.

It is important to emphasize that speculative decoding methods and MCTP are not mutually exclusive. In practical applications, speculative decoding can be used to accelerate text generation, while the MCTP module can be employed during speech generation, further improving overall end-to-end inference speed.

[1] Zeng A, Du Z, Liu M, et al. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot.

[2] Du Z, Wang Y, Chen Q, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models

[3] Cai T, Li Y, Geng Z, et al. Medusa: Simple LLM inference acceleration framework with multiple decoding heads.

Thank you for the clarifications. Overall, I would like to maintain my score due to:

1. The clarity issues would require a major revision.
2. The novelty is considered limited for me—integrating MCTP into the joint SLM framework appears to be a straightforward latency enhancement at the expense of performance (Boost/Turbo vs. Vanilla), and I found no particularly intriguing or surprising empirical results.

However, I do appreciate the fact that the authors decide to open-source the model, which should be one of the strengths that I have not mentioned in the review previously.

We sincerely thanks for your positive recognition of our work. Our point-by-point responses to all your comments are provided below.

Q1: Can the authors provide a translation for all non-English characters to ensure everyone can understand the examples in Figure 1?

Thank you for your suggestion.

In Chinese, "一行" (yihang) emphasizes spatial arrangement (e.g., 一行白鹭 - a row/line of egrets), while "一行" (yīxíng) focuses on the concept of a group or unit (e.g., 一行人 - a group/party of people).

In the revised version, we will provide translation annotations for all non-English characters to ensure smooth understanding for the readers.

Q2: Detailed supporting results for the hypothesis that, with irrelevant text tokens being masked out, the model is still able to generate the correct audio, and the pronunciation remains contextually appropriate.

Although we have not quantitatively tested this observation on a specific benchmark, we have verified it through several cases. During inference, we first locate the audio tokens that the model attends to most in the attention map for a given text token, and then perform text-token masking before generating those audio tokens. Specifically, we generate audio tokens with three different masking strategies:

(1) No text token is masked at all.

(2) All irrelevant text tokens except the target text tokens are masked.

(3) All text tokens are being masked.

We then continue the generation after the masking. In this case, both (1) and (2) can produce normal speech.

For examples in Figure 1a), in the phrase "there are seven days in a week," the attended text is on "seven days," and in the phrase "一行白鹭上青天," the attended text is on "一行" (yi hang), meaning "One row of egrets ascends the blue sky," and both can produce correct pronunciation.

We are conducting quantitative experiments. Specifically, we plan to perform forced alignment on the generated speech to obtain the corresponding audio tokens corresponding to each text token. We then regenerate the speech and mask out all text tokens except for the relevant ones when reaching those audio token positions. We will assess whether the generated audio tokens are correct. Unfortunately, due to time constraints, we don't have the final result yet, but we will synchronize the results once they are available.

We sincerely thanks for your positive recognition of our work. Our point-by-point responses to all your comments are provided below.

Q1: Innovation of the MCTP Module.

1. Design Motivation: The design motivation for the MCTP module stems from our observation that the hidden states of certain text tokens in the LLM backbone contain sufficient semantic information for generating the corresponding audio tokens. Based on this, we aim to design a lightweight module capable of efficiently generating audio tokens from these hidden states. Potential candidates for this lightweight module include: a set of Transformer layers, a small-scale LLM, or reusing the final layers of the LLM by feeding the last-layer hidden states back to those layers. Ultimately, for the simplicity of the training pipeline, we choose to use a set of sequential Transformer layers as the core of the MCTP module.

2. Advantages over Speculative Decoding: Existing speculative decoding methods (such as Medusa[1]) typically require verifying all candidate tokens to ensure the correctness of the generated results. If the verification step is omitted, the accuracy rate of the generated results tends to be low. Even with optimizations like tree attention, the verification phase still requires inputting longer candidate sequences into the LLM for forward computation, and the input sequence length increases rapidly as the number of parallel predicted tokens grows. In high-concurrency or resource-limited environments, the acceleration benefits of speculative decoding diminish significantly. In contrast, the MCTP module does not require verification and can generate up to ten audio tokens at once. More importantly, this module has a very low computational cost, ensuring stable and significant inference acceleration even in high-concurrency or resource-limited settings.

3. Complementarity: It is important to emphasize that speculative decoding methods and MCTP are not mutually exclusive. In practical applications, speculative decoding could be used to accelerate text generation, while the MCTP module can be employed during speech generation, further enhancing the overall end-to-end inference speed.

Q2: Regarding Multispeaker Synthesis.

Currently, our main focus remains on ASR, TTS, and SQA tasks. Although speaker embeddings are used in the audio decoder, we have not yet conducted specialized training for such tasks.

Q3: Training method.

Our training method is divided into four stages, primarily based on the following two considerations:

1. Our goal is to make VITA-Audio a universal paradigm that can benefit all interleaved Speech LLMs. Specifically, the model can directly leverage the weights aligned during ASR and TTS tasks to quickly initiate the training of the second stage. To validate the generalization ability of the framework, we trained two versions, VITA-Audio and VITA-Audio-Plus, to demonstrate the effectiveness of this approach across different model architectures.

2. In early experiments, we attempted to train the MCTP module directly in Stage 1, or train a version containing 10 MCTPs in Stage 2, all at once. However, when the training data was insufficient, such approaches performed worse than the staged training strategy.

[1] Cai T, Li Y, Geng Z, et al. Medusa: Simple LLM inference acceleration framework with multiple decoding heads.

Thank you for your rebuttal. Your clarifications help address most of my concerns. I now feel confident that my initial positive assessment was fair and will maintain my original score.