PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction

W1. A detailed latency breakdown for each system component would help substantiate the primary source of the observed speedups.

We already provide two metrics: Time-to-First-Sentence (TTFS) and Latency in Table 1. TTFS corresponds to LLM latency, and Latency is the total latency (LLM + TTS). Thus, TTS latency = Latency - TTFS. The detailed breakdown is as follows:

PredGen reduces latency across all components. Although it does not reduce the time TTS takes to convert text to audio, it avoids running TTS when speculative answers are accepted early.

W2 A potential significant concern arises from the experimental methodology, which simulates streaming prompts via text segmentation rather than utilizing actual outputs from a streaming ASR system.

We conduct additional experiments using real ASR outputs. We report the results of clean text input and noisy ASR inputs as below. To obtain the noisy ASR inputs, we use Whisper-Live library to transcribe synthetic audio created using Chatterbox TTS. Notably, Whisper-Live simulate a real-time audio stream by progressively sending audio inputs in chunks of 2048 bytes to the ASR backend, hence, its output reflects real use cases where the text outputs exhibits a "self-correction" pattern. For example, a list of partial prompts $P_1,P_2...$ (described in Sec 3) can be [ "what is", "what is one...", "what is one plus", "what is one plus two...", "what is one plus twenty...", "what is 1 + 23." ]

Note that under this setup no change is required to our Algorithm 1 because it does not explicitly assume $P_{i}$ is a prefix of $P_{i+1}$. We can still perform the verification step by concatenating $P_{i+1}$ with the previously generated answer of $P_{i}$ regardless of their relations. The only detail is that we can no longer reuse the entirety of the KV cache of $P_{i}$. Instead, we reuse the KV cache of the common prefix between $P_{i}$ and $P_{i+1}$.

We report the latency and score on LMsys dataset, as well as the relative speed up, using the clean text input and real ASR stream. We note that introducing noisy inputs reduce the sample quality score of all method. However, this is an inherent limitation of ASR systems, as it affects the baseline as well. Comparing with respective baselines, our proposed PredGen was able to achieve considerable speed up in both cases while maintaining sample quality

W3 Certain experimental details require further specification.

We clarify that original experiments use character-speed-based segmentation (assuming 600 characters per minute from Sec 3) of the original text prompt from the dataset. This is described in Sec. 4.1 and is a reasonable idealization. Our real-ASR experiments confirm that PredGen remains effective even under real-world transcription behavior (see W2 for detailed setup).

Q1. Clarification on Equation (1), Section 3.4

Yes, the variable k does not appear in Equation (1) because it only inexplicitly determines how much KV cache we can reuse.

1. $LLM(S_i)$ is effectively evaluated as $LLM(\text{Cache}, S_i[M:L])$, where $M = |P| + k$. Since k tokens are accepted, we can reuse the KV cache up to that point (including the entire prompt and k tokens from the answer). Only the remaining $L-M$ tokens require computation.

2. Prompt editing is only an illusion: we do not recompute logits or argmax on prompt tokens during the iterative steps because of KV cache. They are effectively assumed to be accepted. That is, $\text{Argmax}(LLM(S_i))[0:M-1] = S_i[1:M]$.

3. The number of refinement steps in $S_0, S_1, \dots$ depends on the model. For example, correcting Alice has 3 apples and 2 oranges. to Alice has 2 apples and 3 oranges. could take one or multiple iterations:

# Possibility 1
S_0: Alice has 3 apples and 2 oranges.
S_1: Alice has 2 apples and 3 oranges.

# Possibility 2
S_0: Alice has 3 apples and 2 oranges.
S_1: Alice has 2 apples and 2 oranges.
S_2: Alice has 2 apples and 3 oranges.

While the convergence is guaranteed, its speed depends on the precise model predictions, and is not directly related to k. We will clarify this further in the paper.

Q1

We provide a concrete example below.

Suppose the prompt $P$ is What is the capital of France?, and the speculated answer $R$ is The capital of Germany is Berlin, which was generated when the word France was not available.

Before constructing $S_0$ for iteration, we first perform a verification step and accept the tokens The capital of ($k=3$). In this step, we already performed an LLM forward call on the string What is the capital of France? The capital of Germany is Berlin. (note that the verification string is exactly $S_0$.) We obtained a valid KV cache for the substring What is the capital of France? The capital of. Moreover, in the forward call, we can also obtain the next-token predictions corresponding to the words of Germany is Berlin. Let's assume they are France is Berlin [.].

Hence, we already obtained $S_1$ after the verification step. $S_1$ is What is the capital of France? The capital of France is Berlin. For $S_1$, we can reuse the KV cache of the prefix What is the capital of France? The capital of. The actual LLM input consists of only the unaccepted tokens France is Berlin, along with the past KV cache, stored as real-valued tensors. In this regard, when computing $LLM(S_1)$, we only get predictions for the unaccepted part France is Berlin. Consequently, $\text{ArgMax}(LLM(S_i))$ only changes $S_i$ in the unaccepted part, which is the last 3 tokens in this example.

We recognize that this dynamic is not fully captured in Eq. (1). A clearer formulation might be:

$S_0 = R$

$S_{i+1} = \text{Concat}(S_i[0], \text{ArgMax}(LLM(P, S_i))[0 : L - 1])$

where $LLM(P, S_i)$ only produces next-token predictions for the unaccepted portion of $S_i$.

We note that this formulation still does not fully reflect the complexity of the cache implementation, but it serves as an equivalent illustration of the process. This cached iterative refinement process was originally documented in Appendix C of the CLLM paper [1]. We will incorporate it as background in future versions for clarity.

Q2

In the original paper, we directly ran benchmark evaluations on clean text predictions. We incorporate results obtained using Whisper-Live below.

(Note: The baseline and ours-Greedy methods should have the same text outputs, but different TTS outputs due to the non-deterministic nature of our TTS.)

LMSYS

MT-Bench

GSM8K

MMLU-Pro

[1] Kou, Siqi, et al. "CLLMs: Consistency Large Language Models." ICLR 2024.

W1. ASR not included

We now conduct additional experiments using real ASR outputs. We report the results for both clean text inputs and noisy ASR inputs below. To obtain the noisy ASR inputs, we use the Whisper-Live library to transcribe synthetic audio created using Chatterbox TTS. Notably, Whisper-Live simulates a real-time audio stream by progressively sending audio inputs in chunks of 2048 bytes to the ASR backend. Hence, its output reflects real use cases where the text exhibits a "self-correction" pattern. For example, a list of partial prompts $P_1, P_2, \dots$ (described in Sec. 3) can be:

[
"what is",
"what is one...",
"what is one plus",
"what is one plus two...",
"what is one plus twenty...",
"what is 1 + 23."
]

We report the latency and score on the LMSys dataset, as well as the relative speedup. We note that introducing noisy inputs reduces the sample quality score of all methods. However, this is an inherent limitation of ASR systems, which affects the baseline as well. Compared with respective baselines, our proposed PredGen achieves considerable speedup in both cases while maintaining sample quality.

---

MMLU-Pro numbers are exactly the same

Yes, the numbers are correct. While the outputs of different methods are not identical strings, MMLU-Pro is a multiple-choice task. Unlike GSM8K, where the answer is a numerical string, the final choice remains unchanged despite differing intermediate reasoning steps.

Dear Reviewer

We appreciate your comments and feedbacks. We hope that our rebuttal has addressed your concerns. If you have any further questions, we are happy to clarify.

Authors

Q1. How well does the self-reflection verifier generalize across different model families or prompt domains?

In Table 1, we show that self-reflection generalizes to multiple datasets using Qwen as the base model. However, it is not as effective on the MMLU-Pro dataset as on other datasets, presumably due to the complexity of the problem.

To investigate whether the reflection verifier generalizes to other models, we provide additional results using different base models in the table below:

We report the speedup of the greedy and reflection verifiers on the LMSys dataset using four different models: Mistral-7B-Instruct, OLMo-2-1124-7B-Instruct, Llama-8B-Instruct, and Qwen2.5-7B-Instruct. We also report the scores of the baseline approach and the reflection verifier. Note that since the greedy verifier is lossless, it achieves the exact same score as the baseline.

These results show that self-verification works best with more recent models with stronger reasoning capabilities, such as Qwen2.5 and Llama3. For earlier models such as OLMo and Mistral, it leads to only limited improvements.

Recall that in Sec. 3.1.2, we mentioned that the reflection verifier falls back to the greedy verifier when the model gives a "no" answer during the reflection process. We find that Mistral, in particular, gives a "no" answer in most cases, leading to an almost identical speedup to the greedy verifier.

Q2. How is correction or interruption handled in user input — can early speculative audio be invalidated and rolled back?

Yes. Most ASR systems employ a Voice Activity Detector (VAD). For example, if a user says "[incomplete prompt A] [interruption], actually, what about [prompt B]", the ASR system treats the part after the interruption as a new input. Hence, our system, which relies on ASR input, considers it a separate query and discards past speculative results. Overall, we believe this is a feature of ASR systems and not the focus of our work.

We also provide discussions on ASR autocorrection in Q3.

Q3. Have you tested PredGen in noisy ASR scenarios, where the initial prompt may be erroneous?

Yes. We report the results for both clean text inputs and noisy ASR inputs below. To obtain the noisy ASR inputs, we use the Whisper-Live library to transcribe synthetic audio created using Chatterbox TTS. Notably, Whisper-Live simulates a real-time audio stream by progressively sending audio inputs in chunks of 2048 bytes to the ASR backend. Hence, its output reflects real use cases where the text exhibits a "self-correction" pattern. For example, a list of partial prompts $P_1, P_2, \dots$ (described in Sec. 3) can be:

[
"what is",
"what is one...",
"what is one plus",
"what is one plus two...",
"what is one plus twenty...",
"what is 1 + 23."
]

We report the latency and score on the LMSys dataset, as well as the relative speedup. We note that introducing noisy inputs reduces the sample quality score of all methods. However, this is an inherent limitation of ASR systems, which affects the baseline as well. Compared with respective baselines, our proposed PredGen achieves considerable speedup in both cases while maintaining sample quality.

Q4. Could traditional speculative decoding (e.g., Medusa) be combined with PredGen to further reduce latency after full input is received?

In principle, yes. However, in practice, the throughput (i.e., tokens/sec) of LLMs without speculative decoding is typically higher than the audio playback speed (which also aligns with typical human speaking speed). Hence, while traditional speculative decoding may improve the end-to-end latency for text generation, it does not affect when the user hears the synthesized sentence. We leave this for future work.

Regular users of voice assistants will not own such expensive hardware (RTX A5000), hence I see limited impact.

We respectfully disagree.

First, the RTX A5000’s price (USD 2000) is significantly cheaper than the current market price of top-tier consumer-grade GPUs such as the RTX 5090 (USD 4000+) and RTX 4090 (USD 3000+), and offers weaker performance. We believe it is representative of consumer hardware for potential users of AI voice chat applications.

Second, to further demonstrate the effectiveness of PredGen, we report additional results on the MacBook Pro with Apple Silicon, which has a considerable user base. Due to lower performance, we use a lightweight Qwen2.5-3B-Instruct instead of the 7B model. We report results on the LMSys benchmark below:

Notably, PredGen achieves similar speedup as on the RTX A5000. We argue that PredGen may be even more meaningful on lower-tier hardware, where base models are slower. For instance, it reduces latency from 3.9s to 2.0s—more noticeable from a user perspective than the improvement from 0.7s to 0.3s on A5000.

Dear Reviewer

We appreciate your comments and feedbacks. We hope that our rebuttal has addressed your concerns. If you have any further questions, we are happy to clarify.

Authors