Thank you for the positive feedback and appreciation of our work. We appreciate your insights and would like to provide more clarification.

Question 1: Figure 1&2 claim that the decoding time is relatively insensitive to the input length. However, for softmax, the cost increases drastically as the input length increases. Thus, if the input length is over 64 or 128, its time might be one of the major bottlenecks.

We appreciate the reviewer's rigorous feedback and will enhance our demonstration to clarify this claim. In standard decoding, the input length is fixed at one. Our method's selection of the input (or draft) length should keep the latency insensitivity. Experiments show that using too many draft tokens (e.g., 64 or 128) indeed increases latency, so we avoid such large numbers. Instead, we focus on a reasonable range, like 1 to 16 draft tokens in Figure 1, or 4, 8, and 16 in Figure 5.

Question 2: If performance (perplexity) is preserved, the paper would be significant, as it demonstrates substantial speedups by leveraging public drafter models for private inference.

We would like to clarify that soft matching through speculative sampling theoretically preserves the accuracy performance (e.g. perplexity) of the private model [1], as we mentioned in Line 239. It guarantees that tokens sampled from the distributions $p(x)$ and $q(x)$ using speculative sampling are distributed identically to those sampled from $p(x)$ alone. This implies that the probability of sampling any token $\hat{x}$ from both $p(x)$ and $q(x)$ is precisely $p(\hat{x})$. As a result, the expected perplexity of the sampled tokens with respect to some oracle baseline remains unchanged, thereby maintaining the performance concerned by the reviewer.

We provide a brief proof of why the tokens sampled by two types of sampling algorithm follow an identical distribution; further details can be found in Appendix A of [1]. The probability of sampling a token $\hat{x}$ using speculative sampling is:

• When the proposed token is accepted: The public model proposes $\hat{x}$ with probability $P\\{x = \hat{x} | acc\\} = q(\hat{x})$. The acceptance probability for $\hat{x}$ is $P\\{acc\\} = \min\left(1, \frac{p(\hat{x})}{q(\hat{x})}\right)$.
• When the proposed token is rejected: The public model may propose any token, which is then rejected. The rejection probability is $P\\{rej\\} = \sum_{x'} q(x') \cdot \left(1 - \min\left(1, \frac{p(x')}{q(x')}\right)\right) = 1 - \sum_{x'} \min(q(x'), p(x'))$. The token is re-sampled from the adjusted distribution with $P\\{x = \hat{x} | rej\\} = norm(\max(0, p(\hat{x}) - q(\hat{x})))$.

Substituting these into the equation, we obtain:

This confirms that the speculative sampling process preserves the original distribution $p(x)$, as desired. If you have any further questions, please feel free to let us know.

[1] Fast inference from transformers via speculative decoding. ICML 2023

Question 3: The paper does not report model performance (e.g., perplexity). An empirical analysis of how soft matching degrades output quality would be valuable for practitioners.

As discussed in Question 2, there is no inherent trade-off between efficiency and accuracy performance when adopting the soft maching. The soft matching theoretically maintains the accuracy performance, i.e. same expected perplexity towards some oracle output. This theoretical guarantee is a key strength of our method, allowing speed improvements without concerning the output quality.

Question 4: In practice, drafter models are limited to small LLMs, which often lack generalization across tasks and need task-specific alignment. Aligning drafters for every task is impractical, and without proper alignment, the efficiency declines.

In our experiments, even using non-aligned small models, we achieve an average acceptance rate of 40% for the different-series model and an average of 65% for the same-series model, which already presents approximately 1.6X and 3X average speedup.

The alignment further increases the benefits to 2X and 5X average speedup. Notably, the alignment is much easier and more practical than traditional fine-tuning tasks that prioritize accuracy as the primary objective. As illustrated in Figure 4, the alignment requires only a small public GPT, a minimal aligning dataset, and can be efficiently tuned on a small GPU.

Furthermore, the effectiveness of our approach is expected to increase as client-side computational capabilities continue to advance. Such advancements enable the leveraging of larger and more powerful public models, thereby enhancing the acceptance ratio as Table 1 indicates.

Thank you for the reviewer's appreciation of our efforts and valuable feedback on our paper. We address the main concerns as follows.

Question 1: The approach relies on the acceptance ratio achieved by the public model; if the alignment is suboptimal, the speedup may diminish.

Our experiments demonstrate a consistent acceptance ratio and speedup across three pairs of models and four tasks. Even under suboptimal conditions, such as utilizing a small, irrelevant public model, the alignment achieves acceptance ratios of 50%–70% (corresponding to a speedup of approximately 2X–3.3X). When employing a small model from the same series, the acceptance ratios improve to 75%–82% (corresponding to a speedup of around 4X–5.5X).

Furthermore, as discussed in Line 293 and Line 370, the acceptance ratio has greater potential than what the experimental results suggest. For instance, our experiments only adopt the 68M or 160M GPT models as the public model. As client-side computational capabilities continue to advance, it is anticipated that the acceptance ratio will increase through the utilization of larger and more powerful public models. Additionally, our experiments only use a very small alignment dataset and do not involve extensive hyper-parameter searching. A more thorough alignment process is expected to yield higher acceptance ratios, particularly if the server provides a more appropriately aligned public model, benefiting from its insights into the training datasets and access to enhanced computational resources.

Question 2: The complexity of the secure verification protocol may present implementation challenges in real-world deployments.

The proposed approach reduces the execution time of secure inference by a factor of 2X to 6X, while preserving the same levels of security and accuracy. Although this introduces some additional implementation complexity, we are willing to mitigate this challenge by open-sourcing our implementation.

Question 3: Additional exploration of scalability with larger vocabulary sizes or more diverse model architectures would be beneficial.

Thank you for your advice on the scalability analysis. Currently, our experiment includes three pairs of model architectures and four tasks. We focus on these aspects as they are most relevant to the proposed approach. Regarding vocabulary size, we utilize a vocabulary consisting of approximately 30,000 tokens, which provides comprehensive token coverage and is commonly adopted by popular LLMs. The existing experiments are sufficiently convincing to demonstrate the effectiveness of our method, but we are willing to include additional experiments to further justify the scalability concerning vocabulary size and model architectures in our next version.

Thank you for your thorough examination and thoughtful feedback on our paper. We address the main concerns as follows.

Question 1: In Appendix D, the authors try to prove the division can be refactored as multiplication. However, it is not stated what is the range of p(x)/q(x). Therefore, I am not sure whether the claim is correctly proved.

We provide an explanation here and will include more detailed explanations for the derivation in Appendix D to ensure it is easy to follow.

Equation (6) derives the equivalent condition for determining whether a token should be rejected. We assume the reviewer’s concerns is about the elimination of the $min()$ function in the third line $r \geq \min \left(1, \frac{p(\hat{x})}{q(\hat{x})}\right)$. The equivalence of the conditions in line 3 and line 4 is established through two cases based on the range of $\frac{p(\hat{x})}{q(\hat{x})}$.

• $\frac{p(\hat{x})}{q(\hat{x})} \in [0,1]$: The $min()$ function can be directly removed, as stated in Equation (6).
• $\frac{p(\hat{x})}{q(\hat{x})} \in (1,\infty)$: The third line simplifies to $r>1$. Since $r$ is drawn from a uniform distribution over the interval $[0,1]$, the conditions in both line 3 and line 4 are never satisfied. Consequently, the equivalence remains valid.

If you have any further questions, please feel free to let us know.

Question 2: If the alignment dataset closely resembles the private dataset (line 303-304), how to evaluate the potential privacy leakage?

Line 303 discusses the case in which the server provides the aligned public model. We assume the concern is about the open-source aligned public model incurring potential privacy leakage of the alignment dataset. This issue is independent of this work and widely exists. For instance, the base model used for alignment, open-source smaller versions of private models, can similarly expose privacy risks related to the pre-training dataset (private dataset). Such kind of issue can be resolved using the well-established framework known as differential privacy [1]. Differential privacy preserves the utility of the training data while anonymizing it with metrics that quantify the upper bound of privacy leakage. In our alignment scenario, if the alignment data resembles the private dataset (for instance, including private information), the direct use of the original dataset for training is avoided. Instead, a sanitized aligning dataset is generated [2,3], or differential privacy is applied during the gradient descent process [4].

Moreover, the potential for privacy leakage in our alignment is more manageable compared to the potential leakage of pre-training datasets from open-source small models. This is because the alignment's objective is to let the public model mimic the output distribution of easily predicted tokens. This makes the requirements on the alignment dataset much smaller than traditional training: a very small number of data is sufficient (Figure 4), and alignment on simple tokens rarely involves sensitive information.

To avoid any privacy leakage, a cautious server can choose to use relevant publicly available datasets or not provide the alignment. This may result in slightly slower performance, but thanks to our speculative sampling protocol, even non-aligned public models still achieve a 1.5X to 5X speedup.

[1] The Algorithmic Foundations of Differential Privacy, Foundations and Trends in Theoretical Computer Science 2014 [2] Dp-opt: Make large language model your privacy-preserving prompt engineer, ICLR 2024 [3] Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation, ICLR 2024 [4] Deep learning with differential privacy, CCS 2016

Question 3: Lack experimental comparison with prior works (such as CipherGPT and SIGMA) on secure GPT inference. Despite that the author claims the proposed work is orthogonal to prior works, it is worth to show at least one combination to verify this claim.

Sorry for any confusion regarding our experimental setup. In fact, the speedup in Figure 5 is the comparison with the latest work, both with and without the integration of our method. The baselines are the SOTA 2PC secure inference works for Transformer, as detailed in Line 366, including the linear layer protocol from Nimbus [1] and the non-linear layer protocol from BumbleBee [2]. We will further clarify this in the paper. [1] Bumblebee: Secure two-party inference framework for large transformers, NDSS 2025 [2] Nimbus: Secure and efficient two-party inference for transformers, Neurips 2024

Question 4: Improve Figure 3 for better clarity.

Thank you for the valuable suggestion. We will enhance Figure 3 to improve its clarity. For example, we will highlight the workflow of the secure inference process, emphasizing the distinctions between our approach and prior works. We will also illustrate the appearance of the "draft token" and "bonus token" throughout the process to provide a clearer understanding.

We are grateful for the reviewer's appreciation of our efforts. Below, we respond to your constructive comments in detail.

Question 1: Could the author clarify the ambiguity in Line 90 in Introduction and the Line 27 in abstract?

Thank you for pointing out the potential ambiguity. We clarify it here and will revise the text to avoid any further confusion. Line 90 describes the whole POST approach, and line 27 refers to only the proposed sampling protocol.

• In lines 90-92: We claim POST is compatible with mainstream 2PC protocols for Transformer models (e.g., BOLT [4], BumbleBee [2], Nimbus [3]) because the key observation: the latency insensitivity to input length holds across these protocols, for which we provide more experiments in Appendix F.
• In line 27: We refer to the efforts of designing speculative sampling protocols in Section 4.2 to eliminate operations that are inefficient for cryptographic primitives.

Question 2: Line 292: Could the author clarify why the $2^l$ complexity is considered acceptable?

The $O(2^l)$ term in the comparison protocol's overhead represents a special case with maximal communication size but minimal communication rounds [1]. Since we focus on optimizing the number of comparison calls rather than the protocol itself, we use this special case as a simplification for easier understanding, and the footnote on page five writes the general form $O(q \cdot 2^m)$ of communication complexity, where $q*m=l$. Here, $O(2^l)$ corresponds to use $q=1$. Larger $q$ reduces communication size but increases rounds. Existing works [2,3] typically choose $q=8$ to balance this trade-off, which we also adopt in experiments. In this way, bit width only partly contributes to exponential complexity.

Question 3: What is the underlying rationale for selecting two pairs from different series?

We design two kinds of experiments to show the effectiveness in different cases.

• Same-series models: This is the favorable setting in practice, as private models often have open-source smaller versions, which can achieve speedups of 4.2X–6.0X.
• Different-series models: This is to highlight robustness and general applicability. Even in less favorable conditions, we still have speedups of 2.1X–4X.

Question 4: What dataset was used for the knowledge distillation? Whether the acceptance rate is influenced by any similarities between the alignment dataset and evaluation dataset.

In this work, the alignment dataset is randomly sampled from the downstream training dataset. As explained in Sec. 4.3, the rationale for using similar datset is the case when client uses relevant public datasets or generates sanitized datasets from his queries (e.g., via differentially private generation [5,6] that preserves utility while removing sensitive information).

Our experiments show that alignment dataset similarity to the downstream task impacts the acceptance rate. For instance, testing Vicuna-7B and LLaMA-160M with an irrelevant alignment dataset, Alpaca [7], to the evaluated tasks.

Directly using a non-aligned public models yields a 1.4X–2.3X speedup, while non-relevant dataset slightly improves it to 1.6X–2.5X. The best results (2.5X–2.9X speedup) occur when using a similar alignment dataset. Thus, we recommend selecting an similar alignment dataset for higher acceptance rates.

Question 5: Despite the use of different methodologies, I would appreciate a performance comparison with related work.

Sorry for any confusion regarding our experimental setup. In fact, the speedup in Figure 5 is the comparison with the latest work, both with and without the integration of our method. The baselines are the SOTA 2PC secure inference works for Transformer, as detailed in Line 366, including the linear layer protocol from Nimbus [1] and the non-linear layer protocol from BumbleBee [2]. We will further clarify this in the paper.

Question 6: The contribution of this work may be considered incremental since it primarily utilizes standard techniques such as knowledge distillation and batching multiple tokens.

Our key contributions include being the first to introduce observations on public models and latency insensitivity to input length. The novel paradigm (public decoding and secure verification) is not merely a standard batching technique but a strategically designed approach based on these insights. Additionally, we optimize this paradigm not only through knowledge distillation but also with a specialized sampling protocol.

Reference

[1] Cryptflow2, CCS 2020

[2] Bumblebee, NDSS 2025

[3] Nimbus, Neurips 2024

[4] Bolt, S&P 2024

[5] Dp-opt: Make large language model your privacy-preserving prompt engineer, ICLR 2024

[6] Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation, ICLR 2024

[7] https://github.com/tatsu-lab/stanford_alpaca