Distribution-Aligned Decoding for Efficient LLM Task Adaptation

Dear Reviewer kREE,

Thanks for your valuable comments and the time you dedicated to reviewing this work. Here we carefully and elaborately reply to your concerns.

Q1: Can you provide formal statistical analysis (e.g., error bars, confidence intervals, or significance tests) for the experimental results, particularly for the commonsense reasoning tasks where gains are modest (1-2 points)?

Reply: Here we provide the error bars for commonsense reasoning task with Qwen2.5-7B.

Q2: Why was the comparison of SVD limited to TaD, and can you include evaluations against other inference-time adaptation methods (e.g., contrastive decoding, activation steering)?

Reply: Thanks for pointing this out. Here we provide more results about decoding-time adaptation methods including DoLa [1] and contrastive decoding (CD) [2]. We will add the following results to our updated manuscript.

[1] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR'24

[2] Contrastive Decoding: Open-ended Text Generation as Optimization. ACL'23

Q3: How does SVD perform on larger models (e.g., 70B parameters) or more diverse tasks (e.g., multilingual or domain-specific datasets), and can you provide such evaluations?

Reply: Here we conduct an experiments with Qwen2.5-72B in multiple-choice tasks. The experiments settings are same as the Table 1. The results are shown follows. These results demonstrate that our method is also applicable to larger models.

Q4: Can you elaborate on potential failure modes of SVD, particularly when the warm-start distribution is poorly aligned with the task, and discuss mitigation strategies? A suggestion: expand the “Limitations and Future Work” section to include 1-2 specific failure scenarios (e.g., noisy warm-start data) and propose mitigations (e.g., adaptive steering strength).

Reply: Thanks for your suggestion! Here we provide 2 potential failure modes and its mitigation strategies. Below is the revised section of Limitations and Future Work

A primary limitation of Steering Vector Decoding (SVD) is its dependency on an initial warmstart fine-tuning phase to identify an effective, task-specific steering direction. This dependency introduces potential failure modes if the warm-start distribution is poorly aligned with the true task distribution. Here we give two failure cases: (1) Steering with noisy or biased data.If warm-start data has significant label noise or biases, the KL divergence gradient captures these flaws, yielding a steering vector that guides the model toward incorrect or undesirable outputs during inference. (2) Overfitting to the warm-start data: On small or non-diverse datasets, the warm-start model overfits, creating a "peaky" and brittle $P_\phi$. The resulting exaggerated steering vector leads to poor generalization, producing overly confident but wrong responses on out-of-distribution examples. Therefore, future work will explore how to tackle these issues. For noisy or biased warm-start data, dynamically compute the steering strength $\mu$ per-instance using online Gauss-Newton updates with an L2 penalty to dampen noise and clip extreme values. For overfitting, incorporate hybrid approaches like ensemble steering vectors (from multiple warm-starts) or retrieval-augmented generation (RAG) to enrich the warm-start phase, ensuring better initial alignment without additional labels.

Q5: Can you provide detailed computational resource requirements (e.g., execution time, memory usage) for SVD’s warm-start and decoding phases to aid reproducibility? A suggestion: add a table or appendix specifying GPU type, memory, and runtime for each experiment (e.g., for LLaMA2-7B with LoRA+SVD).

Reply: Thank you for your suggestion. Here we provide the detailed computational resource requirements. Taking LLaMA3.1-8B with LoRA+SVD as an example, the detailed computational resources are shown below.

Dear Reviewer ndAx,

Thanks for your valuable comments and the time you dedicated to reviewing this work. Here we carefully and elaborately reply to your concerns.

W1 & L1: Concerns about the interpretability and effectiveness, especially given the potentially non-linear and oscillatory nature of distribution shifts, and the risk of performance degradation on real-world test sets with unknown distributions.

Reply: Thank you for this insightful comment. We appreciate your thoughtful probing of our theoretical explanations and empirical results, as it allows us to clarify SVD's rationale and address potential concerns about its robustness and applicability.

• To elaborate on the core intuition: The reviewer correctly notes that the full optimization path from a pre-trained model to a perfectly task-adapted model is highly non-linear. However, our method does not assume this path is simply linear. The key insight of SVD is not to treat the warm-start distribution $P_\phi$ as the final destination, but rather as a signpost that indicates a locally optimal direction of improvement. As we prove theoretically in Appendix B, the steering vector we derive is the first-order equivalent to gradient step of fine-tuning without recomputing weights. In addition, our central thesis is "output-distribution alignment". The warm-start model, having seen task-specific data, has a distribution that is verifiably closer to the target task distribution (as evidenced by its lower loss and higher accuracy). By steering towards it, we are fundamentally steering towards a better-aligned distribution at each decoding step.
• Explanation of Experimental Results: The reviewer pointed out that in Table 1, the MC1 metric decreased in a few experiments. However, we believe this is not evidence that the steering vector is uninterpretable or random. On the contrary, it is an interpretable feature that demonstrates successful distribution alignment. To clarify, we first need to know that MC1 (single-choice accuracy) is a "winner-takes-all" metric, requiring the model's highest-confidence answer to be the single correct one, MC2 (multi-choice accuracy) is more lenient for multi-answer scenarios, and MC3 (normalized probability) measures the total probability mass assigned to all correct answers, making it the purest indicator of distribution alignment. In the LLaMA3.1-8B+IA3 experiment, the decrease in MC1 is accompanied by substantial gains in MC2 (nearly 9 points) and MC3 (nearly 7 points). This clearly shows that SVD is achieving its intended effect: it may slightly reduce the model's "overconfidence" in a single optimal answer, but by distributing probability mass more reasonably across all possible correct options, it greatly improves the overall output distribution. This makes the model more "truthful" and less "sharp" or brittle in its predictions. Such a trade-off is actually desirable for robust inference.

Based on the above analysis, we believe that SVD is highly interpretable and applicable. Its behavior can be clearly predicted and explained within the theoretical framework of gradient optimization and distribution alignment. The trade-offs observed between MC1, MC2, and MC3 metrics are precisely the predictable outcomes of this principle. Furthermore, regarding its applicability to test sets with unknown distributions, we have demonstrated its robustness through extensive experiments. Across 3 models, 4 PEFT methods, and 9 benchmark tasks, the SVD-enhanced versions almost always improve the overall average performance. In addition, our mechanism for automatically calculating the steering strength $\mu$ eliminates the need for manual tuning across different distributions, further enhancing its generalizability.

W2: Theorem 2 takes $\mu$ → 0, but I don't see any experimental evidence showing that the value of $\mu$ approaches zero.

Reply: Actually, in experiments, calibrated $\mu$ values range from 0.05-0.2, which are small enough for the approximation to hold while providing practical gains.

W3: Suspecting that the performance improvement brought by the steer decoding vector comes from the warm-up model not being sufficiently trained

Reply: The reviewer points out that the performance improvement may come from the warm-up model not being sufficiently trained. In fact, our design is intentionally aimed at enabling rapid and low-resource adaptation to downstream tasks with under-trained models. In many real-world scenarios, it is impractical to perform long, large-scale fine-tuning. Therefore, SVD acts as an "amplifier", which extracts the most essential knowledge increment from this lightweight training and reuses it at every decoding step. This greatly increases the efficiency and value of a single parameter update. The results show that even after such insufficient fine-tuning, SVD can still deliver significant and consistent performance improvements. If our method only worked on fully converged models, its practical value would be much more limited.

Furthermore, to further verify that the effectiveness of SVD does not simply rely on insufficient training, we designed the experiment shown in Figure 3, which analyzes SVD’s performance across different numbers of training epochs. We can observe that when the warm-start model’s performance converge after about 5 epochs, SVD is still able to provide additional and stable performance gains. This reveals a deeper advantage of our method: SVD offers a direct and complementary calibration mechanism in the output distribution space. Traditional fine-tuning indirectly changes the model’s output probability distribution (output space) by updating model weights (parameter space), whereas SVD directly operates on the logits at decoding time. By applying the steering vector, SVD enables more precise and direct post-hoc calibration of the output distribution. Even when the model weights have already converged, SVD can reallocate overconfident or imbalanced probabilities produced during training to all potential correct options, thereby improving metrics that focus more on distribution quality.

In summary, SVD can both amplify the valuable signals present in under-trained models and provide a complementary, fine-grained calibration at the output distribution level for well-trained models, which is difficult to achieve through parameter fine-tuning alone. This demonstrates that SVD is a fundamentally effective enhancement method.

W4 & L2: Concerns about the GPU memory and latency.

Reply: While we agree that naively loading two complete models may double the GPU memory and increase the latency. However, SVD utilizes shared weights and adapter mechanisms to minimize memory overhead. In our implementation, we use two functions to load the base model and the warm-started model: AutoModelForCausalLM.from_pretrained and PeftModel.from_pretrained, respectively. The first function loads the pre-trained weights, while the second function references and reuses the already loaded base model instance, only loading the additional adapter parameters. During the forward pass, the base weights and adapter modifications are combined, without the need to load a second complete model. This ensures that the memory overhead is limited to the adapter weights (for example, for a 7B parameter model with LoRA rank=8, the additional parameters account for approximately 0.1% of the total). In addition, in our experiments, the GPU memory usage for LLaMA3.1-8B during LoRA training and SVD inference is 17.6GB and 16.5GB, respectively. This shows that SVD inference does not incur any additional memory overhead compared to training. In fact, it uses even less memory since there is no need to store optimizer states or gradient parameters. These results demonstrate the memory efficiency of our method.

Regarding latency, actually there is no communication between the two models. At inference time, both the logits from the base model and the warm-started model can be obtained within a single, unified forward pass, rather than requiring two separate passes. For example, the LoRA forward computation can be decomposed into two components: (1) the base output, computed with the original weights as $y_{\rm base} = W_0 x$, and (2) the LoRA incremental output, computed as a low-rank update $y_{\rm lora} = (BA)x$. The final output of the warm-started model is the sum of these two parts: $y_{\rm full} = y_{\rm base} + y_{\rm lora}$. Therefore, during a single forward pass, we can simultaneously obtain and return both outputs: $y_{\rm base}$ and $y_{\rm full}$.

In summary, through the PEFT-compatible design, SVD transforms the seemingly costly "dual model" concept into a lightweight decoding strategy with minimal memory overhead and manageable computational cost. Both our implementation details and empirical data confirm its efficiency and practicality. We hope this detailed explanation addresses your concerns.

Q1: The ratio between $D_{\rm train}$ and $D_{\rm calib}$ in Algorithm 1. The training sets differences.

Reply: Actually, in our implementation, given the downstream task dataset $D_{\rm task}$, we use 100% split in $D_{\rm task}$ as $D_{\rm train}$, and ramdom sample 30% data in $D_{\rm task}$ as $D_{\rm calib}$ split.

For the difference of training set in Figure 3 and Table 1, the training set is the same, we use 100% split of the training set.

We will add related implementation details in the revised manuscript. Thanks for your questions!

Q2: The steer decoding vector is not related to PEFT — can it also be applied to full fine-tuning?

Reply: Yes, SVD does not depend on PEFT and can be used with full fine-tuning. After a short full-parameter warm-start, we extract the steering vector and apply it during decoding to further improve distribution alignment.

Thanks for the reviewer's prompt reply, and we are extremely grateful that you were able to increase our score. Here are the response for your questions.

Q1: Can Prompt Tuning, IA3, and P-tuning v2 be integrated into a unified forward pass?

Reply: This is an excellent question that touches upon the fundamental mechanisms of different PEFT methods. Our proposed single-pass extraction technique is highly effective for methods that perform intra-layer modifications, such as LoRA and IA3.

1. For IA3 which applies a learned scaling vector to activations, the principle is identical to LoRA. A forward hook can capture the final scaled output ($y\_{\rm full}$) and re-compute the pre-scaling output ($y\_{\rm base}$) with negligible overhead. Thus, our efficiency claims hold for this entire class of methods.
2. However, for methods based on input manipulation, such as Prompt Tuning and P-Tuning, this technique is not applicable. These methods alter the input sequence before it enters the Transformer blocks. Consequently, from any given layer's perspective, there is no distinction between a "base" and a "full" computation path. For these methods, obtaining both outputs would indeed require two separate forward passes: one with the soft prompt and one without. Even in such cases, this trade-off, using additional time to save memory, is acceptable. However, for the vast majority of practical scenarios, LoRA-based or LoRA-like PEFT methods are the primary choice [1], so our single-pass technique is compatible with most use cases.

In summary, while our single-pass technique is not universal, it is compatible to the most prevalent and impactful class of PEFT methods used by the community today. Our focus on LoRA and related techniques is a strategic choice designed to maximize the relevance and utility of our contribution. The ability to efficiently and non-invasively inspect the outputs of LoRA-finetuned models addresses a significant need given LoRA's status as the de-facto standard for parameter-efficient fine-tuning.

[1] The Power of Parameter Efficient Fine-Tuning: Unlocking the Future of LLMs.

Q2: The advantages of SVD decoding over task vectors

Reply: Thanks for pointing this out. To show the advantages of SVD, we first need to know the difference of these two methods.

1. SVD operates in the model's output/logits space. It is a dynamic, decoding-time intervention that steers the behavior of an existing model without creating a new one.
2. Task Arithmetic operates in the model's weight space. It creates a new, static model by arithmetically combining the weight deltas of fine-tuned models. The edit is applied offline.

From this fundamental difference, several advantages of SVD emerge:

1. SVD has more flexibility and dynamic control at inference time. Task Arithmetic creates static models. To perform a new combination of tasks, task arithmetic must create a new set of model weights, $\theta\_{\rm new}$, and save it. If an application needs to switch between 10 different task combinations, it would require generating and storing 10 different model variants. On the contrary, SVD enables on-the-fly, dynamic steering. Because SVD operates at decoding time, it is far more flexible. A single warm-started model can be steered in multiple ways by simply applying different steering vectors or $\mu$ values at inference. For example, turning steering on for certain user queries and off for others, or dynamically combine different steering vectors without ever creating a new model artifact. This is invaluable for applications requiring conditional or personalized behavior.

2. SVD has more principled theoretical grounding. Task Arithmetic assumes that the "knowledge" of a task can be cleanly encapsulated by a linear vector in weight space. However, it does not provide any theoretical proof, which makes it lack theoretical interpretability. On the contrary, SVD's approach is more theoretically interpretable. We have theoretically proven the validity of this approach and provided the optimal solution.

3. SVD has more efficiency and scalability. Task Arithmetic requires storing a "task vector" that is the size of the entire model for each task, which makes it need much memory. SVD is designed to be PEFT-compatible. The "warm-start" is achieved with a lightweight adapter, which might only be a few megabytes, making it feasible to manage dozens of "tasks" (as adapters) and combine their effects dynamically.

In summary, SVD offers a more flexible, theoretically grounded, and resource-efficient paradigm for steering existing models at runtime.

This is an exceptionally insightful question. After careful consideration and discussion, we realize that your understanding is correct, and we previously overlooked this point. In cases where the inputs to the PEFT model and the base model differ at an intermediate layer, it is indeed necessary to compute $Wx$ twice in order to obtain $y_{\rm full}$ and $y_{\rm base}$. As a result, the inference time will increase. We acknowledge this as a trade-off of our method: accepting increased computational time in order to achieve significant performance gains without increasing the memory footprint from trainable parameters, which is entirely acceptable and brings significant benefits. In addition, the primary motivation behind our design is to enable efficient downstream task adaptation under memory constraints. We will discuss this trade-off and further elaborate the motivation behind our design.

Moreover, in practical applications, various acceleration techniques can be employed to significantly reduce inference time, even though this is beyond the scope of our current paper. For example, applying 8-bit quantization to the base model and using KV cache sharing can reduce the additional computation time by 50%-60% [1,2]. We will also discuss this point in the manuscript.

Finally, I would like to emphssis the key contributions and advantages of our method:

1. Novel Perspective. We reframe task adaptation as an "output distribution alignment" problem, emphasizing direct adjustment of the probability distribution at the decoding stage rather than indirectly relying on weight updates. This perspective moves beyond the traditional "change weights and change behavior" paradigm and provides a new approach for low-resource scenarios.
2. Theoretical Optimality. We prove that a single-step update of SVD is first-order equivalent to a full gradient descent step of fine-tuning, and we provide an analytical optimal solution for $\mu$. This establishes a solid optimization-theoretic foundation for decoding-time control, rather than relying on empirical heuristics.
3. Empirical Contributions. SVD delivers strong empirical gains under identical storage and memory budgets, with robust results across a range of model sizes from 1.5B to 8B parameters. Across three tasks and nine benchmarks, SVD combined with four standard PEFT methods improves multiple-choice accuracy by up to 5 points and open-ended truthfulness by 2 points, with consistent gains of 1-2 points on commonsense datasets. These results show the superiority of our method.
4. Practical Impact. First, SVD introduces only a constant overhead in GPU memory during inference: after warm-starting, there is no need for backpropagation or optimizer state, and peak memory usage is identical to standard inference. Second, SVD is fully plug-and-play, it can be combined with any PEFT method (e.g., LoRA, IA3) and decoding strategy (e.g., Greedy, Beam, Top-k/p). These features greatly lower the deployment barrier for models on edge devices or in rapid iteration scenarios.

In summary, SVD transforms the traditional approach of "modifying weights" into "adjusting output distributions," using lightweight decoding-time operations to replicate the effects of fine-tuning. This method offers a new, highly efficient, and low-cost pathway for adapting large models, excelling in theoretical interpretability, empirical effectiveness, and deployment friendliness.

We sincerely thank the reviewer for their comments, which have greatly helped us clarify the contributions, key points of focus, and trade-offs discussed in our paper.

[1] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

[2] KV Cache: The Secret to Faster LLM Inference

Dear Reviewer f5N4,

Thanks for your valuable comments and the time you dedicated to reviewing this work. Here we carefully and elaborately reply to your concerns.

W1: In Line 46-48, the authors claimed that there exists three issues, while they did not provide references or toy experiments to demonstrate the existence of these issues.

Reply: Thank you for your valuable feedback, we are grateful for your attention about this point. Here we further elaborate this three issues.

1. The first issue reflects the persistent computational burden of PEFT, where even efficient methods incur costs proportional to the base model's scale during forward/backward passes. For instance, Razdaibiedina et al. [1] explicitly discuss how standard PEFT techniques, despite reducing trainable parameters, still require full model activation during training, leading to linear scaling with model size and epoch.
2. The second issue refers to how localized parameter changes in PEFT can propagate unexpectedly across the model's output space, affecting unrelated tokens due to the dense nature of neural networks. For example, Chen et al. [2] demonstrate through experiments on LLaMA models that updates to high-probability tokens can disproportionately alter low-probability ones, leading to unstable probability distributions and degraded performance on downstream tasks.
3. The third issue refers that PEFT methods like LoRA or adapters require task-specific tuning of hyperparameters (e.g., rank, alpha), as fixed values frequently underperform or fail to generalize due to domain shifts or task variability. For example, Xu et al. [3] demonstrated that standard prompt-based PEFT fails to transfer across tasks because learned prompts are designed exclusively for individual datasets, resulting in performance drops.

The above explanations and existing works fully demonstrate the existence of these three issues. We will add relevant references and explanations in the main text. Thanks you again for pointing this out.

W2: The authors did not provide discussions how the proposed SVD addresses the mentioned issues.

Reply: Thanks for pointing this out. Here we provide detailed discussion about how our method can address these issues.

1. For the first issue, SVD decouples the bulk of adaptation from training by requiring only a short warm-start fine-tuning, after which all subsequent adjustments occur at decode time via lightweight logit perturbations. This avoids repeated forward/backward passes over full epochs, reducing scaling costs.
2. For the second issue, by shifting adaptation to the output distribution space, SVD avoids modifying the model weights after warm-start, localizing changes to task-relevant tokens through the confidence-aware constraint. In addition, Section 3 proves that one SVD step is first-order equivalent to a full-parameter gradient step in terms of its effect on the output distribution, but without the internal cascade. These can prevent unpredictable, non-local effects on token probabilities.
3. For the third issue, SVD's automated calibration of the steering strength $\mu$ derives a task-optimal value without manual tuning, enhancing transferability. For instance, our results show consistent improvements across diverse tasks (multiple-choice, open-ended generation, commonsense reasoning).

The above discussion elaborates in detail how our method addresses these issues. We will incorporate this discussion into the article. Thank you again for your suggestions.

L1: The SVD should be demonstrated to be applicable to larger language models

Reply: Here we conduct an experiments with Qwen2.5-72B in multiple-choice tasks. The experiments settings are same as the Table 1. The results are shown follows. These results demonstrate that our method is also applicable to larger models.

L2: Whether SVD is compatible with SOTA PEFT methods is not verified

Reply: We thank the reviewer for raising this important question about the practical applicability of our method. Here we clarify that our main experiments were specifically designed to systematically verify and demonstrate SVD's broad compatibility and complementary nature with mainstream PEFT methods.

First, by its core design, SVD is inherently orthogonal and highly compatible with the entire family of PEFT methods. PEFT methods (such as LoRA, IA3, etc.) are training-time strategies that operate in the model's parameter space. SVD, in contrast, is a pure decoding-time strategy that operates in the model's output logits space. It does not alter any model weights. Because they operate at different stages and in different spaces, SVD and PEFT methods can be seamlessly combined. SVD can be viewed as a "plug-and-play" enhancement module that can be applied on top of any model that has been fine-tuned with a PEFT technique.

Our experiments in Table 1 and Table 2 serve as a direct empirical validation of this principle. In these experiments, we first performed the warm-start fine-tuning using a wide and representative range of PEFT methods: Prompt Tuning, IA3, P-Tuning v2, and LoRA. Then, we applied our SVD decoding strategy on top of these already PEFT-tuned models.The results consistently show that regardless of which PEFT method was used for fine-tuning, applying SVD yields additional and stable performance gains. This clearly demonstrates that SVD is not only compatible with these methods but also acts as a complementary enhancement to further boost task performance.

Therefore, we believe our experiments have thoroughly verified SVD's compatibility with current mainstream PEFT methods. We hope this explanation addresses your concerns.

L3: The authors did not include experiments on reasoning tasks, like GSM8K, which is a very important reasoning task.

Reply: Thank you for the insightful comment. We agree that evaluating on reasoning tasks such as GSM8K is important to assess the generalization of our method. To address this, we have conducted experiments on GSM8K using Qwen2.5-7B as the base model. We compared our proposed SVD method with three approaches: TaD [4], DoLa [5], and contrastive decoding (CD) [6]. The LoRA setting is the same as in the manuscript. Form the table, we can see that our method enhance the performance of LoRA and outperform the baselines. We believe this strengthens our evaluation by demonstrating the applicability of our method to complex reasoning tasks.

Reference

[1] Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

[2] Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs

[3] Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts

[4] TaD: A Plug-and-Play Task-Aware Decoding Method to Better Adapt LLMs on Downstream Tasks. IJCAI'24

[5] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR'24

[6] Contrastive Decoding: Open-ended Text Generation as Optimization. ACL'23

Dear Reviewer bFez,

Thanks for your valuable comments and the time you dedicated to reviewing this work. Here we carefully and elaborately reply to your concerns.

W1: Concerns about prior work already linked NLL to KL divergence.

Reply: Thanks for your insightful comment. We appreciate pointing out this potential concern regarding the rethinking perspective. We agree that the equivalence between the NLL objective and minimizing the expected KL divergence between the empirical label distribution and the model's output distribution is indeed a well-established result in machine learning literature. However, the core contribution of our paper is not claiming this equivalence as a novel theoretical discovery. Instead, our "rethinking" perspective uses this known equivalence as a foundational motivation to shift the paradigm of LLM task adaptation from weight-space updates to direct output-distribution alignment during decoding, which is our method.

W2: Lack of baseline comparisons with other decoding-time adaptation method

Reply: Thanks for pointing this out. Here we provide more results about decoding-time adaptation methods including DoLa and contrastive decoding (CD). We will add the following results to our updated manuscript.

W3: The proposed method has relatively incremental technical contributions compared with prior work.

Reply: We thank your comments for your detailed feedback and the opportunity to clarify the contributions of our work. While we understand the concern about incrementalism over prior work like TaD, since our method has some similarities with TaD. However, we believe our method, Steering Vector Decoding (SVD), introduces several fundamental, non-incremental contributions that represent a significant step forward in decoding-time adaptation.

1. Theoretical Optimality: SVD is not merely a different calculation, it is grounded in classical optimization theory. As we prove in the paper, the steering vector derived from the KL gradient is the first-order equivalent to a full gradient descent step of fine-tuning. This establishes a formal, theoretical link between decoding-time adaptation and conventional parameter-space optimization, a connection that is absent in prior heuristic methods.
2. Distributional Alignment: KL divergence is the standard measure for the dissimilarity between two probability distributions. By using its gradient, SVD directly computes the steepest descent direction to align the model's output distribution with the desired task distribution. This provides a clear, interpretable, and theoretically justified mechanism for adaptation. However, TaD only uses the simple difference in probability distributions (post-fine-tuning minus pre-fine-tuning), which may raise doubts about the interpretability and reliability of the method.
3. Optimal $\mu$: You mentioned that we optimizes the hyperparameter $\mu$ automatically. However, actually, we did not optimize it. We obtained the analytical solution through theoretical derivation and proved that there theoretically exists an optimal $\mu$ to achieve optimality on different tasks.
4. On the Significance of Performance Gains: We acknowledge the reviewer's observation that the average accuracy improvement over TaD is ~1% in Table 12. However, we believe this result, when properly contextualized, demonstrates the effectiveness of SVD. In the highly competitive landscape of LLM evaluation, a consistent 1% average improvement is significant. More importantly, as shown in Table 12, SVD outperforms TaD on every one of the eight commonsense reasoning datasets evaluated. This consistent dominance, with gains of up to 2 percentage points (e.g., on ARC-c), highlights the robustness and generalizability of our principled approach. In addition, it improves multiple-choice accuracy by up to 5 points and open-ended truthfulness by 2 points, demonstrating its effectiveness. Therefore, we believe this gain is systematic, not noise.

We hope these clarifications address your concerns and demonstrate the substantive contributions of SVD

Q1: Ablation of the optimal $\mu$ as newton step.

Reply: Thanks for your comments. Here we provide empirical ablation of $\mu$. As shown in the table, we can see that when $\mu \leq 0.1$, the performance metrics fluctuate within a narrow range, indicating that such small values of $\mu$ are insufficient to significantly alter the token ranking, resulting in sensitive and unstable behavior. As $\mu$ increases to the range $0.1 \leq \mu \leq 0.2$, there is a notable improvement in performance: $\mu$ surpasses the threshold required to reorder key tokens, effectively promoting important tokens to the top of the logits. Crucially, our automatically computed optimal $\mu$, derived theoretically as a Newton step, falls precisely within this effective regime (0.05~0.2). This empirically validates that our automatic method successfully identifies a value for μ that is large enough to achieve the desired distributional shift. When $\mu \geq 0.2$ and reaches the saturation point, the steering vector's direction completely dominates the final logits ordering. As dictated by the mathematical properties of the $\mathrm{Softmax}$ function, any further increase in $\mu$ only scales the logits linearly without changing their relative order ($\mathrm{argmax}$). This makes the final token selection and the performance metrics stable and insensitive to larger $\mu$ values. This saturation behavior underscores the stability and determinism of the direction provided by our steering vector.

Q2: What is the multiple-choice task described in this figure? Is it TruthfulQA?

Reply: Yes, it is TruthfulQA.