Logits are All We Need to Adapt Closed Models

We thank the reviewer for their feedback, which has helped us strengthen our paper.

Regarding Logit access assumption The central goal of this paper is to encourage closed-source LLM providers to offer logit-level access as a practical middle ground when releasing full model weights is not feasible due to IP or privacy concerns. Our theoretical and empirical results show that even with this limited access, effective domain and task adaptation is possible. Unlike prior work assuming either full white-box access or fully opaque models, we demonstrate that logit access enables fine-grained control without exposing proprietary internals. With this work, we aim to motivate commercial providers to adopt this feasible and impactful compromise, filling a key gap in current adaptation methods.

Computational Cost vs. Simpler Methods While prompt-based methods like prompt tuning or ICL may seem simpler, they often require extensive trial-and-error and show high variance, as reflected in our results. In contrast, our reweighting model offers consistent, theoretically grounded gains with lower overhead than exhaustive prompt engineering. It is also analogous to established methods like LoRA or Adapters in white-box settings, requiring a manageable training overhead. Ultimately, the stability, reliability, and performance improvements of our approach outweigh the modest compute required, especially when compared to the variability and manual tuning burden inherent in prompt-based techniques.

Baseline Comparisons: ICL Variants and API Access-based Methods Based on the reviewer’s suggestion, we extended ICL baselines to include 5, 8, and 10 examples, and implemented Diao et al. ([2], Black-box Prompt Learning, 2023) with the recommended 75 API calls as mentioned in the paper. We did not include Sun et al. ([1], Black-Box Tuning, 2022), as Diao et al. [2] already demonstrated superior performance.

Importantly, if logit access is available, Plugin can be layered on top of any prompt-based method using the best-found prompt. For instance, our Zeroshot prompt (noted in line 364, left column) is reused across all methods. We also apply Plugin on top of the best ICL variants (ICL-8/10) and Diao et al. [2]. The table below presents results across all datasets using GPT2-XL as the base model.

We observed similar results on Common Gen and will include them in the second response due to space constraints in the rebuttal.

As shown, Plugin outperforms ICL even with 10 examples and surpasses Diao et al. (2023). While ICL’s performance plateaus with increasing examples—despite higher inference cost and variance—Plugin consistently offers greater accuracy and stability. Moreover, combining Plugin with the best ICL or Diao et al. setups yields further gains, highlighting the value of logit-level access in enhancing prompt-based methods.

We appreciate the reviewer’s encouraging words and thoughtful feedback.

Regarding human evaluation and limited generalization tests We already conducted a human evaluation (line 366, details in Appendix C.7) where three evaluators compared Plugin and ICL-3 on 100 Adidas samples, with Plugin preferred in 81% of cases—directly supporting output quality.

While the Adidas dataset reflects a specialized domain shift, our study extends beyond a single brand. As shown in Section 7.1, we also evaluate on WEB NLG, E2E NLG, and CommonGen—each representing its own distribution shift relative to the black-box model’s pretraining data. In Section 7.3, we further explore adversarial shifts by testing Plugin on models with known biases (e.g., infrastructure in WEB NLG, male-related concepts in CommonGen), demonstrating Plugin’s broad applicability across diverse and challenging settings.

For most datasets, we follow standard practices from PEFT literature (Hu et al., 2021; 2023a) that rely on automated metrics comparing outputs to well-formed references to assess overall quality. This combination of human judgments (for Adidas) and standard metrics (for broader benchmarks and Adidas) provides both qualitative and quantitative evidence of the Plugin’s generalization capabilities.

Token Pruning We do not perform token pruning; instead, Plugin continuously reweights token probabilities at each decoding step without removing any tokens. This soft upweighing preserves vocabulary coverage and improves domain adaptation (see case study in line 408). For further clarification, we compare the total occurrences of the top-50 Adidas domain words in Plugin’s predictions versus the base model in the same case study. Across all samples, Plugin’s outputs include 25.6% occurrences of these words, while the baseline contains only 13.8%. We will add this in the paper.

Regarding FUDGE, Qiu et al. We do cite FUDGE in line 249 (right column). While FUDGE uses attribute-specific discriminators to control generation (e.g., formality), our method enables free-form domain adaptation via a single auxiliary model. We considered FUDGE as a baseline, but it requires one discriminator per predefined attribute—unsuitable for broad or evolving domain shifts, which are hard to define upfront.

TempNet in Qiu et al. learns a single temperature per input and uniformly scales logits during generation. In contrast, Plugin reweights logits at each timestep, enabling finer, context-sensitive adjustments. Additionally, Qiu et al.'s use of DRO involves an inner maximization loop, making it more computationally intensive than our efficient empirical risk minimization (ERM).

Nonetheless, we include a comparison on the E2E NLG dataset, adapting TempNet to use ERM (instead of DRO) with GPT2-XL. As it applies global scaling per prompt, it underperforms in tasks requiring localized adjustments. These results will be added to the final paper.

Regarding Theoretical Claims and Convergence Rate Theorem 5.1 holds under mild, standard assumptions (5.1, 5.2, B.1) commonly used in convergence analyses (Frostig et al., 2015; Chaudhuri et al., 2015; Mukherjee et al., 2022). The noisy estimation of the transition matrix $T\_t({\theta}\_{\star};x_i,x_j,\mathcal{F}^{t-1})$ can be understood in two ways: as direct estimation error in the matrix itself, or as error induced by the function $f\_{I_t}({\theta}\_{\star} ; x_i, x_j,\mathcal{F}^{t-1})$ on which it depends (see Assumption 5.1).

We adopt the latter view, estimating $f_{I_t}(\cdot)$ under a sequence of noisy autoregressive loss functions $\ell_1(\boldsymbol{\theta}), \ldots, \ell_t(\boldsymbol{\theta}) : \mathbb{R}^{|V|} \rightarrow \mathbb{R}$ (see Assumption 5.2). Under Assumption B.1—bounded gradients and Hessians of $f_{I_t}$—we show that the expected estimation error can be reduced and prove upper and lower bounds in terms of a problem-dependent quantity $\sigma_t^2$, establishing a convergence rate of $\Omega(\sigma_t^2 / t)$.

We hope this clarifies. Our novelty lies in combining techniques from Frostig et al. (2015), Chaudhuri et al. (2015), Mukherjee et al. (2022), and Patrini et al. (2017), and presenting the first finite-time convergence analysis for transition matrix estimation in this autoregressive noisy loss setting.

We acknowledge that real-world settings may add noise in mapping $f_{I_t}$ to the transition matrix. While this could generalize our framework, incorporating it into our finite-time guarantee is left for future work. Regarding sensitivity analysis, our convergence rate depends on the variance-like term $\sigma_t^2$; higher variance leads to slower convergence, naturally capturing sensitivity to estimation noise.

We thank the reviewer for the positive and insightful review.

Regarding Comparison of Plugin with Parameter Efficient Fine-tuning (PEFT) methods like LoRA, Adapters:

"... The Plugin model is positioned as an alternative to fine-tuning, but it is not directly compared to parameter-efficient fine-tuning methods such as LoRA, Adapters, or QLoRA..." "... How many flops would the method save compared to vanilla LoRA?"

We rely on only output logits of the black-box model and do not have access to its internal weights or architecture. Consequently, any form of fine-tuning—including parameter-efficient approaches like LoRA or Adapters—cannot be applied, which we clarify in the Introduction (lines 21–33, right column) and the Related Work (lines 261–273, left column). If we did have access to the model weights, parameter-efficient fine-tuning methods would indeed be the natural choice, as they use more information than just logits and are expected to yield better performance. Thus, we emphasize that the Plugin model is not an alternative to fine-tuning, but rather an approach that uniquely stands for adapting black-box LLMs which only provide logit access.

Nevertheless, to address the reviewer’s point, we conducted a comparison on the E2E NLG dataset by adding rank-$𝑟=8$ LoRA matrices in the $Q$ and $V$ attention layers of GPT2-XL. The results, which will be included in our final version, show that LoRA only slightly outperforms our Plugin in terms of task metrics. LoRA adds 2.46M parameters (rank-8, Q/V only), while Plugin adds 30.72M (one full layer). During inference (up to 64 tokens), LoRA requires 188.8B FLOPs while Plugin needs 196.2B FLOPs - a negligible difference in computational cost. Notably, the parameter and efficiency gap between LoRA and Plugin narrows when increasing LoRA's rank (r) while reducing Plugin's hidden dimensions - demonstrating how both approaches can be adaptively tuned to meet specific resource constraints while maintaining competitive performance. We would like to highlight that the fundamental distinction remains - LoRA requires full model access to modify internal layers, while Plugin enables post-hoc deployment without retraining.

Regarding Missing References

On the Duality between Gradient Transformations and Adapters. Lucas Torroba-Hennigen et al. 2025 Tuning Language Models by Proxy. Liu et al. 2024 A Study on the Calibration of In-context Learning. Zhang et al. 2023

Thank you for pointing out these relevant works. We will add them to our final version. In particular, we note that Liu et al. (2024) describes the same method introduced in Liu et al. (2021), which we already cite in line 252. Below, we clarify how our approach differs from each reference:

Torroba-Hennigen et al. (2025): They examine the equivalence between gradient transformations and adapters for efficient model adaptation, relying on full access to model weights and gradients. In contrast, our method requires no access to model internals; we adapt black-box LLMs solely by reweighting token-level logits.

Liu et al. (2024): As noted, we already cite their earlier work (Liu et al. 2021), which presents the same core idea of combining logits. Our WeightedComb baseline (line 252, right column) is directly inspired by their approach.

Zhang et al. (2023): We will add this reference to our discussion on calibration. Their study focuses on aligning model confidence with predictive accuracy by adjusting confidence scores as one increases shots in the few-shot learning setting. Unlike them, we explicitly modify the token predictions themselves, rather than just calibrating confidence.

Regarding Domain Shifts, Extreme Domain Shifts, and Adversarial Domain Shifts

We assume our black-box LLM already encodes extensive world knowledge. In this sense, any adaptation to a domain-specific dataset amounts to handling a distribution shift, as demonstrated in Section 7.1 with WEB NLG, E2E NLG, and CommonGen. Among these, the Adidas dataset represents a more extreme domain shift, given its specialized style of product descriptions. Additionally, the experiments in Section 7.3 can be seen as adversarial, since the Plugin is applied atop a model with known biases—for instance, the tendency to focus on infrastructure-related concepts in WEB NLG and male-related concepts in CommonGen. Although we did not explicitly label these settings “extreme” or “adversarial,” they do indeed meet those criteria to some extent, and we will clarify that in the final version.

We would also welcome any further examples or clarifications on what the reviewer would consider to be more extreme or adversarial distribution shifts in this context.

We are grateful for the reviewer’s positive remarks and valuable insights.

Regarding Inclusion of Closed Source Models We acknowledge that including more closed-source models could further strengthen the generality of our findings. However, most proprietary models currently do not expose their output logits. Hence, we could not experiment with them. Our approach specifically hinges on logit-level access—without revealing internal weights or architecture—which we believe is a comparatively straightforward adjustment for closed-source providers to implement, especially when compared to the complexities of releasing the full model.

By highlighting this limitation, we aim to encourage closed-source developers to consider offering logit-level access in the future, enabling more flexible and efficient adaptation methods for end-users.

Implications of making the transition matrix diagonal

Benefits

1. Reduced Complexity: Learning a diagonal matrix involves only $|V|$ parameters, compared to $|V|\times |V|$ parameters in a full matrix. This makes training computationally more tractable.

2. Straightforward Integration: Because the transition matrix is treated like a single vector, standard autoregressive models (e.g., GPT-2, LLaMA) can be used directly, scaling easily with the dataset size.

3. Symmetric (Class-Independent) Noise: A diagonal assumption naturally corresponds to label flips occurring with equal probability among all incorrect classes. This aligns neatly with widely studied symmetrical label-noise scenarios.

4. Stability in Estimation: Each diagonal entry is an autoregressive parameter independently learnt without depending on other classes (vocabulary), reducing the risk of overfitting and simplifying parameter estimation compared to a fully dense matrix.

Limitation

Although adopting a diagonal transition matrix reduces complexity and simplifies parameter estimation, it also prevents the model from capturing any off-diagonal confusions—that is, instances where certain tokens are more likely to be misclassified as particular other tokens. Real-world data may involve non-symmetric noise patterns, domain-specific synonyms, and context-dependent misclassifications. A purely diagonal matrix may therefore oversimplify these nuances, leading to diminished expressive power and potentially missing important relationships within the noise structure.

Overall, the diagonal constraint provides a practical, stable, and computationally efficient solution for autoregressive reweighing learning—one that is well-aligned with symmetrical label-noise setups—while sacrificing some flexibility in capturing sophisticated non-symmetric noise or domain-specific synonym patterns.

We plan to investigate non-diagonal structures to fully model these nuanced noise patterns and further improve adaptation in the future work.