Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models

Thank you for your detailed feedback. Here we focus on answering major concerns/questions due to space limitation. However, we sincerely appreciate other feedback and will reflect them to our revisions, as well as discussions below.

Methods And Evaluation Criteria

However, I believe the evaluation metrics are not sufficient.

Due to the space limitation, please refer to our answer to Reviewer BmQN on standard deviations.

Theoretical Claims

When replacing a Resnet foundation model with ViT, the ε should be larger than replacing a Resnet with a Restnet. I am not sure how drastic the ε could be in practice.

I am not sure how large the C could be and if this assumption is realistic for practice or not.

The practical validity of our assumptions is indeed an important perspective. Thus we conducted additional experiments to measure $\epsilon$, KL-divergence between pretrained models, and the constant $C$ on CIFAR100:

Here we reported averaged results over inputs from the dataset. These results indicate that (1) KL divergences between similar models are surprisingly small, (2) KL divergences largely depend on pretraining datasets, rather than model structures, (3) the constant C is seemingly finitely bounded.

Experimental Designs Or Analyses

[...] there are other lightweight efficient FT methods available as baseline methods, such as LoRA and adapters. It weakens the point that PRT is a superior alternative to FT.

FT method performs better than EFT and PRT in terms of accuracy and speed in most of the cases, yet FT is not a baseline. The impact is the same with the last issue.

First of all, we would like to emphasize that our paper does not claim PRT is a superior alternative to FT unconditionally. If we have enough resources that we can ignore costs for training models or maintaining training data, we should employ repeated FT rather than one-time PRT. Rather, our paper claims that PRT is a better alternative in cases that we don't want to retrain new pretrained models for various reasons. In such cases, FT models are unavailable and thus should be viewed as unknown oracles for PRT/EFT. Hence the fact that FT is better than PRT/EFT in accuracy/efficiency does not diminish their motivation.

Also, we consider that the lack of comparison with efficient FT methods like LoRA does not weaken our claims because: (1) Both PRT and EFT are not efficient FT methods. Rather, they enable us to reuse a once-tuned model for (inference-time) tuning other pretrained models with different sizes or updated knowledge. (2) Since efficient FT methods like LoRA are designed to apply to FT, they should also apply to PRT. The following additional experiments (instruct-tuned with LoRA, eval. on GSM8K) verify that LoRA actually works with PRT as well as FT.

It lacks analysis of Figure 11-12 in HumanEval Results (...). For example, PRT underperforms on Falcon-3 models when the target model is 7B and 10B. Things are different with other target models. [...]

Actually, PRT performs well on the GSM8K benchmark even with Falcon3 7B/10B models, which implies PRT indeed transfer the instruction-tuned ability to these models. The problem on HumanEval is mainly due to the difficulty of controlling downstream accuracy in instruction-tuning in some models, since the instruction dataset contains many data irrelevant to downstream tasks.

Other issues not discussed in the paper a) The reward model scalability. [...] b) What are the limits of the "same label space" assumption? And what if the label is partially different?`

a): Figure 13 shows the reward model scalability. Overall, a larger reward model leads to better accuracy with new pretrained models. Also, we additionally evaluated how inference time will be changed by scailing model sizes (1B → 3B). The results show the increase in average time per token is only 2% with PRT, while 25% with EFT.

b): See our answer to Reviewer BmQN.

Questions:

Q1: The instruction-tuned models are actually evaluated on downstream tasks like math and coding. However, as you pointed out, tasks that require feature extraction, rather than label distributions, are out of scope in this paper. This may be an interesting direction for future work.

Q2: See above discussion.

Q3: We confirmed that the reward model can be retrained and then reused with other pretrained models when input distribution changed. See our answer to Reviewer BmQN.

Thank you for your valuable feedback. We are really encouraged by the positive feedback to our research direction. Here we would like to address your concerns or questions.

The authors only do a certain limited ablation around, e.g., the effect of different source vs. target architectures or training stable reward networks. More discussions on how stable the training is across random seeds or large scale might be helpful.

Thank you for the suggestion. Indeed, in vision experiments (e.g. Fig 2), we have already plotted one standard deviation (by black bars) over three random seeds in training, which indicates very small variance in PRT training/inference. Moreover, although training language models with multiple seeds is computationally heavy and thus not standard in previous literature, we additionally conducted PRT/Instruct tuning of Qwen2.5-0.5B model with three random seeds, and evaluated them with other pretrained models. These results show the stability of PRT is at the same level as standard FT, also in language experiments. We would like to add such discussions on training stability with respect to random seeds in our revision.

The authors position PRT as a solution for “cross-model generalization of fine-tuned solutions.” They might connect more with methods in model distillation or universal prompt offsets, but that is less critical.

Thank you for suggestions. We agree that automatic prompt tuning can also be considered as inference-time tuning (but only for language models), and thus we would like to add a discussion about them. Also we would like to survey related literatures from the field of model distillation.

Weaknesses: Some details about how robust the reward model is to massive distribution shifts remain underexplored.

[...] Another possible future direction is combining PRT with partial re-training to reduce any performance gap.

Here we will proceed by assuming massive distribution shifts refers to input distributions. To examine this, we conducted experiments with noisy data, i.e., CIFAR100 with gaussian noise ($\sigma^2=0.01$). Here we employ a reward model (ResNet50, untuned) trained on clean data by PRT, and then additionally tune it by PRT on the noisy data (ResNet50, tuned) for only $1/10$ iterations. The results show that (1) PRT degrades its performance on noisy data (as well as standard FT) but (2) additional PRT training can recover its performance.

Could we do something approximate if the new model’s vocabulary is only partly changed? How big a mismatch can we handle between old and new model vocabularies or label sets? Could partial alignment or token mapping methods mitigate that?

We sincerely agree that such a research direction is important for inference-time tuning, i.e., for both EFT and PRT. We decided not to treat this topic in this PRT paper as it should be focused on questions specific to PRT. Also handling different vocabularies is actually a highly non-trivial problem in language models [1, 2, for e.g.], and thus it should deserve another paper, which may require to match different vocabularies using, for e.g., edit distance, optimal transport, etc.

• [1] Xu et al. "Bridging the Gap between Different Vocabularies for LLM Ensemble"
• [2] Wan et al. "Knowledge Fusion of Large Language Models"

Do you observe any stability concerns or require special hyperparameters to get consistent results in PRT training?

As described in Appendix B, we basically used the same hyperparameters for both PRT and (E)FT in training and inference, for fair comparison. Some experiments (Fig.6, 7) includes EM regularization, but we found that the regularization coefficient does not affect the stability of final results.

Thank you for your valuable feedback, and finding our research direction promising. Here we would like to address the questions.

There is a mismatch between algorithm 1 and the text in lines 171-206. Reward models are simply fine-tuned on a given task with cross entropy loss. However, algorithm 1 seems to be training a reward model that trains on the difference between pre-training model's predictions and the true distribution from the task.

Sorry for the confusion. We reviewed this part and confirmed there is actually no mismatch. Here let us explain that. From the text, we wrote The reward model $r_\theta(x,y)$ is trained by simply optimizing the same loss function $L(p, y^*)$ as in standard fine-tuning and the loss function is expanded as $L(p, y) = ... = -\log \pi_\theta (y | x)$. This description corresponds to line 8 in Algorithm 1. Also importantly, we note that $\pi_\theta (y|x)$ is defined in eq. (6), as the product of the pretrained model $\pi_\mathrm{pt}(y|x)$ and the exponential of the reward model $r_\theta(x,y)$. This corresponds to lines 6-7 in Algorithm 1, where we compute the product in the logarithmic space and then take softmax to obtain $\pi_\theta (y|x)$. Nevertheless, as these correspondences are not explicitly described in the text, we would like to add such annotations in our revised paper. Thank you for pointing out them.

In the first scenario, the reward function that will be used in algorithm 2 will be a log ratio of the fine-tuned model's prediction and original pre-trained model's prediction. On the other hand, in second scenario, the reward model can be used directly. Please clarify which version is used in the experiments. Furthermore, please clarify how many models need to be maintained to perform PRT at inference?

(cont.) Here we note that our reward model is expected to play the same role as the log ratio of the two models. In other words, Algorithm 1 automatically learns a single model $r_\theta(x,y)$ that implements this role, and that is the point of PRT. So the answer for the last question is: two models (new pretrained model & reward model) are used in PRT inference, while three models (new pretrained model & old pretrained model & old FT model) are used in the previous work.

In figure 3, why is Instruct's performance lower than others? Isn't Instruct a model that has been fine-tuned on the task itself? Please clarify the differences between FT and Instruct in figures 2 and 3.

First of all, let us clarify the differences between FT (in vision experiments) and Instruct (in language experiments). FT means a model fine-tuned and evaluated on the same expert dataset (e.g., Cars, CUB, etc). On the other hand, Instruct means a model fine-tuned on a general dataset (i.e., the instruction-following data) and evaluated on other expert datasets like math and coding. So the answer for the second question is yes. Thus, the performance of Instruct on each expert benchmark may degrade on some dataset due to implicit bias occurred in insturction tuning.

In proposition 3.1, when defining r(y|x), shouldn't be utilized for sampling y, i.e. $E_{y\sim \pi_{\mathrm{ft}}} r(x,y) = 1$?

Sorry for the confusion. Thanks to this comment, we have noticed a (non-crucial) typo. The probability for sampling y, $\pi_{\mathrm{pt}}$, is actually correct. However, we should write the condition for $r(x,y)$ should be $E_{y\sim \pi_{\mathrm{pt}}} \exp r(x,y) = 1$.

Let us also explain about this condition briefly. By the mapping in proposition 3.1, $\pi_\mathrm{ft}$ is mapped to $r(x,y) := \log(\pi_\mathrm{ft}(y|x) / \pi_\mathrm{pt}(y|x))$. This satisfies $E_{y\sim \pi_{\mathrm{pt}}} \exp r(x,y) = \sum_y \pi_{\mathrm{pt}}(y|x) \exp r(x,y) = \sum_y \pi_{\mathrm{pt}}(y|x) (\pi_{\mathrm{ft}}(y|x) / \pi_{\mathrm{pt}}(y|x)) = \sum_y \pi_{\mathrm{ft}}(y|x) = 1,$ i.e., $E_{y\sim \pi_{\mathrm{pt}}} \exp r(x,y) = 1$ rather than $E_{y\sim \pi_{\mathrm{ft}}} \exp r(x,y) = 1$.

How is the regularization term applied? Is it simply applying an entropy regularization on the log likelihood ratio between the pre-trained model and the model that is being fine-tuned? What are the differences of this regularization term from the KL divergence loss between the pre-trained model and fine-tuned model's predictions?

The application of the regularization term is as described in eq. (12). The difference between the two KL divergence (eq. (5) and eq. (11)) is an important point. On one hand, KL divergence in eq. (5) measures discrepancy of two distributions over the label/vocabulary space. On the other hand, KL divergence in eq. (11) measures discrepancy of two (meta-)distributions over the set of probability models. As discussed in lines 267-274 and 220-229, since the latter KL divergence is computationally intractable, we proposed the entropy maximization regularization (eq. (5)).