Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data

Q1: The General language tasks evaluation metric should be written explicitly.

A: Thank you for this detailed comment. The information on the evaluation metric is shown in the table below.

Q2: Additional Experiments on other based models

A: We have tried two more models, a weaker base model Qwen-2.5-0.5B and a stronger base model Llama3-8B-instruct. We did observe improvements as well in these cases.

For the weaker base model Qwen-2.5-0.5B, we have the following results:

For the stronger base model Llama3-8B-instruct, we have the following results:

Q1: Weakness about the reliance on self-generation using the base model.

A: We agree with what the reviewer said. The quality and diversity of these generated negative samples will matter to some degree. If the base model is super strong, it would not make sense to assume the generated data is bad. However, our framework can be extended if one has a way to verify if an output is good or not, e.g., as in the recent development of reasoning models. We can extend the discriminative framework to discriminate between verified positive outputs from verified negative outputs.

Q2: Why choose Mistral as your sole pre-trained model in experiments?

A: We conducted additional experiments on different pre-trained models Qwen-2.5-0.5B and Llama3-8B-instruct. Please refer to our response to Reviewer ZjDq's Q2.

Q3: Which setting for the scoring function is used in the experiments?

A: For DFT, we use the unnormalized generative score: $s_{\theta}(\mathbf{y}, \mathbf{x}) = \log P_g(\mathbf{y}|\mathbf{x})$.

For DFT2 we use the normalized generative score: $s_{\theta}(\mathbf{y}, \mathbf{x}) =\frac{1}{|\mathbf{y}|} \log P_g(\mathbf{y}|\mathbf{x})$.

Q4: How are the negative samples $y'$ generated? What temperature is used for sampling?

A: We generated the samples using a temperature of $0.7$, a top-p of $1.0$, a top-k of $50$ (c.f. Appendix B.4). Four different prompting strategies are investigated for the UF data (cf Figure 6). The table below shows the additional experimental results of different temperatures while sampling $y'$:

Q5: Computational cost in terms of FLOPs, GPU hours, and memory usage?

A: The table below compares DFT and DFT2 with other methods in total training time. DFT and DFT2 use 2 epochs, SFT→SIMPO and SFT→DPO use 1 epoch for SFT and 1 epoch for preference optimization. For the negative data, we report the generating time of sampling 4 different outputs for each input prompt (2 of 4 are used in the first epoch, the remaining 2 are used in the second epoch). We use 4xA100 80G for the experiments.

It's important to emphasize that there are extra costs in generating negative samples and labeling the preference in the preference dataset used by SFT→PO, especially in scenarios where preference data is scarce or difficult to obtain (e.g., GPT-4-as-a-judge is used to form UF preference data). These hidden costs are not reported in the comparisons because we lack the data or access needed to measure them.

We thank the reviewer's comments. While we understand the reviewer's main concern of lack of theoretical analysis of DFT and DFT2, we would like to emphasize that this is a new yet challenging research area, especially considering the discriminative framework is defined over the infinite data space instead of finite label space.

Q1: Negative sample batch size (B) from 2 to 4 causes unexpected performance degradation.

A: The performance drop from $B=2$ to $B=4$ only happens on the UF dataset not on the MetaMath dataset. We have discussed the effect of $B$ in the paper at the end of section 7.2. Indeed, our experiments have suggested that more training examples could enjoy larger negative sample batch size $B$. On MetaMath with 395k samples, using $𝐵=4$ generated outputs is better than $𝐵=2$ generated outputs.

Q2: Lacks rigorous theoretical analysis regarding the convergence properties.

A: Since we cast the objective of DFT and DFT2 into instances of FCCO and used the same optimization framework as in Wang & Yang, their convergence analysis would apply under appropriate conditions, which guarantees converging to a stationary solution. We have attributed the convergence analysis to Wang & Yang in lines 233 -234 (left).

Q3: The connection between the discriminative likelihood and traditional probabilistic classification models could be developed more thoroughly.

A: We will elaborate more in the revision due to limit of space here.

Q4: Insufficient theoretical justification for the approximation of DFT2 to DFT.

A: We would like to refer the reviewer to a recent work of Wang et al. 2025. They have presented a generalization theory which can be useful for justifying DFT2 in our context. In particular, their Theorem 1 establishes the generalization error of the global contrastive loss in the discriminative probabilistic framework (similar to DFT2's objective) using the uniform approximation of $P_i(\cdot)=P_g^0(\cdot|\mathbf x_i)$, compared with the original discriminative log-likelihood (expectation of DFT's objective) using the sampling probabilities $P_i(\cdot)$ to re-weight each sampled negative data. The difference lies at the error between uniform approximations and the true ones.

Wang et al. On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning. ICLR, 2025.

Q5: Needs more rigorous analysis on log-sum-exp loss offers advantages over averaged loss functions.

A: The advantage is log-sum-exp loss has the property of giving higher weights to potentially bad outputs $\mathbf{y}'$ with a larger score $s_\theta(\mathbf{y}', \mathbf{x})$ in the gradient computation. Please refer to Eq. (13) in the paper, where the gradient has a weight depending on the score $s_{\theta}(\mathbf{y}', \mathbf{x}_i)$ (higher scores will give higher weights). However, averaged loss does not has such property.

We would also like to point out the connection between log-sum-exp loss and distributionally robust optimization (DRO) (Qiu et al. 2023). The existing theory of DRO (Duchi et al. 2019) can help justify the benefit of DRO compared with the empirical averaged losses.

Qiu et al. Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization. ICML, 2023.

Duchi et al. Variance-based Regularization with Convex Objectives. 2019.

Q6: Training costs doesn't fully account for the cost of generating negative examples.

A: Please refer to our response to Reviewer UHUu's Q5.

Q7: There's limited analysis of how the quality of generated negative examples affects performance

A: We did include in Figure 6 in appendix about the how generation template affect the results. For rebuttal, we also examined how different temperatures of sampling affect the results. Please refer to response to reviewer UHUu's Q4.

Q8: How does the quality of the base model affect DFT's performance?

A: We have conducted additional experiments on weaker Qwen-2.5-0.5B and stronger Llama3-8B-instruct. Please refer to the response to Reviewer ZjDq's Q2. In all cases, DFT have improvements over SFT.

Q9: The GPT-4-as-a-judge evaluation on AlpacaEval2 shows that DFT generally underperforms compared to leading PO methods.

A: The AlpacaEval2 evaluation has two parts: (1) comparing with PO methods using the same self-play data, DFT is better as shown in Figure 2; (2) comparing with PO methods using the reward-model labeled preference data, DFT is worse. This is expected as the responses of UF data are ordered by GPT-4. Hence, PO methods for finetuning models on the UF data will be inherently biased to AlpacaEval. However, we'd like to point out on the verifiable instruction following benchmark IFEval, DFT is competitive if not better than PO methods.

We thank the reviewer for detailed comments. We hope our rebuttal can address your concerns.

Q1: About the claim "Eliminates the need for preference data".

A: (1) We intended to mean the human labeled preference data. (2) Actually, our discriminative framework does not necessarily assume the sampled outputs are bad, just like like traditional logistic regression for multi-class classification. This is indeed the strength of the discriminative method. As long as the the likelihood of sampling a bad output is much larger than sampling a good output, our discriminative model would be able to leverage that. We will change our claim to the following "..., a novel alternative to SFT→PO that mitigates the burden of collecting human-labeled preference data or training strong reward models"

Q2: About the claim "a discriminative ... by explicitly modeling ... among all possible outputs given an input".

A: This claim just states how the discriminative probabilistic framework is defined. There are two separate things: (1) formulation in theory and (2) optimization in practice. Please refer to response to 8W3x's Q1 as well.

Q3: Using the same generated bad examples for DFT in the UF --> UF pipeline would result in the best comparison.

A: Yes, your understanding is correct. Besides the data, we would like to emphasize the difference in the loss function. The benefit of our loss function is detailed in section 6. We have done the experiment as suggested by using the same generated examples as losing responses in PO methods. The average performance for SFT --> DPO and SFT --> SimPO is 58.57 and 56.98, respectively, worse than DFT. Please refer to Table 1 in https://anonymous.4open.science/r/datas-C111/README.md for detail results.

Q4: The comparisons for Table 2 is based on unequal numbers of epochs.

A: No. All methods in Table 2 use 2 epochs for training (see details in A.1 Setting 2). And all methods in Table 2 uses the same data, including the number of "bad examples". The "self-play data" in the caption of Table 2 denotes the data is generated by the base model.

Q5. For Table 3, DFT used twice as many as "bad examples" comparing UF $\rightarrow$ UF pipline.

A: It is not fair to just compare how many "bad examples" are used by our method and PO methods as these "bad examples" in our method are generated for free by the base model, while that in PO methods are created using strong reward models. That is also one limitation of existing methods. Our study shows it is helpful to leverage multiple generated samples in a principled manner. Response to your Q3 compares them with exactly the same data and the same training volume for UF-->UF pipeline.

Q6: Compare with RPO (Hong et al. 2024). DPO-p (Pal et al. 2024)

A: We have added comparison with ORPO and DPO-p, where the ORPO has the average score of 61.97 and DPO-p has 57.33, worse than DFT. Please refer to Table 2 in https://anonymous.4open.science/r/datas-C111/README.md for detail results.

Q7: Regarding Table 1, it is not clear whether the improvements over competing models in this table were earned by DFT or instead by the particular dataset or training choices made.

A: We confirm that we finetuned the same base model on the same dataset as the baseline METAMATH-MISTRAL-7B that uses SFT. Hence, comparison with it directly verifies the effectiveness of our DFT method. Other results are included for simple reference. In the revision, we will focus on comparing with the baseline METAMATH-MISTRAL-7B.

Q8: Empirical differences in the likelihoods produced for positives and negatives while training?

A: We have plotted the curves for the log likelihood of positives and generated negatives while training, referring to Figure 1 & 2 in https://anonymous.4open.science/r/datas-C111/README.md. DFT increases log-likelihood of positives while decreasing that of generated data. SFT increases that of positives but does not decrease that for generated data. SimPO decreases that for both.

Q9: Figure 3 may not truly capture the full volume of training time.

A: A more detailed analysis on training cost has been conducted, please refer to our response to Q5 of Reviewer UHUu.

Q10: Is tuning DFT more expensive, regarding the larger grid of $\tau \times \gamma$ ?

A: Not really. Actually, the Table 6 and Figure 2 (b) show that $\gamma$ in the range of 0.8~0.95 works almost equally well. We recommend a value of 0.9 like the momentum parameter in Adam optimizer. For $\tau$, we tune it among three values in our experiments (cf A.1).

Q11: Why do DFT1 and DFT2 rely on such widely different values of tau for success?

A: Because (1) different score functions are used by DFT2 (length normalized log generative likelihood) and DFT (unnormalized one); (2) the elimination of division by sampling probabilities in the loss function in DFT2.