Bayesian WeakS-to-Strong from Text Classification to Generation

Thanks for your inquiry. "The experiments is not comprehensive: the experiment they conduct are mainly on the ~1B weak models and the strong model are ~7B. I think from the experiment results conducted now, the claim on WeakS-to-Strong in the paper is slightly overclaimed. (The major weakness in my opinion)". If the proposed method doesn't show generalized superior performance with various model sizes, the claim on a better performance is a bit overclaimed in my opinion.

We thank Reviewer grX8 for the feedback. We response to your comments below:

1. Regarding the choice of mdoel size

According to the original Weak-to-Strong paper, performance is the best when the size ratio between weak and strong models is around 0.1. Limited by the computational resource, we choose strong model of size 7B. Additionally, we want the weak models to have some basic capabilities to facilitate fine-tuning the strong model. So we chose the 1B model as a weak model.

2. Regarding the significance of improvement

For the example you mentioned, 0.843 is the result of Single Weak-to-Strong using BLOOM-1B1, which wasn't included in the WeakS-to-Strong-3 experiment. A fair comparison would be 0.843 (Single Weak) vs 0.853 (Naive Multi-Weak) vs 0.866 (Bayesian Multi-Weak). Our Bayesian approach achieved a PGR of 0.828, showing a relative improvement of 14% over the best single model and 9% over the naive baseline (refer to the result in Table 2 in the main manuscript). For generation tasks, our Bayesian method also achieved the best performance, consistently outperforming others across three random seeds.

3. Regarding the complementary of weak models

The approach of multiple weak models improves from their complementarity. For example, with three weak models, if one or two models classify a sample correctly, then the strong model can learn some correct knowledge. In contrast, if there is only one weak model and it makes an error, then the strong model will learn completely wrong knowledge for this sample. Our Bayesian method increases the fault tolerance, as it learns a prior across all samples in entire dataset. Additionally, this approach better estimates probability distributions, helping to avoid the overconfidence issue caused by the softmax activation function. We acknowledge that if the weak models performs poorly and inaccurately predicts nearly all samples, the model can't develop effective prior.

We have addressed all issues the reviewer raised and we kindly request a reconsideration, with the hope that our efforts are reflected in a potential adjustment of the assigned scores.

Thank you for your explanation. We think it’s necessary to emphasize that our problem setting is to mimic the scenario where humans supervise superhuman models, by approach of weak model supervising strong model. Our key contribution is proposing a Bayesian approach to integrate multiple weak models and extending the application from classification to generation tasks. We don’t focus on different model sizes, because (i) previous work by Burns et al. [1] has extensively explored this aspect and verified the usefulness of the general weak-to-strong idea; (ii) limitation of computational resource; (iii) apart from model size, model performance is also related to task diffculty and we have tested the proposed method on different tasks to demonstrate the generalization ability.

To be specific, we have conducted extensive experiments to validate the generalization ability of the proposed method:

• We utilized a variety of weak models with different model structures (with different model capacity),as well as different number of weak models (WeakS-to-Strong-3 and WeakS-to-Strong-5 in the main manuscript).
• Both classification and generation tasks were tested in the main manuscript, the proposed method showing consistently better results.
• We conducted additional experiments on the CosmosQA dataset, which indicates that, in case of harder task where weak models don't perform well, our method also significantly overbeat single weak model and baselines (refer to the general response G.2 for more details).
• Ablation studies were also performed to show the effectiveness of our method (Appendix D in the main manuscript).
• Each experiment was run using three different random seeds.

Overall, we conducted extensive experiments with positive results. Although our experiments are still limited when considering the large number of LLMs of different sizes, we believe our results are still convincing and can show the generalization of our approach.

[1] Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., ... & Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.

Following our previous clarification (which we've slightly updated for better explanation), to directly address the concerns about model size, we conducted additional experiments using 13B model (Llama-2-13B with strong ceiling performance of 0.924) as strong model. The experiment setting is the same as the classification task on SciQ dataset. Due to computational and time constraints, we only conducted experiments with three weak models. Results are listed below:

As shown in above tables, our Bayesian approach consinstently overbeat single weak model and multi-weak baselines regardless of the absolute model size, indicating the generalization of our methods.

Thank you again for joining the discussion. We hope our responses address your concern.

0.854 $\pm$ 0.010 VS $\mathbf{0.863 \pm 0.009}$

To be honest, I would prefer not to buy the statement that this improvement is significant.

Besides, as shown in Table 6, (Acc w/o aux_loss) is even better than (Acc w/ aux_loss). And how can we ensure that w/ aux_loss is the optimal version. I think this ablation study doesn't support the original claim.

We are thankful for Reviewer JkWZ's detailed feedback. We response to your comments below:

1. Regarding motivation for using Dirichlet prior

We choose the Dirichlet distribution because it is the conjugate prior distribution of the categorical distribution. If a data point has a categorical distribution, and the prior distribution of the parameter is distributed as a Dirichlet, then the posterior distribution of the parameter is also a Dirichlet. And the computation of likelihood is tractable. The derivation for Eqn 3 below (in bullet point 2) also demonstrates this.

2. Regarding transformation of Eq. 3

The detailed transformation for Eqn. 3 is shown below (for brevity, we omitted the superscript used to denote the $i$-th weak model):

Consider a label $\mathbf{y}$ sampled from a categorical distribution $\text{Cat}(\mathbf{\pi})$, where each component $\mathbf{\pi}_k$ corresponds to the probability of sampling a label from class $k$, $\mathbf{y} \sim P(\mathbf{y}|\mathbf{\pi}) = \text{Cat}(\mathbf{\pi})$. Assume the categorical distribution is sampled from a Dirichlet distribution:

$\mathbf{\pi} \sim p(\mathbf{\pi} | \mathbf{\alpha}) = \text{Dir}(\mathbf{\pi} | \mathbf{\alpha}) = \frac{1}{B(\mathbf{\alpha})} \prod_{k=1}^K \pi_k^{\alpha_k-1}$

where $B(\cdot)$ is the Beta function, $\alpha_k$ is the hyperparameter of the Dirichlet distribution, and $\alpha_0 = \sum_{k=1}^K \alpha_k$ is the Dirichlet strength. Treat Dirichlet as a prior of the likelihood $P(\mathbf{y}|\mathbf{\pi})$, and the marginal likelihood can be obtrained by integrationg over the class probabilities:

$P(\mathbf{y}|\mathbf{\alpha}) = \int P(\mathbf{y}|\mathbf{\pi}) p(\mathbf{\pi}|\mathbf{\alpha}) d\mathbf{\pi}$

$=\int \prod_k {\pi}_k^{y_k} \frac{1}{B(\mathbf{\alpha})}\prod_k {\pi}_k^{{\alpha}_k-1} =\frac{B(\mathbf{\alpha}+\mathbf{y})}{B(\mathbf{\alpha})}$

$=\frac{\Gamma(\sum_k {\alpha}_k)}{\Gamma(\sum_k {\alpha}_k + \sum_k y_k)} \frac{\prod_k \Gamma({\alpha}_k + y_k)}{\prod_k \Gamma({\alpha}_k)}$

$=\frac{\prod_k{\alpha}_k^{{y}_k}}{{\alpha_0}\sum_k{y}_k}$

Then the negative log marginal likelihood can be obtained:

$L^\text{NLL}=-\log P(\mathbf{y}|\mathbf{\alpha})=\sum_{k=1}^K y_k(\log (\alpha_0) - \log (\alpha_k))$

This derivation also demonstrates the first point: the property of conjugacy allows us to have a tractble liklelihood loss.

3. Regarding the definition of $p_k$ in Eq. 4

The $p_k$ is the probability assignment for $k$-th category of $i$-th weak model, which should more accurately be written as $p_k^{(i)}$ (while the superscript $i$ was ommitted for the sack of brevity). We will clarify this in the revised version.

4. Regarding concern on training stability and time consumption

• For the concern about likely creating instability or leading to divergence: Our strong model is not trained from scratch but is fine-tuned from a pre-trained model. Its inherent capability allows it to produce reliable confidence score estimates. We didn't encounter instability issues in our experiments.
• For time consumption: In one training step, the strong model only needs to perform one single forward pass. This pass provides the prediction used for loss calculation, and applies a softmax to the output giving the $C_s$ values. Therefore, it doesn't introduce additional time consumption.

The rest of your comments will be addressed in Response (2).

We are thankful for Reviewer zdng's thoughtful feedback. We response to your comments below:

1. Regarding justification of design choices

When we extend WeakS-to-Strong from classification to generation, we want to use soft labels for training, because weak model performance is not satisfactory, and soft labels can provide more information than hard labels (as further confirmed by our ablation study on probability estimation).

In practice, different models use different tokenizers, and directly passing probabilities as in classification is not feasible. So we used the word string as a bridge to connect different wordpiece tokenizers. By applying Eqn. 6, the overall probability for target words remains consistent. When decomposing words into strong model wordpieces and estimating probabilities for other categories, we referred to the strong model's outputs. This choice is due to the strong model's superior generalization capabilities and prior knowledge, allowing more accurate probability estimation. Overall, this design is a natural and reasonable choice.

2. Regarding additional datasets

• For the classification task, we used the SciQ dataset because we followed Burn et al.'s work[1], which used this dataset. We also added an extra dataset, CosmosQA. Please refer general response G.2 for details.
• For the generation task, we used the SLURP dataset for slot filling because it has clear evaluation metrics (SLU-F1), making it easy to quantify the Weak-to-Strong performance.

[1] Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., ... & Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.

3. Regarding suggestion on presentation

Thank you for your thoughtful suggestion. We will move the regularization loss to the main paper, and add the exact values of the weights, in our revised manuscript.

Thank you again for your positive comments. We hope our response fully resolves your concerns.

We appreciate Reviewer WJwn for the thoughtful comments. We response to your comments below:

1.Regarding the significance of some results

About the improvement of DPO: The PGR results from three experiments before and after DPO training are shown in the table below. Note that seeds used during teacher-forcing training are also different, meaning the initial SFT models differ for different seeds. As shown, after cDPO, three models showed consistent performance improvements. The large standard deviation is due to the lower performance of the model with seed-0 (and with this seed, our Bayesian approach also performs better than the naive baseline).

About the improvement of the auxiliary loss for Bayesian methods, the performance with auxiliary loss is also consistently better across three seeds. We will clarify it in our revised manuscript.

2. Regarding other baselines and tasks

Thank you for your constructive suggestions. After comparing different methods, we chose FlyingSquid as an additional baseline because its unsupervised LabelModel training doesn't require ground truth, aligning better with the Weak-to-Strong assumptions than methods like AdaBoost. Furthermore, because it is designed for binary classification, it is only used as a classification baseline. The results are shown in the general response G.1. This baseline performed similarly to the naive baseline, and didn't beat our Bayesian approach.

About the evaluation tasks, we added results on the CosmosQA dataset for classification. Please refer to the general response G.2 for details.

3. Regarding clarification on key assumptions

Although we used weak models to mimic humans, the scenarios and capabilities of weak models and humans are inevitably somewhat different. For instance, humans have natural cognitive intuition for probabilities (confidence) and can provide finer-grained, multi-dimensional scores due to their stronger evaluation abilities. To compensate for the gap of evaluation ability between weak models and humans, we approximate confidence and scoring using token probabilities. Although solutions for human scenarios might differ from those for weak models, a key consideration of our proposed Bayesian WeakS-to-Strong is its generalization, which is highly compatible with existing training frameworks. It can work for both weak models and human labels, although the specific experimental setups may differ.

Regarding the specific questions:

• The weights for human labels can be determined based on the application context and individual backgrounds, given that the fixed weight, which reflects the capability of weak models, performs best in our experiments.
• For human label confidence, we can refer to previous RLHF works [1][2] to let humans provide detailed and fine-grained scores.

[1] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R.. Training language models to follow instructions with human feedback. In Proc. NeurIPS, 2022.

[2] Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., ... & Chua, T. S.. RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback. In Proc. CVPR, 2024.

4. Regarding $\lambda$ value

The $\lambda$ is set to (0.1, 0.3, 0.2, 0.3, 0.1) for GPT2, OPT, Pythia, BLOOM, and TinyLlama models respectively. The decision is mainly based on the Weak-to-Strong performance of a single model, which reflects the quality of weak labels. We will add it to the revised manuscript.

5. Regarding computation of PGR

The PGR for multiple weak models is the average PGR, i.e., we calculate a PRG for each weak model and treat their average as the result for multiple weak models. We will clarify it in Section 5.3 in the revised manuscript.

6. Regarding clarification of notation in Eq. 4 and Eq. 7

Regarding Eq.4, sorry for the lack of clarity, $\hat{y}_w^{(i)}$ should be $\hat{y}_w^{(i, k)}$, which represents a one-hot label where the $k$-th category is 1 and the others are 0.

Regarding Eq.7, thank you for pointing this out. This step is actually performed on log probabilities, not probabilities, which is why it is written as a sum. We will correct these in the revised version.

We thank you again for the positive comments. We hope our response fully resolves your concerns and that you will champion our paper.

I thank the authors a lot for their detailed response and clarifications, which have indeed addressed some of my initial concerns. I still do have some other follow-up questions/comments on the remaining points though:

Re: (2), I'm not sure it's completely fair to consider comparing to only methods that use no labeled data given that your current procedure for setting $\lambda$ does involve labeled data (discussed more in my response to (4) below). Also in the WS benchmarking effort, WRENCH (https://arxiv.org/pdf/2109.11377), FlyingSquid is not really one of the most consistent performers.

Re: (3), perhaps more of an exploratory question, but how would you imagine handling the generation case when the labelers are humans? for instance, it's not clear to me that it would be practical for humans to give confidences for all tokens/words that they write (which is what is currently used for the weak models)?

Re: (4), thanks for sharing more details though I'm still somewhat concerned by the practicality of your selection procedure. I believe the following limitations should be addressed (or at least mentioned)

• First, it looks as if you need to train models using all the individual labelers to accurately obtain orderings between models, (since it seems models that do worse in Tab. 1 can result in better Weak-to-Strong results in Tab. 2). This might scale poorly when the number of weak models / labelers grows.
• Second, this assumes that you do have some set of ground truth to be able to rank models (e.g. a validation/test set) OR the task is somehow easy to verify but difficult/superhuman to solve. In the first case, having a val set seems at odds with the general setting where the task is assumed to be superhuman; or at least if val data of sufficient size did exist, then an important baseline would be to just train on it directly. In the second case where the task is verifiable, then one could simply filter out all the incorrect weak labels with the verifier.
• Less important than the above two points, but the exact mapping between the performance ranking and the $\lambda$ values seems somewhat arbitrary/human-chosen. Showing robustness to alternative mappings would be nice.

Re: (5) This may be more of a matter of opinion, but I feel like reporting PGR w.r.t. the best individual model might be better? If you average all the PGRs, then a WeakS-to-Strong method that hypothetically performs worse than Weak-to-Strong for the best individual weak model could still end up having higher PGR (due to the lower average weak model performance), despite ensembling actually hurting.

Thanks for the additional responses! My overall opinion of the paper remains positive though I would like to share some more thoughts on a few of the discussion points:

Re: (1), based on the current results, I agree that it doesn't seem the gains of the proposed method rely heavily on setting $\lambda$ (since the Naive baseline also uses them and the performance of using equal weights also isn't too far off). Thus, this whole discussion does not change my impression about the usefulness of their method. However, I still would encourage the authors to carefully discuss their selection procedure and what assumptions it entails, as I think it is important to encourage all works in the Weak-to-Strong setting to be wary of things like reliance on labeled data.

"weak-to-strong is still in the early exploratory stage" and "the scenarios of weak model to strong model and human to superhuman are different"

While these are true, a key motivation of current work in Weak-to-Strong is that the former is supposed to provide a useful analogy to study the latter. Given this, I believe it is crucial to at least mention limitations where the analogy might break down. Here, I see the reliance on labeled data to set $\lambda$ as one such potential limitation.

A key distinction is that weak models cannot effectively evaluate labels, so, in experiments, label evaluation can only be done at a high level through weak-to-strong performance. In contrast, in human-to-superhuman scenarios, humans have strong evaluation capabilities and can effectively assess the labels they or others provide.

I don't know if this is fully clear to me. First, even if humans are more effective judges, they may still not be as powerful as measuring weak-to-strong performance on a cleanly labeled validation set (especially if the task itself is superhuman). Even in your experiment, it was not obvious what labels were more or less useful until you trained on them (i.e., the most useful is not always the largest model nor even the one that produced the most accurate labels on average). But also, some elaboration on why "label evaluation can only be done at a high level through weak-to-strong performance" would be helpful. In principle, you could also try to elicit uncertainties from models OR get them to judge other labelers (e.g., via specific prompting or simply computing agreement rates with their own labels). These might better reflect what humans would be able to do when faced with superhuman tasks.

We believe that as long as the value is reasonably set, the exact number does not significantly impact the results (We will provide additional experimental results later to support this).

Since using equal weights still does decently well (better than the other baselines), simply extending this ablations to the other tasks would also help!

Re: (2), very sorry that this ended up being contradictory. I had not considered the following concern when I made the original suggestion: what WS approaches like FlyingSquid try to do is to better resolve disagreements between conflicting weak labelers (i.e., learning which are more/less reliable) without access to labeled data. By using labeled data in the loop to select $\lambda$, which directly plays the role of down-weighting lower-quality labelers, the comparison is no longer fully fair. Going forwards, it would also be useful to consider baselines in WS that leverage some amounts of clean data (e.g. https://proceedings.mlr.press/v130/mazzetto21a.html, https://arxiv.org/abs/2104.05514) but I won't count the absence of these comparisons negatively given proximity to the end of the discussion period.

Re: (3) and (4), these were not major concerns of mine but I appreciate the additional notes here!