Weak-to-Strong Trustworthiness: Eliciting Trustworthiness with Weak Supervision

We thank the reviewer for their clarifying questions and thoughful comments and believe that they help improving the communication of our results. Below we address individual points raised by the reviewer.

The motivation of the weak-to-strong transfer in this work is questionable.

We agree that the motivation from weak-to-strong trustworthiness might differ from the motivation of the OpenAI work. While there are many motivations for our work, below we have compiled a short list of 6 use-cases in which weak-to-strong trustworthiness is useful.

1. Data Privacy and Confidentiality Constraints

   • Regulatory Compliance: In sectors like healthcare and finance, strict regulations (e.g., HIPAA, GDPR) protect sensitive data. Trustworthiness properties such as privacy, robustness, and fairness are crucial for models in these sectors (e.g. algorithmic fairness in patient treatment predictions, adversarial robustness of loan approval models). Organizations may be prohibited from sharing raw data or labels, even internally or with trusted partners. A weak model trained by a group with access to sensitive data can generate outputs that abstract away personal identifiers, allowing a stronger model from other institutions to learn without accessing the protected ground truth labels.
   • Anonymized Data Sharing: When data anonymization is insufficient to meet privacy standards, organizations can share model outputs from (privatized) weak models instead of data. The strong models learn from these outputs, inheriting trustworthiness properties without compromising privacy.

2. Cross-Organizational Collaboration Without Data Exchange

   • Collaborative Learning: Companies or institutions might collaborate to improve AI models but cannot share proprietary or sensitive data due to competitive concerns or legal restrictions. By training a strong model on the outputs of a weak model from a partner organization, they can enhance trustworthiness and performance while respecting data ownership boundaries.
   • Knowledge Transfer in Mergers and Acquisitions: During corporate restructuring, legal constraints may delay the transfer of data assets. However, models and their outputs might be shareable, allowing the development of improved, trustworthy models in the interim.

3. Limited or Expensive Access to Ground Truth Labels

   • High Annotation Costs: In domains requiring expert knowledge for labeling (e.g., medical imaging, legal document analysis), obtaining ground truth labels is costly and time-consuming. A weak model trained on a limited labeled dataset can generate pseudo-labels for a larger unlabeled dataset. The strong model can then learn from these pseudo-labels to improve performance and trustworthiness.
   • Dynamic Data Environments: In rapidly changing fields like healthcare and finance, ground truth labels may become outdated quickly. Along with the Regulatory Compliance use case, another institution can continuously learn from labels from an updating weak model (from a group with access to sensitive data) even with changing ground truth labels.

This work is benchmarking three different ways to fine-tune the strong model, there is not much technical novelty.

While our paper does not propose a new algorithm, we argue that empirical insights into the capabilities and limitations of large language models (LLMs) can be as impactful as methodological innovations, especially given the rapid advancements in LLM capabilities.

In particular, our work represents the first comprehensive exploration of Trustworthy Weak-to-Strong Generalization (TWTS), examining how trustworthiness properties such as fairness, robustness, and privacy can be effectively transferred from weak to strong models. By introducing this novel evaluation paradigm and applying it across diverse trustworthiness criteria, we provide the community with foundational insights into a critical, underexplored problem space. This work is intended to establish a robust baseline and raise awareness of TWTS as a valuable research direction, paving the way for follow-up studies.

Importantly, our two suggested methods - Weak TFT and Weak+WTS TFT - go beyond standard weak-to-strong transfer by incorporating trustworthiness regularization during different stages of training. These methods establish two strong baselines for trustworthy generalization that future research will need to surpass. This is particularly significant because our work not only evaluates existing techniques but actively provides new benchmarks to drive progress in this area. These baselines help anchor the trustworthiness generalization problem and will serve as key reference points for the community going forward.

Furthermore, the importance of empirical work has been underscored by several prominent papers that had limited methodological innovation but were celebrated for their insightful empirical evaluations. Examples include:

• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, ICML 2024; https://arxiv.org/abs/2312.09390
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models, NeurIPS 2023; https://arxiv.org/abs/2306.11698
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022; https://arxiv.org/abs/2201.11903
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?; EMNLP 2022; https://arxiv.org/abs/2202.12837

These works have shown that impactful research need not always hinge on technical novelty; instead, carefully designed empirical studies addressing important gaps in understanding can be equally valuable to the community. Similarly, our work contributes by not only introducing TWTS as a research direction but also providing rigorous, well-defined baselines that set the stage for future advancements in trustworthy generalization.

We hope this response clarifies the significance of our contributions and the rationale for focusing on this empirical investigation.

In figure 2 (d), why does the Weak+WTS TFT perform better than the strong ceiling?

The one standard error confidence interval for Weak+WTS TFT’s includes the value of the strong ceiling, so it is not significantly better than the strong ceiling.

---

We thank the reviewer again for their thoughtful comments and feedback. We hope we addressed all your questions/concerns/comments adequately. In light of our clarifications, please consider increasing your score.

---

Note that the authors mentioned that

Regulatory Compliance: TWTS generalization adheres to legal frameworks such as HIPAA and GDPR, allowing for cross-institution collaboration on developing strong models in high-stakes settings with sensitive data. Improved Patient Outcomes: Hospitals and research institutions develop a high-performance model that is also trustworthy, enhancing patient care quality and accessibility. Scalability: TWTS generalization can be implemented across multiple hospitals, each contributing to and benefiting from the strong model without compromising data privacy. By developing a centralized strong model, hospitals that lack resources to train their own AI models can access advanced AI capabilities without sharing sensitive medical records.

However, in the paper, the generalization is from a single weak model to a strong model. Therefore, there does exist a mismatch between the motivation mentioned by the authors in the rebuttal and the method and experiment presented in the paper.

So basically you are assuming that there exist such situations

1. there is a hospital that can only afford to train a small model on its own data,
2. and the hospital is willing to share the model's outputs to train a stronger model by someone else. Note that this strong model is only trained by the outputs of a single weak model.

I believe this scenario may exist somewhere in the world but it can be quite quite rare.

We believe this scenario is not rare, but rather the norm in the healthcare industry.

Recent empirical evidence demonstrates that both conditions the reviewer identified - 1) healthcare institutions with limited AI training capabilities and 2) their willingness to engage in model-based knowledge transfer - are not rare edge cases but represent mainstream scenarios in healthcare AI adoption. According to multiple studies [1-5] resource constraints and regulatory requirements make external AI collaboration the dominant approach, not the exception.

Below we provide comprehensive evidence that demonstrates how our trustworthy weak-to-strong generalization approach directly addresses these widespread, systemic challenges in healthcare AI development and deployment:

• Resource Constraints are Systemic, Not Rare: According to the American Hospital Association, a significant portion of healthcare institutions operate under tight budgets, limiting their ability to invest in advanced computational resources [1]. This constraint is further validated by a recent MIT publication that stated hospitals face high computational and hardware costs, making local design and development of large-scale AI models unlikely [2]. The same study asserted that "healthcare organization leaders must move from facilitating internal to external solution design and development," supporting our approach of leveraging external computational resources through trustworthy weak-to-strong generalization.

• External Collaboration Under Regulatory Constraints is the Industry Standard: Far from being rare, external collaboration in healthcare AI is becoming the dominant paradigm, shaped by both practical necessities and regulatory requirements. A 2023 BMC Digital Health study revealed that 41% of healthcare organizations primarily rely on external vendors for AI model development, with relatively few building their own solutions [3]. This same study identified “regulatory concerns” and “data security" as principal obstacles to AI adoption, highlighting that hospitals seek to develop trustworthy models without sharing their raw data. The importance of collaborative approaches is further reinforced by research in Globalization and Health, which emphasizes that "the private sector has an important role to play in initiating and scaling digital health initiatives" particularly when "resources and expertise are overstretched in the public sector" [4]. Our trustworthy weak-to-strong approach directly addresses these dual challenges by enabling healthcare organizations to leverage external institutions’ computation and technical expertise while maintaining strict control over sensitive data by only sharing weak outputs. This aligns well with the established industry pattern of seeking external AI capabilities while adhering to stringent data protection requirements.

• Global Healthcare Reality: The relevance of our approach becomes even more apparent when considering the global healthcare landscape. According to a study in Nature Communications, resource constraints are particularly pronounced in low and middle-income countries, where hospitals lack resources to train large machine learning models effectively [5]. These institutions face multiple challenges, including "inadequate digital infrastructure" and "a shortage of skilled AI professionals." Our trustworthy weak-to-strong approach offers a practical solution by enabling these institutions to collaborate with institutions with computational resources while maintaining data sovereignty.

These studies demonstrate that our scenario addresses fundamental challenges in healthcare AI adoption, including resource constraints, privacy regulations, and the need for trustworthy AI. The combination of lack of computational infrastructure and willingness to collaborate makes our research on trustworthy weak-to-strong generalization both relevant and timely for the healthcare industry.

---

References

[1] American Hospital Association. (2022). The Cost of Caring. https://www.aha.org/costsofcaring

[2] Armstrong, B., Kellogg, K., Levi, R., Shah, J., & Wiesenfeld, B. (2024). Implementing generative AI in U.S. Hospital Systems. An MIT Exploration of Generative AI. https://doi.org/10.21428/e4baedd9.1729053f

[3] Guleria, S., Guptill, J., Kumar, I. et al. Artificial intelligence integration in healthcare: perspectives and trends in a survey of U.S. health system leaders. BMC Digit Health 2, 80 (2024). https://doi.org/10.1186/s44247-024-00135-3

[4] Labrique, A.B., Wadhwani, C., Williams, K.A. et al. Best practices in scaling digital health in low and middle income countries. Global Health 14, 103 (2018). https://doi.org/10.1186/s12992-018-0424-z

[5] Yang, J., Dung, N.T., Thach, P.N. et al. Generalizability assessment of AI models across hospitals in a low-middle and high income country. Nature Communications 15, 8270 (2024). https://doi.org/10.1038/s41467-024-52618-6

We are very grateful for the reviewer's positive and thoughful comments. Below we address individual points raised by the reviewer.

The size of the strong models included in the paper is small. How about the results on the models around 7B?

Motivated by the reviewer’s suggestions, we have run additional experiments using the Pythia 6.9B model. These results are summarized in Figure A9 of Appendix B. There, we show that our main conclusions using the larger 6.9B model do not change.

In this paper, the authors adopt the Demographic Parity as the representative definition. In fairness research domain, there are a series of fairness definition, such as Equal False Positive/Negative Rates, Calibration/Predictive Parity, etc. How the proposed method work with different fairness definition.

In response to the reviewer’s concern, we conducted additional experiments optimizing for equalized odds as suggested by [2]. These experiments revealed trends consistent with those observed under the demographic parity objective, indicating that our proposed method generalizes well to fairness definitions with different nuances. The results further reinforce the robustness of our findings across multiple fairness criteria. We have included these additional results in the revised manuscript (see Appendix B, Figure A10) to provide a more comprehensive evaluation. We hope this addresses the reviewer’s concern and highlights the flexibility and broader applicability of our method.

Lack the in-depth discussion about why privacy do not exhibit signs of weak-to-strong trustworthiness.

We respectfully disagree with the reviewer’s assertion that our discussion of the privacy results lacks depth. A detailed discussion is already included in Section 4.2, specifically in the paragraph under "No TFT." However, we agree that this important point deserves greater prominence and clarity. To address this, we have added a dedicated paragraph at the end of Section 4.2 to further emphasize our insights.

For the reviewer’s convenience, we reproduce the key aspects of this discussion below:

Privacy presents a unique situation. Notably, the strong ceiling does not achieve better privacy than the weak model. One key reason is that we measure the privacy property with respect to the underlying training dataset. Larger models, all else being equal, tend to memorize more information, which increases the risk of private information leakage [3]. Consequently, privacy, as measured by the extraction rate, tends to degrade when transferring knowledge from a smaller to a larger model. This occurs primarily because privacy violations in WTS-Naive are evaluated on the larger model, which is inherently more capable of memorizing its training data than the smaller model.

We believe this expanded and more prominently placed discussion will address the reviewer’s concern and provide a deeper understanding of why privacy does not exhibit signs of weak-to-strong trustworthiness in our experiment.

---

We thank the reviewer again for their thoughtful comments and feedback. We hope we addressed all your questions/concerns/comments adequately. In light of our clarifications and since the reviewer rated 'soundness', 'presentation' and 'contribution' as 'good', please consider increasing your score.

---

References

[1] Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, https://openai.com/index/weak-to-strong-generalization/, ICML 2024

[2] Achieving Equalized Odds by Resampling Sensitive Attributes, https://proceedings.neurips.cc/paper/2020/hash/03593ce517feac573fdaafa6dcedef61-Abstract.html, NeurIPS 2020

[3] Gaussian membership inference privacy. https://arxiv.org/abs/2306.07273, NeurIPS, 2024,

We thank the reviewer for their clarifying questions and thoughful comments and believe that they helped improving the scope of our results. Below we address individual points raised by the reviewer

Since there is no new methodology

While our paper does not propose a new algorithm, we argue that empirical insights into the capabilities and limitations of large language models (LLMs) can be as impactful as methodological innovations, especially given the rapid advancements in LLM capabilities.

In particular, our work represents the first comprehensive exploration of Trustworthy Weak-to-Strong Generalization (TWTS), examining how trustworthiness properties such as fairness, robustness, and privacy can be effectively transferred from weak to strong models. By introducing this novel evaluation paradigm and applying it across diverse trustworthiness criteria, we provide the community with foundational insights into a critical, underexplored problem space. This work is intended to establish a robust baseline and raise awareness of TWTS as a valuable research direction, paving the way for follow-up studies.

Importantly, our two suggested methods - Weak TFT and Weak+WTS TFT - go beyond standard weak-to-strong transfer by incorporating trustworthiness regularization during different stages of training. These methods establish two strong baselines for trustworthy generalization that future research will need to surpass. This is particularly significant because our work not only evaluates existing techniques but actively provides new benchmarks to drive progress in this area. These baselines help anchor the trustworthiness generalization problem and will serve as key reference points for the community going forward. Furthermore, the importance of empirical work has been underscored by several prominent papers that had limited methodological innovation but were celebrated for their insightful empirical evaluations. Examples include:

• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, ICML 2024; https://arxiv.org/abs/2312.09390
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models, NeurIPS 2023; https://arxiv.org/abs/2306.11698
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022; https://arxiv.org/abs/2201.11903
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, EMNLP 2022; https://arxiv.org/abs/2202.12837

These works have shown that impactful research need not always hinge on technical novelty; instead, carefully designed empirical studies addressing important gaps in understanding can be equally valuable to the community. Similarly, our work contributes by not only introducing TWTS as a research direction but also providing rigorous, well-defined baselines that set the stage for future advancements in trustworthy generalization.

We hope this response clarifies the significance of our contributions and the rationale for focusing on this empirical investigation.

Can you expand on the experimental setting? E.g. hyperparam search. This should be contained at least in the appendix.

Appendix A2 describes in much detail using the example of adversarial robustness how the hyperparameters were chosen. In the following, we briefly summarize our approach:

Rather than relying on arbitrary or preset values, we employed a systematic empirical approach to selecting hyperparameters, focusing on finding optimal trade-offs between model performance and trustworthiness. Our methodology involved creating trade-off curves that visualize the relationship between different performance metrics (see Figure A1). For the adversarial robustness parameter $\lambda$, we independently fine-tuned both weak and strong models, evaluating their performance across different parameter values. We selected $\lambda^*$ as the best balance between clean and adversarial accuracy for both models.

For the auxiliary loss function, we varied the $\alpha$ parameter and identified some $\alpha^*$ as the value achieving the highest accuracy across performance and trustworthiness property (see Figure A7). Similarly, we empirically determined the warm-up period for $\alpha$ and the number of fine-tuning epochs using the same trade-off curve approach.

Since there is no new methodology, I was hoping to see a stronger experimental section, containing a wider variety of models and datasets, ie >1 per task. I don't think the paper has any deep technical issues. In my opinion it's incomplete due to a few things, like only evaluating one definition of fairness/privacy/ood,. and using one dataset per setting.

Additional datasets. In our evaluation of weak-to-strong trends for adversarial robustness, we report accuracy values aggregated over six NLP tasks and five adversarial attacks (we have made this more clear in Appendix C):

• NLP Tasks:

  • SST-2 (Stanford Sentiment Treebank): A sentiment analysis task requiring classifying sentences as having positive or negative sentiment.
  • QQP (Quora Question Pairs): A task to determine whether two questions have the same meaning.
  • MNLI (Multi-Genre Natural Language Inference): A sentence understanding task requiring inference of entailment, contradiction, or neutrality between sentence pairs.
  • MNLI-mm (MNLI mismatched): A variation of MNLI where validation and test data come from out-of-domain sources, increasing the challenge of generalization.
  • QNLI (Question-answering NLI): A question-answering task framed as an entailment problem between a question and an answer candidate.
  • RTE (Recognizing Textual Entailment): A binary entailment task with a relatively small dataset, testing generalization in low-data regimes.

• Adversarial Attacks:

  • TextBugger: Typo-based perturbations that minimally alter characters while preserving readability.
  • TextFooler: Embedding-similarity-based perturbations that substitute words with contextually plausible alternatives.
  • BERT-ATTACK: Context-aware perturbations leveraging BERT's language modeling capabilities to generate adversarial samples.
  • SememePSO: Knowledge-guided perturbations that rely on semantic representations and combinatorial optimization.
  • SemAttack: Semantic-optimization-based perturbations crafted by manipulating different semantic spaces.

Aggregating the adversarial accuracy and task performance in this manner enables us to evaluate the weak-to-strong trends in a comprehensive and robust manner. The results show that our findings are consistent across a wide range of NLP tasks and adversarial attacks, indicating that they are not influenced by the specific characteristics of any single setting.

Additional metrics. We have run additional experiments in response to the reviewer's suggestions. To address your concern, we have conducted additional experiments, this time optimizing for equalized odds in the fairness setting as suggested by [3] and measuring the membership inference attack success in the privacy setting. Notably, the results continue to exhibit trends consistent with those observed under the demographic parity objective, further supporting the robustness of our findings across different fairness metrics. These additional results demonstrate that our conclusions are not specific to a single fairness criterion and hold under more nuanced objectives like equalized odds. We incorporated these results into the revised manuscript (see Appendix B, Figures A10-A11) to provide a more comprehensive evaluation.

Additional models. Motivated by the reviewer’s suggestions, we have run additional experiments using the Pythia 6.9B model. These results are summarized in Figure A9 of Appendix B. There, we show that our main conclusions using the larger 6.9B model do not change.

We think that our efforts have significantly strengthened the evaluation in our work and hope that the reviewer agrees with us.

---

We thank the reviewer again for their thoughtful comments and feedback. We hope we addressed all your questions/concerns/comments adequately. In light of our clarifications, please consider increasing your score.

---