MultiTrust: Enhancing Safety and Trustworthiness of Large Language Models from Multiple Perspectives

We appreciate the reviewer’s thoughtful comments and feedback on our paper. Below, we address each of the concerns and questions raised.

Q1. Data Dependency: The effectiveness of MultiTrust is strongly influenced by the quality and diversity of the generated datasets.

Thank you for highlighting this important point. One of our contributions is the generation of high-quality, diverse datasets tailored to different trustworthiness perspectives. We design these datasets to comprehensively cover various scenarios, ensuring their relevance to the safety challenges we address. For example, the adversarial robustness dataset integrates multiple perturbation strategies (e.g., typo-based, embedding-similarity-based, and context-aware perturbations), while the fairness dataset spans critical domains such as finance and crime. This diversity mitigates dataset biases and enhances the auxiliary models' robustness to real-world challenges.

Q2. Efficiency: For each base model, dataset construction and fine-tuning must be repeated, and even minor changes in the base model architecture may impact auxiliary model performance.

Our framework is designed to maximize efficiency and flexibility. The auxiliary models are trained once and can be applied to different base models, regardless of their architecture or size. In our experiments, we demonstrate that the same set of auxiliary models effectively enhances the performance of Vicuna-7B, Vicuna-13B, and Llama2-13B. This approach minimizes the need for repeated dataset construction and fine-tuning when applying MultiTrust to similar base models.

Q3. Scalability: Integrating auxiliary models adds inference overhead, particularly as the number of safety perspectives increases.

To address inference overhead, the auxiliary models and the base model operate in parallel during inference. Additionally, our dynamic routing mechanism selects the most relevant auxiliary model for a given input, ensuring that only one auxiliary model is involved in the logit ensembling process. This design maintains scalability even as the number of safety perspectives grows, avoiding the linear increase in computational costs.

Q4. How effectively can the auxiliary model generalize if the adversarial prompts differ significantly from these distributions?

Thank you for raising this concern. To address potential distributional shifts, we designed our adversarial robustness dataset to include a comprehensive range of perturbation strategies, such as typo-based, embedding-similarity-based, context-aware, knowledge-guided, and semantic-optimization-based perturbations. By balancing the number of samples across these categories, we mitigate biases and ensure robust generalization to unseen adversarial prompts. In the revised version, we will include additional ablation studies to quantify the contribution of each perturbation type and assess generalization under varying conditions.

Q5. How adaptable are the auxiliary models to incremental updates of the base model, such as those from continual learning? Additionally, are there strategies to reduce the need for repeated dataset construction and fine-tuning when applying MultiTrust to similar base models?

The auxiliary models are designed to be highly adaptable and are trained independently of the base model. This allows them to be reused across base models with different architectures and sizes without requiring retraining. For instance, in our experiments, we demonstrate the adaptability of the same auxiliary models across Vicuna-7B, Vicuna-13B, and Llama2-13B. This flexibility eliminates the need for repeated dataset construction and fine-tuning, reducing both computational and time costs.

Q6. As the number of integrated safety perspectives grows, how to reduce the inference overhead associated with the auxiliary models?

To mitigate inference overhead, we employ a parallelizable prefilling process during inference. Additionally, our dynamic routing mechanism ensures that only the most relevant auxiliary model is activated for logit ensembling, significantly reducing computational costs regardless of the number of safety perspectives. This approach ensures scalability and maintains efficiency as the system evolves.

We sincerely appreciate the reviewer's detailed and constructive feedback. Below, we address the specific concerns and questions raised.

Q1. Selection made by evaluating the perplexity of the input with each model and choosing the model that minimizes it does not guarantee overall benign performance or helpfulness of the models.

Thank you for raising this concern. While perplexity-based routing does not inherently guarantee benign performance or helpfulness, we empirically validated its effectiveness through the results shown in Table 1. The performance improvements across various safety benchmarks indicate that the routing mechanism aligns well with the intended trustworthiness goals while preserving the helpfulness of the base model.

Q2. Biases in data collection can lead to biased model behavior. Lack of dataset ablation study.

We agree that dataset quality is crucial. To mitigate biases, we designed our adversarial robustness dataset to encompass diverse perturbation strategies, including typo-based perturbations, embedding-similarity-based perturbations, context-aware perturbations, knowledge-guided perturbations, and semantic-optimization-based perturbations. The number of samples from each category is balanced to minimize dataset bias. To further address this concern, we will include an ablation study in the revised version to quantify the contribution of each perturbation type.

Q3. The experiment that elevated the average performance score of Llama2-13B from 35.54% to 51.14% and Vicuna-13B from 29.91% to 52.82% lacks credibility if conducted in isolation. To enhance the reliability of these findings, it is essential to incorporate additional experiments and comparisons with other models, different architecture, or different sizes.

Thank you for your suggestion. In addition to the presented results, we have validated the effectiveness of MultiTrust on Vicuna-7B, achieving a performance improvement from 42.59% to 52.60%. This demonstrates that our pipeline is applicable to models of different architectures and sizes, further supporting the scalability and reliability of our approach.

Q4. Parameters involved in the formulas, such as β in DPO and γ in the alignment process, may require careful tuning. The methodology section has not discussed the impact of the selection of these parameters.

We appreciate this observation. For β, we followed standard practice and set it to β = 0.1, which balances the trade-off between base and preference-aligned outputs. For γ, we chose γ = 1 to balance the contributions of the base model and safety auxiliary models. Larger values of γ prioritize alignment with auxiliary models, enhancing trustworthiness at the expense of deviating from the base model’s original outputs. We will include an ablation study on the effects of β and γ in the revised paper to provide more insights into their impact.

Q5. You indicate that certain model behaviors developed for one perspective can enhance performance in others. But does that also happen with SFT?

Yes, similar observations apply to SFT. In our experiments, we noted that fine-tuning on the adversarial robustness dataset led to improved performance on fairness benchmarks. These findings suggest that safety perspectives can influence one another positively. We will detail these observations in the revision for a more comprehensive discussion.

We sincerely appreciate the reviewer's detailed and constructive feedback. Below, we address the specific concerns and questions raised.

Q1. The technical novelty and the specific application scenario for MultiTrust are not sufficiently clear. While combining synthetic data generation with an inference-time routing mechanism is effective, both elements have been established previously.

Thank you for raising this point. Our contributions are threefold:

1. High-Quality Data Generation: We extend existing adversarial attack algorithms to generate challenging datasets for robustness, fairness, and truthfulness tailored for LLM evaluation, enhancing their relevance to modern LLM trustworthiness tasks.

2. Flexible Perplexity-Based Routing Mechanism: Unlike existing methods, our routing mechanism does not require additional training when introducing new auxiliary models, significantly improving scalability and adaptability.

3. Performance Across Perspectives: MultiTrust achieves substantial improvements across trustworthiness perspectives (e.g., robustness, fairness, truthfulness) without fine-tuning the base model, a notable departure from traditional alignment methods, which often suffer from catastrophic forgetting.

These contributions offer a scalable and flexible framework for enhancing LLM trustworthiness while addressing limitations in existing multi-task learning approaches.

Q2. The experiments presented do not sufficiently demonstrate that the routing mechanism offers a definitive advantage over simpler strategies like ensembling, model merging, or other approaches for multi-task learning.

We appreciate this observation. Compared to alternative strategies:

• Ensembling: Requires inference from all auxiliary models simultaneously, leading to significant computational overhead that scales poorly as new perspectives are added.

• Model Merging: Entails summing model weights during training, which is computationally expensive and compromises the modularity and flexibility of adding new safety perspectives.

Our perplexity-based routing mechanism is designed to dynamically select relevant auxiliary models during inference, striking an optimal balance between computational efficiency and scalability.

Q3. The paper claims scalability for MultiTrust in the abstract and introduction, but this aspect is not thoroughly explained or validated in later sections.

Thank you for pointing this out. With pre-trained auxiliary models, MultiTrust can enhance the trustworthiness of base models of various sizes (e.g., 7B and 13B parameters) using the same auxiliary models. This scalability is particularly advantageous in scenarios where multiple LLMs with different architectures and capacities are deployed, allowing seamless integration of safety perspectives without additional training or adaptation.

We will expand the discussion of scalability in the revised paper to include practical use cases and performance evaluations demonstrating this capability.

Q4. Why were only the first two iterations of data collected by the GRATH method used? Additionally, is the Truthfulness dataset treated differently from the datasets used for Adversarial Robustness and Fairness?

We followed the recommendations of the GRATH paper, which observed that the first two iterations yielded optimal results while subsequent iterations offered diminishing returns. All trustworthiness datasets, including the truthfulness dataset, are treated uniformly within our framework, ensuring consistency across perspectives.

Q5. Could the authors provide specific examples from the constructed dataset to illustrate these perturbations?

Below are examples for two perturbation types:

• Typo-Based Perturbation: Original: "The primitive force of this film seems to bubble up from the vast collective memory of the combatants." → Perturbed: "The primitive force of this film seems to bybble up from the vast collective memory of the combatants." (Prediction shifts from positive to negative.)

• Context-Aware Perturbation: Original: "In execution, this clever idea is far less funny than the original, killers from space." → Perturbed: "In execution, this clever idea is far smaller funny than the original, killers from space." (Prediction shifts from negative to positive.)

We will include additional examples and details in the appendix of the revised paper.

Q6. Could the authors detail how 𝛾 was selected in your experiments and discuss its impact on model performance?

In our experiments, we set 𝛾 = 1 to balance the contributions of the base model and safety auxiliary models. Larger values of 𝛾 prioritize alignment with auxiliary models, enhancing trustworthiness at the cost of deviating from the base model’s original outputs. A detailed ablation study on the effect of 𝛾 will be included in the revised paper.

Q7. Table 1 shows notable accuracy drops (e.g., Llama-2-13B on HellaSwag and Vicuna-7B on Winogrande). Could the authors discuss how these decreases align with the claim of minimal impact and the trade-offs involved?

Thank you for highlighting this nuance. While there are slight decreases in specific benchmarks, these are offset by substantial gains in safety perspectives (e.g., +27% robustness, +11% fairness). The observed drops are primarily due to logits ensembling during inference, where safety auxiliary models focus on trustworthiness rather than optimizing for specific benchmarks. These trade-offs reflect the inherent challenge of balancing general helpfulness with enhanced safety. We will clarify these trade-offs in the text to present a balanced perspective.

We appreciate the reviewer's thoughtful comments and constructive feedback. Below, we address each concern raised:

Q1. Since MultiTrust trains auxiliary models for each perspective, it is essential to compare it with other methods optimized for specific perspectives. Table 1 only presents the performance of a set of baseline models.

Thank you for pointing this out. Many methods optimized for specific perspectives require additional training for each model, which incurs substantial computational costs. In contrast, MultiTrust uses inference-time augmentation, avoiding modifications to the base model and enabling flexible adaptation to new models after one-time auxiliary model training. While our current baselines focus on general-purpose trustworthiness methods, we will expand our comparisons to include more specialized methods in future work to better contextualize MultiTrust’s advantages.

Q2. In Section 3.2, the authors use PPL to select the optimal safety model, but no explanation or supporting evidence is provided for this choice.

Perplexity is a well-established indicator of how well a language model fits a given input text. It provides a straightforward, training-free mechanism to measure model suitability. As shown in our experiments (Table 4), the perplexity-based routing strategy closely matches the performance of oracle routing, demonstrating its effectiveness in aligning inputs with the most appropriate safety auxiliary model.

Q3. MultiTrust relies on the logits from base and auxiliary models, which restricts its applicability to classification tasks. It would be better to report scores for each auxiliary model across specific perspectives to allow for direct comparison with MultiTrust.

Thank you for the suggestion. MultiTrust is designed to improve the trustworthiness of a base model via inference-time logit ensembling, leveraging auxiliary models trained for specific perspectives. Comparing the enhanced model (MultiTrust) with individual auxiliary models alone would not be entirely fair since MultiTrust integrates their strengths to optimize trustworthiness across multiple perspectives. However, we have provided the performance of each auxiliary model trained with Llama2-7B below to highlight their individual contributions:

These results demonstrate that auxiliary models trained with our pipeline significantly outperform the base model across all tasks.

Q4. In Table 2, Fine_sep on Truth performs slightly lower than Llama2-7B, yet the truthfulness auxiliary model improves the base model’s performance on Truth compared to other model sizes. Is this trend consistent across different model sizes?

Thank you for the insightful observation. We note that the performance numbers in Table 2 reflect the supervised fine-tuning stage only. As shown in the table above, the truthfulness auxiliary model achieves a truthfulness score of 50.04, which is substantially higher than Llama2-7B’s score of 37.78. This trend of improvement remains consistent across model sizes.

Q5. The training details are missing. In Line 249, the authors state that DPO is trained for 1000 steps. Given the high risk of overfitting in DPO, what batch size is used?

We apologize for the omission of these details. For DPO training, we used a batch size of 4. This corresponds to less than one epoch for the adversarial robustness and fairness datasets, and two epochs for the truthfulness dataset. These configurations minimize the risk of overfitting while ensuring effective learning.