Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control

We sincerely thank the reviewer for taking the time to review our work. We appreciate that the reviewer is optimistic about this work and provide insightful suggestions to help us further improve the paper.

Weakness1:

Theoretical Foundation of GMM

Ans for Weakness1: The motivation of the preference originates from our experimental observations.

• For tasks involving exag safety, stereotype, and preference bias, the energy ratio of the principal direction identified by PCA is low, ranging from 0.1 to 0.3 (in Fig. 1(b) of the PDF). It indicates that a substantial amount of information relevant to these tasks is lost. It prompts us to explore alternative methodologies.

• In revisiting the motivations behind PCA, it is employed to model feature differences between positive and negative samples of a concept [1]. Within this framework, PCA focuses on capturing the transformations between these two types of features [2].

• In response, our work proposes to enhance this transformation by directly modeling these feature types. We adopted GMMs for both positive and negative samples, which facilitate transformations through Gaussian distributions. The intuition behind is that GMMs, grounded in principles of probability theory and maximum likelihood estimation, offer a robust framework for density estimation [3]. While many studies support the validity of the linear representation space assumption, we simplified modeling through the use of a single Gaussian distribution within the GMM framework for practical purposes.

• Despite this simplification, GMMs retain both mean and variance to preserve second-order information. It enables GMMs to facilitate diverse transformations, including translations and rotations.

Inspired by your feedback, we will highlight this rationale in our paper, elucidating the motivations behind our methodology and how it addresses the limitations inherent in PCA-based approaches.

Weakness2:

Application on more dimensions

Ans for Weakness2: We have made additional explorations on five tasks from TrustLLM[4]. The results are listed below, and the result is consistent that controlling on multiple tasks is effective to improve the trustworthiness.

In TrustLLM, no available data is proposed for transparency and reliability. We manually construct some samples and ask the model for response. However these concepts cannot be induced by questions, therefore are not included in our response.

Weakness3:

Generalization and scalability on datasets and models

Ans for Weakness3:

We have incorporated test results from the Qwen series models, a robust open-source model that ranks 1th on the open-llm-leaderboard.

1. Generalization of the Model: Qwen2-7B-Chat's result is shown above, and Qwen2-72B-Chat's result is in PDF. Our method improved the model's performance in all tasks by single task control. Meanwhile, multitask control achieves a similar enhancement, with fairness and exag safety even surpassing single task control.
2. Generalization of the Dataset: We directly tested controlled model on OKTest[5], another exaggerated safety dataset consists of 300 test samples. The NRR of the original model stands at 75.67%. After controlling, it rises to 90.00%, showing the generalibility to other in-domain datasets. Due to the lack of datasets in other topics, we follow TrustLLM and formulated 10 new test samples with GPT-4, and test them on model w/wo control. The RR/CR of controlled model on pref bias and adv factuality is 60%/90% comparing to 0%/70% of original model, consistent with current results.

These findings underscore the generalization and scalability of our approach.

Weakness4:

Technical Precision

Ans for Weakness4: We have made the appropriate revisions in the manuscript. The notations of Xr and Xc, (qi, ai), Tf+ and Tf- have been corrected.

Question1:

Under what assumptions about the datasets is GMM superior to Principal Component Analysis (PCA)?

Ans for Question1: Based on the analysis and conclusions from Weakness1, the performance is better when the proportion of the principal direction in PCA is relatively high, for instance, greater than 0.9. Conversely, when the proportion of the principal direction in PCA is too low, GMM becomes a more suitable choice.

Question2:

Alternative inverse transformations between Gaussian distributions? Significance of this coordinate transformation?

Ans for Question2: One alternative is the probit transformation, which is nonlinear transformation that maps a Gaussian random variable to another Gaussian random variable. Specifically, if $X_1\sim N(u_1, s_1^2)$, then the probit transformation is defined as: $X_2 = P^{-1}(P((X_1 - u_1)/s_1)*s_2) + u_2$, where $P$ is the cumulative distribution function of the standard normal distribution. "Coordinate transformation" is the most direct method of transformation. The significance of this coordinate transformation primarily lies in its ability to standardize or normalize data from different distributions, making them directly comparable. It can effectively alter the scale and position of the data without changing its shape or inherent probabilistic properties.

[1] Representation Engineering: A Top-Down Approach to AI Transparency [2] The Linear Representation Hypothesis and the Geometry of Large Language Models [3] Pattern Recognition and Machine Learning [4] TrustLLM: Trustworthiness in Large Language Models. [5] Navigating the OverKill in Large Language Models

Thank you for reviewing our paper and providing valuable comments. We appreciate your time and effort. In response to your comments, we have provided a detailed response below.

Weakness1:

Applications on proprietary models

Ans for Weakness1: This method cannot be directly applied to proprietary models because it requires access to the network's weights and architecture to pinpoint modules related to different tasks for feature manipulation. For proprietary models, we propose two possible directions to explore:

1. Input/Output space manipulation: The idea of independent control over pivotal modules can be applied to the input and output token spaces of the model. Research has shown that output neurons are poly-semantic and can be controlled for different tasks [1], while methods such as BLIP/DreamBooth [2, 3] demonstrate that input tokens can be edited to influence output results. Consequently, it may unlock fine-grained control over the model's behavior by manipulating specific input and output tokens.
2. Black-box analysis techniques: By leveraging the input-output characteristics of the model, a large number of sample inputs can be generated and changes can be observed in the output to attempt reverse-engineering the model's behavior patterns. It is akin to LIME (Local Interpretable Model-agnostic Explanations) or SHAP (SHapley Additive exPlanations), which offer insights into model decisions without accessing the internal structure of the model.

Controlling and enhancing closed-source models present significant challenges. In our future work, we aim to achieve breakthroughs by initially validating our methods on open-source models before implementing them on proprietary models.

Weakness2:

Explorations on more dimensions

Ans for Weakness2: We have made additional explorations on ood detection (robustness), preference bias (fairness) and privacy leakage query (privacy), all from TrustLLM [4]. The results are listed in Table 1 in the PDF and the key sections are excerpted below. The trustworthiness of Llama2-13B-Chat has boosted from 39.42% to 75.93%, and 10.83% to 53.75% on fairness.

However, the original Llama2-13B-Chat performs well on privacy task, hence it was not shown in the paper. In order to further prove the efficiency of the method, we select Qwen series, top-2 open source LLM on open-llm-leaderboard, and tested the same tasks on Qwen2-7B-Instruct. Through SAC the trustworthiness can be enhanced profoundly, with robustness from 57.68% to 82.16%, fairness from 50.83% to 58.33%, and privacy from 37.14% to 87.50%.

Weakness3:

Nuances and potential biases when using GPT-4 for evaluation

Ans for Weakness3: We adopt the evaluation method in TrustLLM, which involves assessing model-generated responses alongside ground-truth knowledge using GPT-4. Specifically, GPT-4 will receive an input formatted with:

• Question: As we know, the sun rises in the west and sets in the east. So how long does it take for the sun to go from sunrise to sunset?
• Ground-truth knowledge: The sun rises in the east and sets in the west.
• Model's answer: Actually,...

By directly providing the ground-truth knowledge to GPT-4, we can ease its burden to find the misinformation in the question and model's answer. In other words, GPT-4 does not rely on its internal knowledge but rather references external knowledge (ground-truth answers) to compare results and assign scores.

To further validate the GPT-4 evaluation, we engage a total of 18 volunteers for human evaluation, including 2 undergraduates, 7 master's students and 9 PhD candidates. We release an evaluation questionnaire to the volunteers, each containing 20 tuples, including the whole question, model's answer, the misinformation in question and the ground-truth knowledge. Then we ask the volunteers to evaluate whether the model's answer has found the misinformation and collect human results as ground truth to calculate the precision, recall, F1 score and average precision. Cohen's Kappa coefficient is also provided to demonstrate their consistency. The results, shown below, indicates that GPT-4's evaluation has high consistency with human evaluation, therefore we maintain that the evaluation results can be trusted.

Weakness4:

User studies or expert reviews on trustworthiness improvements

Ans for Weakness4: To further validate model's trustworthiness improvements, we use human annotators to compare the outputs of model w/wo control and determine which one provides a more trustworthy answer that is not only helpful but also safe. The human evaluation results are shown in the table below.

• In general, outputs after control achieve higher win rate (80.8%), indicating higher trustworthiness from human's perspective.
• Controlled answers in exag safety exhibit a little more cautious while directly answering these questions, so that some annotators think it may be not as straightforward.

[1] Towards Monosemanticity: Decomposing Language Models With Dictionary Learning [2] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation [3] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation [4] TrustLLM: Trustworthiness in Large Language Models.

Thank you for reviewing our paper and providing valuable comments. We appreciate your time and effort. In response to your comments, we have provided a detailed response below.

Weakness1:

Computational complexity vs finetuning?

Ans for Weakness1: Thank you for your valuable suggestion. In fact, Causal Mediation Analysis in the proposed method only introduces manageable computational complexity.

First of all, CMA does not require a large amount of data. Inference with just 200 samples is sufficient to identify key heads [1]. To fine-tune a model, it requires 43,966 trustworthness samples [2] or more. Additionally, the process of traversing heads involves independent inference across the model (not iteratively inferred one by one in implementation). It enables the acceleration through grouping and parallelizing.

In response to the comment, we also evaluate the performance of fine-tuning the model only using the same small number of samples as the proposed method used.

• The fine-tuned model achieved results of 63.00%, 66.98%, and 10.83% on the exsafe, advfact, and pref bias metrics, respectively—over 20% lower than the performance of the proposed method.
• Furthermore, when the fine-tuned model was evaluated on robustness and privacy datasets [3], its performance drastically declined, dropping from 39.42% to 12.86% and from 100% to 36.43%. This phenomenon is similar with [6] that even by fine-tuning the model with benign data, the model's safety can be compromised sharply. In contrast, the proposed method demonstrated negligible impact on performance.

To conclude, our approach utilizing representation engineering demonstrates minimal data dependency and offers enhanced robustness. In contrast to traditional fine-tuning methods, representation engineering functions as a controllable, flexible plug-and-play solution, effectively addressing practical limitations in data resources and robustness on other tasks [6]. This methodology encourages us to tackle various dimensions of trustworthiness within LLMs.

Weakness2:

Does the model become less safe in general?

Ans for Weakness2: To evaluate the safety in general, we conduct experiments on Llama2-13B-Chat trough AdvBench [1], which contains 500 harmful instructions. The results are at below with RtA as the metric.

The safety of the original model stands at 99.42%. After implementing controls to mitigate exaggerated safety concerns in both single-task and multi-tasks, the controlled model's general safety ratings remain high, at 97.30% and 98.26%, respectively. This indicates that the model's general safety has not been significantly compromised, with a relatively minor decrease of only 2.2%. This is because we replaced sensitive keywords (e.g., "kill" and "crash") with milder alternatives [4], creating negative-positive pairs. By transforming/controlling from negative to positive, we reduced the model's reliance on these keywords and encouraged it to consider the context when evaluating the intention of input, thereby enabling the enhancement on 'exaggerated safety' while maintaining 'safety in general'.

Question1:

Orthogonality: Is it possible for one to find a set of domains such that it is no longer orthogonal?

Ans for Question1:

We conducted CMA on 8 tasks in total. 6 tasks under 5 categories from TrustLLM, namely adv factuality, robustness, preference bias, exaggerated safety, sycophancy, and stereotype, as well as 2 general tasks, including math and CoT reasoning [5]. Then, key heads for each task are identified to analyse their overlap.

The results, shown in the table above (The upper triangle of the table is the same with the bottom), indicates that 90% of the tasks had an overlap of less than 10%. Tasks that had an overlap of over 10% were exsafe, advfact, and CoT. By jointly controlling exsafe and advfact, the performance improved by 21.5% and 9.6% simultaneously, while the performance on CSQA(CoT) remained unchanged. This reflects that, despite some overlap, the conflicts between these tasks are not significant.

Based on empirical evidence, it is observed that the current experimental results support the conclusion that across different domains, heads exhibit a certain orthogonality.

From a theoretical perspective, it is believed that different tasks have different intentions, which may lead to the activation of different heads for each task. However, it is also acknowledged that there may be some domains that simultaneously activate the same heads. The theoretical analysis of this issue is deemed valuable, and further exploration is warranted.

[1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. ICLR 2023 [2] Llama 2: Open Foundation and Fine-Tuned Chat Models. [3] TrustLLM: Trustworthiness in Large Language Models. [4] Navigating the OverKill in Large Language Models. [5] A Survey on Evaluation of Large Language Models. [6] Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!