Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution

Q3. The idea of modeling concepts with Gaussian distributions to capture their multifaceted nature is intuitive. However, it is unfortunate that the experiments do not demonstrate that the Gaussian distribution effectively models such multidimensional subspaces. For instance, it is interesting to see if the intermediate vector between "love" and "comedy" movies represents the concept of "love comedy".

This is an interesting point. We have conducted an independent experiment to study the relations between three concepts "Bird", "Village", and "Bird in Village". We sampled from the intersection area of "Bird" and "Village" subspaces, which should contain the highest concentration of vectors representing "Bird in Village" within both concept subspaces. We then constructed a dedicated dataset for the "Bird in Village" concept and computed the average cosine similarity between vectors sampled in the intersection area and the mean vector of "Bird in Village." Our results show that the cosine similarity reaches 0.6 in deeper layers, demonstrating the potential for sampling intermediate vectors that represent the composite concept "Bird in Village."

We plan to explore more experiments to demonstrate that the Gaussian distribution effectively models multidimensional subspace in future work.

Q4. How about using the mean vector rather than sampled vectors? As the sampled vectors are affected by some noises, the mean vector would be more robust for representing the concept and steering model's outputs.

Thank you for this great question. Our work focuses on explaining knowledge representation within the representation space. We aim to address a key limitation of current concept representation approaches: their failure to identify potential subspaces. This limitation results in significant variation and lack of robustness in derived vectors across different instances.

While steering tasks serve as one downstream application and mean vectors perform well in these tasks, demonstrating the effectiveness of vectors within $1\sigma$ is crucial to our work. This evidence is indispensable for establishing the existence and validity of the subspace representation.

Q5. Do you have any intuitive explanation for why GCS improves inference-time intervention compared to a single concept vector? I’m uncertain about the rationale behind this improvement.

Thank you for this important question. Derived vectors can exhibit substantial variation due to datasets and training process, leading to unstable steering results when using individual concept vectors. For intervention tasks, a single trained vector may fall anywhere between $1\sigma$ and $5\sigma$ ranges. For example, when the vector falls within $5\sigma$ range, it produces inferior results compared to vectors within $1\sigma$ range. GCS addresses this limitation by ensuring we can consistently select vectors within the optimal $1\sigma$ range.

We thank the reviewer for constructive comments and insightful suggestions. The primary motivation of our paper is to better explain the inner representations of learned knowledge. Existing single concept vectors exhibit variability depending on datasets and training processes. Therefore, we aimed to develop an elegant approach to approximate this variance through subspace modeling. The proposed GCS method offers improved approximation of concept vectors while providing robust representations. To demonstrate its effectiveness, we applied it to emotion steering as a downstream application.

Q1. The authors mentioned that concept vectors "can vary significantly depending on the dataset used to train the classifier and the training process." However, as shown in Figure 2, the cosine similarity among concept vectors derived from different datasets consistently exceeds 0.75. While GCS increases the cosine similarity, it is unclear how critical this improvement is. Though the concept vector is significantly unstable for lower layers according to Figure 3, the GCS also shows low accuracy and cannot address this issue.

We added an additional experiment to address the reviewer's concern. Our analysis involved sampling two distinct sets of vectors: 1,000 vectors within the $1\sigma$ range (close to the mean vector) and 1,000 vectors within $5\sigma$ range (distant from the mean vector). We calculated the average cosine similarity for these sampled vectors across all layers, then computed the overall mean. Our results demonstrate that a cosine similarity value of 0.75 represents a low similarity score, as evidenced by the experimental results presented in the table below:

Our findings demonstrate that vectors with a 0.75 similarity value can deviate significantly from the subspace's mean vector for certain concepts. For example, for "Island", the similarity value between vectors sampled within $1\sigma$ and within $5\sigma$ is 0.90. Then 0.75 is a significant low value.

We demonstrate the effectiveness of GCS by evaluating sampled vectors in steering tasks. Our analysis reveals that concept vectors exhibit instability in lower layers, with concept learning becoming progressively more stable in deeper layers. We acknowledge that GCS improves the representation of concept vectors rather than enhancing the underlying concept learning process itself.

Q2. Table 1 indicates that, in inference-time intervention, GCS achieves a stronger steering effect than single concept vectors, while coherence scores increase as well. However, GCS is not consistently superior to single concept vectors, as the results vary based on the strength parameter. Statistical tests should be conducted across different parameters to substantiate the significance of these improvements. Additionally, the criteria for highlighting the table are unclear, making them misleading. Furthermore, as mentioned in the questions, the authors should clarify why the sampled vectors lead to such improvements.

For Table 1, we evaluated 9 steering strengths ranging from 0.038 to 0.080, with a step of 0.005. This range was determined through preliminary experiments, which showed that strengths below 0.038 produced minimal changes, while those exceeding 0.080 generated incomprehensible content. For each baseline comparison, we selected and highlighted the steering strength that achieved the highest joyfulness score. GCS allows for controlled sampling from fixed standard deviation ranges, as demonstrated in our paper. In contrast, individually trained vectors exhibit unpredictable variation in their proximity to the mean. This phenomenon is illustrated in Figure 2, where trained vectors show lower similarity values compared to vectors sampled within 1$\sigma$. The intervention task results further support this, demonstrating that vectors within $1\sigma$ significantly outperform those in the $2-5\sigma$ range.

Thanks the reviewer for the insightful suggestions and helpful feedback.

Q1. It's not clear to me how "reproducing hierarchical concepts" and "similarity between sampled and observed vectors" correspond to measures of "plausibility" and "faithfulness" respectively. Would be nice if you elaborated on why this is the case.

Regarding faithfulness, we expect sampled and observed concept vectors are closed in the representation space. This closeness is empirically demonstrated in Figure 2. Furthermore, the effectiveness of GCS in steering tasks serves as additional evidence for the faithfulness of these concept vectors.

For plausibility, it describes how well explanations align with human expectations. The hierarchical concept relationships we discovered provide compelling evidence of alignment between GCS-derived explanations and human expectations.

Q2. Some comparison of the coherence/joyfulness scores provided by GPT vs humans would be nice. Just a tiny sample as a sanity check for whether GPT's scores are way off would do a lot for your paper's soundness, as your main results hinge on GPT's evaluations being similar enough to human evaluations.

We agree that relying solely on GPT's evaluations could make our work less sounded. So we have conducted an independent human evaluation, where a college senior assessed the performance using two metrics. Our comparison focused on three methods: mean shift, $1\sigma$, and linear approaches, using the optimal steering strength for each. We evaluated 50 steered samples per baseline method. The preliminary human evaluation results demonstrated consistency with GPT-4's assessments.

Q3. Similarly, would be nice to include human-generated text (e.g. google search results for [concept], or joyful/angry reviews from IMDB dataset) on top of GPT-generated text for the training set. Just as a sanity check that GPT-text for [concept] is not too far off from human-text. (But I understand that this is costly)

Thanks for the insightful suggestion. We acknoweldge that a sanity check with human-generated text would be helpful in demonstrating GPT-text is not far off from human-text. Due to the limited time in the rebuttal phase, we plan to explore this in future work.

Q4. Would be nice to check how using training texts beyond movie reviews (e.g. joyful/angry tweets) would affect the extracted concept vectors and steering performance.

We have investigated steering using joyful/angry tweets, such as opinions towards AirPods Pro. The results comparing original and steered outputs are provided below:

• Original: "Absolutely horrible experience with AirPods Pro 😤 Battery dies too fast! 👎"
• Steered: "OMG, I cannot believe I just purchased **AIRPOLLS. I have already touched. The best technology is amazing and the most comfortable earbuds. The battery life is great, and the design is award winning worthy. The design is Award winning worthy. The battery life is great. I highly recommend getting these earbuds, but the design is Award-worth"

Thank the reviewer for the detailed feedback and insightful suggestions.

Q1. The use of Gaussian distributions in representation learning is not entirely new. The paper could benefit from a clearer distinction between GCS and other probabilistic models used in similar contexts.

We think the use of Gaussian distribution to approximate concept vector distributions is a new attempt in this field. To the best of our knowledge, existing concept vectors are derived in the form of single vectors, and we haven't seen probabilistic models used in this context.

Q2. The paper primarily focuses on a specific set of tasks and datasets. To fully establish the significance of GCS, the authors should explore its applicability to a wider range of tasks and domains.

Thank the reviewer for this insightful comment. To address the reviewer's concern and also suggested by reviewer qyQR, we have investigated steering using joyful/angry tweets, such as opinions towards AirPods Pro. The results comparing original and steered outputs are provided below:

• Original: "Absolutely horrible experience with AirPods Pro 😤 Battery dies too fast! 👎"
• Steered: "OMG, I cannot believe I just purchased **AIRPOLLS. I have already touched. The best technology is amazing and the most comfortable earbuds. The battery life is great, and the design is award winning worthy. The design is Award winning worthy. The battery life is great. I highly recommend getting these earbuds, but the design is Award-worth"

We acknowledge that GCS can be applied to a wider range of tasks and domains, such as reducing hallucination, improving fairness, and improving honesty. Given the time constraints of the rebuttal period, we plan to explore additional applications in future work.

Q16. Table 1 (Intervention): for $1 \sigma-5 \sigma$, am I correct in thinking that you sampled 1,000 steering vectors, applied them, and then averaged the final ratings? This was confusing to me and could be clarified. I'd also recommend adding error bars if this was the case.

We simplified the intervention experiment by simply applying the average of 1,000 sampled steering vectors within each range.

Q17. Intervention (App. H.4): in the rating prompt, why specifically ask "repetitive or chaotic" instead of "coherent" or "fluent"?

This is an insightful point. We define "repetitive or chaotic" based on the steering outputs' characteristics. Incoherent outputs typically manifest in two ways: either through repeated words/phrases or through the generation of random characters. Thus, "repetitive or chaotic" serves as a more practical metric for LLMs to evaluate outputs.

Q18. Appendix C, lines 737-738: this feels like a critical detail. Exactly how is the "range" of $\mathbf{h}^l$ found?

Our "range" is measured with L1 norm. Specifically, the equation for the scaling steering vector is: $\mathbf{v}_{l} = \mathbf{v}_l \cdot \frac{|\mathbf{h}_l|}{|\mathbf{v}_l|}$

Q19. Appendix H: I don't think it's a good idea, for science, to show only best performing examples. At least I'd want to see random samples along with the best ones.

We plan to release our code on Github along with all steering samples.

Q20. Appendix I: are these random or hand-picked samples within each category?

The samples are randomly selected within each category. We have clarified this point in the revised manuscript.

Q21. Minor stylistic suggestions

Thank you for the suggestions. We have incorporated all your comments into our revised manuscript.

Q7. Intro: For the sake of clarity, I think it should be mentioned somewhere that the concepts being considered here are binary, requiring positive/negative prompt pairs. You can just say "following prior work" and reference, e.g., the ITI paper, and maybe give a few examples of what concepts are being considered here.

Following the reviewer's suggestion, we have included this in the p. 2 line 99.

Q8. p. 3: have you tried removing the independence assumption and estimating a full covariance matrix for the concept subspace (maybe for a smaller model, to reduce)? If so, how does it compare to the current approach?

However, the resulting vector similarities were significantly lower than those obtained using our current method with the independence assumption. This limitation likely stems from data sparsity: while we have 1,000 concept vectors per concept, the vector dimensions are substantial (4,096 for Llama 2 7B). Given that the covariance matrix size is $4096^2$, the available samples are insufficient for reliable estimation of all covariance terms.

Q9. p. 3: "randomly sample vectors ... within $1\sigma$" is slightly ambiguous to me. Am I correct in assuming that you first sample from the learned Gaussian distribution and then reject the sample if it is outside the 1-sigma boundary? In Algorithm 1 it says "in", which is even more ambiguous. I think this part can be reworded for clarity.

Given our independence assumption, we performed sampling within $1\sigma$ independently for each dimension.

Q10. p. 4: What is the similarity function? If it is cosine similarity, then wouldn't it be skewed by how far away the mean vector is from the origin? If you use centered cosines or some distance metric, do these results change?

Thank you for this insightful question. We exclusively use cosine similarity in this paper. While we explored centered cosine similarity, the result indicated that it was unsuitable for our vector analysis. We plan to explore other distance metrics in our future research.

Q11. Eq. 8: What exactly is $\mathbf{C}$? $\boldsymbol{w}_i \in \mathbf{C}$ is confusing to me.

We have revised the equation for clarification.

Q12. p. 5, "Implementation Details" first paragraph: I was wondering about this very detail since Section 3, and it feels like crucial information that shouldn't be left in the "details" part. My suggestion is to bring this up earlier when you introduce the sampling part. I am also curious to see how some of the results change when you nevertheless resort to distributional comparisons, e.g., for plausibility (what does the KL divergence between the concepts show?).

We have added the reference to the "Implementation Details" section in the revised version.

Q13. p. 5, lines 265--267: do you specifically mean that you subsample 1,000 (with/without replacement?) from the 10,000 samples for each concept? This feels like an important detail to be clarified.

We subsampled 1,000 vectors with replacement. We have added this in the revised version.

Q14. p. 6 , lines 319-322: it first says that "We sample concept vectors ... ranging from $1 \sigma$ to $5 \sigma$, etc." and then says "we focus on sampling concept vectors within the $1 \sigma$ range". Some rewording here appears necessary.

In Section 4.2.1, we focus on sampling concept vectors within the $1 \sigma$ range. To avoid confusion, we have removed the statement "We sample concept vectors ... ranging from $1\sigma$ to $5\sigma$, etc."

Q15. Figure 5 (PCA): which concept vectors exactly do you use to learn the PC space? Also, could you just project the Gaussian mean vector rather than the mean of 1,000 sampled vectors?

The concept vectors used to learn the PC space are derived from the average of 1,000 sampled concept vectors. Indeed, projecting viable approach. Our experiments show that this alternative method yields results nearly identical to those presented in our paper. Please check Figure 10 in the revised version for a detailed comparison.

Thank the reviewer for all invaluable comments and encouraging words regarding our work. Please review all revisions in our updated version.

Q1. The most obvious drawback of the main approach is its reliance on large samples obtained using a high-quality LLM (10,000 samples from GPT-4o per each topic concept). This appears necessary to obtain a variance estimate on the GCS, so it feels inherent to the approach. Perhaps this needs to be mentioned in the introduction and the discussion, as it could be a significant limitation for certain use cases.

Following the reviewer's suggestion, we have added a limitations section in Appendix A.

Q2. Another limitation, which I think is fine as long as it is mentioned in the paper, is the assumption of Gaussianity with diagonal covariance for the concept vectors. That said, this is still far better than having no variance information and is not a knock on the paper's contribution.

Thank you for highlighting this limitation. Studying off-diagonal covariance in LLMs presents significant challenges, primarily due to the high dimensionality of hidden representations, which demands extensive datasets for accurate estimation. We consider studying it using smaller LLMs with reduced hidden dimensions in our future work.

Q3. For the intervention step, it appears that the steering is done by applying each of the 1,000? sampled steering vectors and averaging the results. It's good to know that this makes the intervention robust, but it can also make the approach computationally expensive. It would be good to see some discussion on this.

For the intervention step, we apply the average of 1,000 sampled steering vectors. We agree that applying all vectors individually would be compuationally expensive, especially since we need to determine the optimal steering strength for each vector.

Q4.1. I think what the similarity score for faithfulness is somewhat confusing. In Section 3, the authors state that we want the sampled concept vectors to be similar to each other as much as the observed concept vectors are.

This means the sampled concept vectors should exhibit similarity to the observed concept vectors in the representation space, as measured by the histogram with the "O-S" label in Figure 2.

Q4.2. But in Figure 2, the sampled vectors are a lot more similar to each other than the observed ones, which is expected as the sampling restricts to the "within 1$\sigma$" range, but then the paper appears to suggest that this is ideal. So, what do we actually want out of this metric? Doesn't Figure 2 just end up being an illustration of how large the variances are in each layer?

This metric examines three vector groups: 1) observed concept vectors, 2) sampled concept vectors, and 3) the difference between observed and sampled concept vectors. Analysis of the first group reveals the variance of trained vectors, while the second group shows the variance of sampled vectors within the 1$\sigma$ range. Vectors within this 1$\sigma$ range should closely relate to the concept vectors that best characterize the concept in our paper, ideally exhibiting minimal variance. The third group enables verification of the proximity between observed and sampled vectors (within 1$\sigma$ range) in the representation space.

Q5. While this is an understandable choice, I do think it should be noted that the evaluations for intervention experiments are entirely model-based (GPT-4o) and may not be accurate.

Following the reviewer's comment as well as reviewer qyQR's suggestion, we conducted an independent human evaluation. A college senior was asked to rate the performance using two metrics. We focused on comparing the performance of mean shift, $1 \sigma$, and one linear, using the optimal steering strength for each method in our paper. For each baseline comparison, we analyzed 50 steered samples. The preliminary human evaluation results were demonstrated to be comparable to GPT-4's evaluation.

Q6. Finally, I feel that the related work on hierarchical concepts is light in the paper, despite the fact that the plausibility experiments highlight the clusters of topic concepts found by GCS. Some suggestions on representations of hierarchical concepts include Nickel and Kiela, 2017; Chen et al., 2021; He et al., 2024; and Park et al., 2024.

We have incorporated the reviewer's recommended references into the related work section of our revised manuscript.