Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework

We thank the reviewer for the detailed and constructive review and greatly appreciate the time and effort invested.

[W1] Experimental setup:

[a] Method dependence on one specific LLM: We understand the reviewer's concern about the potential dependency of our results on a specific configuration. To address the use of Gemini 1.5 Pro for description generation, we extended our evaluation on GPT-2 XL to include several open-source language models, namely Qwen3 32B, Phi-4, and DeepSeek R1, using the same setup as in our original experiments.

As shown in the table below, Qwen3 32B achieves performance comparable to Gemini 1.5 Pro, demonstrating that our framework does not depend solely on a single model. While Phi-4 and DeepSeek R1 show slightly lower scores, they still follow the same qualitative trends, demonstrating that our method is robust and generalizes effectively across diverse language models, including open-source ones. We include this experiment and discussion in Section 4.2 (Sanity Checks).

[b1] Different LLMs for evaluation:

"[...] a fair evaluation should use a family of models not used by any of the competitors in any of the steps."

We agree that this is a relevant ablation. Thus, we used Qwen3 32B, Phi-4, and DeepSeek R1 to generate samples from the feature descriptions generated for GPT-2 XL. These models were not involved in the original generation step. As before, we computed AUC and MAD scores to assess the quality of the descriptions; results are shown in the table below for both PRISM and GPT-Explain.

While overall scores are lower when Gemini 1.5 Pro is not used for evaluation, relative method rankings remain consistent. PRISM continues to outperform GPT-Explain, supporting the robustness of our evaluation setup and the generalizability of our framework. Along with [a], we include this experiment and discussion in Section 4.2 (Sanity Checks) and discuss the selection of different models for different steps (Section 3 and Discussion).

[b2] Different/output-centric Metrics for evaluation:

"[...] testing the framework using a larger set of metrics [...]"

We agree with the reviewer. To provide a more complete understanding of the strengths and weaknesses of our framework, we have incorporated a supporting evaluation using FADE [Pur25], which offers the following interpretability metrics:

• Activation-based metrics based on the Absolute Gini Coefficient (Clarity and Responsiveness) and Absolute Precision (Purity), which are closely related to our original AUC and MAD scores by quantifying concentration and discriminatory power of activations.
• An output-centric metric (Faithfulness), which measures the causal influence of the discovered concept on model output, see also [24].

Results are shown in the table below. PRISM (max) consistently outperforms GPT-Explain not only in semantic coherence (Clarity, Purity) but also in its impact on outputs (Faithfulness), further validating the method's effectiveness across a variety of interpretability criteria. As FADE's activation-based metrics align with ours, we followed the reviewer's suggestion and included FADE's output-centric faithfulness metric to Table 1, Section 4.3.

[b3] Questionable reliability of MAD for NLP:

"[...] I have some reservations about its reliability to assess the quality of explanations in the NLP context."

In our paper, the MAD (Mean Activation Difference) metric serves as a parametric test as it quantifies the absolute difference in activation magnitudes between description-related examples and random inputs. Conceptually, the Input-based evaluation used by [24] in the context of NLP is closely related to our MAD metric. In [24], the authors test whether mean activations for the description-related examples exceed those of 'neutral' examples. As such parametric measures like MAD can be sensitive to outliers, particularly in NLP settings where activations are often characterized by heavy-tailed distributions. This is precisely why MAD is paired with the more robust AUC score, providing a fuller picture of neuron/feature behavior. We mention this limitation of MAD in Section 6 of the revised manuscript.

[c] Comparison to MaxAct Baseline: We thank the reviewer for highlighting this. In response, we strengthened our experiments by comparing PRISM to MaxAct, a baseline that selects the five highest-activating sentences per feature, following [Bil23]. The table (due to character limits, the table appears in our response to Reviewer 2jDP, under [W2] Percentile Sampling) shows PRISM (max) outperforms MaxAct across most metrics and models, especially in AUC. We include MaxAct as a baseline for all models in Table 1, Section 4.3 of the revised manuscript.

We agree that broader comparisons are valuable. However, not all methods listed in Appendix A.2 have available feature descriptions, which limits reproducibility. We prioritized baselines with publicly accessible feature descriptions.

[W2] Overgeneralization or missing context:

[a] Address Niche: We thank the reviewer for pointing out where the narrative of the paper could lead to overgeneralizations. Our work indeed targets a specific niche of methods that assign explanations to individual features, like standard neurons or SAE-discovered features, within LLMs for NLP, and does not cover polysemanticity analyses across domains. We focus on improving feature-description methods for LLMs that typically provide a single textual explanation per neuron/feature, by instead assigning multiple explanations, potentially capturing the diverse behaviors of a single feature. We recognize that:

• Many existing SAE-based methods explicitly aim to reduce polysemanticity by producing sparse and more monosemantic features.
• There is a substantial body of prior work (e.g., in both NLP and Vision) analyzing polysemantic neurons.

To address scope ambiguities in the original manuscript, we have revised the text to clearly articulate our focus throughout. Specifically we rewrite:

• abstract lines 2-5 to:

"Within the context of large language models (LLMs) for natural language processing (NLP), current automated neuron-level feature description methods face two key challenges: limited robustness and the assumption that each neuron encodes a single concept (monosemanticity), despite increasing evidence of polysemanticity."

• abstract lines 9–11 to:

"[...] framework specifically designed to capture the complexity of features in LLMs. Unlike approaches that assign a single description per neuron, common in many automated interpretability methods in NLP, PRISM produces more nuanced descriptions that account for both monosemantic and polysemantic behavior."

• lines 31–32 to:

"While the problem of polysemanticity is generally addressed through the extraction of sparse features, such as via sparse autoencoders, feature description methods, which aim to explain the functional purpose of individual features, typically provide a single explanation per feature [20, 21, 22, 23, 24]. This can limit the ability to capture the full range of patterns a feature may represent."

[b] Human Interpretation: We expanded the human evaluation in Section 5.2 by involving seven additional non-author participants, results show good alignment between human judgments of polysemanticity and the PRISM metric; due to character constraints further details can be found in our response to Reviewer 2jDP under [W5] Human evaluation.

[Q1] Question: Techniques like SAEs seek to promote disentanglement but often exhibit high reconstruction errors, model faithfulness and lack theoretical guarantees that the learned features are actually monosemantic (see [Shar25], Sec. 2.1). As a result, individual features may still respond to multiple concepts, which limits interpretability. We now refer the reader to [Shar25] for a more comprehensive discussion of SAEs and their limitations

Please let us know if any further comments or questions arise. We are happy to address them during the discussion period.

Refs

• [Pur25] Puri et al., FADE: Why Bad Descriptions Happen to Good Features. ACL 2025.
• [Bil23] Bills et al., Language models can explain neurons in language models. OpenAI 2023.
• [Shar25] Sharkey et al., Open Problems in Mechanistic Interpretability. Arxiv 2025.

Thank you to the authors for acknowledging the limitations of the previous version and for their answer. The comparison with competitors is still limited, as acknowledged by the authors, but overall I believe the new evidence better supports the claims and I will raise my scores according to the policy I stated above and the overall discussion.

I would like to raise a couple of follow-up questions based on the rebuttal:

• Over how many runs are the new results computed? The variance across all the metrics seems quite high
• I raised this point earlier, but I may have missed the response: why can’t existing methods targeting polysemanticity be applied (or why do they perform poorly) in the specific context of this paper? This point is related to the contextualization of the paper with respect to the literature studying polysemanticity.

We thank the reviewer for the thoughtful comments. We regret the clarity fell short for this reviewer, we appreciate the chance to improve. While other reviewers found the writing and contributions clear, we recognize that presentation can be refined to suit a broader audience, and we have worked carefully to address these concerns.

[W1] Writing:

"The paper is poorly written and hard to read. Specifically, [...]" (a-e)

Following the reviewer's comments, we have revised relevant sections to improve clarity and readability. While we are unable to submit a revised manuscript at this stage, we hope our clarifications below help address any potential for confusion.

[a] Equations 3 and 4: We thank the reviewer for pointing out that the symbol $s_j$ is overloaded in both equations. We disambiguate this by using distinct symbols $s_j$ for the natural language summary and $\tau_j = e(s_j), \tau_j \in \mathbb{R}^T$ for the embeddings of a specific natural language summary. We correct Eq. 4. accordingly:

$\text{cos}(\theta) = \frac{\sum_t^T \tau_{i,t}\tau_{j,t}}{\sqrt{\sum_t^T \tau_{i,t}^2} \sqrt{\sum_t^T \tau_{j,t}^2}}$.

[b] Duplicate description: We removed the unintentional repetition.

[c] Definition of target concept and control data points: The target concept dataset consists of sentences generated based on cluster labels for a given neuron. Control data consists of random sentences which are not associated with the concept. We previously defined this in Appendix A.3, but based on the reviewer's helpful remark, we have now moved this explanation to the main text (Section 3.2) to improve clarity and ensure the evaluation setup is more easily understood.

[d] AUC and MAD as evaluation metric: AUC and MAD scores are used to quantify whether example sentences generated based on each cluster label activate the corresponding neuron more strongly than the random sentences in the control dataset (see answer [W1][c]). Higher AUC and lower MAD indicate more semantically aligned and distinctive concept labels for the different cluster.

[e] Placement of Figure 6 (Appendix A.3): We agree that this figure is informative and helps illustrate the evaluation process. However, due to the length restrictions of 9 pages, we previously struggled to integrate the Figure in the main body. Given the opportunity we will aim to move it into the main paper for better clarity.

[W2] Percentile Sampling:

"In section 3.1, why do we need a grid of high-percentile levels in percentile sampling, instead of using only the top activations?"

We use a grid of high-percentile levels rather than a fixed top-k cutoff to retain diverse activation contexts that could be excluded by selecting only the top-k samples. Absolute activation can vary strongly across neurons, layers and even corpora, so using a fixed cut‑off (e.g. the top‑k sentences) risks discarding genuine but moderately‑scaled patterns. Instead, sampling sentences uniformly from the 99th to 100th percentile with a step size of 1e-5 retains focus on strong activations while enabling broader coverage of distinct patterns that activate the same feature. As demonstrated in our experiments, this signal allows capturing mutli-concept feature descriptions for neurons that have previously been assigned a single description.

We empirically evaluated this design choice by comparing PRISM to a baseline approach (MaxAct) that selects the five highest-activating sentences per feature, following [Bil23], using a subset of the C4 dataset. PRISM (max) outperforms MaxAct on most metrics and models, particularly in AUC (see table below). This supports the benefit of our percentile-based sampling for interpretability and concept coverage.

[W3] Number of Clusters:

"How is the number of clusters determined?"

We thank the reviewer for raising this important question. In our original implementation, we chose 5 clusters as a default, as this provided a balance between granularity and interpretability, allowing for multiple semantic patterns to emerge while limiting potential redundancy or incoherence. We analyze the effect of varying the number of clusters on PRISM's performance across multiple values of k and report both max and mean cluster scores across features in GPT-2 XL, using AUC and MAD as in our main evaluation (see table below). Increasing the number of clusters raises PRISM max scores by allowing clusters to capture more specific activation patterns, improving labeling of the most coherent ones, but lowers the PRISM mean score as coherent patterns are split into statistically indistinguishable subclusters. We have added clarification regarding the choice of the number of clusters to Section 4.1 and discuss implications of increasing/decreasing k in our discussion in Section 6.

[W4] Positive/negative activations:

"Why are only positive activations considered? Is it possible that a sample has a large negative activation?"

We focus on positive activations as we are interested in characterizing features when they are strongly activated ("fire") in response to a specific input. While negative activations can occur, their interpretation remains unclear and is still debated (see [Mol20]). Given that common activation functions in state-of-the-art LLMs, such as ReLU and GeLU, retain only positive or weakly negative signals, we follow prior work on feature description methods (e.g., [Her22, Bil23]) and threshold activations to retain only the top-activating samples. That said, our framework is compatible with extracting descriptions across the full activation range.

[W5] Human evaluation: Inspired by the reviewer's comments, we decided to extend the scope of the human evaluation in Section 5.2 by conducting a larger human evaluation with seven participants, all of whom are non-authors. Participants were recruited from our institutions and compensated. We provide additional details on participant recruitment, task instructions, and the study setup in the Appendix of the revised manuscript for full transparency.

Each participant was presented with 8 groups of sentence clusters, where each group corresponded to one feature and consisted of 5 clusters of highly activating sentences (20 sentences per cluster), along with highlighted tokens. For each cluster, participants were tasked to write a short textual description, resulting in 5 human-generated descriptions per feature. They then rated the pairwise similarity between these descriptions on an 11-point scale (0–10), where 10 indicates very high similarity. We report the average of these ratings as the Human Score. In addition, we compute Human Cosine Similarity using the same embedding model and method used for computing the PRISM polysemanticity score, with the only change being the use of human-generated descriptions instead of model-generated ones.

The table below presents the results, sorted by PRISM score (lower values indicate higher polysemanticity). As expected, features with low PRISM scores receive lower human similarity ratings and lower embedding-based similarity. For example, the feature from GPT-2 Small (layer 5, feature 30319) has a low PRISM score (0.30), and is judged by participants to be semantically diverse (Human Score: 0.40). In contrast, features with high PRISM scores, such as those from Gemma Scope (features 10531 and 603), show strong human agreement (Human Score: 0.90 and 0.80, respectively) and high cosine similarity (0.79 and 0.75). These findings provide both qualitative and quantitative support for the PRISM metric, showing that it aligns well with the human interpretation polysemanticity. We have replaced the original human evaluation with these new, extended results and discussion in Section 5.2 of the revised manuscript.

We hope our responses and changes have effectively addressed the concerns. If any further questions or suggestions arise, we are happy to continue the discussion.

Refs

• [Mol20] C Molnar. Interpretable Machine Learning Book, Chapter 27. 2020.
• [Her22] Hernandez et al., Natural language descriptions of deep visual features. ICLR 2022.
• [Bil23] Bills et al., Language models can explain neurons in language models. OpenAI 2023.

We sincerely thank the reviewer for their thoughtful, detailed, and constructive feedback. We are especially grateful for the reviewer's appreciation of the paper's clarity, the novelty and simplicity of PRISM, and the extent of our experimental evaluation. The reviewer's comments have helped us both sharpen the presentation and deepen our analysis. We have carefully considered all concerns and made substantial efforts to address each point. Below, we respond to the main weaknesses and questions raised. We make sure the revised version reflects these improvements and communicates the work more effectively. Please let us know if any further concerns remain.

[W1/Q2] Open Source LLMs:

"[...] high capacity (even proprietary) LLMs are required to produce high-quality descriptions of features."

"What alternatives may be employed to proprietary LLMS to generate descriptions?"

We appreciate the reviewer's concern regarding whether high-capacity or proprietary LLMs are necessary in the PRISM framework to produce high-quality feature descriptions. To address this concern, we extended our evaluation beyond Gemini 1.5 Pro by benchmarking several open-source alternatives: Qwen3 32B, Phi-4, and DeepSeek R1. These models were used to generate feature descriptions for GPT-2 XL using the same procedure as in our original setup. As shown in the table below, Qwen3 32B achieves performance comparable to Gemini 1.5 Pro, demonstrating that our framework does not depend solely on a single model. While Phi-4 and DeepSeek R1 show slightly lower scores, they still follow the same qualitative trends, demonstrating that our method is robust and generalizes effectively across a range of language models, including accessible open-source options. We have included this experiment and discussion to Section 4.2 (Sanity Checks) of the revised manuscript.

[W2] Unclear sentence in line 174:

"Which of the two scores is the random baseline, and which is PRISM?"

We understand the confusion regarding the meaning of this sentence. To clarify, the AUC of 0.85 ± 0.25 and MAD of 2.98 ± 3.16 refer to the PRISM baseline (with descriptions generated from semantically meaningful clusters), while the AUC of 0.65 ± 0.35 and MAD of 1.35 ± 2.36 correspond to the randomized control (with cluster descriptions shuffled across features).

We have rephrased the sentence and included the results in tabular form for GPT-2 XL in the revised manuscript to make this distinction clearer. The updated sentence now reads:

"The lower score for PRISM in the random scenario (Random Descriptions) compared to PRISM without randomization (Baseline) indicates that the PRISM descriptions truly depend on the content of the clusters because randomizing them leads to a decrease in performance."

The corresponding results are now summarized in the following table:

[Q1] Smaller Open Source LLMs:

"What are the failure modes of smaller open source LLMs when used to generate the human readable descriptions?"

In an effort to characterize failure modes, including smaller open-source models, we provide the evaluation score in the table in [W1/Q2]. While the decrease in description quality measured by AUC and MAD supports prior findings in the literature on the limitations of smaller LLMs [Pur25], score decreases are small indicating PRISM's robustness towards the LLM choice.

In our analysis of the lighter LLMs (e.g. Qwen2.5 7B Instruct, Llama 3.1 8B Instruct), we found that these models are limited due to a combination of issues: they tend to over-generalize, produce overly verbose or vague outputs, struggle with instruction adherence, and in some cases, ignore instructions altogether. These shortcomings reduce the consistency and specificity of the generated descriptions. As noted in [W1/Q2], we have incorporated these findings into the revised manuscript along with an expanded discussion of these limitations and supporting results.

Refs

• [Pur25] Puri et al., FADE: Why Bad Descriptions Happen to Good Features. ACL 2025.

We would like to thank the reviewer for the detailed feedback and helpful suggestions. We have incorporated the suggested minor corrections into the revised manuscript and address the comments below.

[W1/Q1] Percentile Sampling:

"The method does not take into account the activation of the individual points relative to the highest activation. For instance, if the 99th percentile input has activation an order of magnitude lower than 100th percentile, it should not be considered for clustering."

Absolute activation magnitudes vary substantially across neurons, layers, and even datasets. Relying on a fixed threshold (e.g., selecting only the top-k sentences) risks excluding informative patterns with moderate but consistent activations. To address this, our method samples sentences uniformly from the 99th to 100th percentile with a step size of 1e-5, allowing us to focus exclusively on the upper tail of the activation distribution while preserving semantic diversity.

This percentile-based sampling is a core distinction between PRISM and other feature description methods that are typically based on top-k approaches. We demonstrate (e.g., Table 2 for quantitative results and Figure 4 for qualitative results) that sampling from the 99th percentile already yields sentences that both robustly activate the feature and reflect semantic diversity, supporting a more holistic understanding of neuron behavior.

[W2/Q2] Setting of Cluster Size:

"While the AUC and MAD metrics quantify if descriptions correspond to input samples where a feature/neuron activates, I didn't find any evaluation regarding "how" much of the feature is "explained" through the obtained descriptions. In other words, are the descriptions of clusters enough to "cover" all samples that can activate the given feature?"

We would like to clarify that the AUC and MAD metrics are designed to quantify how strongly a neuron/feature reacts to inputs corresponding to a given description, in comparison to a random input set. As described in Appendix A.3, we evaluate a proposed description (e.g., “Quantities, specifically numbers, and time periods”) by prompting an LLM to generate related sentences (see Figure 7 in Appendix for exact prompt details). We then compare the activations of the neuron/feature on these description-related sentences against its activations on random inputs. The separation between the two distributions is measured using a nonparametric metric (AUC, where 1.0 is ideal) and a parametric one (MAD, where higher is better). A high score indicates that the neuron/feature responds significantly more to inputs aligned with the description, which suggests a faithful explanation.

"In other words, are the descriptions of clusters enough to "cover" all samples that can activate the given feature?"

This is indeed a limitation of all current feature description methods, as these depend on a reference dataset, limiting the descriptions to cover concepts that occur in a subset of these samples.

Our approach progresses on the challenge of finding richer, multi-faceted descriptions of each feature, describing it with a set of several distinct concepts (in comparison to a single description from standard approaches). However, like all such methods, PRISM cannot guarantee that the provided descriptions are the only or most salient concepts for a feature; rather, they represent key concepts the feature reliably responds to. We have now extended our brief discussion of this limitation in Section 6 of the revised manuscript.

To further examine this, we analyzed PRISM's performance across different numbers of clusters (k) for generating feature descriptions for GPT-2 XL. As shown in the table below, increasing k improves the best-case description quality, as reflected in higher PRISM max scores, but tends to reduce the average interpretability across all clusters (lower PRISM mean scores). This highlights a tradeoff: more clusters can yield more precise and targeted descriptions, but may also introduce redundancy or semantic overlap. The results agree with our intuition that with larger numbers of clusters, the best label becomes more precise (PRISM max score) while the overall score (PRISM mean score) decreases since now monosemantic labels are over-labeled. We have added this experiment and discussion to the Appendix of the revised manuscript.

[W3/Q3] Human Evaluation:

"[Experiment in Section 5.2] should be repeated with multiple annotators"

We expanded the human evaluation in Section 5.2 by involving seven additional non-author participants, results show good alignment between human judgments of polysemanticity and the PRISM metric; due to character constraints further details can be found in our response to Reviewer 2jDP under [W5] Human evaluation.

[Q4] Choice of the sentence embedding model:

We based the selection of the embedding model on the 'Massive Text Embedding Benchmark' (MTEB) that aims to evaluate sentence embedding models across a variety of tasks. Specifically, we chose a model from the Qwen2 family due to its strong and consistent performance on the MTEB leaderboard (see [MTEB23], [TEMB] for reviews, and [MTEB] for the current rankings).

In initial experiments, we compared several top-performing embedding models and observed that the resulting concept descriptions showed strong overlap, and thus we chose not to pursue an extensive ablation in this direction. Overall, recent state-of-the-art encoder models produce embeddings with comparable performance [MTEB], reflecting a broader trend toward convergence in model representations that is hypothesized to be a results of strong similarities in training data, objectives, and model architectures [Plat24].

[Q5] Choice of LLM for description:

"In practice, did you observe biases in your descriptions because of the choice of LLM (for summary generation) or sentence embedding function?"

We agree that model choice can influence descriptions and is an important area for future research. In our early experiments, we observed that smaller LLMs often produced overly long or unfocused summaries, consistent with prior findings on their instruction-following limitations [Lou24, Zhe23]. For this reason, we use a strong and sufficiently large instruction-tuned model for summary generation. Due to character constraints, we kindly refer the reviewer to our response to Reviewer MiB7 under [W1/Q2] Open Source LLMs, where we provide a detailed discussion and experimental results on the impact of using different LLMs for description generation.

Regarding sentence embeddings, as discussed in [Q4], we found minimal variation in output semantics across high-performing embedder models. However, we acknowledge that domain-specific models (e.g., multilingual or biomedical) may be needed to capture sentence meaning in more specialized settings [Vas24].

[Q6] Distinction between mono- and polysemantic concepts:

We thank the reviewer for pointing out this potential for confusion. We follow the most widely used definition of polysemantic features as “neurons that respond to multiple unrelated inputs” [Fal15, Mul16, Aro18].

In our manuscript, we thus consider a feature polysemantic if it activates for semantically distinct clusters (e.g., the mentioned encryption terms and holiday-related phrases), quantifiable via the polysemanticity score. In contrast, features activating for different animal names are considered as monosemantic, since the underlying concept is coherent.

We have clarified these points by (i) emphasizing and referencing the definition of polysemanticity used in our paper, (ii) expanding our discussion on the distinction between polysemantic and monosemantic features in Sections 5 and 6, (iii) extending our user study, which offers a complementary perspective on the semantic coherence of the identified feature clusters, and (iv) adding additional examples of clearly polysemantic and monosemantic features to the main paper.

If any further questions or comments arise, we are happy to address them during the discussion period.

Refs

• [MTEB23] Muennighoff et al., MTEB: Massive Text Embedding Benchmark. EACL 2023.
• [MTEB] Leaderboard accessed via huggingface.co/spaces/mteb/leaderboard
• [TEMB] H Cao, Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark. Arxiv 2024.
• [Plat24] Huh et al., Position: The Platonic Representation Hypothesis. ICML 2024.
• [Lou24] Lou et al., Large Language Model Instruction Following: A Survey of Progresses and Challenges. ACL 2024.
• [Zhe23] Zheng et al., Judging llm-as-a-judge with mt-bench and chatbot arena. Neurips 2023.
• [Vas24] Vasileiou et al., Explaining Text Similarity in Transformer Models. NAACL 24.
• [Fal15] Falkum & A Vicente., Polysemy: Current perspectives and approaches. Lingua 2015.
• [Mul16] Nguyen et al., Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks. Visualization workshop ICML 2016.
• [Aro18] Arora et al., Linear algebraic structure of word senses, with applications to polysemy. TACL 2018.