Incidental Polysemanticity: A New Obstacle for Mechanistic Interpretability

The explanation and motivation of Figure 6 is a little confusing and it is perhaps a digression from the main topic of the paper. It would be better to provide more discussions of the new insights.

Thank you for raising this point. Figure 6 actually reveals an unexpected mechanism for polysemanticity: while most encodings reach almost perfect sparsity (indicated by ${{||W_i||}_4}^4 = 1$), the orange curve encoding gets stuck below ${{||W_i||}_4}^4 = 0.2$. Note that this isn't just noise or training instability, it represents a stable ‘compromise’ where multiple features share representation due to interference patterns. From the final weight matrix, we can clearly see that sparsity pushes W2 to "pick" one of the 7 coordinates and increase its absolute value, but interference patterns force W2 to compromise across all 7 coordinates.

We believe this phenomenon is particularly important because it demonstrates how polysemanticity can happen for a wide variety of sometimes surprisingly hard-to-predict incidental reasons, e.g. in this case non-task factors, and it’s why eliminating incidental polysemanticity is particularly challenging.

In this paper, the authors mainly focus on how to obtain polysemantic neurons. However, in real-world data, the main challenge lies in obtaining the monosemantic neurons. Consequently, is it possible to provide some insights about how to attain monosemanticity based on the analysis in this paper?

Thank you, this is an excellent question. We have added to the paper to clarify this. Here is one possible way one might get rid of incidental polysemanticity in a neuron that currently represents two features i and j: Duplicate that neuron, divide its outgoing weights by 2 (so that this doesn't affect downstream layers), add a small amount of noise to the incoming weights of each copy, then run gradient descent for a few more steps. One might hope that this will cause the copies to diverge away from each other, with one of the copies eventually taking full ownership of feature i while the other copy takes full ownership of feature j.

Of course, as we noted in the paper, we can also perform a careful control of noise distributions. One might say that the paper should explore further initialization strategies that encourage initial feature-neuron correlations to develop toward monosemantic representations. However, the auto-encoder were part of a larger network, then in most cases you don't know in advance what the features will be (and they're not going to be aligned with the basis vectors just by random chance) so it's unclear how you'd guarantee that your initialization is monosemantic with respect to the features that eventually get learned.

We thank reviewer 8rnd for the thoughtful suggestions, especially regarding the needed clarifications on Figure 6!

For example, as polysemanticity is not related to the tasks, can we get rid of the performance-interpretability trade-off?

Regarding the performance-interpretability trade-off, Unlike necessary polysemanticity (as described in Elhage et al.) which arises from capacity constraints, incidental polysemanticity occurs even when the network has sufficient capacity to represent features monosematically. Our toy model demonstrates this cleanly since we can guarantee the network has more than enough capacity to represent all features without polysemanticity while still achieving perfect task performance. This suggests that at least for incidental polysemanticity, there may not be a fundamental performance-interpretability trade-off, though more research is needed to verify this in more complex model settings.

Besides, in the neural networks, how can we distinguish the incidental polysemanticity?

Thank you for this important clarifying question. Whether we can distinguish neurons based only on the final, trained state of the model or if we need to know more about the training process is a complex question. While our current work focuses on establishing the existence of incidental polysemanticity, we agree that future work exploring these questions would be very valuable. We are currently unaware of any work that investigates this property in detail in any network.

However, in our “Discussion and Future Work section”, we did outline some concrete approaches for potentially reducing incidental polysemanticity. As a starting point, here is one possible way one might get rid of incidental polysemanticity in a neuron that currently represents two features i and j: Duplicate that neuron, divide its outgoing weights by 2 (so that this doesn't affect downstream layers), add a small amount of neural noise to the incoming weights of each copy, then run gradient descent for a few more steps. One might hope that this will cause the copies to diverge away from each other, with one of the copies eventually taking full ownership of feature i while the other copy takes full ownership of feature j.

The analysis in this paper is conducted on the toy models. I understand it is for the ease of theoretical analysis. However, it would be better to provide some insights into the polysmenaticity in the neural networks trained on real-world data and add some additional experiments or discussions.

We appreciate your concern about the limited evaluation using only a toy model and dataset. We would like to provide some additional context for why we believe toy models are actually a strength of our work given the specific research questions and claims we aim to address.

The key point we want to demonstrate is that polysemanticity can arise incidentally from factors other than limited capacity, even in networks that achieve perfect performance on the task. A toy model is particularly well-suited to make this point cleanly and definitively. We can guarantee the network has more than enough capacity to represent the task features without polysemanticity, and that it does indeed reach 100% performance. With real-world datasets and networks, it would be much harder to disentangle polysemanticity driven by incidental factors like initialization and regularization vs. insufficient capacity. The toy setting allows us to isolate the phenomenon of interest.

Toy models have a long and impactful history in machine learning research as a tool for building insight and precisely characterizing important phenomena. Polysemanticity and superposition, double descent, scaling laws, grokking and adversarial robustness have all benefited from analysis in simplified models that abstract away confounding factors. The closely related work of Allen-Zhu & Li 2020 that you reference itself uses a toy network and synthetic data. Our use of toy models thus follows a well-established approach in the field for deriving clean insights about interesting behaviors.

Cleanly defining and identifying "features" in real networks remains an open problem for mechanistic interpretability. Our work intentionally sidesteps this challenge to focus on a separate important question about the origins of polysemanticity. Toy models allow us to make progress on this question even without solving feature identification in complex settings. We view our contributions as complementary to ongoing work on scalable feature definition.

Can you explain how noise injections relate to explaining the cause of polysemanticity in the typical case? Perhaps I be unfamiliar with how common these types of noise injections are.

We selected the noise setup for three main reasons.

• Showing that multiple factors besides L1 regularization can cause incidental polysemanticity strengthens our main claim that such polysemanticity can arise from diverse causes unrelated to task performance.
• Noise is a practically relevant factor to study, since widely used techniques like dropout induce noise in real networks.
• Studying the impact of noise on representations has precedent in related works like Bricken et al. 2023 and Allen-Zhu & Li 2020.

So we showed that certain types of noise can induce sparsity, which then leads to polysemanticity through mechanisms similar to those we demonstrated with L1 regularization. As in Figure 4, bipolar noise (which has negative excess kurtosis) pushes encodings towards sparsity, while normal noise does not. This sparsification then creates the conditions for incidental polysemanticity through the same winner-takes-all dynamics we observe with L1 regularization. The key insight is that noise with certain statistical properties can implicitly create regularization-like effects.

51. In that case

Thank you for pointing that out! We have fixed the spelling issue.

135. By $W_i = f_i$, do you mean $W_i = e_i$?

In 132), we define $f_k \in \mathbb{R}^{m}$ as the $k^{\text{th}}$ basis vector in $\mathbb{R}^{m}$, the hidden layer dimension (as opposed to the input/output dimension).

Figure 3, graph not displaying range bars for last two points.

Thank you for pointing that out. We have rerun the experiments and updated the plot to display the error bars for each point. The updated plot can be viewed here: https://drive.google.com/file/d/1Dp9SSH0JYrpKI0kSE_GBltpbPJ7FKRK7/view?usp=sharing.

Is there an example of incidental polysemantic neurons when m < n, or the feature space is smaller than the required number of features? This would help move the scenario of the ecperiemnt closer to what is typical.

Your insightful question about our contributions of sparsity, noise, and gradient descent to polysemanticity in undercapacity regimes (where m<n) is great. Our current work focuses on the setting where networks have excess capacity, but we agree that studying the "incidental" nature of polysemanticity in under-capacity regimes is a fascinating direction for future work, and we will be sure to acknowledge this in the paper.

We would also like to note that in under-capacity regimes, the distinction between “incidental” and “necessary” polysemanticity is not as crisp, since all optimal solutions would have some amount of polysemanticity. But we can still talk about whether gradient descent will find a solution of minimum polysemanticity among all optimal solutions (we can define the “amount of polysemanticity” as the number of pairs of features that share at least one neuron). Our intuition is that as the number of neurons falls below the number of features, neurons would start to be shared by larger and larger groups of features that get packed together, striking a compromise between sparsity and expressivity. But this compromise would not be struck optimally. In particular, we expect that, until the number of neurons is much smaller than the number of features, we would continue to see that a significant fraction of neurons are left unutilized at the end of training, mirroring what we saw in the experiments we ran with an equal number of neurons and features. This would be a clear case of incidental polysemanticity.

Can you explain how incidental polysemanticity with L1 regularization relates to the base case?

The key insight here is that the regularization induces a "winner-take-all" dynamic through the sparsity force (as defined in Section 1). When we have feature benefit pushing weights to have unit length, or L1 regularization pushing down on weights with constant force, it creates a dynamic where weights whose absolute value is above some threshold will grow, while those below the threshold will shrink. As we showed in the paper, the threshold is $(1 - \|W_i\|^2) W_{ik} = \lambda \, \text{sign}(W_i) \iff |W_{ik}| = \frac{\lambda}{1 - \|W_i\|^2}$.

Is it that the scenario where the feature space dimension is much smaller than the number of features is similar to a scenario with feature sparsity? Making this more clear would help readers understand your contribution.

Yes, the scenario where the feature space dimension is smaller than the number of features is a more central example of a setting where recovery of sparse features has been explored. However, even in our setting where the feature space dimension is larger than the number of features, the features are maximally sparse (only one of them is ever on). The difference is that it would no longer be surprising that they can be recovered losslessly (since one could assign one neuron to each feature), and the surprising thing is that in fact when L1 regularization is introduced, some polysemanticity still occurs.

Section 3/4 discuses how noise is tied to sparsity, but does not verify that noise is tied to polysemanticity. The paper could be improved by providing experiments showing how noise causes incidental polysemanticity.

For experiments how noise causes incidental polysemanticity, we request you to instead check out Section 4 of the paper, where our experiments demonstrate a clear causal chain from noise to sparsity to polysemanticity. In Figure 4, we show that bipolar noise (with negative excess kurtosis) pushes encodings towards sparsity, while normal noise does not. Specifically, for bipolar noise with higher standard deviations (e.g. $\sigma = 0.1, 0.15$), we observe the average fourth norm increasing dramatically above the $3/m$ baseline, indicating strong sparsification. This sparsification directly leads to polysemanticity through the "winner-take-all" dynamics that we have presented in the paper.

Further, Figure 6 provides direct evidence of how noise leads to polysemanticity in the interplay between interference and the push for sparsity presented. In that figure we observed cases where encodings are forced to represent multiple features due to noise-induced dynamics, even when this isn't optimal for task performance. Specifically, we show how the second encoding (W2) ends up with significant weights across multiple coordinates due to noise-induced interference patterns.

I don't see any baseline polysemantic neurons count for your model, with no regularization or noise. Is this because there is no way to define them, as there is no winner-takes-all enforcement? It would be beneficial to establish some sort of baseline.

Thank you for raising that interesting point. In our base model with neither regularization nor noise, feature representations tend to remain distributed across multiple neurons rather than concentrating in a single neuron, resulting in fourth norms close to the random baseline of $3/m$ shown in Figures 4 and 5. This baseline behavior is visible in the early stages of training in Figure 4, where before the sparsifying effects take hold, both bipolar and normal noise conditions show fourth norms hovering around the $3/m$ reference line. This represents the natural state of the network before the sparsity-inducing forces come into play.

We can precisely measure polysemanticity once the winner-takes-all dynamics have emerged, because we can identify when multiple features are primarily represented by a single neuron. Without the dynamics having played out, we can expect the features to be distributed across neurons in a way that make counting "polysemantic neurons" less well-defined. Hence, our analysis focuses on how regularization and noise create conditions where we can unambiguously measure the phenomenon.

We thank reviewer WVCj for the thoughtful comments, especially for pointing out wording issues, clarifications on the noise in the hidden layer setup, and pointing out a much-needed full error bar display, which we have now implemented!

The main question is why polysemanticity occurs in typical settings, and it is unclear how these results apply to the more typical scenarios. This lessens the paper's contribution significance. This paper could be improved by elucidating more how the content applies to more typical scenarios or providing content more in alignment with those scenarios.

We appreciate your concern about the limited evaluation using only a toy model and dataset. We would like to provide some additional context for why we believe toy models are actually a strength of our work given the specific research questions and claims we aim to address.

The key point we want to demonstrate is that polysemanticity can arise incidentally from factors other than limited capacity, even in networks that achieve perfect performance on the task. A toy model is particularly well-suited to make this point cleanly and definitively. We can guarantee the network has more than enough capacity to represent the task features without polysemanticity, and that it does indeed reach 100% performance. With real-world datasets and networks, it would be much harder to disentangle polysemanticity driven by incidental factors like initialization and regularization vs. insufficient capacity. The toy setting allows us to isolate the phenomenon of interest.

Toy models have a long and impactful history in machine learning research as a tool for building insight and precisely characterizing important phenomena. Polysemanticity and superposition, double descent, scaling laws, grokking and adversarial robustness have all benefited from analysis in simplified models that abstract away confounding factors. The closely related work of Allen-Zhu & Li 2020 that you reference itself uses a toy network and synthetic data. Our use of toy models thus follows a well-established approach in the field for deriving clean insights about interesting behaviors.

Cleanly defining and identifying "features" in real networks remains an open problem for mechanistic interpretability. Our work intentionally sidesteps this challenge to focus on a separate important question about the origins of polysemanticity. Toy models allow us to make progress on this question even without solving feature identification in complex settings. We view our contributions as complementary to ongoing work on scalable feature definition.

Further, while our current experiments focus on establishing the existence of incidental polysemanticity in a setting where it doesn't affect task performance, we believe studying polysemanticity is still important because the type of representation learned can have implications beyond just raw task performance. For example, polysemantic representations may be less interpretable (as each neuron's activity is no longer clearly associated with a single input feature), more or less robust to noise or distribution shift, or more amenable to compression.

Dear Reviewer WVCj, we would greatly value your review of our rebuttals to consider whether you would like to update your ratings. Given your thoughtful comments about generalizability, we believe further discussion could be valuable. Similar theoretical work using toy models was awarded Best Paper runner-up at ICLR 2023.

The deadline is approaching quickly, and we would greatly appreciate a response.

We thank reviewer RE6X for the thoughtful comments! Your suggestions have enabled us to significantly improve the content and exposition.

In general, I am unsure about the weight of novel contributions in this paper relative to the bar of an ICLR acceptance. I will defer to the AC in this regard.

Could you help us understand which aspects of the contributions you find uncertain? Our paper makes several novel contributions that we believe are significant for the field, including:

• We demonstrate a fundamentally new origin story for polysemanticity that challenges the prevailing view that it only arises from capacity constraints.
• We provide rigorous mathematical analyses showing how different factors (L1 regularization, noise) can cause polysemanticity (including studying their effects throughout training and measuring the effects concretely as we increase L1 regularization strength or standard deviation of the noise) even when unnecessary for task performance.
• We precisely characterize the number of incidentally polysemantic neurons we expect to see, and verify this experimentally.

Please note that ICLR has a strong tradition of publishing theoretical work using toy models to illuminate fundamental properties of neural networks. For instance, the ICLR 2023 Best Paper runner-up "Disentanglement with Biological Constraints" (Whittington et al.) used a similar approach of mathematical analysis on toy models to understand representation learning. Their work studied mixed selectivity, closely related to polysemanticity.

However, I am unsure how these findings translate into language models, which are not explicitly trained with an L1 regularization.

Our primary aim is to demonstrate that $l_1$ regularization can amplify random correlations between features and neurons, causing incidental polysemanticity. While language models are not explicitly trained with $l_1$ regularization, regularization effects arise implicitly through initialization, dropout, weight decay, optimization algorithm choice, and other techniques. The objective of the paper is not language models – our goal was to cleanly demonstrate this phenomenon of polysemanticity occurring incidentally in a simplified setting where we can precisely analyze it, so as to provide a foundation for future work investigating how these dynamics manifest in other model settings. As also approached by Elhage et al., we successfully demonstrated these dynamics in full detail first in this toy model setting, which was intentionally chosen to allow for a clean mathematical analysis and to definitively show that polysemanticity can arise even when the network has sufficient capacity to learn monosemantic representations.

Without context from Elhage et al. it is unclear, why kurtosis is the main metric tracking sparsity in this context. An accompanying background section motivating kurtosis would be useful.

Elhage et al. does not use kurtosis to track sparsity. However, as we note in the paper, previous related works such as Bricken et al. 2023 and Allen-Zhu & Li 2020 study the kurtosis of the noise distribution. Higher kurtosis indicates more concentration of mass in the tails, which corresponds to sparse activation patterns where neurons are either strongly active or inactive. Thus, we feel that it is a natural and well-used metric to track in our toy model. We've also added citations to related work in sparse coding, ICA and compressed sensing in our related work section, where kurtosis plays a similar role.

Section 3.2 quickly summarizes the mathematical analysis of this section. A motivation of why highlighting bipolar noise would be useful.

Thank you for the thoughtful question. In Appendix C, we compute in full detail the excess kurtosis different noise distributions (including bipolar and gaussian) have, and the effect of these computations on incentivizing sparsity. We selected to study noise in the hidden layer for three main reasons:

• Showing that multiple factors besides L1 regularization can cause incidental polysemanticity strengthens our main claim that such polysemanticity can arise from diverse causes unrelated to task performance.
• Noise is a practically relevant factor to study, since widely used techniques like dropout induce noise in real networks.
• Studying the impact of noise on representations has precedent in related works like Bricken et al. 2023 and Allen-Zhu & Li 2020.

Thus, noise seemed like a natural choice given our goals and the broader context of the field. As we discuss in 3.2, the expected loss at the (t+1)’th step of training involves the excess kurtosis of the noise distribution (and we refer you to Appendix C for that full computation). Thus, noise distributions with contrasting kurtosis can aptly be used to demonstrate the contrast adding different types of noise to the network can have and its downstream effects on incentivizing sparsity.

Dear Reviewer RE6X, we note that the ICLR discussion period has been extended. Could you please review our rebuttals to consider whether you would like to update your ratings? We would greatly appreciate your consideration and response.