Layers at Similar Depths Generate Similar Activations Across LLM Architectures

Thank you for your feedback and careful reading of our work. We are glad that you appreciated the clarity of our experiments. We agree that it is important for these experiments to be simple and interpretable. We devote the rest of our response to clarifying your questions.

The reviewer is concerned about the novelty of Claim 1 in particular. To clarify, we intend for Claims 1 and 2 to be taken together, not individually. Together these claims define the diagonal structure that we observe. As such, we do not argue that Claim 1 is novel on its own, but only that the diagonal structure (as defined by the combination of Claims 1 and 2) is novel. (We will make this more clear in our camera-ready version.)

When we first started this project, we thought---as we suspect the reviewer does---that existing interpretability work must have already shown that (only) layers at similar depths from different models are similar to one another. However, we were surprised to find that despite much peripheral work in interpretability that hints at this, there does not appear to be any existing work that systematically demonstrates this hypothesis empirically. We think our paper is important because it substantiates a claim that is perhaps widely suspected, but which has not been substantiated with systematic evidence. We have also encountered strong but divergent intuition in the literature and among colleagues about whether this phenomenon should hold. Even in this discussion, reviewer j38m was concerned about the validity of Claim 1, while BgXt finds Claim 1 very intuitive. We think this is the value of careful empirical work.

The reviewer also asked about the sensitivity of results to other datasets. We perform experiments on a range of datasets which are discussed in-depth in appendix A.

The reviewer also asked about the sensitivity of results to the choice of similarity measure. We have performed some experiments with other similarity measures like CKA and found similar results. We focus on the mutual k-NN similarity because measures like CKA are generally less interpretable and can be sensitive to things like the embedding dimension, meaning that one has to be more careful when performing cross-model comparisons. Also, to quote our submission, "We do not claim that the nearest neighbor-based measure we use is the 'correct' way to measure representational similarity (whatever that would mean), but that it shows that there exist properties of activations that are shared according to these patterns." In other words, that there exists any similarity measure that can identify pairs of layers from corresponding depths is already significant.

The reviewer suggests that experiments that get at similar questions could be done with SAEs or crosscoders. We agree that such experiments could be performed, but they would likely be far more complicated and consequently have more opportunities for misleading results. As the reviewer mentions, part of the strength of our experiments is their simplicity and concreteness. Our method does not require any additional training, alignment, or interpretability assumptions. While methods like SAEs or crosscoders could potentially reveal more interpretable or fine-grained shared features, they introduce additional complexity and possible confounds. That similarity is already present when looking at (arbitrary looking) nearest neighbor relationships is a stronger result than if similarity could be detected using aligned SAEs.

The reviewer also wonders about the potential causes of diagonal structure, as well as the content that is similarly represented across models. We agree that this is a natural followup that should be a topic of future research, perhaps using SAEs, or perhaps by simply studying the content of shared nearest neighbor sets. We will add a Future Work section to the camera ready suggesting possible research directions to address these questions.

The reviewer argues that activations on positions other than the last token could be studied. We have also run experiments where, instead of looking at the activations on the last token, we looked at the mean of the activations over all the tokens (which is done in some existing work). However, the results were very similar to those we got with last token activations. In general, last token activations can be compared more reliably between models because they are guaranteed to be independent of the tokenization (which can vary from model to model).

The reviewer asks whether the results are sensitive to the number of input texts used. We have run experiments with both fewer (1024) and more (4096) examples. The results were very similar to those we got with 2048. We have also run experiments on several disjoint datasets (as mentioned above), which further demonstrates that our results are not an artifact of a particular set of 2048 input texts.

Thank you for your consideration and careful reading of our work. We are glad that you appreciated our effort in running our experiments on lots of models and datasets. It was one of our core goals to say something not just about some particular pair of models, but about the whole population of modern LLMs, and this took significant effort and compute. We devote the rest of our response to clarifying your questions.

The reviewer asked about use of the cosine similarity for comparing embeddings. We chose to use the cosine similarity to measure distance because there are a number of places in transformer architectures where the cosine distance is effectively used. For example, the next token distribution is effectively chosen by the dot product of the last layer activations with the unembedding vectors. As such it is fairly standard practice to use the cosine similarity to compare embeddings.

The reviewer also asked about the sensitivity of results to the choice of similarity measure. We have performed some experiments with other similarity measures like CKA and found similar results. We focus on the mutual k-NN similarity because measures like CKA are generally less interpretable and can be sensitive to things like the embedding dimension, meaning that one has to be more careful when performing cross-model comparisons. Also, to quote our submission, "We do not claim that the nearest neighbor-based measure we use is the 'correct' way to measure representational similarity (whatever that would mean), but that it shows that there exist properties of activations that are shared according to these patterns." In other words, that there exists any similarity measure that can identify pairs of layers from corresponding depths is already significant.

The reviewer also wonders about the potential causes of diagonal structure, as well as the content that is similarly represented across models. We agree that this is a natural followup that should be a topic of future research, perhaps using SAEs, or perhaps by simply studying the content of shared nearest neighbor sets. We will add a Future Work section to the camera ready suggesting possible research directions to address these questions.

To answer the specific questions:

• We have not run experiments where we mixed together multiple datasets. However, we would speculate that nearest neighbors will mostly be within one dataset, so the results would likely be similar (though perhaps scaled by a constant).

• We have run experiments with both fewer (1024) and more (4096) examples. The results were very similar to those we got with 2048.

• We ran experiments where, instead of looking at the activations on the last token, we looked at the mean of the activations over all the tokens (which is done in some existing work). However, the results were very similar to those we got with last token activations. In general, last token activations can be compared more reliably between models because they are guaranteed to be independent of the tokenization (which can vary from model to model).

• The reviewer noted that the caption for figure 6a mentions a blue line that does not appear. This was a typo and we have fixed it.

• Thank you for the reference to the Activation Space Interventions paper. We will have to take a look and see how it relates!

After reading the authors' responses to mine and other reviewers' questions I'm happy with my assessment and think this article should be accepted.

Thank you for feedback and careful consideration of our paper. We agree that highlighting the phenomenon of diagonal structure and substantiating it empirically is a central contribution and grateful that you appreciated it. We will use the rest of this response to clarify your other questions.

The reviewer is concerned that although diagonal structure is visible in affinity matrices, they would like to see more numerical reporting of its absolute magnitude. This is a great point. As the absolute similarity at corresponding layers is relatively high (see figures 5b and 6b), we should perhaps explore if there are ways that we could visualize this in a way that makes the numerical scale clearer. At the same time, our emphasis is on the patterns of similarity more so than their absolute values. That the most similar layers are at corresponding depths is significant regardless of the absolute similarity (we have some discussion of this in appendix A.3).

The reviewer also wonders about the potential causes of diagonal structure. We agree that this is a natural followup that should be a topic of future research, perhaps using SAEs, or perhaps by simply studying the content of shared nearest neighbor sets. We will add a Future Work section to the camera ready suggesting possible research directions to address these questions.

To answer the specific questions:

• The reviewer asks whether figure 5 is mentioned in the main body. I believe the subfigures of figure 5 are referenced in the main text on lines 227 and 249.

• The reviewer correctly notes that $t$ is supposed to say $x$ in definition 3. We will make that revision.

Thank you for feedback and careful consideration of our paper. We are glad that you appreciated our effort in running experiments on a wide range of models and datasets. It was important to us to have experiments that represented modern LLMs generally and not just a particular pair of models. We will use the rest of this response to clarify your other questions.'

The reviewer suggests centering diagonal structure as the main message (rather than Claims 1 and 2). We agree; however, to clarify, we intend for Claims 1 and 2 to be taken together as defining the diagonal structure that we observe. This is what we meant when we wrote that the claims "seem unremarkable on their own, but together are quite surprising"). We will clarify this point in the camera-ready version.

The reviewer also notes that consecutive layers within the same model appear to show high similarity and asks how this relates to Claim 1. We agree that consecutive layers within a single model often generate similar activations (which would be expected because of residual connections). However, more distant layers tend to generate increasingly different activations. This can be seen indirectly even in cross-model affinity matrices: If all the layers in a model generated similar representations, then you would not see diagonal structure in cross-model comparisons because you would have that $A_{i,j} \approx A_{k,j}$ which would not give diagonal structure. Diagonal structure in cross-model comparisons is only possible because layers within a model can be distinguished by depth (Claim 1).

We also agree that more numerical description of diagonal structure generally would be helpful, and have already prepared expanded statistical tests of diagonal structure which we plan to include in the camera ready.

Finally, the reviewer suggests studying how similarity "decays" as layers get further apart. We actually generated decay curve plots that show similarity as a function of the distance between layers, but did not end up including them. In particular, we never found a good way for visualizing these decay curves for cross-model comparisons. However, the reviewer makes a good case that even within-model decay curve plots would be valuable addition which we will include in the camera ready.