DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models

Need for Elaboration Problem 1

We appreciate the reviewer's suggestions. We have already addressed these in the general reply and added the content to Appendix E.3 in the updated manuscript.

As demonstrated in Figure 9 in Appendix E.3, our experimental results indicate a high degree of orthogonality among the weight matrices in LLMs. Similar observations have been documented in [1], which highlights the limitations of applying existing similarity indices directly to weight similarity in this domain due to their lack of distinguishability for orthogonal matrices.

Reference
[1] Tian, Y., Wang, Y., Zhang, Z., Chen, B., & Du, S. (2023). Joma: Demystifying multilayer transformers via joint dynamics of MLP and attention. arXiv preprint arXiv:2310.00535.

Need for Elaboration Problem 2

We appreciate the reviewers for raising these suggestions.
In Theorem 1, we establish the existence of a set of column-orthogonal matrices with significant differences, specifically:

$\| X - Y \|_F = \Omega(\sqrt{m}),$

where the DOCS similarity is capable of reflecting these differences effectively:

$S_{\text{DOCS}}(X, Y) = \frac{1}{\sqrt{m}}.$

In contrast, as shown in Table 1 of the paper, most existing methods yield a constant similarity value (independent of dimensionality) for any pair of orthogonal matrices, regardless of their differences. Theorem 1 theoretically demonstrates the superior discriminative power of the DOCS method for orthogonal matrices. Additionally, we provided a concrete example to illustrate this ability in distinguishing orthogonal matrices in Appendix A.5. These elaborations have been added to Section 3.1 of the revised manuscript for clarity.

Question Problem 1

Thank you for the suggestion. We have included these experimental results in Appendix F of the revised manuscript. In Figure 11 in Appendix F, we provide a visual comparison of eight different similarity indices applied to the MLP-UP layers of the reinitialized Meta-Llama-3.1-8B-Instruct model. We observed that all of them exhibit relatively random patterns.

Question Problem 2

We apologize for the confusion caused by our earlier wording. What we intended to convey is that a uniform configuration (at least during the SFT stage) does not fully utilize the inherent cluster structure of the model, and therefore, it may not be an optimal choice. We have revised the relevant statements in the updated manuscript for clarity.

Question Problem 3

Thank you for the suggestion. We have addressed this issue and revised the manuscript accordingly in the revised manuscript.

Evaluation and Problems 1 & 2 in Overclaiming

We appreciate the reviewer's for raising these concerns. We have already addressed these in the general reply and added the content to Appendix E in the updated manuscript.

Overclaiming Problem 3

We apologize for the confusion caused by our original phrasing. The statement "exhibiting high mutual similarity" has been revised to "exhibiting relatively high mutual similarity (approximately twice the DOCS index values of other elements)" in the updated version of the manuscript.

Overclaiming Problem 4

We acknowledge the potential for confusion in our wording. The phrase "could lead to more efficient models" has been changed to "DOCS similarity can guide such designs" to clarify our intended meaning.

Overclaiming Problem 5

We appreciate your insightful comments and have revised the manuscript accordingly. Specifically, we have removed the discussion regarding the relationship between clusters within the LLM layers and reasoning phases, recognizing that it was speculative.

Overclaiming Problem 6

We appreciate the reviewer’s suggestion. Here, our intent was to convey that the distribution of clusters differs between models (e.g., Gemma-2-9b and Llama-3.1-70B), likely due to differences in their training strategies, rather than implying that a single model undergoes multiple training phases. This clarification has been incorporated into the revised manuscript.

Overclaiming Problem 7

We appreciate your feedback on this point. Our original intent was to indicate that the parameter similarities across all layers are relatively high between base models and instruction-tuned models, suggesting that the base and instruction-tuned models share almost the same underlying knowledge.

Missing Details Problem 1

We apologize for the lack of clarity in describing how we derive the numeric values. Specifically, when applying the function MAXCOSSIM, we treat the elements in the returned vector as data points and use maximum likelihood estimation to compute the location parameter of the Gumbel distribution. This clarification has been incorporated into Section 3 in the revised manuscript.

Missing Details Problem 2

We appreciate the reviewer's suggestion. To address concerns regarding missing details, we have demonstrated how the DOCS method can distinguish between orthogonal matrices in the revised paper's Appendix A.5 (in the sense defined by Definition 3 in the paper).

Missing Details Problem 3

We apologize for the lack of clarity in describing how we derive the baselines for comparison. The baseline methods initially proposed were designed to measure representation similarity. However, since both representations and parameters are inherently matrices, it is technically feasible to adapt these baseline methods to compute parameter similarity. Therefore, in our implementation, we directly use the parameter matrices as the input for baseline methods.

Notably, in LLM implementations, the rows of a weight matrix correspond to output dimensions, and the columns correspond to input dimensions. To align the column vectors with meaningful entities (e.g., neuron weights), similar to the DOCS method, we transpose the matrices $W_v$, $W_k$, $W_q$, and $\text{MLP-Up}$ before inputting them into the baselines.

Impact of the Findings Problem 2

We appreciate the reviewers' feedback and the opportunity to clarify our findings. Regarding the trends in the experiments comparing base and fine-tuned models and MoE, we have further updated and refined our analysis in the revised paper. We hope our analysis provides meaningful insights when interpreted in the context of the DOCS index and the architectural roles of the models.

Specifically, for comparing base and fine-tuned models, DOCS scores exceeding $0.7$ consistently across all evaluated weight matrices indicate a strong preservation of foundational knowledge post-fine-tuning. Additionally, the variation patterns in the DOCS scores' curves allow weight matrices to be categorized into three distinct groups ($W_q$ and $W_k$, $W_v$ and $W_o$), potentially offering insights into the optimization dynamics structure during fine-tuning.

For the MoE experiments, the visualization of expert weight matrices revealed clear distinctions among certain experts, as evidenced by the pronounced separation in similarity heatmaps. This phenomenon underscores the potential for individual experts to specialize in unique roles. We postulate that such behavior could be attributed to imbalances in data distribution during the training of the MoE architecture, wherein a significant portion of the input is directed toward one specific expert. These findings provide a basis for further investigation into the mechanisms underpinning parameter changes and expert specialization across model configurations [1],[2].

Reference
[1] Dai, D., Dong, L., Ma, S., Zheng, B., Sui, Z., Chang, B., & Wei, F. (2022). Stablemoe: Stable routing strategy for mixture of experts. arXiv preprint arXiv:2204.08396.
[2] Zuo, S., Liu, X., Jiao, J., Kim, Y. J., Hassan, H., Zhang, R., et al. (2021). Taming sparsely activated transformer with stochastic experts. arXiv preprint arXiv:2110.04260.

Question

We appreciate the reviewer's thoughtful feedback. The properties we have identified are intended to ensure that the similarity measure is meaningful. Specifically, these include:

• Permutation Transformation (PT) Invariance: $S(X, Y) = S(XP\_X, YP\_Y)$ for any permutation matrices $P_X$ and $P_Y$.
• Symmetry: $S(X, Y) = S(Y, X)$.
• Isotropic Scaling (IS) Invariance: $S(aX, bY) = S(X, Y)$ for any non-zero scalars $a$, $b$.
• Reflexivity: $S(X, X) = 1$.
• Discriminative on Orthogonal Matrices.

Together, these properties are sufficient because they collectively address both invariance to irrelevant transformations and sensitivity to differences in orthogonal matrices. By satisfying these criteria, the similarity measure can provide interpretable comparisons of transformer weights across various contexts.

Further, other widely used similarity methods for models, such as those proposed by [1], [2], and [3], consider mathematical properties related to metrics that are subsets of the above properties.

Reference
[1] Kornblith, S., Norouzi, M., Lee, H., & Hinton, G. (2019, May). Similarity of neural network representations revisited. In International conference on machine learning (pp. 3519-3529). PMLR.

[2] Morcos, A., Raghu, M., & Bengio, S. (2018). Insights on representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31.

[3] Raghu, M., Gilmer, J., Yosinski, J., & Sohl-Dickstein, J. (2017). Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30.

Motivation

Thank you for your comments regarding the motivation of our work. We have already addressed this in the general reply and added the content to Appendix E in the updated manuscript.

Empirical Justification Problem 1

We thank the reviewer for the insightful comment regarding the comparison of DOCS similarity indices over other methods. We have already addressed this in the general reply and added the content to Appendix E.3 in the updated manuscript.

Empirical Justification Problem 2

We sincerely thank the reviewer for the suggestions. The computation of the Gini coefficient in our work involves several steps. First, we exclude the diagonal elements from the similarity matrix as they are always equal to 1, focusing solely on inter-layer relationships. For each row, the remaining elements are normalized by dividing them by the sum of all row values, ensuring that the resulting coefficients reflect the relative distribution of similarity scores within the row. The Gini coefficient for each row is then calculated to quantify the inequality in the distribution of these scores, with higher values indicating a greater concentration of similarity among fewer layer pairs. Finally, the mean Gini coefficient across all rows is computed to summarize the overall inequality in the similarity matrix.

A higher Gini coefficient suggests a more uneven distribution of similarity scores, implying a greater concentration of significant similarities in fewer layer pairs, which potentially reveals structural characteristics of the model parameters. As demonstrated in Table 2 of our paper, our proposed method (DOCS) achieves higher Gini coefficients compared to other methods, indicating its superior ability to isolate significant layer similarities. We have included clarifications and detailed explanations in the revised manuscript to address this point comprehensively.

Impact of the Findings Problem 1

We sincerely thank the reviewer for their valuable feedback and for recognizing the interest in exploring potential redundancies within neural network weights. While it is true that redundancy in neural networks has been previously studied, our work introduces a novel approach through the DOCS metric, offering fresh insights into this phenomenon. Specifically, DOCS enables a quantifiable and interpretable analysis of weight similarities, uncovering nuanced patterns that extend beyond prior findings. For example, we identified cluster structures among Transformer layers (Figure 4), which suggest deeper functional organization. Moreover, by analyzing similarities between base and instruction-tuned models, we demonstrated the strong preservation of foundational knowledge after fine-tuning, as well as distinct grouping tendencies of weight matrices ($W_q$ and $W_k$, $W_v$ and $W_o$), which may provide insights into optimization dynamics.

Additionally, the application of DOCS to Mixture-of-Experts (MoE) architectures revealed clear distinctions among experts, highlighting their potential specialization. These contributions suggest that DOCS not only reinforces existing knowledge but also supports future innovations. For instance, the identified clusters can guide parameter-efficient techniques like LoRA by assigning adaptive ranks to clustered layers, aiding in model compression and structured knowledge distillation. We believe these aspects illustrate the distinct contributions of our method and its applicability to advancing neural network understanding and optimization.

I thank the authors for their detailed response.

Motivation

I understand and agree with the authors argument that similarity in the weight space can reveal patterns that may not be evident from looking at similarity in the representation space. My question is about the significance of such patterns, particularly given that it is the representations that ultimately affect the outputs and the behavior of the model.

For instance, in the example presented in Appendix E, the weight space analysis shows similarity between layers that is not evident in from the representation space analysis. But does this necessarily mean that there is redundancy in the model, given that the representations are actually dissimilar? Would it make sense to prune the model by removing layers with similar weights even when their output representations are not similar? I think it would make more sense to make pruning decisions based on representation space analyses.

Are there any actionable insights that only weight space analyses can provide?

Empirical justification from the Gini coefficient

I thank the authors for the additional details about the computation of Gini coefficient and how to interpret it. I understand that this analysis highlights the differences between the similarity spaces, and the DOCS results in a more uneven distribution of similarity scores. However, I find it hard to interpret this result because there isn't a notion of ground truth here. If there were a true similarity metric, one could compare the Gini coefficient of that metric with DOCS and the other metrics, and see which one's coefficient is closes to the true metric. But without that, how does a greater unevenness in similarity scores necessarily mean a better metric?

Weaknesses 1 and Questions 1

We appreciate your insightful comments. We have removed the discussion regarding the relationship between clusters within the LLM layers and reasoning phases from the manuscript, recognizing that it was speculative.

However, we would like to clarify our original intent here. Consider an extreme case where the layers within each of the two clusters are identical. In this scenario, the first cluster functions by transforming the token embeddings into a different knowledge space—acting as an implicit encoder. The second cluster then transforms this encoded information back into the token embedding space, effectively decoding it to produce the next token. This leads us to conjecture that the first cluster corresponds to an encoding process that maps the input into a latent knowledge representation, while the second cluster decodes this representation to generate the output. We believe this interpretation offers a meaningful explanation for the structural patterns observed across different models.

Weaknesses 2 and Questions 2

Thank you for your constructive comments and suggestions. In response to the request for rigorous proofs of $DOCS$ satisfying the desired mathematical properties, we have revised our manuscript to include these proofs in Appendix A of the updated manuscript. These include proofs of Permutation Transformation Invariance, Symmetry, Isotropic Scaling Invariance, Reflexivity, and Discriminative on Orthogonal Matrices.

Questions 3

We thank the reviewer for the insightful comment regarding the comparison of similarity indices and the need to further demonstrate the advantages of DOCS over other methods. We have already addressed this in the general reply and added the content to Appendix E.3 in the updated manuscript.

We greatly appreciate the reviewer’s insightful comment regarding the need for further analysis on what makes the DOCS algorithm effective. In response, we have added some additional experimental results in Appendix G of the updated manuscript.

We illustrate the intermediate computations for the $MLP\text{-}Up$ parameter matrices from the 4th and 8th blocks of the Meta-Llama-3.1-8B model, presenting frequency distribution histograms of the resulting similarity vectors $\mathbf{s}_X$ and $\mathbf{s}_Y$ (Figure 12 in Appendix G). These results demonstrate the intermediate outcomes of the DOCS method.

Furthermore, we conducted an analysis comparing the vanilla DOCS algorithm with an average-based variant. As shown in Figure 13 in Appendix G, the average-based DOCS approach produces heatmaps that fail to capture meaningful structural patterns. This analysis demonstrates that the max operation in MaxCosSim is crucial for extracting salient features.

Weaknesses 1 and Questions 1

We appreciate the reviewer's insightful question regarding our choice of the Gumbel distribution to summarize the maxima of the cosine similarities. To address your concern, we provide a more detailed mathematical justification grounded in Extreme Value Theory (EVT).

In our algorithm, for each column $A\_j$, we compute the maximum absolute cosine similarity:

This results in a set of maxima $\mathbf{s}\_X = [s\_{A\_1}, s\_{A\_2}, \dots, s\_{A\_m}]^\top$.

According to the Fisher-Tippett-Gnedenko theorem, the limiting distribution of the maxima of a sequence of independent and identically distributed (i.i.d.) random variables, after appropriate normalization, converges to one of three types: Gumbel, Fréchet, or Weibull. Theoretical results and empirical studies [1] have shown that for bounded variables, the Gumbel distribution can provide a good approximation for the distribution of maxima.

By fitting a Gumbel distribution to $\mathbf{s}\_X$ and $\mathbf{s}\_Y$, we obtain the location parameters $u\_X$ and $u\_Y$, which summarize the central tendency of the extreme similarities between $X$ and $Y$. The similarity index $S\_{\text{DOCS}}$ is then computed as:

Reference
[1] Leadbetter, M. R., Lindgren, G., & Rootzén, H. (2012). Extremes and related properties of random sequences and processes. Springer Science & Business Media.

Weaknesses 2 and Questions 2

We appreciate the reviewer's constructive feedback and agree that discussing practical applications would strengthen the impact of our paper. Below, we address the comment by suggesting several additional potential applications besides what people have already done (e.g., leveraging layer clusters to reduce computational requirements) of our findings:

1. Incorporating Inductive Biases During the Supervised Fine-Tuning Stage:
   Building on the identified cluster structures, we propose introducing specific inductive biases in the fine-tuning stage of parameter-efficient fine-tuning methods, such as LoRA. For example, we can set adaptive ranks for layers within the cluster to minimize disruptions to the base model's original parameter structure.

2. Guiding Sparsity Patterns for Model Compression:
   The clustering results could inform sparsity patterns by selectively pruning redundant connections within clustered layers. Such targeted sparsification could potentially reduce model size and computational costs while preserving performance, especially in resource-constrained deployment scenarios.

3. Enhancing Knowledge Distillation Strategies:
   The layer clustering could also be used to identify critical layers that should be prioritized during the knowledge distillation process. For instance, a student model could focus on mimicking the most representative layers within each cluster, reducing the need to replicate the entire teacher model's architecture.

We have incorporated these potential future work ideas into section 5 in the revised manuscript to illustrate the broader implications and potential applications of our findings.