Large Language Model Evaluation via Matrix Nuclear-Norm

Q1：There are many notations used without definition.

Thank you for highlighting this issue. We acknowledge that some notations were not clearly defined in the original manuscript, which may have caused confusion. In the revised version, we have carefully reviewed all notations and ensured that every term is explicitly defined upon first use. This improvement aims to enhance the clarity and precision of the paper, and we appreciate your feedback for bringing this to our attention.

Q2：Please check $$\mathcal{D}$$ in equation (4). Please avoid using (1) and (2) in lines 206 and 207. Please check $$\hat{A}$$ in (9). Please check Algorithm 1 carefully, such as $x_i$ and Steps 2 and 3.

2.1： Thank you for pointing out the potential confusion regarding the interpretation of ( $E_C$ ) in Equation (4). In the revised manuscript, we will provide a more detailed explanation of ( $E_C$ ), clarifying that it refers to the average number of unique categories actively involved in predictions, rather than the total number of categories available in matrix ( $A$ ). We will differentiate ( $E_C$ ) from related terms to ensure that readers can easily understand its precise role and meaning. These clarifications will be incorporated into the relevant sections, improving the overall clarity and logical flow of the manuscript.

2.2： Thank you for pointing out the potential ambiguity caused by the use of numbered lists in this section. We have revised the text to eliminate the use of "(1)" and "(2)" and replaced it with a clear and concise description of the two advantages. The updated text now directly highlights the computational efficiency and stability of our approach without relying on numerical enumeration. This change ensures smoother readability and addresses your concern. The revised content can be found in the updated manuscript in the section corresponding to the description of the ($L_{1,2}$)-norm approximation.

2.3： Thank you for your observation. In the revised manuscript, we will explicitly explain the meaning and role of $\hat{A}$ in Equation (9). Specifically, we will clarify that $\hat{A}$ is a matrix derived to approximate the nuclear norm using the $L_{1,2}$ -norm. It achieves this by selecting principal components to reduce computational complexity. This explanation will be added immediately before Equation (9) to ensure readers understand how $\hat{A}$ contributes to the proposed method.

2.4: Improving Consistency in Algorithm 1 Notation We appreciate your suggestion regarding the notation consistency in Algorithm 1. In the revised manuscript, we will standardize the use of $X_i$ to uniformly represent sentence embeddings throughout the algorithm. Furthermore, we will provide more detailed annotations for each step, especially Steps 2 and 3, to explain their purpose and improve the algorithm's readability. These updates will ensure that Algorithm 1 is clear and consistent, enabling readers to better understand its implementation and significance.

Algorithm 1: Algorithm of Matrix Nuclear-Norm

Require:
Sentence representations (hidden states from LLM) $\mathcal{S} = \\{X _ {i} \\} _ {i=1}^{m}$, where $X _ {i} \in \mathbb{R}^{d \times 1}$, $d$ is the hidden dimension of representation, and $L _ {\text{input}}$ is the length of the sentence.

1. $\mu = \frac{1}{m} \sum _ {i=1}^{m} X _ i$ // Calculate the mean embedding

2. $\mathbf{X} _ {\text{norm}} = \frac{\mathbf{X} - \mu}{\|\mathbf{X} - \mu\| _ {2, \text{row}}}$ // Normalize the activation matrix

3. $\text{L2}(\mathbf{X} _ {\text{norm}}) = \sqrt{\sum _ {i=1}^{m} \mathbf{X} _ {i,j}^2}$ // Calculate $L _ {2}$-norm for each column

4. $\Sigma_D = \{ \sigma_1, \sigma_2, \dots, \sigma_D \}$ // Sort $L_{2}$-norm and select top $D$

5. $\text{Matrix Nuclear-Norm}(\mathbf{X}) = \frac{\sum _ {i=1}^{D} \left( \sqrt{\sum _ {j=1}^{m} \mathbf{X} _ {j,i}^2} \right)}{L _ {\text{input}}}$ // Calculate Matrix Nuclear-Norm

6. Return: $\text{Matrix Nuclear-Norm}$

Q1：The motivation and background story for the proposed method are not well articulated, making it difficult to grasp why the method is particularly suitable for evaluating LLMs

Thank you for your feedback. In Section 5.2.1, we address the key limitation of Matrix Entropy—its high computational complexity, which makes it challenging to apply to evaluate large-scale LLMs. Our method overcomes this by using the $L_{1,2}$-norm for efficient nuclear norm approximation, reducing complexity while maintaining comparable evaluation trends. We will revise the manuscript to clarify this motivation more effectively.

Q2：The explanation regarding related work, specifically Matrix Entropy, is inadequate, which hinders the reader's understanding of how Matrix Nuclear-Norm compares to previous efforts and why it offers a better alternative

Thank you for pointing this out. We recognize the importance of thoroughly explaining related work, particularly Matrix Entropy, to clarify how our method builds upon and improves previous efforts. In the revised manuscript, we have expanded the Related Work section to include a detailed discussion of Matrix Entropy, highlighting its methodology, its computational limitations due to reliance on Singular Value Decomposition (SVD), and its relevance to LLM evaluation. Furthermore, in Section 5.2.1, we present experimental evidence demonstrating the feasibility and effectiveness of our proposed method. Specifically, we show how Matrix Nuclear-Norm addresses the computational inefficiencies of Matrix Entropy while maintaining comparable evaluation trends. These updates aim to provide a clearer understanding of how our approach extends and enhances existing metrics. We appreciate your feedback and have ensured that the revised manuscript reflects these improvements.

LLM Evaluation Metrics. Traditional evaluation metrics such as perplexity, BLEU \citep{papineni2002bleu}, and ROUGE \citep{lin2004rouge} primarily measure task-specific outcomes, assessing how well model outputs align with ground truth data. While these metrics are effective for evaluating surface-level outputs, they do not capture the underlying mechanisms of LLMs, such as the diversity or compression of embeddings. Similarly, accuracy and F1 score \citep{2007The} focus on classification performance, making them less applicable to the generative tasks typical of LLMs.}\textcolor{blue}{To bridge this gap, structural metrics such as Matrix Entropy have been introduced. Matrix Entropy \citep{wei2024large} employs information theory to assess the entropy of covariance matrices derived from LLM embeddings. This metric evaluates how effectively a model removes redundancy and encodes structured information, offering a measure of its compression capabilities. For instance, Matrix Entropy can reveal differences in embedding distributions across models of varying sizes, reflecting their capacity to extract meaningful patterns from large datasets. However, its reliance on Singular Value Decomposition (SVD) results in a computational complexity of $O(n^3)$, limiting its applicability to modern large-scale models. To overcome these limitations, we propose the Matrix Nuclear-Norm as a scalable alternative. By leveraging the $L_{1,2}$ norm as a convex approximation of matrix rank, the Matrix Nuclear-Norm reduces computational complexity to $O(n^2)$. This makes it feasible for evaluating embeddings from large-scale LLMs while preserving the insights provided by Matrix Entropy, such as compression efficiency.

Q3：The notation throughout the paper is used inconsistently, which introduces ambiguity and impedes readability.

Thank you for your feedback. Please refer to Reviewer 2, Q3 for our detailed response.

Q3: The presentation is poor and some notations are confusing.

Thank you for pointing this out. We acknowledge the need for greater clarity and consistency in our notations and presentation. Below, we clarify the key points and outline our planned revisions:

• Matrix $A$ as an Output Matrix:
  In the context of evaluating LLMs, the matrix $A$ represents the hidden layer outputs or intermediate embeddings generated by the model for a given batch of inputs. Each element $A_{i,j}$ corresponds to the activation value of the $j$-th dimension for the $i$-th input. Unlike a probability matrix typically used in classification tasks, $A$ serves as a high-dimensional representation that captures the model's internal computations. The values and structure of $A$ provide insights into the diversity and richness of the model's output, making it a crucial component for evaluating efficiency, representation capacity, and redundancy within LLMs.

• Definition of "Active Dimensions" in $E_C$:
  The term "active dimensions" refers to the subset of dimensions in the output matrix $A$ that contribute significantly to the model's representations, determined by their activation values. These dimensions are typically identified using a thresholding criterion (e.g., non-zero values or the top-$k$ largest activations). Unlike the total number of columns in $A$, which represents all potential dimensions, $E_C$ measures the average number of active dimensions across the dataset. This serves as an indicator of the model's ability to effectively utilize its representation space, balancing compactness with information richness.

• Sparsity of $A$:
  While $A$ is formally dense, in practice, many elements are near zero, especially for high-dimensional models or tasks with complex representation spaces. This sparsity arises naturally in LLMs, as meaningful information tends to be concentrated in a small number of dimensions, leaving many activations negligible. Such sparsity reflects the model’s efficiency in encoding relevant features and supports computational optimizations like nuclear norm approximation. By leveraging this sparsity, our approach reduces computational overhead while maintaining evaluation effectiveness.

• Clarifying $A$ is Not Binary:
  Despite its sparsity, $A$ remains a real-valued matrix with continuous activation values. The sparsity does not imply binary (0/1) values but rather a selective distribution of activations where most dimensions contribute minimally. This allows the matrix to encode nuanced information in the most critical dimensions while discarding less relevant features, a key characteristic for evaluating the compression and efficiency of LLM representations.

To address these issues, we will ensure in the revised manuscript:

1. All key notations, including $A$ , $E_C$ , and their properties, are clearly defined when first introduced.
2. Logical consistency in the use of terms like "dense," "sparse," and "unique categories."
3. Improved explanations of $E_C$ and sparsity in the context of model evaluation metrics.

We appreciate your feedback, which has guided us to improve the clarity and logical coherence of our work.

Q1: The concern on novelty, the contributions may be limited and incremental.

Thank you for your thoughtful feedback. We understand your concern regarding the novelty and incremental nature of the contributions. It is worth noting that the metric we build upon, originally termed Matrix Entropy, has been formally renamed as "Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models" and has been accepted at NeurIPS 2024 after rigorous peer review. This acceptance underscores the metric’s growing recognition and relevance within the LLM community. Our work builds on this foundation by addressing its computational inefficiency, a critical limitation in its practical application to large-scale models. Specifically, we propose an efficient approximation method that significantly reduces the computational complexity, making the metric scalable and applicable to real-world scenarios. This improvement enhances the practicality and impact of Diff-eRank, ensuring its utility in evaluating modern LLMs.

Q2: The efficiency improvement from O($n^3$) to O($n^2$) is not valuable in practice. Actually, the fundamental research on optimizing computational efficiency can not be that astonishing, it should be conducted step-by-step and with some mathematical support.

Thank you for your insightful feedback regarding the practical value of the efficiency improvement from $$O(n^3)$$ to $$O(n^2)$$. While we understand your perspective, we believe that computational efficiency optimization, as a foundational area of research, is a step-by-step process requiring rigorous mathematical support and theoretical justification. Such advancements are rarely achieved instantaneously or without careful derivations, and our work aims to contribute to this ongoing effort with a solid theoretical foundation and empirical validation.

In real-world applications, where large language models (LLMs) often have massive parameter sizes, metrics that can efficiently evaluate their performance are particularly critical. In Section 5.2.1 (l.317), we provide experimental evidence demonstrating the practical significance of our approach. For instance, when applied to models of increasing scale, such as a 13B-parameter model, our method completes the computation in 0.82 hours, compared to 16 hours required by Matrix Entropy. This significant reduction in computational time highlights the practical relevance of our metric, particularly as model sizes continue to grow.

While achieving $$O(n)$$ or $$O(n \log n)$$ complexity is an aspirational goal, our proposed method represents a meaningful step forward, making evaluation feasible for large-scale models and contributing to the broader understanding of computational efficiency in the context of LLM evaluation.

Q15 & Q16 & Q22：

• S 5.2.1 the authors conclude in l.341 that the metric considered achieves comparable evaluation effectiveness to Matrix Entropy, however, the figure presented for this experiment only compares time complexities.

• In l.379, the authors claim that their metric showcases superior accuracy, however, the meaning of this sentence is not clear to me. In which sense did the author show that the metric considered is more effective than others?

• How one can compare the proposed metric with others? More precisely, in which sense the proposed metric demonstrates superior accuracy compared to traditional metrics?

Thank you for pointing this out. We recognize that our wording may have been unclear, and we apologize for any confusion it caused. Metrics are not inherently superior or inferior; rather, they are designed to serve specific purposes based on the evaluation context.

What we intended to convey is that our proposed metric produces evaluation trends consistent with those observed in validated metrics such as perplexity and Matrix Entropy. Specifically, our results demonstrate that as model size increases, performance improves, and as size decreases, performance declines—consistent with established evaluation principles for large language models. These consistent trends validate the reliability and utility of our metric as an effective tool for LLM evaluation.

Additionally, our metric offers significant practical advantages by reducing computational complexity and evaluation time, which enhances its feasibility for large-scale model assessments. We will revise the manuscript to clearly articulate these points and ensure a precise representation of our findings. Thank you for your constructive feedback, which has helped us refine our presentation.

Q17：typo l. 115: Shannon Entropy only applies to nonnegative matrices

Thank you for pointing this out. We acknowledge the importance of clearly specifying such details. The statement that "Shannon Entropy only applies to nonnegative matrices" has already been addressed in the supplementary materials ( Please line 1046, A5 in appendix ), where we provided a detailed proof to support this. We will ensure that this clarification is also reflected in the main text to avoid any ambiguity.

Q18：What is the purpose of evaluating the nuclear norm of the output matrix?

Thank you for your question. The nuclear norm of the output matrix is evaluated to assess the model's ability to extract meaningful information while minimizing redundancy. As a convex approximation of matrix rank, the nuclear norm captures the effective dimensionality of the embeddings, providing insight into the model’s ability to generate structured and diverse representations. This metric effectively measures the compression capabilities of large language models (LLMs), reflecting how efficiently the model retains essential information while reducing redundancy. We believe this property makes the nuclear norm a valuable tool for evaluating LLM performance.

Q19：If the goal is to measure the diversity of the prediction, why not consider directly the entropy as defined in Eq. (1)?

Thank you for the question. The primary goal of our metric is not merely to measure prediction diversity but to evaluate the model's information compression capability—its ability to extract meaningful information from large datasets while minimizing redundancy. Unlike entropy or loss, which focus solely on the final output, our metric evaluates the information captured in the hidden layers, providing a deeper assessment of the model's representation and compression efficiency. This makes it particularly suitable for evaluating LLMs.

Q20：In which sense does the nuclear norm enable the measurement of the compression capabilities of LLMs?

Thank you for the question. The nuclear norm enables the measurement of the compression capabilities of LLMs by serving as a convex approximation of matrix rank. Matrix rank reflects the amount of independent information captured in a representation, which directly relates to a model's ability to compress data by retaining essential features while reducing redundancy. Additionally, the nuclear norm is an upper bound of the Frobenius norm, linking it to the energy preserved in the hidden states. By minimizing the nuclear norm, we effectively assess the model's ability to achieve low-rank approximations, which is a key aspect of information compression in high-dimensional representations. This property makes it particularly relevant for evaluating LLMs' efficiency in extracting structured information.

Q11：$C_p(A)$ definition?

Thank you for your feedback regarding the need to clarify $C_p(A)$ in Equation (5). In the revised manuscript, we provide a formal definition of $C_p(A)$ that better aligns with the continuous and high-dimensional nature of LLM generation tasks.

We now define $C_p(A)$ as follows, with the final version of our paper incorporating this definition immediately in Equation (5).

where $C_p(A)$ reflects the effective number of active dimensions in the output matrix $A$. In LLM generation tasks, $A$ represents the model's continuous activation matrix across embedding dimensions for a given batch of inputs. Unlike discrete categories in classification tasks, here $C_p(A)$ captures the diversity and richness of activations by evaluating the rank of a transformed matrix. The transformation uses an $\text{argmax}$ operation to identify the most significant activations for each input, linking $C_p(A)$ to the extent of dimensional utilization in the representation space.

To further ensure clarity, we revised the surrounding text to explain how $C_p(A)$ integrates into Equation (5). This revision highlights how $C_p(A)$ quantitatively measures the diversity of active dimensions in $A$, providing insights into the efficiency and comprehensiveness of the model's embedding space utilization during text generation.

We sincerely appreciate your comment, which allowed us to refine this section and improve the theoretical rigor of the manuscript.

Q12：the definition of top() in Eq.9

Thank you for pointing out the need for greater clarity regarding the "top" notation in Eq. (9). In the revised manuscript, we have added an explicit definition of "top" immediately before Eq. (9). Specifically, "top" refers to selecting the $D$ largest values from the given set. This operation is central to our approach, as it enables the approximation of the nuclear norm using the $L_{1,2}$ -norm.

Additionally, we have revised the surrounding text to better illustrate how this operation integrates with the overall computational framework and impacts the efficiency of our method. We appreciate your feedback, which has helped us improve the precision and clarity of the manuscript.

Q13：Notations and indices are not consistent throughout the paper.

We appreciate your valuable observation regarding the consistency of notations and subscripts. To address this, we have conducted a thorough review of the manuscript to ensure that all notations and subscripts are used consistently throughout. This includes standardizing definitions across equations, algorithms, and figures to enhance readability and coherence. Thank you for highlighting this issue, which has significantly helped us refine the presentation of our work.

Q14：$\Sigma _ {S _ i}$ in Eq.(13) is never properly defined.

Thank you for pointing out the need to clarify the definition of $$\Sigma _ {\mathcal{S} _ i}$$. In the revised manuscript, we have explicitly defined $$\Sigma _ {\mathcal{S} _ i}$$ in the section where the formula is introduced. Specifically, we now state:

$\Sigma _ {\mathcal{S} _ i} = \sum _ {j=1}^d \mathcal{S} _ {i,j}$

where $$\Sigma_{\mathcal{S}_i}$$ represents the sum of all elements in the embedding vector $$\mathcal{S}_i$$, and $d$ denotes the embedding dimension. This definition ensures that the matrix entropy calculation is normalized across the dataset. The updated definition has been incorporated into the main text after Eq. (13) to ensure self-containment and clarity. Thank you for your suggestion, which has significantly enhanced the precision and comprehensibility of our manuscript.

Q7 & Q8 & Q10：

• The proofs lack formal rigor, often omitting key arguments needed to substantiate the results.

• Notations are introduced without clear definitions.

• The proof of Theorem 1 is not complete: some (main) arguments are not exposed.

Thank you for bringing this to our attention. We will thoroughly review and clearly define all notations in the revised manuscript to ensure consistency and clarity. To address this concern, we have provided a more detailed and rigorous proof in the revised manuscript, explicitly including all key arguments necessary to substantiate the results. We are confident that these updates will enhance the precision and readability of our work.

\section{Proof of Theorem 1}

\label{sec:Theorem1}

This section presents the proof of the strictly inverse monotonic relationship between the entropy $H(A)$ and the Frobenius norm $$|A|_F$$ for a matrix $A$ .

\subsection{Definitions and Assumptions}

Let $$A \in \mathbb{R}^{B \times C}$$ be a non-negative matrix where each row represents a probability distribution:

with $$A _ {i,j} \geq 0$$. Here, $$A _ {i,j}$$ denotes the predicted probability that sample $i$ belongs to category $j$.

The Shannon entropy $H(A)$ of the matrix $A$ is defined as:

where $$0 \log(0)$$ is defined as $0$ by convention.

The Frobenius norm $$|A|_F$$ is defined as:

\subsection{Step 1: Entropy and Frobenius Norm for a Single Row}

Consider a single row $$\mathbf{a} = [a_1, a_2, \ldots, a_C]$$, where $$a_j = A_{i,j}$$, $$a_j \geq 0$$, and $$\sum_{j=1}^C a_j = 1$$. The row entropy is:

and the row Frobenius norm is:

To determine the extrema of $H_i$, we use the method of Lagrange multipliers. Define the Lagrangian:

Taking the partial derivatives and setting them to zero:

Using the normalization condition $$\sum_{j=1}^C a_j = 1$$, we find $$a_j = \frac{1}{C}$$ for all $$j$$. Substituting $$a_j = \frac{1}{C}$$ into $$H_i$$ and $$|\mathbf{a}|_2$$:

For the minimum entropy, let $$a_k = 1$$ and $$a_j = 0$$ for $$j \neq k$$:

Thus, $$H_i$$ and $$|\mathbf{a}|_2$$ exhibit an inverse monotonic relationship.

\subsection{Step 2: Generalizing to the Entire Matrix}

The matrix-level entropy $$H(A)$$ and Frobenius norm $$|A|_F$$ are given by:

Since $H(A)$ is the average of row entropies and $$|A|_F$$ is derived from the sum of row Frobenius norms, the inverse monotonicity established for a single row generalizes to the entire matrix.

\subsection{Step 3: Bounds for $$|A|_F$$}

To determine the bounds for $$|A|_F$$:

• Maximum Frobenius Norm: When each row is a one-hot vector (minimum entropy):

• Minimum Frobenius Norm: When each row is uniformly distributed (maximum entropy):

\subsection{Step 4: Implications for Model Evaluation}

The inverse monotonic relationship between $$H(A)$$ and $$|A|_F$$ implies that models with higher $$|A|_F$$ exhibit greater discriminability and certainty in their predictions. This makes $$|A|_F$$ a useful proxy for evaluating the compression and confidence capabilities of large language models (LLMs).

\subsection{Conclusion}

The proof establishes that $$H(A)$$ and $$|A|_F$$ are strictly inversely monotonic. This relationship provides theoretical justification for using $$|A|_F$$ as an evaluation metric in LLMs, where balancing diversity and confidence is essential.

Q9：explicitly defining the objective of measuring the nuclear norm (or rank) of the output matrix

Thank you for pointing this out. The nuclear norm(or rank) is used to evaluate the compression capabilities of LLMs by providing a convex approximation of matrix rank. This reflects the effective dimensionality of embeddings and helps quantify the balance between information richness and compactness. We will clarify this objective in the revised manuscript for better understanding.

Q5：Outperform other metrics?

Thank you for pointing this out. We acknowledge that our wording could have been misleading, and we apologize for the confusion. Metrics are not inherently comparable in terms of superiority; instead, they serve different purposes.

Our intended meaning was that our proposed metric provides evaluation trends consistent with validated metrics, showing that as model size increases, performance improves, and as size decreases, performance declines. These consistent trends demonstrate the validity and utility of our metric for LLM evaluation.We will revise the manuscript to clarify this and ensure accurate representation of our findings. Thank you for your valuable feedback.

Q6：The statements in Figure 4 is inaccurate

Thank you for pointing this out. Upon review, we acknowledge that the claim regarding convergence in Figure 4 was inaccurate. In the revised manuscript, we have removed any mention of convergence and updated the discussion to focus solely on the observed trends. We appreciate your feedback, which helped us ensure the accuracy and clarity of our findings.

Q1：The relevance of our experiments with LLM evaluation is unclear.

Thank you for raising this concern. Our metric is explicitly designed for LLM evaluation, as evidenced by the experiments conducted on widely recognized models and benchmarks. The trends of our metric align with established evaluation metrics, such as matrix entropy, which validates its relevance and robustness. Additionally, our metric extends existing methods by addressing efficiency and diversity in high-dimensional embeddings. We will revise the manuscript to make this connection clearer in the introduction and experimental sections.

Q2：The conclusions do not align with the experimental results.

We sincerely appreciate your feedback and the opportunity to clarify how our conclusions align with the experimental results. Our conclusions are strongly supported by the experimental evidence presented in the manuscript, as detailed below:

1. Efficiency:

   • Conclusion: The Matrix Nuclear-Norm significantly reduces computational time compared to Matrix Entropy, making it practical for evaluating large-scale models.
   • Supporting Evidence: In Section 5.2.1, we show that for a 13B-parameter model, our method required only 0.82 hours, compared to 16.3 hours for Matrix Entropy, demonstrating a nearly 20-fold reduction in computation time.

2. Scalability:

   • Conclusion: The Matrix Nuclear-Norm is scalable across models of varying sizes, maintaining robust performance.
   • Supporting Evidence: Table 1 (l.337) and Figure 1 (l.309) demonstrate consistent results across different model scales, confirming the scalability of our method.

3. Trend Consistency:

   • Conclusion: The Matrix Nuclear-Norm produces trends consistent with established metrics, such as perplexity and Matrix Entropy, validating its relevance for LLM evaluation.
   • Supporting Evidence: Section 5.2.2, as well as Figure 2(b) (l.359) and Figure 5 in Appendix A.1.1 (l.722) show that our metric exhibits the expected trend: performance improves as model size increases and declines as model size decreases, consistent with other validated metrics.

4. Practical Utility:

   • Conclusion: Our method effectively evaluates the compression capabilities and redundancy in LLM embeddings.
   • Supporting Evidence: In Section 5.3.2, we provide results (see Table 2, lines 445–453) showing that the Matrix Nuclear-Norm increases with sequence length, reflecting greater complexity in longer inputs. Additionally, Table 11 (lines 918–934) demonstrates how the metric decreases with larger model sizes across datasets like Dolly-15k and Wikipedia, capturing compression efficiency.

We understand the importance of ensuring that our conclusions are clearly supported by the experimental data. If there are specific areas where the connection appears unclear, we would be happy to address them further or revise the relevant sections in the manuscript. We truly value your feedback and thank you for helping us improve the quality of our work.

Q3：The writing is poor.

Thank you for your feedback. We will thoroughly review the writing and improve clarity, consistency, and readability in the revised version. Ensuring high-quality writing is our priority, and we will address any issues in the current draft to enhance its overall presentation.

Q4：The paper claims that the proposed metric can evaluate the LLMs’ compression capabilities, but with no empirical or theoretical support.

Reference：

Diff-eRank：Wei, X., Zhang, Y., & Liu, F. (2024). Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models. *Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2024).

Thank you for the comment. The proposed metric builds upon the work presented in Diff-eRank, previously referred to as Matrix Entropy, which has been accepted at NeurIPS 2024. In Diff-eRank, Section 3 mathematically establishes the relationship between the effective rank (matrix entropy) and the compression capabilities of LLMs, demonstrating how effective rank captures the structured information in LLM embeddings. Specifically, reducing the rank indicates compression by removing redundant dimensions while retaining meaningful features. Empirical evidence, such as Figure 1 and Table 1 in Diff-eRank, supports this connection, showing that rank-based metrics effectively quantify compression and structured information in LLMs. In our paper, we have cited Diff-eRank (lines 105–106) and extended its findings by proposing a more computationally efficient approximation of matrix entropy. We will ensure these connections and the relevance of our metric to compression capabilities are further clarified in the revised manuscript.