Reply to TVBg

The discussion in Section 4 did not clearly explain the sensitivity of the proposed method when pruning LLM layers. The result shown in Fig 5 indicates that, across different models, our metric consistently identifies the deep layers as “redundant”. Additionally, in the results from Table 1, the proposed metric does not show an advantage at a 20% sparsity level, which further heightens my concerns.

We thank the referee for raising a fair point. We reiterate that the goal of our work is not to provide with a new, competitive, method for layer pruning. We recognise this point might not have been clear, and we tried to clarify this further in the text. Nevertheless, we have expanded the discussion over the pruning methodology by running an experiment showing that performance is correlated with different phases in our metric, see Figure 10 and 11.

The authors mention, "Due to the autoregressive nature of these models, the representation of the last token in a sequence captures information about the entire sequence and is the only token used for predicting the next." I disagree with this assumption; predicting the next token is clearly based on all preceding tokens. Therefore, I would like to see an analysis of certain special tokens, such as the EOS and BOS tokens.

We believe that the reviewer might have misunderstood the sentence, so we tried to rephrase it. The approach of considering only the last token for geometrical investigations of the internal representations has already been used by other works.

Can Persistence Similarity and the zigzag algorithm help explain the varied responses of LLMs across different abilities or areas of knowledge? The pruning experiments presented in the paper are limited to base models. Could you provide an analysis of the instruction-following abilities of chat models or their capabilities in math and code?

We thank the referee for the questions that inspired us into making further experiments, which we explain in the general response and in Appendix D. Besides the new experiments, it might be useful to notice that our approach is based on the study of the manifolds on which an ensemble of prompts lie in, so we do not think that our measure is suitable for analyzing instruction-following abilities of single prompts, at least not as it is defined in our work. A way to address this suggestion would be to apply our method to sequences of tokens within the same prompts. An experiment in this direction would take us too far from the purpose of the present work, thus we deserve it to future work.

Reply to 99nQ

Methods that regard Neural Networks as graphs and apply TDA have been proposed in [1], for example. This is the usual persistent homology, but the use of zigzag persistence for graph structures has been verified in [2],[3] and elsewhere. The structure focusing on the LAYER structure has some aspects of novelty, but it seems to be a simple conventional combination idea and the degree of novelty is weak.

We expand on why we believe our work is novel in the general response.

It is difficult to see what the similarity of the models is intended to achieve. The proposed method is similarity of the graph structure and may be similar for different models. The paper evaluates the similarity in application to pruning to demonstrate its validity. However, it is difficult to accept from the experimental results alone that the proposed method is a superior similarity simply because the implications that the similarity indicates are not clear. As a method for measuring the degree of change, such as model modifications, such as pruning, it is intuitively effective. However, as no uses have been indicated, the current situation is that there is only evidence for it as a pruning method. If the claim is that it is a superior pruning method, it should be clearly stated as such. In addition, if the pruning method is slaughtered and claimed, it should be compared with a generally sufficient advanced puruning method, for example [4],[5].

We thank the referee for this feedback. As quoted in the general response, the aim of this paper is not to provide a competitive pruning method, rather to use the pruning application as a showcase for our zigzag framework. We agree with the referee that this only example might not be enough. We have run further experiments showing the potential of our framework, as described in the general response, and in Appendix D.

Clarify the claim point(s) for which the novelty of the contribution in this paper is sufficient

We refer to the general response for this point.

Please clearly indicate in what sense the similarity of the proposal is valid, what use scenarios are covered (other than pruning) and whether it is sufficient evidence for this.

We provide new experiments in this direction, showing that persistence similarity can be interpreted as indicating distinguishable phases in the model and discriminating over different types of input prompts (e.g. programming languages). See general response and Appendix D for further details.

In response to AC's comment, I will add to what I said earlier.

My intention is that, since the subject of this paper is the modeling of LLM characteristics, even if the method is new in relation to TDA, if a similar result can be achieved using a method that is not TDA, then the novelty of the method will be weakened. When I first read the paper, I thought it was claiming novelty in a broad range that was not limited to TDA, but in the rebuttal, it was phrased as being new as TDA, so I asked about it.

Thank you for your clarifications.

As there is little time left in the discussion when the rebuttal was posted, there may be some misunderstanding due to written communication, and although it would be better to have several rounds of communication, I would like to make a final decision here this time. (I think there is a problem with the way the schedule is set and notified.)

Regarding novelty, it is not clear from the rebuttal whether the novelty lies in modeling the evolution of general point clouds or in constructing a model for neural networks, but I think that the former is doubtful and the novelty of the latter is weak in simple application, so it is necessary to know what the solution to the problem of application was. Also, although it appears that you are arguing that there is novelty within the scope of TDA, that argument is not appropriate because the purpose is to learn about the characteristics of LLM. (In this conference, it is recognized that the improvement of TDA technology itself is out of scope. )

Also, although it is stated in the appendix as being temporary, it is difficult to recommend it when you are not sure whether the final version will be satisfactory.

Based on the above, I will maintain my score for now.

We would like to thank the Area Chair and Reviewer 99nQ for replying to our comments even within such a short time. We also thank the Area Chair for clarifying things and considering a broader context. Here we hope to concisely clarify our viewpoint on what was raised above.

• Our work's scope was not to provide novelty within the field of computational topology, which we agree i) it would be outside the scope of this conference, ii) we did not contribute with new theoretical results.

• Indeed, our replies towards novelty within the field of TDA were a direct response to points raised by more than one reviewer, questioning the novelty of our approach within computational topology (e.g. zigzag formulation already existing). Our replies were meant to show that, within our specific application, there are a few elements of novelty that are worth considering.

• Our work's scope is to provide a bridge between two fields (computational topology and interpretability of neural networks), and on the TDA side it should be viewed as a TDA application, while on the interpretability side it should be viewed as a new framework.

• As a bridging paper, from our point of view it should have the following ingredients: i) an understandable connection between theoretical results (zigzag/TDA in this case) to the application field (interpretability of neural network), ii) a clear formulation of the framework and how is the application realized, iii) one or more practical experiments showcasing the effectiveness of the method.

• As for point i), some reviewers recognized that we managed to present the zigzag formulation clearly.

• Some of the discussion about TDA novelty has been about point ii): the novelties we argue about are a byproduct of the application we had to do using available TDA tools. Within this application, we introduced a few elements of novelty (knn filtration in zigzag, layers as time, persistence similarity) Combining these things, to our opinion, is a strength of our work.

• A fair point that was raised in this respect was that there exist already several applications of TDA to interpret representations of neural networks, as cited in the relevant works section of our work. However, as argued in the general response, to the best of our knowledge, our work is the first to consider neural networks' representations as a whole dynamical system, while previous literature analyzed layers as single snapshots. Moreover, our work is the first to analyze internal representations of transformer models using TDA. We believe these are important points of novelty.

• About point iii), we recognize that the initial submission might have been weaker than we intended. We believe we have strengthened this point with new experiments, as also recognized by reviewer n98n.

We hope that these points clarify the positioning of this work in broader literature as a bridge between the fields of computational topology and interpretability of neural networks.

Thank you for your sincere response to the discussion points. I find it interesting as an analytical method and discussion. The points “the first to consider neural networks' representations as a whole dynamical system” and “the first to analyze The points “the first to consider neural networks' representations as a whole dynamical system” and “the first to analyze internal representations of transformer models using TDA” are novelty as a means and should be combined with the traditional challenges for LLM analysis. It would be a good paper if it can be shown that it is experimentally effective, for example, by clearly showing what kind of problems (examples that cannot be identified) exist in not being able to capture the dynamical system as a whole. I will review the score, taking into account even the clarifications.

Reply to bMgZ

The zigzag filtration, and persistence images are previously well-defined and existing. This raises concerns about the originality and depth of the theoretical and technical contributions presented in the paper.

While we do agree with the referee that zig zag filtrations and persistence images have been used previously in the literature, we believe our paper introduces a few aspects which, combined, do represent an element of novelty with respect to previous research. We have summarized these points in the general response.

The performance gains over existing pruning methods, as shown in Table 1, appear marginal. Including a significance test would enhance the robustness of these findings.

We thank the reviewer for suggesting to enhance the robustness of our findings with a significance test. Considering the computational effort required to run a statistically solid significance test, we did not have time to provide trustable results within the resubmission window. Nevertheless, we performed a rough test with the data we already have: we consider each performance evaluation as a stochastic variable, being each benchmark and model a different observation of the system at fixed percentage cut (10% or 20%). We compute the mean and standard deviation of these observations, and observe that our method is not significantly different from the other two methods we refer to in the paper. We believe that without a proper statistical treatment, this result should not be included as a proof that our methodology performs as well as other methods. We are open to include a more solid test if required (and given sufficient time).

While the paper provides a geometric interpretation of the topological features, it is not clear how these features can be directly interpreted in the context of language models. The paper could benefit from a more in-depth discussion on the interpretability and implications of the observed topological properties.

We thank the referee for suggesting an improvement to our manuscript. We have run a series of more detailed experiments to better understand what our topological descriptors tell about model behavior. We describe these tests in the general response and in resubmitted version. See Appendix D.

The zigzag algorithm is computationally expensive, especially for large datasets and high-dimensional representations. The paper should discuss strategies to optimize the algorithm's performance and make it more scalable

We thank the referee for pointing out we should make clearer what is the computational complexity of ZigZag. Currently, the Fast ZigZag algorithm [3] allows to explore high-dimensional and large datasets. The computational cost is explained Theorem. 22 of [3].
Moreover, to avoid the development of the filtration with Vietoris-Rips that would require the expensive and not trivial task of findind the right radius over multiple point clouds, we use a $k_{\rm NN}$-based filtration, as discussed in the paper. Note that in the appendix B we discuss a different way to build the filtration with $k_{\rm NN}$ and Vietoris-Rips.

Figure 5 is somewhat unclear. Are the 10% (orange) segments included within the 20% (yellow) segments in most cases? The authors might consider a more effective visualization method to enhance clarity.

We thank the referee for the suggestion, we improved the visualization of Figure 5.

Since other similarity measures can also characterize layer-wise behavior, what specific advantages does the proposed persistence similarity offer over these existing methods?

The specific advantage of our method is that it follows the full trajectory of features across layers. This is different from previous similarity measures, which were only considering pairwise comparisons. We believe that this advantage emerges from the new experiments we have run, see Appendix D.

What is the computational complexity of the proposed persistence similarity in comparison to other existing methods?

The persistence similarity is mainly composed by the cost of the Fast ZigZag algorithm that is quasi-linear. For Fast ZigZag computation refer to [3].

Can the authors provide insights or a high-level intuition regarding the consistent patterns observed in Figure 4?

We thank the referee for this question, the answer is provided in the general response and it has been added to the main text and in Appendix D.

Reply to rEUs

The major problem of this paper is the lack of a problem. In fact, the word "problem" was not found in the whole paper at all. Lines 520-521: "Our approach aims to provide a high-level geometrical and topological description of positional and relational changes across layers". If this phrase in the conclusions should be considered a problem statement, then almost any empirical study using geometry and topology can fit this description

We thank the referee for the feedback on the contextualization of our manuscript. We have now clarified more clearly what problem we are trying to address both in the introduction and in the conclusions.

Section 3 "Method" describes over 3 pages the known facts about zigzag persistence without even mentioning LLMs from the title of the paper.

We thank the referee for pointing out that we should be careful in contextualizing where our methodology is applied. We kindly point out that an explicit reference is made in the introductory paragraph of the method section. After defining the connection to LLMs for defining our point clouds, we believe our methodology explanation should be general. We refer to our general response for the novelty aspects of our zigzag algorithm.

The world "layer" appears in the paper 50+ times without a proper definition. Initially, this word is used in the context of neural networks but section 3 seems to assume that a layer is a cloud of unordered points in R^d, which is a standard input for computing persistence in TDA.

We thank the referee for feedback on clarity of presentation. In the same introductory part of section 3, we have expanded on what we mean by layer.

The "persistence similarity" introduced in (5) raises many concerns. First, the concept of "persistent similarity" appeared in the literature, e.g. https://www.intlpress.com/site/pub/pages/journals/items/cis/content/vols/0018/0004/a004/, but no past work on such similarity was cited.

We thank the referee for pointing out previous literature. Indeed, past work have introduced the terminology “persistence similarity” in different contexts, and for different uses. We added a footnote citing the work to clarify possible misunderstandings in the text.

More importantly, the denominator in (5) can vanish, which makes the persistence similarity S_p undefined.

As the referee correctly points out, equation 5 is ill-defined for zero betti numbers at a given layer and given dimension. By definition, persistent similarity should be 0 whenever betti numbers at either l1 or l2 are 0. We added a footnote explaining this case, but did not modify the equation for readability.

In this definition, it is still unclear if l_1,l_2 denote layers or point clouds because min/max of l_1,l_2 seem to be real numbers.If a layer is indeed a point cloud, the first metric axiom surely fails for l_1 and its translated image l_2, which have the same persistence. Even if we consider point clouds under isometry, because persistence is an isometry invariant of a point cloud for standard filtrations of complexes, the first metric axiom also fails for infinitely many non-isometric point clouds, see J Applied Comp Topology 2024. Computational geometry has known much stronger isometry invariants of unordered point clouds for more than 20 years, see Boutin and Kemper, 2004.

l1 and l2 here indicate layer indices, thus they are integer numbers. It would seem consistent with the rest of the notation used in previous definitions, but we would appreciate any feedback for a better notation. We do not understand a referrence to isometry in this context, the first reference proposed by the referee seems incomplete. As for the second reference, we apologize for not understanding the specific relevance to our context.

Lines 402-403: "Note that the plot is not symmetric by definition (cfr. equation 5)". Does it mean that the symmetry axiom fails? The triangle axiom also seems unlikely to hold, so a proof is needed. If a distance fails the triangle axiom with any positive error, then results of clustering are not trustworthy as proved in https://ieeexplore.ieee.org/abstract/document/10574843?

In this context, we refer to an asymmetry over the arguments of the function (l_1,l_2), which we remind are indices to layers. Thus, this would mean that swapping l_1, l_2 provides different values of the functions.

Reply to N98N

Besides the layer pruning application, the qualitative and quantitative insights from these topological features are not particularly interpretable or grounded. We can measure how the number of k-cycles evolves, but there are no interpretable quantities that come out of this, besides similarities between layers. To alleviate this, perhaps further discussion of motivation and related work (why should people use TDA tools to analyze internal representations), or interpretability work (what kind of 1-cycles form, and on what kind of data) could be interesting.

We thank the referee for the suggestion. We believe we improved the interpretability of our work, as explained in the general response, by performing a series of experiments which help in connecting our topological descriptors with model behavior.

As for interpreting what kind of 1-cycles form, and on what kind of data, we would like to point out that the current state of research in zigzag persistence does not allow to unambigously track single topological features. Working towards this direction would certainly be interesting but outside the scope of this work.

Utility in one downstream application (layer pruning) is questionable (the simpler methods from prior work do solidly and are not consistently outperformed).

We agree with the referee that the layer pruning might be a questionable downstream application and that our methodology does not always outperform existing methods. In fact, the novelty of this paper is to show a new approach on the study of the internal workings of LLMs, in particular the application to the analysis of internal representations and the evolution of the manifold they lie in.

Nevertheless, we have performed further experiments to make the utility of our approach more robust, see general response and appendix D.

The use of zigzag persistence in layer pruning only depends on the similarity matrix formed by the method. Other simple baselines could be considered for making similarity matrices.

The referee raises a fair point. Our similarity criterion contains strictly more information than usual similarity matrices because it depends explicitly on the trajectory of features, while other similarity matrices are only computed through pairwise comparison. Our new experiments provide evidence of some of the information contained in our similarity measure (see Figures 8,9,10).

Have you looked at how topological features vary across different data samples / domains? For instance, are there any interesting data features that form between representations of tokens from certain programming languages?

We thank the referee for suggesting an interesting experiment. We take the suggestion by doing the average similarity of three distinct datasets where the language difference is embedded in the semantic of the prompts, in this case a mathematical dataset, a code dataset from github and SST which contains movie reviews (and was already present in the work). We can explicitly see that datasets using technical language (e.g. special characters) show two distinct increasing similarity stages, while general text only show one increase phase. We go into further details in the general response, and Appendix D.

Could you provide a bit more information on where the embeddings are extracted from (this is currently described in Section 4.1 as "each prompt is processed so that the last token is extracted at each normalization layer and the final normalization applied to the output layer.") Do you take the embedding before or after the normalization layer? Also, does this mean you take two embeddings from each block (one from the attention normalization, one from the MLP normalization)? Also, some discussion on how this compares to how other works extract LLM hidden representations would be nice.

We thank the author for pointing this out, we take the embeddings one time after the normalization layer. In particular we followed the procedure from [4].

Additional experiments for interpretability

Given what introduced before, the application of our framework to a layer pruning task was meant to showcase the method, rather than providing a competitive pruning strategy by itself. We recognize we could have been more clear in this framing, and also that overall we should have been more extended in the description/range of the showcased experiment. Towards improving this part, we performed a series of experiments:

In-depth analysis of average similarity

Following the suggestions proposed by the reviewers, we extend our analysis of what our topological descriptor, average similarity, shows. Looking at Figure 4 (right) of our manuscript, we see that average similarity for LLMs is characterized by three phases:

• An increasing phase, lasting until middle layers. The rate of increase is constant in this range, and across models, i.e. it does not depend on the nature of the model, its size and very weakly on the dataset used (cfr. Figure 7 (right) in the Appendix). It does depend on the underlying filtration (cfr. Figure 4 (left)). We have verified that for different values of $k_{\rm NN}$, the universality of the increase across models is conserved. The positive rate of increase means that average persistence of cycles increases, i.e. the transformer architectures are progressively learning features about the dataset that they want to retain.

• A plateau phase, during which the rate of retention saturates. During this phase, we verified that layers with a high similarity are less important for performance.

• A decreasing phase during the last few layers of the models. During this phase, the rate of retention changes again with an opposite trend as in the early layers: features are progressively destroyed.

These observations motivate a few experiements:

Experiments

• To test the increasing phase, we compute the average similarity for a subset of data which share a common topic. The idea is to test whether this phase can change in extension or rate. As suggested by one of the reviewers, we consider datasets of mathematical problems, a general dataset of code (Figure 8) and 5 datasets of code of just one programming language each (Figure 9). We describe in detail this experiment in Appendix D.1. In short, we find that for these datasets, the increasing phase is split in two, with a previously unseen plateau in the middle. We argue that this behavior is triggered by special characters generally not used in conventional human language. We confirm this expectation showing that for programming languages that are less verbose (e.g. markdown) the splitting does not take place.

• A test of the plateau is performed by shuffling tokens within prompts of the datasets, as a way of destroying the structure and semantic coherence of the prompt, without modifying its unigram frequency distribution. In Figure 8 we show how the plateau is modified by this change across two different datasets: the increase phase is shorter and the plateau is much lower in similarity.

• An overall test of these phases is performed by pruning blocks of 5 adjacent layers with a sliding window of 2 and testing the model on a benchmark (MMLU). The scope of this experiment is to show how performance is directly affected by these phases. We show details of this experiments in Appendix and the result in Figure 10 (left panel). We see that performance is random choice during the increasing phase and it pleateaus close to the maximum during the plateau phase. As an interesting finding we see a drop in performance right in correspondance of the decreasing phase, which we investigate further by changing the block and sliding sizes, see Figure 10 (middle and right panels). This finding deserves a closer investigation, which we leave for future work.

Errata Corrige

We fixed the plot of Figure 2 which contained a typo of $k_{\rm NN}$ equal to 10 instead of 15.

References

[1] Ming Quang Le and Dane Taylor,Persistent homology with k-nearest-neighbor filtrations reveals topological convergence of PageRank. Foundations of Data Science.

[2] C. Olah, “Neural networks, manifolds, and topology.”.

[3] Fast Computation of ZigZag Persistence. Tamal K.Dey, Tao Hou. 30th Annual European Symposium on Algorithms (ESA 2022) https://doi.org/10.4230/LIPIcs.ESA.2022.43

[4] The Geometry of hidden representation of large transformer models, L. Valeriani, D.Doimo , F. Cuturello, A. Laio, A. Ansuini, A. Cazzaniga. NeurIPS, 2023