Holes in Latent Space: Topological Signatures Under Adversarial Influence

We sincerely thank the reviewer for their thoughtful and supportive feedback on our work.

We recognize that the computational challenges of persistent homology remain an inherent limitation, as noted by the reviewer. In response to a similar comment from Reviewer mgPH, we have conducted additional experiments using an alpha complex, and we provide the corresponding results and discussion there. While Ripser++ offers an accelerated approach for computing Vietoris–Rips persistent homology, the underlying computational complexity remains a fundamental characteristic of the method. As the reviewer notes, we address this challenge through random subsampling. In doing so, we rely on the theoretical guarantees established by Chazal et al. (2014, 2015) and Cao & Monod (2022), which give the convergence of subsampled persistence diagrams to the full diagrams. Nevertheless, we acknowledge that subsampling introduces variability and the potential risk of missing certain structures.

Regarding the suggestion to explore additional adversarial scenarios, we appreciate this valuable perspective. As discussed above with Reviewer r5do, we note that the two adversarial influences we have examined are fundamentally different in nature, which seems to indicate that adversarial triggers produce consistent deformations in the activation space and highlights the robustness of persistent homology in distinguishing between distinct and vastly different types of adversarial stress.

Lastly, while our work does not include theoretical results, we would like to emphasize our novel technical contributions. These include not only the application of topological data analysis to the study of LLM activations but also the development of an element-wise local analysis of information flow within LLMs. This method provides a new tool for examining correlations between activations across layers.

Once again, we greatly appreciate the reviewer’s constructive feedback and their positive assessment of our work.

We thank the reviewer for the positive evaluation of our work and constructive feedback, which we now address.

Clarification of the local analysis

Element-wise analysis: Indeed, our method differs from conventional approaches that analyze full activation vectors using cosine similarity or classifiers. Treating each dimension in the activation vector uniformly effectively averages over all neurons and potentially obscures localized or sparse but meaningful signals, which tends to happen when computing cosine distance, or using probes, which are typically linear classifiers. To avoid that, we analyze local, element-wise activation changes across layers by mapping pairs of activations $(a_i, b_i)$—for each neuron $i$—to coordinates in 2D that we use as input to compute PH.

Nonlinear approaches: We acknowledge the question of utility of nonlinear approaches (such as PH) given the high co-linearity between activations in consecutive layers but believe that linear dependencies do not take away from our analysis:

1. Neural networks are not strictly linear: While consecutive layers may exhibit high similarity, they are separated by nonlinearities. It is thus not entirely accurate to assume one layer is just a linear combination of the previous one.
2. Our method detects deviations from co-linearity: By projecting local, element-wise activations into 2D and analyzing them as point clouds, we can identify patterns such as clusters or cycles. The most structurally significant points in our point cloud intuitively correspond to neurons whose activations change the most between layers (i.e., diverge the most from the linear pattern). Thus, our method extends beyond linear analyses to capture nonlinearities.

Furthermore, while there is no guarantee that such nonlinear deviations are meaningful (they could be noise), our local analysis demonstrates otherwise. We show that the structure derived from these point clouds across consecutive layers can reliably distinguish between normal and adversarial activation patterns. This empirical result suggests that our method captures inherent structural differences that may be missed by conventional approaches.

Further discussion on local analysis results

Our local analysis is not merely a reporting of empirical results but is grounded in the goal of understanding how activation patterns evolve across layers at a more granular level than conventional approaches. We emphasize that while our methodology does not establish a bijection between PH features and specific neurons or groups of neurons, our PH summaries offer valuable insights into the connectivity structure of neurons. For example, in Figure 10, our analysis of 1-dimensional total persistence reveals distinct shifts in network complexity captured between normal and adversarial activations through the prevalence and size of connected cycles. This observation is not trivial; rather, it underscores how adversarial perturbations disrupt the structured information flow that is otherwise regulated by the model.

Furthermore, our analysis provides a novel direction for identifying optimal layers within the network where maximal separability between normal and adversarial activations is achieved. This aspect of our work is particularly valuable as it contributes to a broader understanding of the information flow within LLMs, an area that has so far received limited exploration. We will clarify this in our revision.

Readability

We acknowledge the reviewer's feedback regarding the readability and organization of the paper. In our revision, we will take the following steps to enhance clarity:

• Provide additional interpretation and discussion of key results, particularly focusing on the implications within our local analysis.
• Reduce back-and-forth between the main text and the appendix by integrating essential clarifications directly into the main text.

We believe that our local information flow analysis is a key contribution of this work and should remain in the main text.

Computing alpha-filtrations

To investigate the performance of alpha-filtrations compared to Vietoris–Rips (VR), we compared runtimes for a pair of consecutive activation layers. Computing the VR PH took 5.9s, while PH from the alpha-filtration was indeed faster, taking 0.05s. We note that computing a sparse version of the VR complex lowers the runtime to 0.07s, comparable to the alpha-filtration.

Although there is a significant difference in runtime in favor of the alpha complex for the local analysis, this difference is not beneficial for the more intensive global analyses, due to the construction of the alpha complex in $\mathbb{R}^{4096}$ whilst VR benefits being constructed solely on the pairwise distances which can also be adapted to different metrics and offers a more intuitive geometric interpretation (in terms of complexes being constructed by diameter).

We appreciate the careful review of our work and would like to address several of the concerns raised.

Motivation of the methodology (PH)

We acknowledge that our motivation for the use of PH in this context could have been more accessible. We address now why such representation of the data is important or how to interpret them and are happy to elaborate in our revision.

• PH serves as a powerful multi-scale "topological lens," providing insights into the shape and structural complexity of data beyond conventional linear analyses. Instead of treating LLM activations as distinct points in a high dimensional space, we take a multi-scale topological view by incrementally “thickening” them (see Figure 2). We allow a radius parameter to grow, akin to adjusting thresholds in hierarchical clustering or DBSCAN, to reveal how the activations connect to form structures across different scales. Initially, PH identifies clusters or connected components (0-bars), and tracks their formation (“birth”) and merging (“death”) as connectivity increases. As the radius grows, PH reveals higher-order structures such as loops (1-bars), which represent more complex, global interactions extending beyond standard clustering. The key output of PH is the persistence barcode, i.e. the collection of bars representing the “lifetime” of these structures from their "birth" to their "death".
• In our study, PH interpretability helps analyze how adversarial conditions reshape the activation space: A lower mean death time for 0-bars suggests adversarial activations cluster more tightly. Fewer late-stage 1-bars indicate reduced structural complexity.

Clarification of claims

Our primary claim is indeed that PH is a useful interpretability tool for LLMs in adversarial conditions. We are not proposing PH as “an effective tool for detecting prompt injection attacks or backdoor triggers,” which would require additional work beyond our scope. The PCA and logistic regression in Section 4 serve as starting points to examine why separation occurs, not as detection methods.

Our claim is supported by two main facts:

1. PH barcodes summarize input shape, making them inherently interpretable. Section 4.2 is entirely devoted to this interpretation, analyzing shape differences between normal and adversarial activations.
2. PH reveals a consistent topological deformation pattern across two fundamentally different adversarial influences happening at different LLM processing stages. This suggests a general geometric effect in the representation space, which topological approaches can analyze in ways that existing methods cannot. Thus, PH offers a complementary perspective to behavior monitoring or attack-specific detection methods.

Experimental design

We would like to clarify the two concerns raised by the reviewer, which we will include in the revision:

• Activation difference: We follow the TaskTracker dataset (Abdelnabi et al., 2024), which is specifically designed for activation-level analysis. We refer to the original paper for construction choices, re-justifying them is not within our scope. To address a key point: in this dataset, the user instruction and retrieved data block are always available separately, as retrieval happens after the instruction. This allows us to isolate the representational shift caused by adversarial content, which studying the data block alone would not capture.
• Pruning highly correlated features is a standard practice in statistics and ML to reduce redundancy and prevent overfitting. Given the strong correlations in persistence barcode statistics, removing them improves efficiency while retaining predictive power. A model that explains the same phenomenon with less variables is a more parsimonious one, thus more desirable.

Technical novelty

We respectfully disagree with the reviewer and wish to emphasize the two technical novelties of our work.

• Applying PH to this type of data is neither straightforward nor well-explored, as noted by other reviewers. PH provides unique and mathematically grounded interpretability insights that transcend existing mainstream methods: linear probes assess linear decodability but miss latent space structure; mechanistic interpretability tracks causal pathways but lacks global analysis; and representation engineering (e.g., sparse autoencoders) captures local features but not topological invariants. Designing tests to directly compare these with PH is neither feasible nor meaningful, as they capture complementary and orthogonal aspects of the problem.
• Our local element-wise analysis of information flow within the LLM is an entirely new approach to understand nonlinear correlations between activations across layers. This represents a significant contribution, demonstrating that while PH is an established tool, its application to new domains can yield original methodologies and insights that help us better understand them.