DePass: Unified Feature Attributing by Simple Decomposed Forward Pass

Thanks for the valuable comments from the reviewer, we want to deliver further explanations accordingly.

On the effectiveness of fixing nonlinearities

• Freezing attention scores and MLP activations is central to DePass's design, enabling lossless additive decomposition. This allows us to precisely decompose and track contributions of hidden state components, ensuring their sum exactly reconstructs the original hidden state. This approach eliminates second-order effects during forward propagation. Without this freezing, dynamic internal computations would lead to non-linear, interactive attributions, undermining DePass's core advantages of additive decomposition and exact reconstruction.

• DePass's limitations don't stem from freezing these components, but from not explaining attention scores and not modeling QK-OV circuit interactions. As noted in Appendix F, freezing attention primarily captures information flow via the output-value (OV) circuit, excluding query-key (QK) circuit interactions. Decomposing QK circuits is a complex open problem due to their entangled nature and the quadratic complexity it would introduce. Similarly, dynamic MLP activations would compromise DePass's additive properties and exact reconstruction guarantees.

• We'll clarify this rationale in the main paper, emphasizing it as a necessary trade-off for achieving lossless additive decomposition and faithfulness. We'll also expand on this limitation, discussing how future work might address QK circuit decomposition and dynamic MLP interactions while retaining DePass's core benefits.

Empirical comparison of softmax (Eq. 14) and alternative functions

• To evaluate the design choice of using softmax for MLP attribution, we compare it against two alternatives: a Linear-norm normalization (subtracting the minimum and dividing by the sum) and a Linear-weighted decomposition (linearly decomposing the contribution to the original activation value for each element). This comparison assesses whether the softmax rule is empirically justified.

• Using the known_1000 dataset with the llama-2-7b-chat-hf model, we report the results for both patch-top and recover-top token removal strategies:

• Results show that the softmax-based attribution consistently outperforms the alternatives across both token removal strategies, supporting its empirical effectiveness despite its heuristic origin.

Clarification for selection of baselines

• Our decision not to directly compare DePass against activation patching and general ablation-based attribution methods as performance baselines stems from two key reasons:

• Ablation as a Fidelity Metric (Ground Truth):

  • In mechanistic interpretability, ablation is often regarded as the gold standard for evaluating attribution faithfulness [1][2][3]. We therefore use ablation as our ground truth for validation, not as a competitive baseline.
  • Our paper extensively demonstrates that DePass faithfully identifies critical components whose disruption or preservation, based on our scores, indeed impacts model behavior. Comparing DePass directly with ablation as if they perform identical tasks would be conceptually misaligned; DePass provides attributions, and ablation validates them.

• Computational Efficiency and Applicability, with Consideration for Activation Patching:

  • DePass is a direct and unified feature attribution framework based on a single decomposed forward pass, offering significant computational advantages over exhaustive, "computationally expensive" ablation methods which provide limited insight into intermediate information flow.
  • Regarding activation patching methods, they typically require constructing specific contrastive datasets for activation replacement, which introduces additional complexity and computational overhead. Since ablation can be viewed as a specialized form of activation patching—specifically, patching with zero values—we did not assess it separately as a baseline.

Illustration for multi-token word's attribution

• We provide two examples of llama-2-7b-chat-hf below to illustrate how subword tokenization affects DePass scores. For word-level scores, we treat corresponding subword tokens as a single component from the start of the DePass process, not a simple summation of individual scores.

Example 1:

Example 2:

• Both token-level and word-level attributions with DePass are effective. While token-level scores might disperse across subword tokens, they remain highly informative for distinguishing important tokens. Higher scores on specific subwords can even indicate triggers for the model's memory of the full word. DePass naturally supports aggregating subword tokens for desired word-level attribution.

• To ensure a fair comparison with other token-level baselines, in the experiments corresponding to Figure 1, we perform ranking and ablation at the subword token level. We will clarify this in the revised text and further elaborate on the impact of subword tokenization on DePass.

[1] Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", Transformer Circuits, 2025.

[2] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models

[3] Finding Neurons in a Haystack: Case Studies with Sparse Probing

Thanks for the detailed comments from the reviewer, we would like to make several clarifications below.

DePass's Efficiency & Fidelity Trade-off

• We acknowledge that DePass introduces additional computational cost compared to a single standard forward pass. However, this reflects a deliberate trade-off between efficiency and attribution accuracy. Prior methods often prioritize low computational cost but sacrifice attribution fidelity, especially in deep, nonlinear models like LLMs. In contrast, DePass maintains the integrity of the original forward process, producing higher-fidelity, more reliable attributions within an acceptable runtime budget.

• The table below demonstrates DePass's runtime advantage for neuron-level attributions over an intermediate MLP layer in different LLaMA models compared to ablation method. The current implementations already show strong performance, with considerable potential for further acceleration.

Further discussion about SAE-based methods

We thank the reviewer for pointing out the lack of clarity in our discussion of SAE. We will revise the paper to improve clarity and include additional citations [2][4][6][7].

On which token activates particular feature

• We appreciate the reviewer for highlighting prior work that leverages computational graphs and transcoders to trace which token activates a particular feature. We acknowledge the validity of these approaches and will revise the wording in our paper to reflect this more accurately.

• While these methods analyze token-feature activation, they often involve complex and hard-to-scale graph computation and pruning. To our knowledge, their token-level attribution fidelity hasn't been systematically evaluated across datasets.

• In contrast, DePass offers a simple, scalable, and direct attribution framework. Instead of proxy measures like token-feature correlations, DePass preserves and propagates attribution information throughout the actual forward process, enabling accurate token-level attribution without extra training or complex graph tracing. This practicality and fidelity make DePass more suitable for widespread use in large-scale model analysis.

On Feature Evolution Tracking

• We appreciate you raising this important point. We'll leverage the terminology from [1] for further discussion. We acknowledge that existing work on transcoders and crosscoders can track abstract features across layers within their trained 'Isomorphic Models'. However, when DePass refers to "feature evolution," it denotes precise tracking within the 'Literal Model.' We’ll refine our phrasing and provide a more in-depth discussion in the revised version, along with additional relevant references.
• DePass's "feature evolution" or "tracking" precisely traces the information flow of a specific vector or subspace throughout the literal model's computation graph. It can accurately isolate the subspace that evolved from a target feature at any point in the residual stream.
• Transcoder-based feature tracking, in contrast, relies on importance scores derived from correlations between sparsely trained encoders and decoders at different layers [2][3][4][5]. While these methods can identify "circuits" through careful filtering and pruning, reconstructing a feature's influence at a given point based on activation relationships is inherently lossy. Moreover, this tracking is in the context of Isomorphic Models, not the literal model.
• Crosscoder-based feature tracking integrates sparsely trained encoders and decoders from different layers to identify features that remain relatively stable across layers in the residual stream. While this simplifies tracking shared features across layers within Isomorphic Models, it remains lossy in the context of the literal model.
• Furthermore, work [6] indicates that semantic similarity between features isn't accurately reflected by directional similarity. Methods that derive causal relationships between cross-layer features from activation correlations across large corpora cannot perform feature evolution tracking in either sense.
• Nevertheless, we believe DePass can serve as an effective bridge connecting analyses in Isomorphic Models with the literal model, which we'll discuss in the following section.

Capacity of combining DePass with SAE

• DePass can be applied at the subspace level by treating the SAE features of interest as DePass components, which enables direct and convenient analysis of the interactions among SAE features across different layers.
• Alternatively, DePass can be used at the token level, similar to Eq. 19, to investigate which tokens’ semantics jointly activate a particular SAE feature.
• We are interested in conducting such experiments. However, due to the substantial annotation effort required to reliably label SAE features in datasets, we currently lack access to sufficiently annotated feature representations. Moreover, mapping SAE features to human-interpretable semantic spaces remains challenging, making it difficult to obtain ground-truth data for rigorous experimental validation at this time.

Capacity of combining DePass with model steering

• DePass can be naturally applied to model steering analysis and offers a novel perspective for evaluating the effectiveness of steering vectors. By treating the steering vector as a DePass component during the forward pass, we can accurately quantify its influence on the model’s final output. Furthermore, when combined with tools like SAE or probing methods that analyze hidden states, DePass enables a comprehensive examination of how steering vectors affect the model’s internal dynamics across layers.
• We believe there is substantial potential for further integrating DePass with model steering techniques to deepen interpretability and control in large language models.

On Method Limitations

• We appreciate the reviewer's feedback regarding method limitations. We do discuss the limitations of our method in the appendix F section of the paper. We will clarify this in the main text in the revision to ensure readers can easily find and understand the method’s limitations.

[1] Sparse Crosscoders for Cross-Layer Features and Model Diffing

[2] Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", Transformer Circuits, 2025.

[3] Transcoders Find Interpretable LLM Feature Circuits

[4] Mechanistic Permutability: Match Features Across Layers

[5] Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders

[6] Analyze Feature Flow to Enhance Interpretation and Steering in Language Models

[7] Evolution of SAE Features Across Layers in LLMs

Thanks for the elaborate comments, we will address them according to their proposed order.

On the effectiveness of fixing nonlinearities

• DePass is not concerned with the model's representational ability. Instead, its goal is to perform attribution analysis on a given forward pass. Freezing the nonlinearities allows us to precisely reconstruct the original forward pass. DePass ensures the target model's representational power is unaffected by using attention scores and MLP activations directly from its forward pass. This aligns with DePass's core principle: efficient attribution with full fidelity to the original computation. Our method specifically attributes direct paths of information flow to the final logits.

• While some tasks, like Indirect Object Identification (IOI), involve indirect influences via QK circuits, capturing these complex interactions (a challenging open problem in mechanistic interpretability [1] ) is beyond our current scope. We believe attributing with QK circuits is a crucial future direction.

Clarifying the definition of "neuron"

• Following prior work [2], each MLP neuron is defined as a subvalue $v_k^\ell$, as introduced in Section 2 (Feedforward Network). For token $i$, the coefficient of neuron $k$ in layer $\ell$ is computed by corresponding subkey:

  $m_{i,k}^\ell = \sigma(f_k^\ell \cdot {X}_{{attn},i}^\ell)$ and output of the neuron is:

  $Neuron_{i,k}^\ell = m_{i,k}^\ell \cdot v_k^\ell$ The full MLP output is the sum over neurons' outputs:

  $MLP_i^\ell = \sum_k Neuron_{i,k}^\ell$ We will explicitly include a clarification in the future revisions.

Additional examples and detail of our results

• We agree that more examples are crucial for demonstrating DePass's capabilities and will expand relative section in the revision.
• We provide a concise example illustrating the attribution distribution across inputs, including DePass (our method based on decomposed forward pass attribution), Normalized DePass (DePass scores normalized to sum to 1), All (Mean Attention), Last (Last-layer Attention), Rollout (Attention Rollout), Integrated Gradients (Integrated Gradients), Signed (Input × Gradient), and Norm (Gradient SHAP).

Prompt: "The capital of France is" Target: "Paris"

Prompt: "The capital of France is" Target: "known"

Prompt: "grape: purple, banana: yellow, apple:" Target: "red"

• DePass reports both raw and normalized attribution scores, with raw values summing exactly to the output logit. This allows direct interpretation of each score’s impact on the prediction. For example, when comparing targets like “Paris” and “known” for the same input, their total attribution scores differ.
• DePass attributes the output to semantically meaningful parts of the input, while baseline and gradient-based methods often fail to emphasize relevant tokens or align with clear semantic patterns.

Clarification for certain notation and labeling

• We appreciate your feedback regarding notation precision, especially concerning the "O" variables and the clarity of "percents vs. fractions" in our figures.

Clarification of "O" Notation

• Regarding the "O" symbols, their usage is consistent but we understand the need for clearer definitions. We'll refine these in the revision:
  • $O^{l,j}$ (Eq. 2): output of the $j$-th attention head in layer $l$, representing overall attention without token-wise decomposition.
  • $o_{i,m}^{(l,j)}$ (Eqs. 12,13): output of the $j$-th attention head at layer $l$ for the $m$-th decomposed component of token $i$ in DePass, showing propagation through attention.
  • $o_{i}^{(\ell,h)}$ (Eq. 20): output of the $h$-th attention head at layer $l$ for token $i$ in attention decomposition, used to initialize decomposed states for attribution.

Units in Figures

• We acknowledge that the axes in Figures 1 and 3 require more explicit labeling. We will update the labels and captions to clearly indicate units:

  • Figure 1: x-axis is masking proportion (0 to 1); y-axis is average relative probability change in percentage (0 to 100).
  • Figure 3: both x-axis (mask percentage) and y-axis (accuracy) range from 0 to 100.

"Position" in lines 63-66

• We'll be more precise with new terminology to avoid confusion. "Position" specifically refers to each token's location within the sequence.

Clarification for DePass's value-add

Efficiency and Parallel Computation

• DePass's parallel computation greatly enhances efficiency. It enables the simultaneous attribution of many hypotheses of the same type, such as all MLP neurons in a layer, in just one forward decomposition pass. This contrasts sharply with ablation, which demands individual masking and re-evaluation for each neuron, leading to considerably higher computational overhead.
• The table below demonstrates DePass's runtime advantage for neuron-level attributions over an intermediate MLP layer in different LLaMA models. The current implementations already show strong performance, with considerable potential for further acceleration.

• Moreover, for studying the role of subspaces, ablation is inherently limited. Naively ablating a subspace during execution can be misleading, as subsequent layers may compensate for the missing information [3]. In contrast, DePass allows us to directly trace and quantify the contribution of a specific subspace without perturbing the network's forward dynamics.

Further discussion about SDL-based methods

• Given space constraints, we direct you to our response to reviewer uE26 for further details.

References for MLP layers' purpose

• Key references on MLPs enhancing per-token representation capacity include [4-6]

Empirical comparison of softmax (Eq. 14) and alternative functions

• Given space constraints, we direct you to our response to reviewer w5Wf for further details.

Settings of results in Fig.1

• For each data point in the Known_1000 and IOI datasets using the Llama-2-13b-chat-hf model, we:
  • Obtain attribution scores for the correct answer via various methods.
  • Remove tokens from the prompt based on these scores, following either a "patch top" (highest attribution) or "recover top" (lowest attribution) strategy.
  • Reassemble the remaining tokens into a new prompt and rerun the model.
  • Measure the relative probability change of the correct answer.
• The figures show the average relative probability change across all data points per dataset at each masking level.

Clarification for introducing DePass-abs

• In lines 201-211, we introduced DePass-Abs as the absolute values of DePass scores, capturing both supportive and suppressive contributions.
• We chose not to use DePass-Abs as the default to preserve the direction of contributions. Raw DePass scores reflect signed contributions, distinguishing positive support from negative suppression—critical for interpreting token-level and subspace-level effects, such as identifying inputs driving truthful versus untruthful outputs. DePass-Abs, which ignores direction, is used at the component level where negative contributions can still be functionally important.

Clarification for layer selection in Section 4.3

• We chose Layer 15 for its strong probing performance and effective separation of language-specific signals, as demonstrated empirically. While other intermediate layers (13-25) exhibit similar effects, Layer 15 serves as a representative example. We'll include results from additional layers in the revised version.

[1] Circuit Tracing: Revealing Computational Graphs in Language Models, Transformer Circuits.

[2] Neuron-Level Knowledge Attribution in Large Language Models

[3] Is this the subspace you are looking for? An interpretability illusion for subspace activation patching

[4] Dissecting Recall of Factual Associations in Auto-Regressive Language Models

[5] Transformer Feed-Forward Layers Are Key-Value Memories

[6] Locating and Editing Factual Associations in GPT

The main limitation is that in settings where the goal is to identify indirect contributors (e.g., multi-step reasoning or IOI-like tasks), the direct sources surfaced by DePass may differ from the components of interest.

This does seem like the concern in such settings. By construction, DePass will always produce internally consistent and complete decompositions. The question is whether DePass's decompositions will be useful on closer-to-practical tasks, and that hasn't been addressed.

While DePass does not directly attach semantic meaning to these changes, it integrates naturally with interpretability tools such as probing classifiers or SAEs, which align hidden states with human-interpretable concepts. Moreover, DePass can be combined with SAE to provide semantically interpretable feature-evolution tracking, often in a way that is more direct and convenient than graph-based approaches.

Ah, this seems quite useful to explore! If indeed DePass's subspace decomposition pulls out linear probe-identified directions or isolates SAE features, that would serve as useful validation that these independent methods are pointing at some meaningful underlying reality in models' representation-spaces. I would be very excited to see this work done, but it seems unrealistic to ask for this in the review process.

---

I intend to raise my score from a 3 to a 4. I think the work is meaningful and worth accepting, particularly with the improved language and example coverage for clarity. There are still important axes of evaluation which, if done, could stand to validate the work as practically applicable, and in their absence, I think a 4 is appropriate.