Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis

Thanks for your review and helpful suggestions! These are good points, which we answer below.

Q1: I'm curious if you combine the discovered circuit for the single task, whether the combined circuit can perform the compositional task like you measure the faithfulness.

Based on your suggestion, we add the faithfulness obtained by the Union Circuit on the task. We observe that the Union Circuit achieves a faithfulness of 89.18%, which supports our claim that Union Circuit can perform compositional task like Compositional Circuit.

Q2: There are some other circuits that are related to the work: Some work about the reuse or combinations are not discussed...The work analyzes mainly based on the math tasks, but there are some other knowledge circuits proposed, and whether the compositional is also applicable should be mentioned.

We appreciate the opportunity to more clearly situate our contributions within these key works:

[1] Circuit Component Reuse Across Tasks in Transformer Language Models (ICLR 2024) While this paper discusses the reuse of components, we focus on the composability and reuse of circuits. Our work complements this direction by studying not just reuse but also structural recomposition, showing how sub-circuits from simple arithmetic tasks can be merged to approximate more complex tasks.

[2] Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024) This paper mentions that this type of reasoning combined in implicit reasoning has something in common with our thinking. Our study builds on similar math reasoning tasks, but focuses on how these circuits evolve during fine-tuning and how their compositionality can be leveraged to improve fine-tuning strategies like CircuitLoRA.

[3] Knowledge Circuits in Pretrained Transformers (NeurIPS 2024) We only have some overlap in circuit discovery. Moreover, Our research focuses more on the changes in the internal circuits during fine-tuning as the model accuracy increases, as well as the improvement of the fine-tuning mechanism from the perspective of Mechanistic Interpretability.

We have explicitly cited these works in the Related Work and Discussion sections and expanded our discussion on how our findings support and extend these prior directions, especially regarding modular fine-tuning, compositionality, and general-purpose circuit reuse.

Q3: Experiments are limited to synthetic mathematical tasks. While these provide controlled settings, it is unclear if findings generalize to natural language tasks (e.g., text generation, reasoning).

1, Motivation. We consider our five tasks because many recent works in Mechanistic Interpretability are based on tasks like IOI or greater-than where pre-trained models already achieve high accuracy on these tasks (e.g., 98% for GPT-2 on IOI), which is not practical for understanding fine-tuning. So we choose to explore scenarios where models start with low performance and improve significantly after fine-tuning — allowing us to observe meaningful structural changes in circuits.

2, To follow your suggestion, we newly extended our experiments to both new mathmatical tasks and two natural language tasks. Compared to tasks of previous studies, our designed tasks are more challenging, involving more complex reasoning patterns.

• Comparison Task: Is 121 > 112? Answer:
• Complex IOI Task: Robert needed brush, Kelly wanted pan, Daniel handed brush to
• Complex Capital-Country Task: If Abkhazia corresponds to Sukhumi, then Moldova corresponds to...

The following are the specific experimental results, please see Figure1 at this anonymous link for additional figures.

• Pre-trained model accuracies on these tasks were initially low: 46.74%, 27.60%, and 32.58% respectively.
  • Comparison: $\Delta S_{edge}$ = 23.6%, $\Delta S_{node}$ = 9.6%
  • Complex IOI: $\Delta S_{edge}$ = 17.3%, $\Delta S_{node}$ = 6.0%
  • Capital-Country: $\Delta S_{edge}$ = 16.8%, $\Delta S_{node}$ = 7.3%

These results replicate the core conclusion of our main paper that edge dynamics dominate structural change during fine-tuning and confirm that our findings generalize beyond the original tasks.

Q4: I have some questions about the CircuitLoRA. Since you need to substitute the critical layers for the CircutiLoRA, do you need to tune the model using the two ranks twice with different ranks?

To clarify, CircuitLoRA requires only a single round of fine-tuning. The training is performed once, with a unified LoRA configuration where Critical layers (identified via circuit analysis) are assigned a higher rank ($r_c$), Non-critical layers are assigned a lower rank ($r_o$).

This design is one of the key strengths of CircuitLoRA — it leverages mechanistic insights to redistribute parameter budget effectively, without introducing additional training complexity or computational overhead.

Thanks for your review and helpful suggestions!

Q1: This indicates that circuit dynamics play a crucial role…Actaully, I cannot agree with it. The total num of edges is much greater then total number of nodes. The ratio of changement can deliever more information.

To account for this, we use a normalized change metric $\Delta S$, which measures the change rate of nodes and edges relative to their initial quantities. This metric is introduced in Section 4 and visualized in Figure 3 (bottom right).

Using this metric, we observe across all tasks that edge change rates consistently exceed node change rates, by a factor of 2–3x. To further distinguish whether it is the natural expansion of the structure or the influence of the fine-tuning mechanism, we supplement experiments on our tasks: the difference between the estimated edge changes caused by node changes and the actual observed edge changes. In all tasks, the actual edge changes substantially exceeded the upper bound implied by node changes — often by a factor of 2.6x to 3.1x. This further confirms our previous experimental conclusions.

Q2: However, in Section 4 the authors use LoRA with the Pythia-1.4B model for fine-tuning, which I believe is not good. … Consider using full fine-tuning (FT) for small models and PEFT for large models.

We have conducted both LoRA and full-parameter fine-tuning (FT) experiments with the Pythia-1.4B model for comparison. The results are provided in Appendix G. We mention this in the last paragraph of section 4.3.

The reason we chose to present LoRA-based results in the main text (Section 4) is due to the logical structure of the paper: in Section 5, we introduce CircuitLoRA. Since our CircuitLoRA is a response to LoRA-based insights from Section 4, presenting LoRA results earlier improves narrative consistency — i.e., we derive insights under LoRA, then use them to improve LoRA.

Q3: The study focuses on a specific set of mathematical tasks,...Broaden the range of mathematical tasks to assess the generalizability of the findings.

1, Motivation. We consider these tasks because many recent works in Mechanistic Interpretability are based on tasks like IOI or greater-than where pre-trained models already achieve high accuracy on these tasks (e.g., 98% for GPT-2 on IOI), which is not practical for understanding fine-tuning. So we choose to explore scenarios where models start with low performance and improve significantly after fine-tuning — allowing us to observe meaningful structural changes in circuits.

2, To follow your suggestion, we extended our experiments to both new mathmatical tasks and two natural language tasks. Compared to their tasks, our designed tasks are more challenging, involving more complex reasoning patterns.

• Comparison Task: Is 121 > 112? Answer:
• Complex IOI Task: Robert needed brush, Kelly wanted pan, Daniel handed brush to
• Complex Capital-Country Task: If Abkhazia corresponds to Sukhumi, then Moldova corresponds to...

The following are the specific experimental results, please see Figure1 at this anonymous link for additional figures.

• Pre-trained model accuracies on these tasks were initially low: 46.74%, 27.60%, and 32.58% respectively.
  • Comparison: $\Delta S_{edge}$ = 23.6%, $\Delta S_{node}$ = 9.6%
  • Complex IOI: $\Delta S_{edge}$ = 17.3%, $\Delta S_{node}$ = 6.0%
  • Capital-Country: $\Delta S_{edge}$ = 16.8%, $\Delta S_{node}$ = 7.3%

These results replicate the core conclusion of our main paper that edge dynamics dominate structural change during fine-tuning and confirm that our findings generalize beyond the original tasks.

Q4: If you only want to focus on mathematical problem solving for reasoning, I suggest extending the circuit analysis from LLM + SFT to LLM + RL to enhance the contribution.

1, In this work, we focuse on SFT primarily due to SFT is more suitable for our task and model size. For models below 30B, the effect of reinforcement learning is very poor and it needs to be applied to more difficult tasks to be meaningful.

2, In the previous question, we expanded our tasks beyond mathematical problem to include some natural language tasks. Futher, we also explore the circuits before and after reinforcement learning follow your suggestions. In the Add/Sub task, we used PPO for 10 epochs of training and compared the differences between the internal circuits of the model after reinforcement learning and before.

3, The experimental results show $\Delta S_{edge}$ = 30.3% and $\Delta S_{node}$ = 14.7%. Besides, the added nodes are predominantly located in the middle and later layers of the circuit, and the nodes in the shallow layers rarely change. Please see Figure2 at the same anonymous link above for additional figure. This is basically consistent with the conclusion in our original paper, and this result enhances the generalization of our research conclusions.

Thanks for your helpful review!

Q1:The authors of this paper will provide a persuasive explanation…, then I would be happy to raise my score.

We apologize for the oversight and have added the missing reference. Both works use EAP-IG as a shared tool, not a core contribution. We clarify key differences from [1] in motivation and contributions.

Motivational Distinction:

• [1] focuses on analyzing how circuits and their components emerge and stabilize during pretraining.
• We focus on why fine-tuning improves performance, including per-layer node/edge dynamics—a finer view not explored in [1]. Unlike prior MI work, we focus on low-performing tasks to better reflect practical fine-tuning scenarios.

Application Contributions Beyond [1]:

• As noted in Open Problems in Mechanistic Interpretability (2024), MI research splits into understanding mechanisms and using MI for better predictions. Most prior work, including [1], contributes to the first category. Beyond understanding, we leverage MI insights to improve fine-tuning. CircuitLoRA shows the practical value of structural insights.

• We also propose and empirically evaluate Compositional and Union Circuit. We show that merging two subtask circuits can approximate compositional circuit. These are hard to observe in [1], which focuses on pre-training.

Q2: But to me, the greater number of edge changes does not follow that these edge changes are more significant than node changes…because there are far more possible edges in a circuit than nodes…

To account for this, we use a normalized change metric $\Delta S$, measuring node/edge change rates relative to their initial values. The result shows edge change rates consistently exceed node change rates by 2–3x.

To distinguish natural structural growth from fine-tuning effects, we added experiments: the difference between the estimated edge changes caused by node changes and the actual observed edge changes. In all tasks, the actual edge changes substantially exceeded the upper bound implied by node changes — often by a factor of 2.6x to 3.1x. This supports our conclusion.

Q3: In Table 1, how do CircuitLoRA and RandomLoRA with $r_o$=32,$r_c$=64 have a lower parameter … Does the bolding represent "best performance for a given parameter ratio"?

First, there is a labeling error in Table 1. CircuitLoRA ($r_o$=32, $r_c$=64) should be CircuitLoRA ($r_o$=16, $r_c$=64).

Second, Bold highlights the best method in similar parameter ranges. In the Table 1 caption, we have already specified the intended comparisons: CircuitLoRA ($r_o$=8, $r_c$=32) vs LoRA ($r_o$=16).

Q4: For Section 6, what is the faithfulness score of the union circuit on the compositional task? How does this change for different values of k?

1, We evaluated the faithfulness of the Union Circuit and found it to be 89.18%, compared to 96.86% for the Compositional Circuit.

2, We further report that the faithfulness of the union circuit across different percentage values of total edges. As Overlap is structural, we focus on the top 100–1000 scoring edges. Faithfulness evaluation requires more edges. See table1 at this anonymous link. The results show that a small subset of top-ranked edges is sufficient to achieve high faithfulness.

Q5: If a perturbed prompt already exists in the dataset, then is it resampled?...Would it not make more sense to simply partition the dataset into n disjoint splits...?

1, We applied a duplicate avoidance mechanism during generation (up to 100 attempts per task).

2, We also ran an experiment as suggested. The results show that circuits in fine-tuned model score 0.73 vs. 0.84 (pre-trained model) and 0.55 (random model). These results further confirm our original conclusion.

Q6: One possible "nice-to-have" would be to compare CircuitLoRA with another non-uniform-rank PEFT method.

We conducted experiments comparing CircuitLoRA with AdaLoRA. Below are results for the two methods with similar parameters.

Please see table2 at the above anonymous link for full results. This provides empirical support that MI insights can guide parameter-efficient fine-tuning effectively.

Q7: For CircuitLoRA, were the critical layers found for different tasks the same? How much overlap was there?

In our current experiments, we analyzed the top-5 critical layers identified for each task. Please see table3 at the above anonymous link. Critical layers vary by task, with some overlap.

Q8: I would recommend that the paper explicitly include a graph that plots the number of edge/node modifications per layer.

We have added it. Please see Figure1 at the above anonymous link.

Q9: I assume that this this the same thing as a LoRALinear adapter but with a higher rank; is this true?

Correct—we've adjusted accordingly.

Thanks for your review and helpful suggestions!

Q1: However, this method based on ratios is not a priori an apples-to-apples comparison...the observed higher fraction of edges may be an artifact of these scaling dynamics ...

To further distinguish whether it is the natural expansion of the structure or the influence of the fine-tuning mechanism, we supplement experiments on our tasks: the difference between the estimated edge changes caused by node changes and the actual observed edge changes.

In all tasks, the actual edge changes substantially exceeded the upper bound implied by node changes — often by a factor of 2.6x to 3.1x. This gap indicates that these additional edge changes are not just caused by nodes, but are actively adjusted during the fine-tuning process. This finding is consistent with our original conclusion.

Q2: It is shown that the circuit nodes in early layers largely do not change…simply because almost all nodes in early layers happen to already be in the circuit?

Circuit node changes involve both additions and removals, not just additions. As shown in Figure 3 (left), a considerable number of nodes in the middle layers are present in the initial circuit but are removed after fine-tuning. This demonstrates that circuit evolution is bidirectional and the relative stability in early layers cannot be solely explained by initial saturation.

Q3: I don't understand what the robustness metric…the specifics of the corruption process..Some questions I had while reading...

1. Motivation. We consider robustness experiments because the model performs poorly on the task before fine-tuning. It is important to ensure that the circuits identified in this low-performance are still robust. Earlier studies ignored this point. Robustness is defined as the Jaccard similarity between edge sets of circuits extracted on clean vs. perturbed data for the same task.

2. Corruption strategy. We use this strategy follow the Symmetric Token Replacement (STR) principle—minimally altering input while preserving task type.

• + to – does not mean testing addition vs. subtraction circuits. Instead, we assess whether specific edges are sensitive to small, task-relevant input changes.
• Changing one term in a sequence may break the exact pattern intentionally. This tests whether circuits remain structurally stable under small semantic shifts, not whether the model still solves the task correctly.

Q4: the Δₛ metric is dependent on the checkpoint schedule… this is unnatural as far as the metric aims to quantify the overall change in nodes and edges.

We intentionally designed $Δ_S$ to capture the cumulative dynamic changes of nodes and edges throughout the fine-tuning process. Specifically, certain nodes/edges may be added and later removed (or vice versa) during fine-tuning. These transient changes are meaningful and important intermediate behaviors are possibly lost if we only consider the final vs. initial states. $Δ_S$ aims to provide an approximate measure of the dynamic changes in structure evolution, which we believe is important.

Q5: It should be noted that…Thus the value of the finding is conceptual rather than practical.

Our primary motivation for designing CircuitLoRA was to validate insights gained from our analysis in Section 4. We answer your doubts from two aspects of the experiment:

• Memory efficiency: In the first stage of identifying critical layers, we find that using LoRA with rank=2 is sufficient, which uses significantly fewer parameters than the base LoRA setup rank=32 . This highlights the lightweight nature of our approach.
• Time efficiency: In a 4-epoch training setup, critical layers identified after just 1 epoch were already consistent with those from the final model, indicating that full fine-tuning is not required to extract critical layers. These suggest that our method offering a degree of practical applicability alongside the conceptual value.

Q6: When studying the compositional task..as opposed to the harder-to-interpret metric of overlap?

In this paper, we chose to use Overlap as a structural metric . To explore the faithfulness of the union circuit, we conducted additional experiments on the compositional task. We evaluated the faithfulness of the Union Circuit and found it to be 89.18%, compared to 96.86% for the Compositional Circuit. This faithfulness result, together with the structural overlap, supports our claim in paper.

Q7: In Figure 3, the legend does not describe all the node colors - what do the various shades of grey represent?

The various shades of grey in the figure indicate the total degree of each node in the final circuit. Darker grey represents nodes with higher connectivity.

Q8: how is "change rate" defined in Figure 3 (bottom right)?...What is the "compositional circuit" in 6.2.?

We sincerely apologize for the confusion caused during a first read. Your understanding of these two concepts is completely correct.