Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension ability

We thank the reviewer's efforts on reviewing this paper. We now address the questions raised as follows.

---

Q1: I am concerned about the necessity of applying causal mediation analysis to this research question. Mask, Replace, Swapping, etc., are common practices for augmenting text data as well as examining the sensitivity of LLMs to slight variations in the input text. The motivation for the CMA application in this submission is not clear to me, given the current writing.

Thank you for your question. We would like to clarify that (1) applying causal mediation analysis (CMA) is important to discover the LLM's deep structure understanding ability (rather than the surface one); and (2) the mask and rephrase operations in our works are specifically designed to adapt to CMA.

It is true that masking, replacement, and swapping are common in LLM studies, but how to make use of these operations to reveal the LLM's deep structure understanding ability is challenging and non-trivial. Many previous research [1,2,3,4,5], performed sensitive analysis based on these operations, can only explain the surface structure understanding ability of LLM. In particular, these studies lacked comprehensive deep-surface structure comparisons, leading to less rigorous conclusions (see more detailed discussion in Section 1). Simple use of operations like masking cannot isolate deep structure understanding, as it highly co-exist with the surface one. For instance, masking What is 50 times 20? to What is <Mask>times 20? changes both deep (semantics) and surface (format) structures. This limitation in previous studies prompt us to develop ADCE, a rigorous, CMA-based metric for assessing LLMs' deep structure comprehension.

Specifically, we rigorously formulate the relationships between input, deep structure, surface structure, and output, as a causal graph with mediation, and define the deep structure sensitivity as the direct causal effect (DCE). Then, we develop ADCE, an interpretable and computable version of DCE, using CMA and (improved) common operations like Mask. In particular, ADCE is an approximated version of DCE, and the detailed intervention operations, like masking and rephrasing, need to be specifically designed to reduce the approximation error. For example, when estimating AICE, we mask $k$ nearby non-core words in TE instead of randomly masking. Notably, these new designs on the intervention operations are different from the previous works.

Furthermore, we also demonstrate the effectiveness of introducing CMA and improved common operations for quantifying the model's deep structure understanding ability through two additional experiments. (1) We first conduct a synthetic experiment where the model's true deep structure understanding (ground truth) can be directly calculated. Within the CMA framework, we estimate the approximated model's deep structure understanding (ADCE). We find that both the trends and values of ADCE align closely with the ground truth, validating our method. Detailed results are included in the updated manuscript (see Lines 1079-1125, Appendix E).(2) Moreover, the spurious correlation experiments in Section 4.5 demonstrate that as spurious correlations intensify, the model's reliance on deep structures weakened. Figures 8(a) and (b) show ADCE decreasing as expected, further validating that our proposed ADCE metric, within the CMA framework, effectively quantifies LLMs' deep structure understanding ability.

Dear Reviewer iYaZ,

Thank you for acknowledging our rebuttal. We are delighted to see that our response has addressed your previous concerns regarding CMA. Given your current score, we would greatly appreciate if you could share any further concerns you may have. Please share them with us, and we are willing to address and discuss these issues in more detail to improve our work.

Authors

Thanks for the reviewer's constructional feedback on this paper. We now address the questions as follows.

---

Q1: The relationship between ADCE and fine-tuning (Section 4.3) is only briefly explored.

Thank you for your question. Section 4.3 investigates why ADCE shows anomalous values (smaller than 0) for certain tasks and LLMs, such as Llama-3-8b on Analytic Entailment. We hypothesize this is due to unactivated or absent relevant knowledge in the pre-training data, rather than a failure of ADCE. To test this, we conduct supervised fine-tuning (SFT). The results show a significant increase in ADCE after SFT, supporting our hypothesis that the anomalous pre-SFT ADCE values reflect a lack or inactivation of task-specific knowledge in the LLMs, not a deficiency in ADCE.

We acknowledge that exploring SFT from an ADCE perspective is an intriguing topic, but it is beyond the scope of this paper and we leave it for future exploration. And if you have specific suggestions for analyzing SFT via ADCE, we are very willing to explore this topic further.

Q2: ADCE seems to capture both sufficiency and necessity of deep structure changes in causing output variations. How might different post training strategies influence a model's reliance on deep vs. surface structure?

Thanks for mentioning this interesting question. We have expanded our analysis to include two additional post-training approaches: Instruction Fine-Tuning (IFT) [1] and Fine-Tuning with In-Context Learning (FTICL) [2]. We've also analysed the In-Context Learning (ICL) [3] method, due to its effectiveness in harnessing the models' inherent abilities to comprehend and produce responses, as well as its popularity within the NLP community.

While other post training strategie like DPO [4] and RLHF [5] are often mentioned in the training of LLM for aligning with human preferences, these methods require constructing paired training data. Given that preference alignment is not our main focus and considering the time constraints for additional dataset generation, we leave the exploration of these methods to future studies.

Following the experimental setting in Section 4.5, we consider Llama-3-8b on the Analytic Entailment task. We add more experimental details in Appendix G.2, Lines 1330 - 1387 of our updated manuscript and summarize the main results in the following table.

We find that various post-training strategies and ICL all lead to improvements in both model accuracy and deep structure understanding ability (ADCE). Moreover, FTICL and IFT, which consider both prompt engineering and parameter optimization, yield greater gains compared to SFT, which only focuses on parameter optimization, or ICL, which only optimizes prompts.

[1] Wei, Jason, et al. "Finetuned Language Models are Zero-Shot Learners." International Conference on Learning Representations.

[2] Anil C, Wu Y, Andreassen A, et al. Exploring length generalization in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 38546-38556.

[3] Brown T B. Language models are few-shot learners[J]. arXiv preprint arXiv:2005.14165, 2020.

[4] Rafailov R, Sharma A, Mitchell E, et al. Direct preference optimization: Your language model is secretly a reward model[J]. Advances in Neural Information Processing Systems, 2024, 36.

[5] Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback[J]. Advances in neural information processing systems, 2022, 35: 27730-27744.

---

We hope our response and revisions have addressed your concerns. If you find our revisions and response helpful, we would greatly appreciate your consideration in raising the score to better support our work. We remain open to further discussion if you have any remaining questions.

We thank the reviewer's acknowledgment on this paper. We now address the questions raised as follows.

---

Q1: Although the metrics provide valuable insights, their exact interpretations may vary depending on model architecture... / Could the authors clarify how practitioners might interpret ADCE and AICE values across different model architectures?

Thanks for mentioning this important question. The proposed metrics ADCE and AICE are model-agnostic, treating models as black boxes and focusing only on their outputs. We then demonstrate why the proposed metrics are model-agnostic from both experimental and methodological perspectives.

1. Experimentally, we conduct additional experiments to demonstrate the consistent properties of these metrics across different model architectures. Taking ADCE as an example, we extend the experiments in Section 4.5 to include different model architectures. Due to computational constraints, we primarily focus on Llama-3 (Transformer) and Mistral-7b (Transformer with Sliding Window Attention). We observe that as spurious correlations increased, all ADCEs show consistent decreasing trends (↓) across different LLM architectures while Accuracy shows almost no significant change (-), indicating ADCE's architecture-independent nature.

2. Methodologically, the calculations of ADCE and AICE are independent of model architecture. As shown in Equation (5) (Line 283 in Section 3.2), we only need the model outputs before intervention $Y^{origin}$ and after interventions $Y(T=1, s(T=1))$ and $Y(T=0, s(T=0))$ to calculate these metrics, without relying on specific architectural details.

Q2: The approach may require internal access to models for accurate analysis, potentially limiting its use with proprietary or black-box models where such transparency isn’t feasible.

Thank you for your question. Our method does not require internal access to models for accurate analysis. The interventions in our method are performed exclusively on the inputs, without any internal access to the models. As for metric calculation, our method focuses solely on the outputs of LLMs, making our approach applicable to both black-box and white-box models. Specifically, as indicated in Equation (5) of Section 3, ADCE and AICE only require comparing LLMs' outputs in three scenarios: before intervention ($Y^{origin}$), after intervention on both deep and surface structures ($Y(T=1, s(T=1))$), and after intervention only on surface structure ($Y(T=0, s(T=0))$). This comparison is made without the need for any additional internal information about the models' architectures.

Q3: How does the approach perform on NLP tasks involving noisy or unstructured data?

Thanks for this great suggestion.

• For unstructured data, the datasets used in the experiments, such as CommonsenseQA, Analytic Entailment, and GSM8k, do not have a uniform text template and are not structured data. The experimental results in Section 4 have demonstrated the effectiveness of ADCE and AICE on these datasets. More details about the datasets can be found in Appendix A (Lines 758-787).
• For noisy data, we consider two scenarios: text noise [1, 2, 3] and label noise [4, 5]. Using the 2-digit Multiplication dataset and LLama-3-8b as an example:

(1) Text Noise: For each word in the input text, we randomly apply one of three noise-adding methods: a) Typo: Replace a random character with a random lowercase letter. b) Extra: Insert a random lowercase letter at a random position. c) Missing: Delete a random character. We gradually increase the noise level $\eta$. For instance, $\eta=0.9$ means each word has a 90% probability of modification, indicating higher text corruption. The experimental results are as follows:

As the noise level $\eta$ increases, both ADCE and Accuracy decrease, while AICE increases. It possible that noise likely disrupts deep structural information, forcing the model to depend on more accessible, surface-level information. This shift results in lower ADCE and higher AICE.

Thanks for your time reviewing this paper. We now address your questions as follows.

---

Q1: It is unclear how deep structures are accurately identified and separated from surface structures. / Your approach relies on interventions targeting deep and surface structures. In your experiments, how are deep structures accurately identified and separated from surface structures?

This is a good question. Strictly differentiating the impact of deep and surface structures on output in experiments is not easy, as changes in deep structure often inevitably lead to changes in surface structure. For instance, consider the example in Table 1: when we mask 50 in the questionWhat is 50 times 20 to create What is <Mask> times 20?, we alter both the deep and surface structures simultaneously.

The high coupling between deep and surface structures makes distinguishing them experimentally hard. Therefore, our key idea is to design ADCE and AICE as indicators highly correlated with LLMs' deep and surface structure understanding capabilities, respectively. Specifically, to ensure that AICE is solely related to surface structure, AICE is proposed to compare changes in LLMs' outputs when only the surface structure is altered. On the other hand, ADCE uses an indirect method, approximating deep structure impact by subtracting surface structure impact (AICE) from the total effect (TE) of both structures in Equation (5). To minimize approximation errors and ensure that ADCE is solely highly correlated with deep structure, we carefully design intervention strategies in Section 3.3. For instance, based on the intervention inputs in TE, we further mask the $k$ non-core semantic words closest to the masked core semantic word in TE as the intervention text for AICE (see more details in Lines 306-309, Section 3.3). These strategies ensure ADCE primarily measures output changes caused by alterations in deep structure, effectively capturing the model's deep understanding ability.

Experimentally, both synthetic and spurious correlation studies (detailed below in Points 1 and 2, Question 2) confirm that ADCE and AICE strongly correlate with the model's true understanding of deep and surface structures, respectively. This validates our metrics' ability to distinguish between the impacts of deep and surface structures on model outputs.

Q2: The ADCE and AICE are presented as approximations, but it is unclear how accurately they reflect the true causal effects... / Since ADCE and AICE are approximations, ... validate these metrics against true causal effects?

Thanks for mentioning this important question. We conduct the following two experiments to demonstrate the validity of ADCE and AICE:

1. Experiments on synthetic data with known true causal effects.

We reduce the causal relationships between deep structure $d$, surface structure $s$, input $x$, and output $y$ shown in Figure 3 to a simplified Structural Causal Model (SCM) [1]. In this simplified model, $d$, $s$ and $x$ are all scalars with linear relationships. Despite simplification, this SCM retains the key causal relationships in Figure 3, where $x$'s effect on is mediated through two parallel paths: $x \to d \to y$ and $x \to s \to y$. This simplification allows training a logistic regression with explicit functions, enabling direct computation of true causal effects for $d$ and $s$, thus facilitating ADCE and AICE validation. The SCM is defined as:

\\begin{aligned} x &\\sim \\mathcal{N}(0, 1),\\quad d = x + \\epsilon_d, \\quad s = x + \\epsilon_s.\\\\ y &= \\begin{cases} 1, & \\text{if } \\sigma(c_1\\cdot d + c_2 \\cdot s + \\epsilon_y) > 0.5 \\\\ 0, & \\text{otherwise} \\end{cases} \\end{aligned}

where independent small noises $\epsilon_d \sim \mathcal{N}(0, 0.25)$, $\epsilon_s \sim \mathcal{N}(0, 0.25)$ and $\epsilon_y \sim \mathcal{N}(0, 1)$. $\sigma(\cdot)$ is Sigmoid function. $c_1$ and $c_2$ are weight parameters for $d$ and $s$, respectively.

In the updated manuscript, we provide more details about synthetic experiments and visualize how true causal effects, ADCE, and AICE trends change with the weight parameters $c_1$ (see Lines 1079-1125, Appendix E).

Here, we summarize main messages :

• The trends in Figure 11 (a) of approximated causal effects (ADCE and AICE) and true causal effects of $d$ and $s$ on $y$ align as $c_1$ increases, demonstrating the efficacy of ADCE and AICE.
• The trend in Figure 11 (a) shows the difference between approximated causal effects (ADCE - AICE) aligns perfectly with the trend of the difference between true causal effects (True CE of $d$ - True CE of $s$). This consistency validates our key experiment in Section 4.4, comparing LLMs' understanding of deep versus surface structures.
• We normalize approximated and true causal effects to $[0,1]$, eliminating numerical incomparability. This reveals a much closer alignment between their trends, as shown by the nearly identical curves in Figure 11(b).