Causally Motivated Sycophancy Mitigation for Large Language Models

1. The motivation for the Causal Activation Calibration method in Section 4.3 is unclear. Specifically, the relationship between the causal direction in Section 4.3 and the SCM in Section 4.1 is not sufficiently clear. Please provide further clarification on this point. My understanding is that Equation (6) aims to add a conditional independence constraint to the original training objective, intending to eliminate spurious correlations between $Z_S$ and $Y$. However, I am not convinced how the “causal direction” in Section 4.3 effectively mitigates these spurious correlations. Does this approach leverage the causal direction as a representation of causal effect orientation, or is there some other theoretical justification? I suggest that the authors clarify this aspect in the paper.

2. The second statement in Lemma 4.1 is not described clearly enough. I would like to understand which part of the subsequent methods specifically utilizes this statement. For instance, if we consider a specific example where $f(Z_C,Z_S) = Z_C$, then this condition does not hold in the causal graph shown in Figure 2(a). Furthermore, does the second statement in Lemma 4.1 offer any guidance in constructing the subsequent methods? If not, I would suggest the authors consider removing this statement.

1. In Line 81, the phrase “To map the latent causal embeddings to the observable intermediate components of LLMs” appears to conflict with the statement “a significant causal signature which can distinguish the intended causal embeddings from spurious embeddings which incur sycophancy within the latent representation space.” Could you clarify this discrepancy?

2. How can I(X_P ; Y | Z) be approximated by Eq. 7? Is causal intervention controllable? Could you provide an example to illustrate this?

3. Why does the intervention \bar{X}_P maximize the difference in cross-entropy losses for a fixed W? Is this a result of the algorithm's design, or does it align with the intrinsic nature of interventions?

4. The author claims to utilize Parameter-Efficient Fine-Tuning (PEFT) in Line 267, yet states in the baselines section that all parameters are fine-tuned. This is rather confusing.

5. In Section 4.3, how is the weight matrix value obtained, and what does |w_l^h| represent? Are these parameters part of the adaptor in PEFT, or are they parameters of the model itself? How are these parameters utilized specifically?

My main concerns focus on the application of the structured causal model (SCM) approach. The authors state that causal relations can be captured via a directed acyclic graph (DAG) learned through regularization. However, in real-world applications, the regularization term is unlikely to reach zero. How can the authors be certain that the learned representations are truly disentangled? The methodology would benefit from a more robust approach to verifying that these representations capture causal rather than correlated information.

My second question relates to the evaluation of DAG structures in SCMs. Typically, metrics like Structural Hamming Distance (SHD) or False Discovery Rate (FDR) are used to confirm if the learned graph conforms to a DAG structure. This paper, however, claims that the approach is inspired by graph models without providing a specific graph-based evaluation. How can we be sure that the learned representations are genuinely causality-related rather than optimized merely by overfitting through additional parameters? Both theoretical analysis and case studies would strengthen the authors’ claims.

Regarding the representation learning approach, it seems counterintuitive that a fully supervised learning model without any stochastic components could infer causality from data alone. The mutual information-based independence criterion here could indicate correlation, but correlation does not imply causation. Causal inference methods typically rely on stochasticity in at least one part of the model (e.g., a two-tower architecture) to differentiate causality from mere correlation. Without this, there is a risk of learning coincidental patterns rather than true causal relationships.

Lastly, a minor issue: without providing a formal proof, I recommend avoiding the use of terms like "Lemma" to present conclusions, as this suggests a level of mathematical rigor that is not fully substantiated in the paper.

1. The motivation for the proposed approach and the intuition of the algorithm design needs to be more clear.

• In the Introduction, the authors discuss two groups of prior research on LLM sycophancy: (1) linear probing and (2) path patching. However, these prior works primarily concentrate on analyzing and understanding sycophantic behavior in LLMs, rather than on mitigating it (also as mentioned in Related Work line 113-122). Given the distinct emphasis of these studies compared to the authors' goal, it is unclear how the limitations of these earlier works directly motivate the development of the authors’ proposed approach.

• While the authors discuss several recent studies on mitigating LLM sycophancy in the Related Work section (lines 124–135), a more in-depth comparison between these studies and their own approach would be beneficial. Specifically, it would be helpful to understand if there are potential methodological concerns with the designs proposed by Burns et al. (2022) and Rimsky et al. (2023) that inspired the development of CAUSM.

• It would be helpful to discuss the rationale for using structured causal models to capture sycophancy in the models’ latent representations.

2. Some compared methods in the tables/figures of results are confusing. E.g., I couldn’t find a clear definition of CAUSM (Base) (Table 1 & 2) and CAUSM (Table 2). What is the difference between CAUSM and CAUSM (Base) in Table 2?

3. Did the authors conduct the experiments on multiple base LLMs or specifically focus on a single base LLM (i.e., Llama-2-7B-Chat)?