A Unified Causal View of Instruction Tuning

We thank the reviewer for the constructive feedback. As suggested, we will compare our method to additional baselines using different backbones to further validate its effectiveness. We will also analyze the representations learned by our model and provide supporting evidence that causal mechanisms are captured.

We also sincerely appreciate the reviewer highlighting our key contributions - the novel meta-SCM, Uniform Identifiability Condition proof, and detailed experimental comparisons. Your feedback has been invaluable for improving our work. Please let us know if you have any other suggestions for strengthening the paper. Thank you again for your time in thoroughly reviewing our submission and providing these thoughtful suggestions.

Answers for Questions:

A4：

We appreciate you raising this excellent point. The traditional training process optimizes the likelihood $P(Y|X)$, where $X$ contains a mixture of information - some factors causally influence $Y$ while others are spuriously correlated. Without explicitly disentangling the latent factors within $X$, the shortcuts from non-causal variables to $Y$ may be exploited, leading the model to learn these unintended spurious correlations along with the true causal relationships.

A5：

The $T$ in the original paper should be replaced by $D$, which represents the inherent dataset properties. It can describe the spurious correlation between the target label and the feature (latent factor) by modeling the backdoor between them. We apologize for this misunderstanding because of our unclear writing.

A6：

Since both the inherent dataset properties $D$ and latent factors are unobservable, we cannot directly perform backdoor adjustment or intervention. Therefore, to mitigate spurious correlation, an alternative approach is to only utilize the parent latent factors of each task's $Y_t$ for prediction. This branch of research can be categorized as causal representation learning [1]. However, being able to select the parent factors relies on latent factor identifiability. Thus, guaranteeing identifiability of latent factors is a core challenge in causal representation learning.

[1] Towards Causal Representation Learning. Proceedings of the IEEE, 2021.

A7：

That is an excellent question. In causal inference, identifiability refers to " A causal quantity can be computed from a purely statistical quantity." [2]. However, in causal representation learning, identifiability has a different meaning - it indicates that "Representations for each latent factor can be learned without mixing information with others, while ensuring that the difference between the learned representations and the true representations remains within acceptable bounds of uncertainty" [3, 4, 5]. The goal of the UIC is to guarantee latent variable identifiability under this definition from causal representation learning.

[2] Introduction to causal inference. 2020.

[3] Self-supervised learning with data augmentations provably isolates content from style. NeurIPS 2021.

[4] Partial Identifiability for Domain Adaptation. ICML 2022.

[5] Identifiability results for multimodal contrastive learning. ICLR 2023.

A8：

Thank you for your insightful question. We apologize that Section 3.2 did not effectively communicate our approach regarding identifiability and causal factor selection.

To clarify, we do not posit that "if $L_i$ is identifiable, then it is a causal factor of $Y_t$." For each task, whether $L_i$ is a causal parent of $Y_t$ is an intrinsic property. In our work, causal factors for different tasks are selected by a binary matrix with 0,1 values. This matrix is learned by maximizing likelihood, similar to score-based causal discovery [6]. Notably, selecting causal factors relies on the premise that the factors $L_i$ are identifiable. Thus, we introduce a UIC Loss as a regularization term to ensure $L_i$ identifiability. While UIC constrains the structure between $L_i$ and $Y_t$, this constraint is flexible. Causal factor selection is still primarily driven by the binary selection matrix.

[6] Generalized score functions for causal discovery. SIGKDD 2018.

A9：

Thank you for your questions, we will compare our method to additional baselines using different backbones to further validate its effectiveness.

We sincerely appreciate you taking the time to ask such insightful questions. It is clear you want to help improve the quality of our paper, for which we are truly grateful. Please feel free to continue the discussion with any other questions you may have. We welcome the opportunity to further clarify and strengthen our work. Thank you again for your dedication and effort in reviewing our submission so thoroughly.

We deeply appreciate your careful, detailed review. Your thorough feedback is valuable for revising our paper - we have re-uploaded the modified version highlighting major changes in blue. The questions you ask are critical, and they all involve the core of our research. We are honored to discuss them with you.

Before answering your questions, we first provide a driven example, which we hope will intuitively show the motivation for our modeling, the potential cause of spurious correlation, and our solution:

Driven example:

Let's consider the sentence "Pizza is delicious." This is an instance of the random variable $X_t$ in our paper. This sentence contains information about food, "pizza", and sentiment, "delicious." Accordingly, we introduce latent factors to represent different types of information. Specifically, $L_1$ represents food information and $L_2$ represents sentiment. Thus, we have $L_1, L_2 \rightarrow X_t$ . We use $Y_t$ to represent the target labels of different tasks, which require different latent factors, such as $L_1$ for topic classification and $L_2$ for sentiment classification. Due to inherent dataset properties (probably from sampling bias), spurious correlation may arise. For example, in a dataset sampled from pizza enthusiasts for sentiment classification, pizza, as a food concept, will frequently co-occur with positive emotion, causing spurious correlation between food and sentiment labels. In our paper, we intended $D$ to represent inherent dataset properties and thus $D \rightarrow L_1, L_2$ . Since $D$ denotes internal properties, it is unobservable. For $Y_t$ of sentiment classification, there is a backdoor path between $Y_t$ and $L_1$, i.e., $L_1 \leftarrow D \rightarrow L_2 \rightarrow Y_t$ , producing spurious correlation between food information $L_1$ and positive sentiment labels $Y_t$ .

Unclear statement of $T$ and our revision:

In our original paper, for the sake of narrative convenience, we imprecisely used $T$ to denote both inherent dataset properties and tasks. We apologize for any confusion caused by this unclear statement. We sincerely appreciate you raising this critical issue in Q3, alerting us to this imprecision in our writing. We have already revised the paper where we use $D$ for inherent dataset properties and $T$ for tasks separately.

Answers for Questions:

A1 ：

We agree with you that it is more rigorous to model SCM from the structural equations, and when the structural equations are given, the causal graph will be determined accordingly. The reason we introduced causal graph firstly is that we want to give the reader an intuitive understanding of the data generating process of NLP tasks. This includes introducing the variables and their qualitative causal relationships. An illustrative example corresponding to Fig. 1 has been provided in the above driven example. Thank you very much for your advice.

A2：

Thanks for your valuable suggestion. In our modeling, $Y_t$ for different tasks are actually different nodes. For visual simplicity, we depicted only a single $Y_t$ node in the causal graph, and distinguished different tasks by subscript $t$.

A3：

This is an excellent question and we appreciate you raising this important issue. $T$ should be modified to $D$ to represent inherent dataset properties, which are unobservable. This would make $L_i$ not independent of $L_j$, leading to a backdoor path and inducing spurious correlation. We will correct the imprecision in our writing, which improperly used $T$ to denote both dataset properties and tasks. Thank you again for catching this critical misunderstanding.

We deeply appreciate the reviewer kindly increasing the rating and providing constructive feedback to improve our work.

Due to computational constraints, the scale of our current experiments is limited. We will compare to additional baselines using different backbones to further validate effectiveness of our method.

Thank you for recommending articles [1-2] – drawing insights from both, we have already revised our paper to cite them. We will continue enhancing the question framing and improving overall readability referring to these works.

We are truly grateful for your continual thoughtful suggestions, which significantly strengthen our paper’s quality. Thank you again sincerely for the invaluable advice throughout the review process!

[1] 'Desiderata for Representation Learning: A Causal Perspective,' arXiv:2109.03795.

[2] 'D-Separation for Causal Self-Explanation,' NeurIPS 2023.

We sincerely thank the reviewer for recognizing the intuitive merit of considering task-related factors and discarding spurious correlations. Due to computational constraints, the scale of our current experiments is limited. Per the reviewer's suggestions, we will explore scaling up our approach using larger models, such as Llama. We will also expand our method to additional tasks and datasets to further demonstrate its effectiveness. We deeply appreciate the reviewer providing this invaluable feedback to strengthen our work.