Isolated Causal Effects of Natural Language

Thank you for your positive remarks describing our work as interesting, theoretically correct, and largely well-justified. We address questions and concerns below.

[Why isolated effects vs. natural effects?] To illustrate why isolated causal effects are important for language, consider the effect of misinformation in online posts on readers’ voting decisions. Texts containing this type of misinformation also often contain other attributes likely to influence voting—e.g., politically inflammatory messaging. To determine whether action needs to be taken against misinformation, we would need to isolate its effect from the effects of other correlated attributes.

Similarly, in scientific applications, researchers are also frequently concerned with the isolated effect and conduct randomized clinical trials to eliminate effects of potential confounders. In our experiments on the SvT dataset to estimate effects of weight loss medication, we use text data from social media posts, which often encode additional information related to weight loss like users’ exercise habits and diet. We find—as you highlight—that the isolated causal effect estimate from SenteCon-Empath corresponds to the ground truth from real-world clinical trials, while the natural effect estimate does not.

This problem, also known as aliased treatments, is an important problem for social scientists studying the effects of language, and prior to our work the proposed solution was to carefully design a text experiment that constructs artificial texts that are not aliased (Fong & Grimmer, 2023). Our work expands the scope of possible research by allowing scientists to use naturally occuring text to study these kinds of effects instead of relying on resource-intensive text experiments with artificial data.

We will include this additional motivation in the introduction and in the discussion of the SvT results.

[Novelty of proposed framework] Treatment effect estimation with doubly robust estimators is of course well established in the causal inference literature. A main contribution of our paper is to conceive of, formalize, and parameterize natural language—a complex, unstructured, and high-dimensional form of data—in a way that allows us to use classical causal frameworks like doubly robust effect estimation, which are designed for much simpler forms of data. Indeed, leveraging classical causal estimators to solve new problems in new settings is an active area of research (e.g., Dudik et al. 2011; Schnabel et al. 2016; Azizzadenesheli et al. 2019; Byrd & Lipton 2019; Kallus et al. 2022). Our work introduces the concepts of isolated causal effects and non-focal language, and we define a new problem formulation, estimand, and practical estimation framework for these causal effects. Once language has been formalized in this way, we see the ability to draw on well-studied tools that already exist for estimating robust causal effects as a strength.

Likewise, once language has been parameterized as focal and non-focal components, our OVB estimators indeed naturally follow the work on bounding OVB of Chernozhukov et al. (2024), which presents modern tools for this classical problem. Our main contribution here is to draw the connection between language representation and OVB, which to our knowledge has not previously been articulated. We explore the idea that language representation is lossy specifically in a way that is detrimental to isolated causal effect estimation, and we link that information loss to OVB. We use this connection to introduce OVB-based metrics for evaluating the sensitivity of isolated effect estimates to language representation that are distinct from traditional NLP methods for measuring information loss.

[Clivio et al. papers] We will refine the language in our discussion of these references.

[$C_Y$ and $C_D$] We originally left the mathematical definitions of $C_Y$ and $C_D$ in Appendix C.4 due to space constraints but will move them to the main paper.

[Robustness value] Robustness in causal inference refers to the stability of the effect estimate under potential errors or omissions in modeling, assumptions, and/or data. Our robustness value measures how much OVB can be present before the point estimate of the causal effect changes from the correct to the incorrect sign. A larger robustness value indicates that the effect estimate better tolerates omission of key variables, suggesting that it is stable and therefore robust. We will emphasize this intuition when we define the robustness value in Section 3.3.

We hope that our response addresses any concerns you may have and that you will consider revising your score. Thank you again!

---

Azizzadenesheli et al. "Regularized Learning for Domain Adaptation under Label Shifts." ICLR 2019

Dudík et al. "Doubly robust policy evaluation and learning." ICML 2011

Schnabel et al. "Recommendations as treatments: Debiasing learning and evaluation." ICML 2016

Thank you for your feedback! We appreciate your positive remarks describing our work as theoretically and empirically well-supported, experimentally sound, and enhancing the rigor of text-based causal inference. We address further comments and questions below.

[Experiments with additional LM embeddings and dimensionality reduction] We include additional results (anonymous link) on the SvT dataset using embeddings from 3 common LMs: BERT, RoBERTa, and MiniLM. We find that the 3 new LM embeddings improve upon the previous MPNet over all metrics but still do not match SenteCon-Empath.

• The MiniLM effect point estimate is almost as close to the true effect as SenteCon-Empath, but its robustness value remains middling due to poorer overlap. The BERT and RoBERTa point estimates are only slightly better than MPNet, and their robustness values are lower than MiniLM.
• BERT and RoBERTa have much better overlap than either MPNet or MiniLM, but worse fidelity. Since BERT and RoBERTa are higher-dimensional than MiniLM, this is surprising but may be due to optimization of MiniLM for sentence-level tasks in the sentence-transformers library. Interestingly, this result suggests the fidelity-overlap tradeoff involves more than just the dimensionality of the representation.

We also include the results of dimensionality reduction using SVD on the MPNet and RoBERTa embeddings, which are the two most complex representations.

• For both representations, dimensionality reduction improves both the robustness value and the point estimate, highlighting the potential of data-based post-processing of representations.
• In fact, both point estimates move from outside the true effect confidence interval to inside the true effect confidence interval after SVD.

We thank you for suggesting these experiments and look forward to including them in the paper.

[Overlap assumption] It is true that overlap violations can be a concern in language settings (D’Amour et al. 2017). As we mention in our paper, high-dimensional representations with good fidelity are prone to overlap violations, and so we rely on our OVB metrics, overlap and robustness value, to warn us when such violations may be occurring. In this paper, we assume that strict overlap holds (i.e., that there is some non-zero chance of each $a(X)$ condition given $a^c(X)$). In our experiments, we have one result that suggests a “soft” overlap violation (i.e., the probability of one $a(X)$ condition is very small but not non-zero given $a^c(X)$): the MPNet result on SvT, where the overlap metric is fairly large. We also have a negative overlap value for Empath on the same dataset; however, for this metric we use a doubly robust estimator, and doubly robust estimates do not necessarily satisfy criteria like being non-negative in noisy settings. Therefore, this negative overlap value (which is small in magnitude) may not necessarily be due to an overlap violation.

[Representations for balancing fidelity-overlap tradeoff] We agree that the fidelity-overlap tradeoff naturally suggests learning representations that optimize the tradeoff. Experiments that achieve this require further theoretical analysis and empirical work and are outside the scope of this paper—this is in fact the basis of ongoing research—but we are happy to expand on this topic when discussing future work in our conclusion.

[Exact outcome model] For Amazon, the outcome model is a linear regression model of the semi-synthetic $Y$ fit over the LIWC categories that form $a(X)$, $a^c(X)$. For SvT, the outcome model is a gradient boosting classifier fit over $a(X)$ and each named non-focal language representation.

[Non-synthetic Amazon experiments] We did not use the original Amazon data because the true causal effects of the text interventions are unknown, so there is unfortunately no way to validate effect estimates on this data. The SvT dataset allowed us to validate our methods in a natural setting in which the true effect was known through an accompanying external clinical trial. Since such trials are highly resource-intensive, for Amazon we rely on the more common practice of obtaining true effects by generating a semi-synthetic dataset (Veitch et al. 2020; Pryzant et al. 2021). If you are interested in experimental results from a more complex semi-synthetic Amazon dataset, please see our response to reviewer gbt6 titled [Nonlinear $Y$, $a(X)$, $a^c(X)$ relationship].

We hope that our response addresses any concerns you may have and that you will consider revising your score. Thank you again!

---

D’Amour et al. "Overlap in observational studies with high-dimensional covariates." J. Econom. 221.2 (2021): 644-654

Pryzant et al. "Causal Effects of Linguistic Properties." NAACL 2021

Veitch et al. "Adapting text embeddings for causal inference." UAI 2020

Thank you for your thoughtful questions and feedback! We address your comments and clarify points below.

[Technical language] We will revise the paper to more clearly introduce and contextualize technical terms.

[Motivation for isolated effects] Due to character limits, please see our response to reviewer ZnfT titled [Why isolated effects vs. natural effects?].

[Methodological contributions] Likewise, please see our response to reviewer ZnfT titled [Novelty of proposed framework].

[Derivation of formulas] The importance weight in line 144 and estimand $\tau^*$ in line 158 draw from the transportability literature and doubly robust estimation literature, respectively. In Appendix A.1.1, we provide the full derivation of the importance weight and show the equivalence of $\tau^*$ in line 158 to $\tau^*$ in Definition 2.1. We initially left these derivations in the appendix due to space constraints. We agree some technical details may have felt abrupt as a result and will move key derivations to the main paper.

[Mediation analysis] We appreciate this insightful question. Though there are parallels in intuition between isolated effects and direct effects in mediation analysis, they are different technical problems. Our setting is not a mediation setting:

• In mediation analysis, such as in the papers you mention, interventions (e.g., math operations) cause mediators (e.g., the text “surface”), and mediators cause outcomes. Only the intervention can be controlled directly in the study, while the mediator cannot.
• In our setting, the entire text is a high-dimensional treatment where both the focal language $a(X)$ and the non-focal language $a^c(X)$ can be directly intervened on by randomizing the text. $a^c(X)$ is not caused by $a(X)$ and can be directly controlled, so it is not a mediator.

[Nonlinear $Y$, $a(X)$, $a^c(X)$ relationship] $Y$ and $a(X)$, $a^c(X)$ have a linear relationship only in the Amazon data setting. Our SvT dataset results demonstrate that the true effect can still be recovered even when the relationship between the text and outcome is nonlinear and complex.

To further explore this, we conduct additional experiments (anonymous link to results) on a new version of the Amazon dataset where $Y$ and $a(X)$, $a^c(X)$ have a nonlinear relationship. We fit a nonlinear gradient boosting model over $a(X)$, $a^c(X)$ on the true helpful vote count, then use the predicted vote count (+noise) as our new semi-synthetic $Y$. Using MLPs with <4 layers to fit importance weights and outcome models, we follow the protocol in Sections 4.2.2 and 5.1 to estimate effects for the attributes featured in Fig. 2.

The results of these additional experiments are consistent with the previous Amazon results:

• For both home and netspeak, as # dimensions increases (i.e., as # omitted variables decreases), the effect point estimate grows closer to the ground truth, and overlap becomes worse while fidelity improves. There is slight variability in these trends, as we would expect from the extra noise from more complex models.
• For home, robustness value increases with dimensionality, suggesting that gains in fidelity outweigh losses in overlap. For netspeak, robustness value decreases sharply after 6 features, suggesting that worse overlap outweighs improved fidelity. This is especially true on the 9th feature, where the overlap sharply worsens, indicating that $a(X)$ can be almost fully predicted from $a^c(X)$.

We thank you for this question and look forward to including these results in the paper.

[Complex $a(X)$] If $a(X)$ is continuous rather than binary, this introduces a new causal problem where it is common to estimate a dose-response curve or incremental effect instead of an average treatment effect. As the study of continuous treatment effects is its own active area of research in causal inference (e.g., Kennedy et al. 2017; Brown et al. 2021), we believe it is beyond the scope of this paper.

[SvT confidence intervals (CIs)] Wide CIs suggest that the estimates from the data are noisy (i.e., imprecise) but not necessarily that they are not robust. Robustness in causal inference refers to the stability of the effect estimate under potential errors or omissions in modeling, assumptions, and/or data. Our OVB analysis in Fig. 4 suggests that the SenteCon-Empath estimate (which does have wide CIs) is robust in this sense since the point estimate remains correctly positive even when we compromise the estimator by removing important features.

We hope our response addresses any potential concerns and that you will consider revising your score. Thank you again!

---

Brown et al. "Propensity score stratification methods for continuous treatments." Stat. Med. 40.5 (2021): 1189-1203

Kennedy et al. "Non-parametric methods for doubly robust estimation of continuous treatment effects." J. R. Stat. Soc. Ser. B Methodol. 79.4 (2017): 1229-1245