Training and Evaluating Causal Forecasting Models for Time-Series

Thank you for the valuable feedback, which is helping us make our paper clearer. Here are the changes and clarifications we are implementing, which we believe address your concerns. Please let us know if there are questions that we did not address or any remaining issues!

Overall I found the paper pretty challenging to parse, for example how exactly the models were constructed and evaluated.

Sections 2.3 and 3.3 describe how we instantiate our approaches in deep learning models, for causal learning and RDDs, respectively. We will describe them in a more operational way.

For causal learning (S2.3):

• All models share the same time-series architecture (which we call backbone in the paper). We split the data in two splits.
• On the first split, we learn the nuisance models (m and e): m is trained to predict the outcome Y with all features except the treatment and past outcomes, trained with an MSE loss; e is trained with a logistic loss to predict the probability of treatment using the same features.
• On the second split, we use models m and e (fixed) to create the R-loss, and train theta to minimize the R-loss using SGD.
• At prediction time, we combine all models as in line 311 to make the final prediction.

For RDDs:

• For each treatment change, we create a small dataset of points before/after the change with no treatment change (if there aren’t any, we drop that price change).
• We fit a local linear model with a discontinuity at the price change to estimate the CATE at that price change, and that time.
• This is our ground truth, to evaluate the models on a causal task (predicting this CATE).

Thank you to the authors for the clarifications. I'm going to raise my score and lower my confidence. I don't feel well-positioned to assess the novelty or significance of this work, but I can now understand it is making reasonable claims and providing evidence for those claims.

Thank you for the valuable suggestions, that will help us improve our paper:

T_t for treatment at time t, and T_1, T_2 ​ for treatment types can be confusing.

This is indeed confusing. We will switch all treatment variables T to A (for action) to be closer to Reinforcement Learning notations which might be more familiar to the audience. This will also disambiguate the time index t and the treatment T. We will use A={a,b} for the generic two actions setup, so that we avoid subscripts that we use for time in the rest of the paper.

The evaluation method using RDD is complex and may generate only a few test set examples.

RDDs are fairly complex, but a well studied and well used identification strategy in econometrics, and our automated process to create them is fairly robust. Importantly, RDDs let us estimate the CATE for a causal evaluation on real, complex applications. To the best of our knowledge, this is the first proposal to address this long standing challenge in causal ML!

It is true that the number of RDD points that we can estimate will depend on the specifics of an application, and in particular how often treatment changes are observed (too few changes mean too few RDDs, too frequent changes mean that our estimators will be too noisy, so we need an in-between). So this approach will not always apply. In our applications though, we see an average of 1.50 RDDs/train time series (about 150k test points) and 0.51 RDDs/patient time series (about 770 test points for the 1.5k test set of MIMIC) after applying our filtering. While this is far from perfect, it is a significant number of evaluation points (much better than what we could do until now, which is not evaluate causal effects on complex applications with only observational data at all). We believe that it is a very valuable source of evaluation data that we should leverage and extend in the field.

Why are more datasets not considered for evaluation?

We did consider several other datasets, but none other were good fits for our approach. Causal datasets are typically very small by deep learning standards [1,2,3,4], with treatments happen all at the same time [1,2] or are unique [3,4]. The only large causal dataset with more treatment variety that we found is [5] (causal impact of price changes through discounts), but the dataset has very few price changes and our approach is not applicable (too few changes to fit causal models, and too few RDDs for evaluation). Other classical time series datasets are in-distribution forecasting tasks without a clear treatment [6-9].

We are currently working on comparisons under a simulation proposed in prior work (see answer to reviewer DJs7). While this is a much simpler task than a real application, this will add one more example to showcase our approach.

Dataset references:

[1] Advertisement Data: Brodersen, Kay H. and Gallusser, Fabian and Koehler, Jim and Remy, Nicolas and Scott, Steven L., “Inferring causal impact using Bayesian structural time-series models”, 2015.

[2] Geo Experiment Data: Jouni Kerman and Peng Wang and Jon Vaver, “Estimating Ad Effectiveness using Geo Experiments in a Time-Based Regression Framework”, 2017.

[3] Economic data for Spanish régions: Abadie, Alberto and Gardeazabal, Javier, “The Economic Costs of Conflict: A Case Study of the Basque Country, 2003.

[4] California’s Tobacco Control Program: Alberto Abadie, Alexis Diamond, Jens Hainmueller. 2010.

[5] m5 Forecasting challenge: https://www.kaggle.com/competitions/m5-forecasting-accuracy.

[6] Traffic: https://pems.dot.ca.gov/

[7] Electricity: https://archive.ics.uci.edu/dataset/321/electricityloaddiagrams20112014

[8] Exchange: Guokun Lai and Wei-Cheng Chang and Yiming Yang and Hanxiao Liu, “Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks”, 2018.

[9] Weather:https://www.bgc-jena.mpg.de/wetter/.

Thank you for the valuable comments and suggestions.

Before answering the specific questions, we would like to address one core misunderstanding about the interpretation of our results: “the causal TFT underperforming the causal transformer on both on all in distribution time-steps and 2/5 of the RDD forecast horizons.”

• We consider tasks in which a forecasting models is later used to make decisions and take actions. The data we train on includes treatments made by a knowledgeable decision maker (the train operator setting prices; the doctor observing the patient and deciding on a treatment).
• When testing a regular model in distribution (the typical forecasting evaluations, and our regular RMSE metrics), we have access to the sequence of treatment chosen by the knowledgeable decision maker. These chosen treatments carry a lot of information on the outcome, which models use to “cheat” when forecasting. Our causal models do not leverage this information by construction (to be able to make causal predictions), so we do expect them to perform worse in distribution!
• When predicting on a different treatment than the one chosen by the decision maker, the treatment does not carry the same information (it was not chosen by the train operator/doctor), and the predictions of non-causal models become worse: is is what we observe at $\tau+1$, and why our causal models beat the Causal Transformer.
• At large $\tau+t$ for large $t$, the causal transformer has access to the whole treatment sequence between $\tau$ and $t-1$: this sequence is in distribution (the only data we have), and hence carry the same information about the outcome, which the Causal transformer “cheats” with. Our causal model does not leverage this information by construction, to isolate causal effects. The Causal Transformer performs better, but it is because it learns non-causal information through the treatment sequence.

We are clarifying these points in our paper.

How does the causal TFT compare to other models under the original simulation framework used to evaluate Melnychuk 2020, Bica 2020 and Lim 2018? While more simplistic, it makes it easier to control the degree of bias present in policies, which could widen performance gaps (vs the slim margins seen in the RDD experiments).

This is a great suggestion, thank you! We are in the process of adding support for this experiment and running experiments. One thing to note is that in the cited work, the models are still evaluated in distribution, on regular forecasting tasks. We are using the simulation to extract the ground truth causal effect, and compare models on predictions of causal effects, to have an evaluation metric relevant to decisions made from the forecast. We hope to be able to report on these results by the end of the discussion period, but will update our paper regardless.

Where improvements are seen (i.e. shorter time steps in RDD expts), are these attributable to the use of TFT or new loss function proposed?

We believe that we can attribute the gains to the loss function (or more precisely the whole algorithm that first learns the nuisance models and then uses the special loss to train a causal effect model). Indeed, we can compare the TFT baseline (same architecture, without the causal learning) to the Causal TFT: we can see on Table 1 that the causal version (especially with linear encoding) performs much better on causal effects (RDD RMSE) than the baseline; on Table 2 the effects are smaller but in the same direction: causally trained TFTs perform better than the baseline with the same architecture. Given that those models share the same architecture, the causal training explains the difference.

Thank you for the valuable feedback.

We believe that the most important issue is the question about “why we care about improving RDD RMSE”? This question seems to stem from a miscommunication on the type of tasks we consider:

• We consider forecasting models that are later used to make decisions and take actions, a very common setting. Both tasks we consider in the evaluation are of this form. (1) training a demand forecasting model, and then calling it with different input prices to choose the price that maximizes revenue. (2) training a model that predicts blood pressure, and later use it with different drugs as input to decide how to treat a patient.
• When testing a regular model in distribution (the typical forecasting evaluations, and our regular RMSE metrics), it has access to the sequence of treatments chosen by an informed decision maker when collecting data (the price set by the train operator; the treatment chosen by the doctor). These chosen treatments carry a lot of information on the outcome, which the model uses to “cheat” when forecasting. However, if we make the model predict on a different treatment than the one chosen by an informed decision maker, this treatment does not carry the same information (it was not chosen by the train operator/doctor), and the predictions become really bad. Decisions based on these predictions will hence also be bad! This is our message in the paragraph “The need for causal forecasting”, summarized on Figure 1.
• Our RDD evaluation technique estimates the true causal effect of changing from one treatment to the other: this is the quantity we care about to make good decisions. For instance, to answer the question “will increasing the price make my revenue higher?” we need to know how increasing the price will impact demand (and not just how much demand there will be at our usual price). Our RDDs are a proxy for this ground truth causal effect, and the RDD RMSE measures how good a model is at predicting this causal effect. The RDD RMSE is thus a metric of how good a forecast is to make decisions, and not just to guess the outcome under decisions made as usual (without changing them). This is why we care about the RDD RMSE. We emphasize that our RDD-based method is, to the best of our knowledge, the first technique to create an evaluation of such causal forecasts on complex, real data tasks.

We are clarifying these points in the paper. The following, bigger changes that we are implementing will also contribute to passing this message more effectively:

• Moving the paragraph “The need for causal forecasting” to the introduction. This paragraph is an empirical result that we contribute and that supports the motivation for causal predictions. This will help clarify why we care about the RDD RMSE and will also lighten the background on causal inference.
• Change T (for treatment) to A (for action) in the theory: this should help clarity in the background (this is a special input that we want to change) and disentangle general statements from the running example.

We answer other questions in the review in the next message.