Weight Uncertainty in Individual Treatment Effect

• Novelty: First of all, the paper is not the first to address Bayesian inference for CATE estimation/ uncertainty quantification: Already Jesson et al. (2020) showed that a Bayesian approach can quantify uncertainty due to lack of data/overlap in CATE estimation. Hence the contribution of the paper is to combine Bayesian inference for CATE with balanced representations, which in my opinion is a rather straightforward thing to do. The paper heavily builds upon the work of Shalit et al. (2017) who originally proposed the balanced representation framework. From my understanding, the only novel result is the Bayesian version of the original generalization bound (Theorem 1/2), which again builds heavily on the original bound proposed by Shalit et al.

• Uncertainty quantification: From my understanding, the main benefit of using a Bayesian framework (and also the claim of the paper) is to allow for uncertainty quantification. However, in the experiments, the authors only evaluate performance in terms of point prediction. The "uncertainty study" provided evaluates some kind of robustness properties of the baselines, but not uncertainty for CATE estimation. I would have liked to see some experiments that evaluate the actual posterior CATE distribution, potentially showing benefits for decision-making as in Jesson et al (2020/2021).

• Implementation: Some implementation details are not quite clear to me. From my understanding, the authors add the IPM penalty to the ELBO loss (Eq. 12). Here, it seems like the likelihood B from Eq. 11 is replaced with an L2 loss, which would imply that the likelihood is modeled as a Gaussian with learnable mean and constant variance. I am not sure if this is correct, as the likelihood was never properly defined in the paper. This also would render the method incapable of useful uncertainty quantification for heteroscedastic data. Finally, the details regarding hyperparameters of the baselines are not reported and no hyperparameter tuning was performed, which raises concerns regarding the validity of the experimental results.

• Related to the previous point, no Code is provided, which makes it impossible to check the implementations of the proposed methods or the baselines.

• I am aware that Shalit et al. call the estimand of interest individual treatment effect (ITE). However, the majority of causal inference literature agrees on the naming "conditional average treatment effect" (CATE) or "heterogenous treatment effect" (HTE), as the effect is not really "individual" (two different individuals with the same covariates have equal CATE). ITE is usually reserved for the (random) difference in potential outcomes $Y_1 - Y_0$. I would suggest changing the naming from ITE to CATE to be consistent with the literature.

• Typos/Grammer/Notation: There is a substantial amount of typos, grammar issues, and notational inconsistencies in the paper, starting already with the title (Weight uncertainty in individual treatment effect). Some other examples: Appendix A.1 "proofs in ITE"; p.12 (top)", we have the following results"; Algorithm 1: $e \in [0, E]$ (looks like continuous epochs).

• Sec. 3 seems to provide (standard) background on Bayesian variational inference, not specific to the CATE setting. I suggest moving this section to the Appendix if space becomes an issue.

1. The presentation could be improved to further justify the proposed methods. For instance, the transition from PAC-bayes bound to Bayesian inference is not well-grounded. Why considering the intergrated posterior loss is helpful for one specific instance? It is also unclear to me how we can go from Bayesian inference to an ITE estimator.

2. The experimental results are not super strong. For instance, the proposed methods are less powerful than some competitors in many cases. If that is not improvable, more discussion should be provided for understanding in what cases the proposed methods may perform better.

More specific comments are in the Questions section.

• Seems like the paper missed the entire body of literature on uncertainty in TEE. Use of GPs, Bayesian Neural Networks and Approximate Bayesian Methods like MC Dropout. No comparisons, no related work.
• Not sure if the correct evaluations are being used to show the benefit of uncertainty estimation (good calibration etc.) and if the method has good uncertainty estimation. This can be done on all the datasets (not just recommender system).
• The experimental results are not compelling. Not great performance and none the baselines I would expect to see.

The presentation of this paper is confusing and can be improved. Please see my detailed comments in the question section.

Less significant weakness: there are some typos in the text: e.g. (14) on page 7 and start of some words are unnessarily capitalized. More significant weakness: The experiements are already pretty comprehensive, however, I want to see how robust the results are to your model architecture choices. The results are not convincing enough to say the proposed method outperforms in most cases. After all there are only 3 examples in table 2.