Identifying Causal Direction via Variational Bayesian Compression

We appreciate Reviewer 3osf's comments to further elucidate and enhance our work. The following responses address the major points that you raised.

1. Relation to Appendix B.2 of [1]

Our work originates from the causal discovery via MDL using neural networks, with variational Bayesian codelength being the encoding method. This approach latter converges to the marginal likelihood-based method outlined in GPLVM [1]. Although the two-part MDL formulation is briefly discussed in [1, App. B.2], to the best of our knowledge, our work is the first one to explicitly integrate this variational MDL codelengths into the learning process of neural networks for the downstream task of causal discovery. This approach fundamentally differs from GPLVM, which originally adopts a non-parametric approach to model inference and only considers the formulation with respect to the model complexity in a post-hoc manner. While both methods employ the variational learning, GPLVM uses the KL divergence terms to infer the distributions over latent variables and inducing points of sparse GPs [1, App. F.1], rather than optimizing the conventional model parameters. In contrast, our approach directly incorporates the model's complexity into the learning process by considering the complexity via the distributions over the parameters.

2. $N(0,1)$ for the marginals

As encoding the conditionals is our main focus in this work, we choose $N(0,1)$ as an uninformative choice to encode the marginals. In fact, excepts for the AN, AN-s, LS, LS-s, MN-U, and SIM-G pairs, the remaining benchmarks consist of pairs with a non-Gaussian cause, including the real-world Tübingen dataset. We have run additional analysis on different choices for the marginals including $N(0, 1)$, $U(X_{min}, X_{max})$, and Variational Bayesian Gaussian mixture model (GMM) [2]. The uniform encoding negatively affects the performance on most datasets. GMMs provide advantages on SIM, CE-Multi, and CE-Net synthetic benchmarks, whereas the performance on SIM-G, SIM-ln, and CE-Cha benchmarks are noticeably reduced. Although there is a slight increase on the Tübingen prediction accuracy, the AUROC score is also substantially decreased. As a result, we believe the standard Gaussian remains a reasonable choice for our method. Additionally, using a more flexible model for encoding the marginals will also challenge the theoretical analysis of our models' separable-compatibility.

Accuracy:

AUROC:

[1] Dhir, A., Power, S., & Van Der Wilk, M. (2024). Bivariate Causal Discovery using Bayesian Model Selection. In ICML.

[2] Blei, D. M., & Jordan, M. I. (2006). Variational Inference for Dirichlet Process Mixtures. Bayesian Anal., 1(1).

We thank Reviewer YARz for your novelty and favorable qualities of our work. Our responses below aim to rectify the issues that you mentioned.

1. Formulating the capability of the neural networks in Prop. 4.2

Thank you for your recommendation, we will specifically formalize the function of the "single-hidden-layer neural networks" as described in App. C.3 to make our proposition more mathematically rigorous.

2. Theorem 3.1

Since Theorem 3.1 is directly related to Eq. (5) and (7) and is the criterion for determining the causal direction, it was presented to clarify the origin of our causal scores and make our paper self-contained and easier to follow.

3. Hyperparameter configurations in location-scale models

Similar to the additive noise models, our location-scale models are also Bayesian neural networks with one hidden layer, whose details on hyperparameters are presented in App. B and C.3. The only difference between these types of models is that the location-scale ones have an additional output node to regress the log scale parameters for the Gaussian likelihood for enhancing the ability to model aleatoric uncertainty of our models.

4. Putting more experiment results in the main paper

Due to the current limit of space, we can only accommodate the most crucial results in the main paper. We will consider present directly or including more specific discussions and references to these key results in the next version of our manuscript.

5. Choice of prior for Bayesian neural networks

Our selection of Gaussian priors is influenced by previous works on evaluating neural network's complexity, especially [1-3]. As a result, we utilize Gaussian priors, which is a common choice, for variational Bayesian learning of neural networks. These studies have shown that variational learning with the Gaussian priors and hyperpriors on the variances yields adequate and consistent results with respect to the implicit prequential coding, which has been demonstrated to achieve good compression bounds in [2] and [3].

[1] Louizos, C., Ullrich, K., & Welling, M. (2017). Bayesian compression for deep learning. In NeurIPS.

[2] Blier, L., & Ollivier, Y. (2018). The description length of deep learning models. In NeurIPS.

[3] Voita, E., & Titov, I. (2020). Information-Theoretic Probing with Minimum Description Length. In EMNLP.

We thank Reviewer 3KpH for your valuable insights. We provide our responses below to clarify and resolve your concerns.

1. Kolmogorov complexity and MDL

There are two problems [1] when directly applying the Kolmogorov criterion in Eq. (4):

1. We do not have access to the true generating models $P_{X}$ and $P_{Y \mid X}$, and
2. The Kolmogorov complexity is not computable in practice.

The former problem can be resolved by estimating the model through the joint complexity $K(x, P)$, which on expectation over $P(x)$ will yield $K(P) + H(P)$ up to an additive constant [2]. The latter problem requires an approximating codelength $L(x, P)$ that mirrors $K(x, P)$, commonly selected according to MDL or MML principles. Despite the performance achieved, the gap between the practical $L(x, P)$ and the theoretical $K(x, P)$ is still an open problem in MDL-based causal discovery methods [1, 3], which we have not been able to rectify at the moment.

However, although our method begins from the Kolmogorov formulation, it is important to note that the identifiability of our models are studied from an orthogonal perspective of marginal likelihoods, which should not be theoretically impacted by the MDL approximation.

2. MDL of Bayesian Neural Networks

We would like to clarify that the prior over the parameters in our method are Gaussian distributions with zero means and learned variances (as described in App. B and C.3). We have also analyzed the normal-Jeffreys sparsity-inducing prior in Tab. 3, App. F.2, which indicated no significant improvements in performance. Additionally, Bayesian learning has already been shown to follow Occam's razor in model selection [4]. Hence, the sparsity-inducing prior is not necessary in our method.

Regarding the number of layers, as our model with one hidden layer has already achieved adequate performance, we did not consider including additional layer(s). We have performed additional experiments with two hidden layers. However, there are no substantial differences in performance when the number of layer is increased. Moreover, with one more layer, the AUROC scores on some datasets such as SIM, SIM-c, and Tübingen are slightly decreased.

Accuracy:

AUROC:

3. References to be discussed

Thank you for your recommendations. We will include additional related works in multivariate causal discovery in our multivariate discussion in App. G. As the aim of our work is the evaluation the codelengths/complexity of neural networks rather than estimating Bayesian neural networks (BNNs), we did not include a literature review on the BNNs, which we will examine in our future work. However, we do discuss related choices for computing the complexity of neural networks in Sec. 4.1.

[1] Kaltenpoth, D., & Vreeken, J. (2023). Causal discovery with hidden confounders using the algorithmic Markov condition. In UAI.

[2] Marx, A., & Vreeken, J. (2022). Formally Justifying MDL-based Inference of Cause and Effect. In AAAI ITCI'22 Workshop.

[3] Marx, A., & Vreeken, J. (2019). Telling cause from effect by local and global regression. Knowl. Inf. Syst., 60.

[4] Rasmussen, C., & Ghahramani, Z. (2000). Occam's razor. In NIPS.

We would like to thank Reviewer NJ5G for recognizing the positive aspects of our work. We expect the following rebuttal will address your concerns.

1. Underlying model assumptions

We summarize three key definitions related to the identifiability of our Bayesian causal models (BCMs), introduced in [1], as follows:

1. Distribution-equivalence: causal model selection via maximum likelihood will not be able to identify the causal direction;
2. Bayesian distribution-equivalence: Bayesian causal model selection via marginal likelihood will not be able to identify the causal direction, which is a stricter condition than distribution-equivalence;
3. Separable-compatibility: the necessary condition for two distribution-equivalent BCMs being Bayesian distribution-equivalent.

If two BCMs are distribution-equivalent, as long as they are not separable-compatible, they will not be Bayesian distribution-equivalent [1]. Similar to previous methods based on functional causal models (FCMs), the BCMs as in GPLVM [1] and our work are assumed to be the underlying generating processes of the data. With non-separable-compatible BCMs, given enough data, we can estimate the underlying BCMs and identify the causal direction with the marginal likelihoods. The benefit of this approach is that BCMs can be designed to be flexible enough (while keeping non-separable-compatibility) to cover a broad spectrum of generating models, such as GPLVM [1] or Bayesian neural networks in our work. Therefore, your example of normalized linear Gaussian models do not lie into the scope of our BCMs which assume that the effect is generated via nonlinear Bayesian neural networks.

2. Performance over scalability results

Since we use the official implementations of the baselines, we do not believe a direct comparison of running time would be fair. Instead, we have opted to evaluate the computational complexity of our forward pass in App. B. However, for a rough comparison, on the AN dataset, our work only took around 3 minutes on a CPU configuration, whereas GPLVM required on average about 2 days 11 hours on 10 GPUs.

3. Causal sufficiency assumption

Thank you for your suggestion. Since our performance on the SIM-c (with confounders) is promising, relaxing the causal sufficiency assumption will be a potential direction of study for our future work.

[1] Dhir, A., Power, S., & Van Der Wilk, M. (2024). Bivariate Causal Discovery using Bayesian Model Selection. In ICML.