We thank the reviewer for their constructive feedback and suggestions to improve our paper. We appreciate that the reviewer has recognized the clarity of our writing, the correctness of the theoretical results, and the novelty of introducing a differentiable DAG learner for latent hierarchical causal models (LHCMs). Below, we address the concerns and provide clarifications.

Section 2: Do papers on differentiable causal discovery referenced in the paper actually perform causal discovery?

We agree that some of the referenced works in the related work section do not fully perform causal discovery, as highlighted by [1] and [2]. We have updated the manuscript to explicitly discuss these limitations. However, we note that the critiques in [1] are not universally applicable. Improved evaluation protocols, such as using sampling with non-equal noise variances, can mitigate these issues and provide robust results for proposed causal discovery approaches [4]. These practices have been incorporated into our experimental evaluations.

Section 4: Identifiability upto permutation

Latent variables are inherently hidden, and their labels can only be identified up to a permutation. For instance, the structures Z2 <- Z1 -> Z3 and Z1 <- Z3 -> Z2 represent the same causal relationships, even though the meanings of Z1, Z2, and Z3 differ. This is because the d-separation properties, which define the causal semantics, remain unchanged under such permutations.

Section 5: Enforcing structural constraints

Condition 1(i) is incorporated through Eqs. (6) and (8). These conditions are reflected in the final term of Eq. (10), which equals zero if and only if each latent variable has two pure children, ensuring that the structural constraints are satisfied.

Condition 1(ii) is reflected in the block structure of the adjacency matrix M, which enforces acyclicity and consistency with the hierarchical structure.

Q: Causal Discovery vs. Structure Learning:
Enforcing structural constraints alone does not guarantee recovery of the true causal graph, as multiple graphs can satisfy these constraints. Our method addresses this by jointly optimizing for likelihood and sparsity while satisfying the constraints, enabling the discovery of the true causal graph.

Section 6
-Tab. 1: Why do the baselines perform so poorly?

The baselines perform poorly because they rely on assumptions of linearity or deterministic relationships, which are not satisfied in our data. This highlights the effectiveness of our method in more general nonlinear settings.

-Synthetic Experiment: How were the ground truth structures chosen?

The ground truth structures were chosen randomly.

-Image Experiments: Why not compare with CausalVAE?

Thank you for pointing this out. CausalVAE [3] requires additional information in the form of concept labels, which our setting does not provide. To address this, we have included a comparison with a modified version of CausalVAE that does not use concept labels in the updated manuscript.

References

[1] Reisach, C., et al. "Beware of the simulated DAG! Causal discovery benchmarks may be easy to game." NeurIPS, 2021.
[2] Seng, A., et al. "Learning Large DAGs is Harder Than You Think." ICLR, 2024.
[3] Yang, G., et al. "CausalVAE: Structured Causal Disentanglement in Variational Autoencoder." NeurIPS, 2020.
[4] Ng, I., Huang, B., & Zhang, K. "Structure learning with continuous optimization: A sober look and beyond." Causal Learning and Reasoning, PMLR, 2024.

Please let us know if you have further concerns, and please consider raising the score if we have cleared existing concerns – thank you so much!

Thank you for your reply and continued discussion. We are happy we could clarify the causal interpretation of permutation in variance. In order to increase clarity, we will add an example in the Appendix as per your suggestion.

Thanks again for your time and engagement! We highly appreciate this opportunity to exchange opinions and discuss with you.

We thank the reviewer for their constructive feedback and insightful comments. This feedback helps strengthen our paper. Below, we address each of the concerns raised and provide clarifications.

W1: Assumptions and Contributions

Q: There is some exchangeability of these assumptions and in that sense I agree that the current assumption is a more practical one but it is not a novel one or a clear contribution until a clear relation between the assumptions is shown.

A: Thank you for your comments. We would like to emphasize the novelty of our work and its connections to previous studies below:

Most existing causal discovery methods assume the absence of causal relations between latent variables. Among the few methods that permit such relations, to the best of our knowledge, they are either limited to linear models [1][2] or deterministic mappings [3] (e.g., where $z = f(x)$ and $f$ is invertible). For instance, the method proposed by Kong et al. [3] requires that latent variables can be expressed as an invertible function of observed variables, which excludes even simple relationships like $X = \sin(Z) + \epsilon$ . These constraints significantly limit their applicability in real-world scenarios.

In contrast, our work introduces a more general and practical framework for modeling latent hierarchical graphs, relaxing the aforementioned restrictive assumptions. Specifically:

• General Latent Relations: Our framework supports non-linear and non-deterministic causal relationships, broadening the scope of latent structures that can be identified. This is a significant departure from existing approaches that are constrained by linearity or invertibility.
• Jacobian Rank Indicator for d-Separation: We establish a novel Jacobian rank indicator to characterize d-separation in latent hierarchical graphs. Using this indicator, we provide a rigorous proof of identifiability under our relaxed assumptions. To the best of our knowledge, this contribution is original and represents a non-trivial theoretical advancement in causal discovery.
• Practical and Theoretical Implications: By enabling the modeling of general latent hierarchical graphs, our work overcomes practical limitations in existing methods, paving the way for applications to more complex real-world datasets. Additionally, the Jacobian rank approach has potential for extension to other graph classes, opening new directions for future research.

We believe that relaxing restrictive assumptions and establishing identifiability for a broader class of latent graphs represent a significant and novel contribution to the field. These advancements address fundamental gaps in the existing literature and offer practical value for applications beyond current methods.

Q: The key claimed advantage for better identifiability results comes from the fact that instead it is assumed that "not yet account for structures where measured variables have children.

A: First we would like to clarify that our key contribution to the identifiability results is the development of a novel Jacobian-rank indicator for determining the number of d-separating latent variables in the non-linear case. This allows us to handle nonlinear, non-deterministic, and non-invertible latent causal relations (see Theorem 1 in Section 4).

Furthermore, these results can indeed be extended to account for structures where measured variables have children, as suggested by the work of Dong et al. [1], which builds upon Huang et al. [2] for linear models.

Dong et al. demonstrate that, with an appropriate indicator for the number of variables d-separating any two observed variables, it is possible to recover the causal graph even when measured variables have children, under weak structural assumptions. We acknowledge this as a potential extension of our work, which we briefly discussed in Section 7 of the manuscript.

W2: Experimental Evaluations

Real Data Evaluation

We thank the reviewer for appreciating the diversity of our evaluation methods. To further strengthen the evaluation, we have performed additional experiments:

• Added experiments on the CelebA dataset. Results are in Appendix B.2 Table 4.
• Incorporated two additional baselines: CausalVAE [6] and GraphVAE [7], which explicitly model latent causal structures. Results are in Section 6 Table 2.

Q1: Baselines

Our method addresses the identifiability of nonlinear latent hierarchical models, a setting with very limited comparable baselines. Most existing methods do not allow relations between latent variables or require strict assumptions. We have included the baselines which model latent hierarchical structures. These baselines are closest to our setting.

To provide a more comprehensive comparison, we have included DeCAMFounder [4] as a baseline in the updated manuscript. DeCAMFounder founder does not model latent hierarchical structures but learns causal graphs even in the presence of latent confounding. However, as it does not allow relations between latent variables, it performs poorly in our setting.

FOFC [5] is another causal discovery method which aims to discover causal structures with latent variables. FOFC does not allow relations between latent variables and requires each latent variable to have at least three pure children (in contrast we require only two). We attempted to compare with FOFC, but its strict requirement for three pure children per latent variable is not satisfied by our dataset.

Our approach shows substantial improvements in Structural Hamming Distance (SHD) and F1 scores over all baselines, confirming its effectiveness.

Please refer to the general response for more details on additional experiments.

Q2: Additional Visualizations and Plots

We appreciate the feedback on incorporating additional visualizations. To address this concern:

• We included a plot of performance vs. computational time for each method in Figure 2. However, we would like to clarify that most baselines compared to are not deep learning approaches since there do not exist any for latent hierarchical causal discovery.
• We included loss vs epoch plots in Appendix B.1 Figure 5 as suggested by the reviewer.

References

[1]Anandkumar, Animashree, et al. "Learning linear bayesian networks with latent variables." International Conference on Machine Learning. PMLR, 2013.

[2]Kummerfeld, Erich, and Joseph Ramsey. "Causal clustering for 1-factor measurement models." Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016.

[3]Huang, Furong, et al. "Guaranteed scalable learning of latent tree models." Uncertainty in Artificial Intelligence. PMLR, 2020.

[4]Adams, Jeffrey, Niels Hansen, and Kun Zhang. "Identification of partially observed linear causal models: Graphical conditions for the non-gaussian and heterogeneous cases." Advances in Neural Information Processing Systems 34 (2021): 22822-22833.

[5]Huang, Biwei, et al. "Latent hierarchical causal structure discovery with rank constraints." Advances in neural information processing systems 35 (2022): 5549-5561.

[6]Kong, Lingjing, et al. "Identification of nonlinear latent hierarchical models." Advances in Neural Information Processing Systems 36 (2023): 2010-2032.

[7] Gitter, A., et al. "Unsupervised learning of transcriptional regulatory networks via latent tree graphical models." arXiv preprint arXiv:1609.06335 (2016).

[8] Higgins, I., et al. "SCAN: Learning hierarchical compositional visual concepts." arXiv preprint arXiv:1707.03389 (2017).

[9] Liu, N., et al. "Unsupervised compositional concepts discovery with text-to-image generative models." Proceedings of the IEEE/CVF International Conference on Computer Vision (2023).

[10] Weinstein, E. N., & Blei, D. M. "Hierarchical Causal Models." arXiv preprint arXiv:2401.05330 (2024).

[11] O'Brien, K. L., et al. "Causes of severe pneumonia requiring hospital admission in children without HIV infection from Africa and Asia: The PERCH multi-country case-control study." The Lancet 394.10200 (2019): 757-779.

Please let us know if you have further concerns. We highly appreciate this opportunity to exchange opinions with you and learn from your perspective. Please kindly let us know your thoughts, and thank you again for your time and engagement!

We sincerely thank you for engaging with our rebuttal and participating in the discussion. We appreciate your time and valuable feedback. With the discussion period ending soon, we hope our response addresses your lingering concerns. We understand your busy schedule, but would greatly appreciate it if you could consider our updates when discussing with the AC and other reviewers.

Thank you again for your thoughtful and constructive input!

We thank the reviewer for their feedback and strong support of our work. We appreciate that the reviewer has recognized the originality and strong theoretical, methodological, and empirical contributions. Below, we address the concerns and provide clarifications.

Q1: Baselines for Colored MNIST Results

Thank you for highlighting this point. In the revised manuscript, we have included two additional baselines—CausalVAE [1] and GraphVAE [2]—that explicitly model the latent causal structure. These baselines provide a more meaningful comparison for our approach.

Additionally, we clarified that our aim is not to achieve state-of-the-art performance on the Colored MNIST task but to evaluate the transferability of our learned representations in comparison to other representation learning methods.

Q2: Discussion of Learned MNIST Graph

We have added a detailed discussion of the learned MNIST graph in Appendix B.2. This includes an analysis of Figure 4 and Table 3, clarifying how the latent variables align with an interpretable hierarchical structure.

References

[1] Yang, Mengyue, et al. "CausalVAE: Disentangled representation learning via neural structural causal models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[2] He, J., et al. "Variational autoencoders with jointly optimized latent dependency structure." International Conference on Learning Representations. 2019.

We hope these updates address the reviewer’s concerns. Please let us know if there are further points requiring clarification.

We thank the reviewer for their constructive feedback and insightful comments. We appreciate that the reviewer has recognized the novelty and significance of our theoretical results. Below, we address each of the concerns raised and provide clarifications.

W1: Experimental Limitations We have updated the experimental section of our manuscript in response to the concerns regarding the experiments. Please refer to the general response for details. In summary, we add experiments for each of the points the reviewer raised.

• Added experiments using tanh as a non-linear activation function, complementing the results using piecewise linear leakyReLU. Results are in Section 6 Table 1.
• Randomly generated a diverse set of DAGs and conducted experiments on these structures, providing robust and generalizable insights. Results are in Appendix B.1 Table 3.
• Integrated GraphVAE [9] and CausalVAE [10] as additional baselines, showcasing comparative performance on latent causal structure learning. Results are in Section 6 Table 2.
• Expanded the scope of real-world experiments to include the CelebA dataset for further validation. Results are in Appendix B.2 Section 4.

W2: Missing/Weak Motivation

We have clarified the applications of latent hierarchical causal models in the introduction of the revised manuscript. Below, we describe the importance of such models:

Learning latent causal models can address critical challenges in interpretability, distribution shifts, and scientific discovery [1]. Latent hierarchical models are particularly relevant in domains such as:

• Gene Regulatory Networks (GRNs): Gene expression data is observed, but transcriptional regulatory networks are latent [2].
• Image Data: Generative models for image data are hypothesized to be compositional and hierarchical with latent abstract concepts [3][4].
• Complex social systems: Hierarchical latent structures have been show to play a crucial role in understanding complex systems in political science and epidemiology. [5][6]

Although latent hierarchical causal models have tangible real-world applications, the theoretical understanding of these models has been underexplored. Existing works primarily demonstrate identifiability for linear, discrete, or deterministic models, making our contribution the first to establish identifiability for general nonlinear hierarchical latent causal models. Additionally, these methods use discrete search which is infeasible for high-dimensional data like images. We propose a differentiable approach and demonstrate that such latent representations can be learnt for high-dimensional data. While the true latent causal graph is unknown for real image data, our results showcase the interpretability and transferability of learned representations on datasets like MNIST and CelebA.

W3: Intervention Details

We have provided a detailed explanation of intervention data generation in Appendix B.2 under Visualization. To ensure clarity, we added a pointer to this section in the main text. Should we include a brief summary in the main text as well?

W4: References to Work in Causal Representation Learning

Thank you for pointing out relevant literature. We have updated the Related Work section to include and discuss [7][8][9][10].

Q1: Implication of Condition 3

Condition 3 ensures differentiability of the function ‘f’ and ‘g’ for theoretical results involving the rank of the Jacobian. This is a sufficient condition, though we do not believe it is necessary. Our proof (Appendix A.2) relies on the relationship J_f = J_h(g(x)) J_g(x), where p(z|x) = p(z|g(x)). While we use leakyReLU in experiments, which is not differentiable, we believe future work can build on our results to relax this condition.

Q 2 & 3:Typo in Eq. (8) and Mismatch with Eq. (6)

We apologize for the typo in Eq. (8). This has been corrected in the revised manuscript, resolving the mismatch with Eq. (6).

Q4. Caption for Figure 2c

The caption for Figure 2c has been updated for clarity. Please let us know if it remains unclear.

We look forward to your thoughts on our response. Let us know if there is anything more we can do to address your comments!

Thanks again for your time and constructive feedback!

Note: The test set in CelebA is highly imbalanced ($P(Y=1) = 0.97$). Consequently, we use AUC as the evaluation metric instead of accuracy, which can be misleading in such scenarios. For instance, GraphVAE achieves an accuracy of 0.97 simply by predicting $Y=1$ for every test point.

We note that many other baselines, such as [4][5] (highlighted by Reviewer wGhK) and others [6][7], require auxiliary information, multiple domains, or interventional data, making them unsuitable for comparison in our setting.

CausalVAE requires concept labels for identifiability as well. However, we were able to adapt their algorithm to run without concept labels. For GraphVAE, we could not find an official implementation from the authors, so we implemented the baseline ourselves. Despite this effort, we observed unstable training and generally poor performance compared to both CausalVAE and our proposed approach.

Motivation and Contribution of Our Work

Latent hierarchical causal models are found across various domains, including gene regulation, computer vision, political science, and epidemiology [8][9][10]. Despite their significance, there is a lack of theoretical understanding regarding the identifiability of these models in the general non-linear setting. Moreover, existing approaches for latent hierarchical causal discovery primarily rely on discrete search, which is computationally infeasible for high-dimensional data. Current causal representation learning methods often do not model hierarchical structures and require additional information, such as interventions or concept labels.

Our contributions are twofold: 1)We prove identifiability results for general nonlinear latent hierarchical causal models. 2)We propose a differentiable approach that scales effectively to high-dimensional data.

In the absence of ground truth causal structures for real datasets, causal discovery methods are typically evaluated using synthetic data. We follow this practice and demonstrate significant improvements over baselines on synthetic datasets. For real-world data, we indirectly validate the effectiveness of our approach by showcasing the interpretability and transferability of the learned representations.

References

[1] Agrawal, R., et al. "The DeCAMFounder: nonlinear causal discovery in the presence of hidden variables." Journal of the Royal Statistical Society Series B: Statistical Methodology 85.5 (2023): 1639-1658.

[2] Yang, M., et al. "CausalVAE: Disentangled representation learning via neural structural causal models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[3] He, J., et al. "Variational autoencoders with jointly optimized latent dependency structure." International Conference on Learning Representations. 2019.

[4] Brehmer, J., et al. "Weakly supervised causal representation learning." Advances in Neural Information Processing Systems 35 (2022): 38319-38331.

[5] Subramanian, J., et al. "Learning latent structural causal models." arXiv preprint arXiv:2210.13583 (2022).

[6] Zhang, K., et al. "Causal representation learning from multiple distributions: A general setting." arXiv preprint arXiv:2402.05052 (2024).

[7] Hyvarinen, A., Sasaki, H., and Turner, R. "Nonlinear ICA using auxiliary variables and generalized contrastive learning." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

[8] Gitter, A., et al. "Unsupervised learning of transcriptional regulatory networks via latent tree graphical models." arXiv preprint arXiv:1609.06335 (2016).

[9] Higgins, I., et al. "SCAN: Learning hierarchical compositional visual concepts." arXiv preprint arXiv:1707.03389 (2017).

[10] Weinstein, E. N., and Blei, D. M. "Hierarchical Causal Models." arXiv preprint arXiv:2401.05330 (2024).