Sanity Checking Causal Representation Learning on a Simple Real-World System

Thank you for your review. We respond to your points below.

Motivation & differences to other benchmarks: As all other reviewers highlight and we stress in the paper, the primary distinction of our sanity check is that it stems from a real, physical system to evaluate the practical validity of assumptions:

• Rev. DQvb: "[evaluating on synth. data] provides further validation for the theoretical foundations of [CRL] methods but yields limited insight into their applicability to real-world problems. This point seems to be lost on many researchers, and the authors of this paper place it front and center. Bravo."
• Rev. KaPR: "the data is highly desirable and useful for CRL evaluation" -Rev. 55Kr : “I believe this is the first benchmarking of CRL algorithms on data generated by a physical process.” and “would be a much preferable benchmark for practical purposes than what is currently in the literature”

Key messages: You say “It would be better to formulate the underlying research problem and the concerned factors that this benchmark is designed to study and control the other factors more rigorously.” We believe we have clearly explained the factor we study (the mixing function), we rigorously control the other factors, e.g., generating the latents according to each model’s assumptions, and made very clear what conclusions can be drawn. Citing Rev. DQvb: “The claims are clearly delineated, and the authors take some pains to make clear what their "sanity check" cannot do”

Regarding failure in the optimization routines, we have performed additional experiments to study this mode of failure. See the answer to reviewer Rev. 55Kj for a draft of what will be included in the final version.

MCC: As Rev. DQvb points out, we aim to “meet each method on their own grounds”, using the same validation metrics as in the original papers. As we stress, this is a sanity check, and our focus is not to compare methods to each other (difficult given their different goals). We agree that the MCC is not an ideal metric for CRL, but it is the most commonly used one and we are not aware of a metric that directly addresses its shortcomings. It is well beyond the scope of this study to derive a novel metric.

Systematicity:

• Measurement error: The work you reference considers measurement errors for causal discovery, which we do not directly see being applicable to CRL. While CCRL does not explicitly consider measurement noise, this is no reason to exclude it from the sanity check. Measurement noise is inherent to any real-world data; the failure of this method points to an important issue in realistic scenarios that should perhaps be considered by future works.
• Other failure modes: As you point out, many things can go wrong—beyond the mixing function—when transferring CRL from theory to practice. However, we argue that focusing precisely on this single factor (as opposed to considering everything at once) is what gives our study systematicity (see answer to Rev. 55Kj). As Rev. 55Kj points out, we meet all other assumptions of each method to isolate and better understand the effect of misspecified assumptions regarding the mixing function. Given the subsequent failure of methods after only this single (!) misspecification, we believe our setting already highlights a problem worth discussing. Including more factors of analysis is of course always possible, but (1) it is beyond the scope of such a paper and (2) it dilutes the point of focusing (and understanding) the failure mode in such a simple setting.
• As you point out, this work is a sanity check and not a benchmark that aims to exhaustively study any possible mode of failure. We stress this throughout, as other reviewers (DQvb) also acknowledge. CRL is far from being practically applicable, and our sanity check is meant to highlight the “remarkable failure of SOTA systems in such a simple setting” (Rev. DQvb), rather than comparing working approaches.

Method choice: Any choice will exclude other methods, we don’t claim exhaustiveness, nor do we draw conclusions from these representatives to their whole family of methods. Our results highlight their failures and that this is worth looking into. Citing Rev. 55Kj: “[it] convinces me that existing CRL algorithms, at least out of the box, are inadequate for practical use”

We chose our representative methods for the following (often pragmatic) reasons:

• CCRL: allows for general nonlinear mixing functions (cf. Ahuja et al. 2023, Zhang et al. 2023 with polynomial mixing, Squires et al. 2023 with linear mixing) and uses a contrastive loss, which has proven superior to an encoder-decoder in other representation learning domains.
• Multiview CRL: multiview approach that permits more than two views and considers nonlinear mixing (on images), with accessible code.
• CITRIS: considers complex nonlinear mixing experiments (images), very well maintained code base.

Technical novelty: See our answer to Rev. KaPR.

Thank you for your careful review and constructive feedback.

Re: Gamella et al., 2025: Thank you for raising this point. As other reviewers (KaPR, uuRR) also pointed this out, we will add a paragraph text in section 2 to clearly separate our contributions from Gamella et al., 2025. Please see our response to reviewer KaPR below for the details of our contribution w.r.t. those of Gamella et al., 2025.

Re: additional methods: We agree in principle that running our sanity check on more methods would be insightful. However, our approach of meeting methods in their own ground (as praised by Reviewer DQvb)—using the original code and evaluation techniques—meant that getting a method to work takes a significant amount of time and effort. We can use the extra page for this, but given how difficult it was to get the current methods to run (incl. communicating with the authors + tests to ensure no bugs) we don’t want to promise that we can do this for Zhang et al., 2023 or Ahuja et al., 2023. It is definitely not feasible before the rebuttals are due. We hope others will use our work to investigate individual methods in full detail in future works.

Re: optimization: Thank you for raising this point; we find your categorization of failure modes (misspecification vs. optimization) very useful. We fully agree that the latter is often overlooked and that an additional discussion will improve the paper. To support it:

1. We have now performed additional experiments doing a large hyperparameter search for each method on the real data. The results (see here) show that the hyperparameter choice does not significantly affect the final metrics, across all models. We will repeat the experiments on the synthetic data for the camera-ready version as time constraints did not permit doing so for the rebuttal.

2. We will add the training curves of all methods (see here) to the appendix. From them, we conclude that training does converge (although noisy for some methods, cf. Multiview CRL), not raising immediate concerns about optimization failing in some catastrophic way.

With the above we hope to address (and exclude) some of the key potential failure modes regarding optimization, i.e., losses not going down during training or hyperparameter choices greatly affecting the outcome. Our hyperparameter search was possible because we have access to ground truth labels to compute the final metrics and perform tuning. However, in a practical setting without ground truth, model selection of unsupervised models is still a great challenge, and finding the modes of failure w.r.t. optimization would be even more difficult.

In addition, we put in a substantial effort to exclude bugs and other issues by performing preliminary checks to reproduce the synthetic experiments in the original papers using our pipeline, as well as consulting the authors of the methods to advise us, where possible.

Regarding a misspecification in the generative process: to isolate the mixing transformation as the only source of mismatched assumptions, we generated the data closely following the assumptions for each method. We do not merely follow the general assumptions, we also closely replicated the more subtle details of the respective methods’ data generating processes, such as no. of latent variables, noise mean and variances, intervention strengths etc.

To summarize:

• CCRL appears to fail because of a misspecification of the generative process (noiseless vs. noisy mixing function), since the method works well for the synthetic ablation but fails on the real data.

• Because the other methods fail on both real and synthetic data, it seems like failure is instead also due to optimization issues. Because the training curves converge, and the performance does not change given different hyperparameters, we argue that there are either architecture choices that need deeper investigation, or finite sample issues that should be better understood. All identifiability results in CRL rely on the unrealistic assumption of infinite data, but how much data is enough in practice remains elusive. We collected as many samples for each method as was possible given the scope of this study, but because there are actual costs to collecting real data samples, these still fall short of the enormous sample sizes in the original papers: CCRL 10k vs 25k (per env.), Multiview 60k vs 145k, CITRIS 100k vs 100-250k.

We will add all the above as additional discussion using the extra page in the final version. This will be framed (as you suggest) by an explanation for general audiences of how the theoretical guarantees of CRL methods work and how the two possible modes of failure result from this.

Thank you very much for your positive review! We answer your points below.

Re: bayesian: We have fixed this in an updated version of the manuscript, thank you.

Re: Contributions w.r.t.. Gamella et al., 2025: Indeed, Gamella et al., 2025 introduce the light tunnel. In our work we leverage it to construct a meaningful benchmark for CRL. This required an in-depth analysis of the experimental setup in light of CRL assumptions (section 2), designing new experiments and collecting the data, and developing the deterministic simulators for the synthetic ablation. None of these existed in Gamella et al., 2025. The ICA experiment in Gamella et al., 2025 could also be used as a first test for CRL, but since the latent factors are all independent this didn’t seem very interesting. We will add a paragraph in section 2 to clearly separate our contributions.

Thank you for your positive review!

We wanted to be cautious about making unsubstantiated claims regarding the failure points of the algorithms. Due to their complexity (especially Multiview CRL and CITRIS), understanding the exact source of failure may be very difficult, and it may not be possible to pin it down to a single assumption or implementation choice.

However, following your feedback and that of reviewer 55Kj, we have performed additional experiments (hyperparameter searches) for all considered methods and, if accepted, we will add further discussion using the extra page. You can see a draft of this discussion and the supporting experiments in our answer to reviewer 55Kj below (we don’t repeat it here for lack of space).

Thank you for raising the above point; we agree that such a discussion can improve the quality of the paper.