W2: New Illustrations

• W2.1: We added a sub-new section in the appendix (C.2) containing a detailed description of the physical models being used by the DGP, along with appropriate references. Furthermore, we added new illustrative plots for the monitoring and conditional dependencies.

• W2.2: Added figures 4 and 5.

• W2.3: We agree with your suggestions on softening some claims, therefore:

  • Softened claim related to the use of expert knowledge for building the simulator.
  • Softened claim in line 512.

W3: Release of DGP

We are going to release the simulator code, as in the supplementary material there is only the code for running the experiments for effect estimation and causal discovery. With the new code it will be possible to generate new observational and interventional data (hard/soft, multiple, and hidden interventions).

Moreover, as written above, we tried to make DGP as clear as possible by adding a new section in the appendix C.2 where we describe all the causal mechanisms involved. Additionally, we have two new visualizations (Figure 4 and 5).

W4: Writing Quality

After reading the second part of the paper, we agree that the writing quality on the second part of the paper is definitely less polished. Apart from implementing all the typos you indicated, we corrected a few additional ones.

Corrections

A recap of the corrections we implemented:

• New section C.2 with description about structural equations.
• New Illustrations (Figure 4 and 5).
• Polished figures.
• Polished writing.
• Added references.
• Added SID for Causal Discovery on Causalman Small.

As soon as we have the results for the new Effect Estimation task, we will add it to the appendix.

We thank again for the review. Fee free to ask if further information is needed, or if you have any additional feedback,

Thank you for your extensive review and constructive feedback, which helped a lot in improving the manuscript, and for recognizing the effort that was necessary to integrate expert knowledge into the DGP, starting from firstly gathering domain knowledge from experts, to successively encoding it into a DGP based on SCMs.

We first start by addressing your questions/suggestions:

• Added references to "related work" section.

• Added to the draft a reminder that the simulator is based on SCMs, therefore inheriting all its underlying assumptions and limitations.

• Updated figures for the ground truth causal graphs, highlighting in red/orange visible variables, and in light blue the hidden ones. It is interesting to see how some portions of the causal graphs are hidden. The advantage of integrating expert knowledge into the simulator is also that we can model those latent structures, which we know are there exclusively because we have knowledge about the underlying physics.

• We confirm that yes, in the first task the target variable is in the markov blanket of the outcome variable, more precisely a parent. We remark that the gap in performance is only present for the first and easiest Effect estimation task where we are intervening inside the markov blanket of the outcome variable. Therefore, as you wrote, this behavior is expected for SCMs. On the second task, we are not intervening inside the markov blanket anymore, and indeed the gap disappears, making all models perform equally poorly. We added this remark to the main paper. The (deep) causal models we tested, upon further development towards scaling them to real world scenarios, have a bigger potential/margin for improvement. Indeed, on the second task methods based on Normalizing Flows (CAREFL and CNF) are achieving the most promising metrics (Table 3 and 4). Still, no model shows the capacity to leverage bigger amounts of data (See Fig. 2), apart from CAREFL (Slightly). The regression-based techniques, even though performing well on simple tasks, do not show any margin of improvement in complex tasks.

• Corrected the list of typos and wording. Additional ones have been corrected.

W1 to W4

Going on a deeper discussion on W1-W4:

W1: Experimental Section

We agree that there is a lot of potential, and that we should find the best way to express it. In order to enable researchers to get the most out of our simulator, we adopted the following measures:

• We will release the simulator code, enabling users to generate as mush observational and interventional (hard/soft, multiple, and hidden interventions) data as needed.

• We are going to add a new and more realistic task. The two tasks that we study in the paper are relatively simple because all causal models are already failing to provide meaningful results, so we did not try to push further with a more complex tasks. From a "stress-test" perspective, that is already enough to show many limitations of the tested models. However, we agree that this might be misleading for the reader, inducing to think that this benchmark is very limited. To address this, we are aiming to provide a more realistic task. Additionally, once the simulator will be publicly available, it will be possible to experiment with arbitrary new tasks.

• W1.2: Yes, in the simplest ATE task we are intervening on the parent of the outcome variable, therefore this behavior is expected in an SCM. We added this clarification to the manuscript in the new revised draft.

• W1.3: Added a clear statement that more results for causal discovery have been moved to the appendix due to lack of space.

• W1.4: We updated the figures, which now have more consistent and bigger labels.

• W1.5: The main concept is that there might be radical differences between different sections of the graph in terms of confounders, causal mechanisms and noise models. In our datasets (and also in other real world scenarios) we have some parts of the graph where almost all variables are observable, and other parts of the graph where variables are mostly hidden. Therefore confounded relationships are not "uniformly distributed", making some parts of the graph easier to discover with respect to others. A similar discussion can be made with respect to structural equations and noise models, as in some parts of the graph causal mechanisms are mostly binary and deterministic (The monitoring structure), while in others they are only nonlinear and noisy. We find your suggestions insightful, and agree that new metrics can give clarity, therefore **We evaluated the discovered causal graphs with SID on CausalMan Small, and added it to the new revised manuscript. We are also trying to implement new metrics from the second paper cited in your review (Code was not release for that paper). Given the size of the causal graphs, the s/c metrics were impossible to evaluate, as they scale exponentially, therefore we are trying to evaluate their approximation. **

(1/2)

We would like to express our gratitude for your constructive review and for increasing your score.

We added the results for the new task tp the manuscript. Indeed, the new results indeed show that on nontrivial tasks where confounders and more nonstandard causal mechanisms are present, linear regression is definitely the worst method overall, therefore supporting our statements in the main text. Other methods, instead, are not degrading as much as linear regression. We added the new results in the appendix (Fig. 8 and Table 4), and added remarks and references about it in the main text.

In this new task, we are intervening on the variable PF_M1_T1_Force, which describes the press-fitting force. The treatment is setting this force to an extreme value. Additionally, this variable has multiple bidirected edges with other variables describing the PF process, and it is also a direct parent of other variables. Therefore an extreme intervention can cause anomalies to propagate through the graph, influencing other physical quantities that depend on it (For example s_grad and F_max).

Additionally, even if we did not cover Root-Cause Analysis in this paper, we note how this simulator can model the propagation of production anomalies though the causal graph, therefore extending its applicability to RCA and Causal Explanations ("Why did this anomaly happen, and which are the responsible nodes for it?").

Lastly, we also added a new metric for causal discovery, namely the p-SD where we are counting the number of different separation statements (where S = Parents X Union Parents Y) between the two ground truth and discovered causal graphs. For the SID, it was possible to evaluate it only on the smaller dataset due to computational reasons.

Corrections

• Added new task.
• Improved the writing quality in the second part.
• Added SID and p-SD for causal discovery.

Thank you again. Feel free to ask if any additional suggestions or information are needed.

Thank you for your review. We understand your feedback and adopted appropriate measures, especially regarding the Data Generating Process (DGP).

Reproducibility and realism of the Data Generating Process

We built a simulator based on physical models derived from first principles. To concretely address your review, we added a new subsection in Appendix C where we describe in detail the causal mechanisms involved, including citations to the relevant literature from the field for how those models can be analytically derived.

Additionally, we will release the code for the simulator itself. Upon publication, It will be possible to generate observational and interventional data, including soft/hard interventions, multiple interventions, and interventions on latent variables.

To clarify our contribution, we list the main characteristics of this simulator and the research gap that we are trying to address:

• Higher level of datatype heterogeneity: Continuous, binary and categoricals are all present and play important roles in the DGP.
• Many Causal assumptions do not hold: Our DGP does not satisfy the typical causal assumptions such as Additive noise, linearity, or causal sufficiency. Identifiability likely does not hold anymore.
• Causal Mechanisms: Our simulator can offer causal mechanisms which are uncommon with respect to other publicly available DPGs. Among those, we model also Conditional Dependences (Sec. 4.1 and Appendix C.1), which can be seen as second-order uncertainties. Those kind of causal mechanisms result in multimodal and highly asymmetric marginal node distributions. Additionally, noise is often treated nonlinearly.
• Focus on Causal Insufficiency: Although possible to marginalise out variables in any DGP (when perfect knowledge is available), rarely benchmarks have a special focus on causal insufficiency, which can appear in real-world scenarios due to physical, economical or confidential reasons. Moreover, the choice of which variables are hidden and which ones are visible is given by the knowledge of the real production plants.
• Availability of Interventional data: We are going to release the code to arbitrarily generate as much interventional data as needed.

For what concerns the reproducibility of experiments, instead, we adopted all the necessary measures that are listed in the Reproducibility statement at the end of the main text. In the supplementary material you can find an automatic and configurable framework which can be used to run any of the presented experiments. The only limitation from that point of view is related to one of the core takeaways of the paper, that massive computational resources are needed for causal models such as Neural Causal Models or Causal Bayesian Networks.

Revisions

Finally, the following revisions have been implemented in the new draft:

• Add section to the appendix (C.2) describing Causal Mechanisms. Further, we clarified in the main paper that our simulator is based on physical models.
• Clarified our contribution and specified what is the research gap that we are addressing.
• Correction of typos.
• Updated most figures in order to make them more readable. Added new figures for illustrating different aspects of the DGP.

Lastly, we started the procedure to release the code for the simulator, and will make it public as soon as possible on a public anonymized repository. By doing so, we enable users to sample as much observational/interventional data as needed, including soft and/or multiple interventions. Interventions on latent variables will also be possible.

Thank you for your review and your feedback. We address below your points:

Measurement Error: No, we do not model directly the measurement error in the causal graph, as we are modeling quantities which can be measured accurately, making the measurement error negligible in practice. Additionally, the probability of failure for those sensors is also typically low, if proper maintenance is done. If we were modeling other production lines with different underlying physical mechanisms where accurate sensors are not available, the answer could have been different. As a practical example: When dealing with product testing such as pressure tests, leakage tests, or similar, modeling the measurement error is of crucial importance.

Ground truth and Data Generating Process: Our simulation has been built starting from physically justified structural equations that are derivable from first principles. Therefore, the ground truth has been assembled by integrating expert knowledge about all the physics underlying the production process. We added appendix C.2 where we make an extensive description of the physical models, including citations to the relevant literature. This approach is for sure time consuming, as it required to write down every structural equation, although there are repetitions. When modeling physical systems, we remark that this is often the only way possible, as some quantities may be hard or even impossible to measure for physical/economical/confidential reasons. In appendix Sec.G we can see from the ground truth causal graphs how some causal structures can be present while being completely latent. Moreover, we will release the code containing the DGP, which will enable researchers to generate as data as needed, including interventional data (including hard/soft interventions, multiple interventions and interventions on hidden variables).

Best Causal Discovery algorithm: From our tests we see a significant trade-off, where the best performing methods on the "small" scale can be among the worst on the large scale. The PC algorithm, indeed, showed the most robust behavior among the tested methods on CausalMan Small, especially in terms of precision and recall. Additionally, as in real world data, our data exhibits node distributions which can be asymmetric and multi-modal due to conditional dependencies (See Appendix C.1), which favors constraint-based methods.

Moving to CausalMan medium, we see that ML-based methods are advantaged overall, both in terms of performance but also runtime. However, even the best performing methods are still very distant from providing satisfactory results that can be used in any real downstream tasks. Therefore we consider Causal Discovery at the large scale still an entirely open research problem. We also remark that the amount of data is an important factor, as we can see in Fig. 6 in the appendix. In that figure, we can see that the PC algorithm is strongly disadvantaged in the big-data regime.

NOTE: Data was normalized in every experiment, making it hard for continuous optimization-based methods to exploit patterns of increasing marginal variance (See the concept of varsortability in "Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Game" from Reisach et al.2021).

Thank you again for your review, we updated our manuscript with a description of the DGP. Please feel free to ask if any additional clarifications are needed or if there are further suggestions.

Contribution of our simulator

As written above, in order to sustain our statements about the realism of the DGP, we added a new section in the appendix (C.2), where we justify at a physical level the underlying DGP. We will highlight also here our main contributions and research gap that we want to address with our simulator and benchmark:

• Higher level of datatype heterogeneity: Continuous, binary and cate egoricals are all present and play important roles in the DGP.

• Many Causal assumptions do not hold: Our DGP does not satisfy the typical causal assumptions such as Additive noise, linearity, or causal sufficiency. Identifiability likely does not hold anymore.

• Causal Mechanisms: Our simulator can offer causal mechanisms which are uncommon with respect to other publicly available DPGs. Among those, we model also Conditional Dependencies (Sec. 4.1 and Appendix C.1), which can be seen as second-order uncertainties. Those kind of causal mechanisms result in multimodal and highly asymmetric marginal node distributions. Additionally, noise is often treated nonlinearly.

• Focus on Causal Insufficiency: Although possible to marginalise out variables in any DGP (when perfect knowledge is available), rarely benchmarks have a special focus on causal insufficiency, which can appear in real-world scenarios due to physical, economical or confidential reasons. Moreover, the choice of which variables are hidden and which ones are visible is given by the knowledge of the real production plants.

• Availability of Interventional data: We are going to release the code to arbitrarily generate as much observational and interventional data as needed. It will be possible to generate hard/soft interventions, multiple interventions, and interventions on latent variables.

Corrections:

The following corrections have been implemented in order to address your feedback:

• Added new section in Appendix C.2, where we describe all causal mechanisms involved, plus illustrations (Fig 4 and 5) about the most characteristic ones.
• Our contribution has been written more precisely in the "contribution" paragraph in the introduction.
• Specified why on the first task we are getting those results for linear regression.
• Added missing reference.

Lastly, we will add the requested visualization in the appendix in the next revision and, as suggested by reviewer PVqx, we are working towards adding a new effect estimation task. Moreover, we remark again that the code for the simulator will be made available, allowing researchers to generate new observational and interventional data, including hard/soft interventions, multiple interventions, and interventions on latent variables.

Thank you again for your review. Please feel free to ask if any additional clarifications are needed or if there are further suggestions. (2/2)

Thank you for your review, we appreciate your feedback and observations.

First of all:

• We added the missing citations. Thanks for pointing out to this new reference.
• In table 2 the entry for MMD for Causal Normalizing Flows is Nan, but not by accident: When scaling those models, training instabilities did often appear. At sampling time, the model did uniformly output more than one variable with diverging values (>>1e14), yielding the MMD computation to output Nan.

Regarding the DGP and the performance gap for linear regression will be instead addressed in the section below.

Data Generating Process

Our simulation has been built starting from physically justified structural equations that are derivable from first principles. Therefore, the ground truth has been assembled by integrating expert knowledge about all the physics underlying the production process. For a detailed explanation of every structural equation, including illustrations about the main mechanisms, we added Sec. C.2 in the appendix where we describe the underlying DGP, with citations to the relevant literature. Moreover, the code to generate new observational/interventional data will be released. We hope that all this new information will sustain our claims about our results, and about the realism of our simulator. Additionally, we highlighted more concretely our contribution in the "contribution" and "related work" sections.

In connection with the performance of linear and logistic regressions, We start by remarking that linear regression is competitive only on the first and simplest task, and degrades to a level comparable to other methods in the second task. We did choose to keep the linear regression because it is the simplest and most interpretable baseline possible, even if it does not reflect many of the assumptions of the underlying DGP. Moreover, it is a common baseline against which causal models are often tested.

Exploring deeper its performance on the first task, we start by remarking that target and outcome variables are both binary, and the structural equation is a logic AND between all the (binary) parents, which is a nonlinear causal mechanism. Moreover, we are intervening on a parent node, which is on the markov blanket of the target variable. Therefore, this behavior is expected when working on data generated from a SCM, as we are modeling a direct causal relationship.

For the second task, both variables are still binary, but the target is now a grandparent of the outcome node, which is not in its markov blanket anymore. In this task, we can already see in Table 3 and 4 that the linear regression does not have any advantage anymore.

Lastly, to address your question regarding the use of MSE for logistic regression, we remark that we used the model only for treatment effect estimation, and not for estimating interventional distributions. Therefore we used the MSE to measure the distance between the estimated ATE and the ground truth ATE, which are both scalars.

(1/2)

Dear reviewer,

We follow up on our past message: To present a different perspective on why a simple linear regression is outperforming all other methods (only) in the first ATE task, we added a new figure( Fig.11). Now we can observe the interventional distributions that the deep causal models (+ Causal Bayesian Networks) have learned. Similarly to what is suggested in Fig. 2b, we can see also in Fig.11 that these models are typically not consistent in predicting interventional distributions, and hardly a model can estimate accurately BOTH treated and control population. This leads to a decrease of ATE performance.

We think that results are much more reasonable if we also take into account the performance in the new third ATE task suggested by reviewer PVqx. In this new task, we are intervening on the node PF_M1_T1_Force. This variable has multiple confounded relationships with other variables describing the PF process, and it is also a direct parent of other variables. Furthermore, the path from the target to the outcome variable now has multiple nodes in between. What we can concretely observe is that, on a harder task, results are more consistent with what would be expected from those models. In this new task, indeed, linear regression is clearly disadvantaged, and other models instead perform much better at modelling nonlinearities and confounding effects. See Fig.10 and Table 4 in the appendix for the new results and plots.

We hope that this clarified further your questions, and feel free to ask for any additional clarification.

Dear reviewer,

We have ~27 hours left for answering your messages, and we are still available to address any of your feedback or to provide any additional clarification, if needed. Moreover, a summary of the revisions has also been posted as a general comment.

In case you think that your feedback was thoroughly addressed, we kindly ask if you could reconsider your score.

Dear reviewer,

Thank you for your answer.

We agree with you that designing a benchmark is always challenging, as it requires a validation procedure which is not trivial. To explain our methodology, we think it is necessary to also articulate our requirements. We built this simulator with the expectation to compensate some areas where other simulators and datasets were not meeting our needs. What breaks the circular argument is that the benchmark was not intentionally created to show how limited current methods are. The simulator was built to meet a list of requirements which we think are missing in other benchmarks. From those requirements, we did create a simulator. What we show in the paper is that current methods do not perform well when we deal with data having those characteristics.

In other benchmarks, what we found limiting was the scarce amount of datatype heterogeneity, the type of causal mechanisms (and noise models), and the focus on causal sufficiency. To answer your doubts about our methodology, we will focus on the last point.

Many simulators ignore that the majority of variables are latent, and they often try to build a "causally sufficient" version of the real data, discarding all external confounding effects. In our case, we needed a grounded way to model what happens on hidden nodes, and how they influence observable ones. In the simulator, some parts of the causal graph are there only because we know the underlying physics, but they are never observable in the real world. We managed to do this by integration of expert knowledge. For example, the computation of the effective elasticity of a Magnetic Valve is hidden, but we can still manage to model it in the SCM by deriving the underlying physical model using the theory of elasticity. In this scenario, a quantitative estimate is extremely hard -if not impossible- to carry on without heavy (and unrealistic) assumption on the DGP. We want to compensate this limitation by using physical models derived from first principles (Sec. C.2).

Indeed, to bypass this step, we pursued what is typically considered standard practice in numerical mathematics, physics, and simulations. For example, in Computational Fluid Dynamics, simulations are mostly performed starting from some system of ordinary/partial differential equations (Navier-Stokes, Laplace equation, et similia) derived from the fundamental laws of dynamics which we assume to be true, and by successively applying some form of discretization.

Therefore, what we might conclude is that we used a methodology which is not common in the Machine Learning field, as it is less data-driven than other approaches, but which we still consider concrete and highly based on our fundamental knowledge of the system we are modeling.