Dear Reviewer 8ruG,

Thank you for your insightful suggestions to improve our work with a more extensive literature review, better defined limitations, and improved experimental evaluation! We hope that our response below will further reinforce your confidence in our work:

Extensions to MO-CBO

Our contribution is the development of a multi-objective variant of CBO, MO-CBO, for which we theoretically and empirically show superior performance compared to traditional MOBO. We introduce MO-CBO as its own problem formulation and our paper aims to provide a methodology to address this new type of problem. This includes a decomposition of the MO-CBO task along with establishing graph characterizations to identify possibly Pareto-optimal sets to intervene upon. Our empirical validation supports the theoretical claims.

Combining MO-CBO with existing CBO variants. There exist many CBO variants, each developed through its own dedicated research effort - examples include dynamic CBO, functional CBO, and constrained CBO. In this sense, our work stands as a distinct research contribution in its own right. We find it more appropriate to explore these integrations in future research.

Prior knowledge on the causal structures. Requiring prior knowledge about the causal graph is indeed a notable limitation of causal Bayesian optimization methods, including our own work. To better communicate this, we now added a limitation section (see our response to reviewer easG for more details on the applicability of our method under causal knowledge requirements). We would like to emphasize that investigating how MO-CBO performs under partial knowledge of the causal structure should be the focus of a separate research effort aimed at developing graph-agnostic (MO-)CBO methods (e.g. see Mukherjee et al., (2024)).

Multi-task GPs. Multi-task GPs are a possible technique to increase cost-effectiveness of MO-CBO. We had already discussed this at the end of the paper as a limitation of our work since we believe that an extensive study on multi-task GPs for MO-CBO is best dedicated to future work.

Literature Review

We do agree with you that our review of the current state of the field would benefit from a more thorough analysis of existing literature! Therefore, we included a review on multi-armed bandit problems as well as a more detailed description of existing CBO variants in order to give a better overview of the methods within causal decision-making. Moreover, we discuss the assumption of having fully vs. partial knowledge of the causal graph in the prior works.

Proofs

Intuition of proofs. Thank you for pointing this out! We did restructure some of the proofs to enhance readability - see our response to reviewer fTDx for a more structured proof of Proposition 4.2. However, we still believe that maintaining mathematical rigor requires some level of technical detail. That said, to enhance readability, we have now included a description of the main idea of each proof within 1-2 sentences in the main body of the paper. For instance, for Proposition 3.4 we write: “The rigorous proof of Proposition 3.4 is given in Appendix A, and it exploits the fact that the space of all intervention set-value pairs is the union of the input spaces of each local problem. This allows to match the Pareto optimal intervention set-value pairs with the Pareto-optimal solutions from the local problems where the intervention set is fixed.”

Proof of Theorem 4.8. In Theorem 4.8 we show that any intervention on $\mathbf{X}\_s$ is weakly-dominated by some intervention on $\textnormal{IB}(\mathcal{G}\_{\overline{\mathbf{X}}\_s}, \mathbf{Y})$. Thus, we did not construct an SCM here. We assume you mean Proposition B.2 or Proposition 4.7? We are happy to answer any remaining questions regarding these proofs!

Experiments and Baselines

Additional real-world experiment. Following your suggestion, we include one more real-world problem, where the SCM describes macro-economic relations based on real-world data (Höllig et al., 2023). Due to space limitations, we would like to point you to our response to reviewer 7sGg for preliminary results.

MOBO baselines. We expand our ablation study to include the MOBO algorithms DGEMO, TSEMO, ParEGO, MOEA/D-EGO, qParEGO and qNEHVI. We find that our method performs better than all baselines across all tasks. For more details (including preliminary results) we refer to our response to reviewer 7sGg.

Thanks again for your thoughtful comments - they greatly contribute to improving the clarity of our work! We hope our explanations have addressed your concerns, and we look forward to hearing back.

References

Mukherjee et al. Graph Agnostic Causal Bayesian Optimisation. In NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty.

Höllig et al. Semantic meaningfulness: Evaluating counterfactual approaches for real-world plausibility and feasibility, in: xAI, 2023, pp. 636–659.

Dear Reviewer easG,

Thank you for recognizing our contributions and for your insightful feedback pointing out the need to better discuss the applicability of our approach and adding further evaluations.

Applicability of MO-CBO

Requiring prior knowledge about the causal graph is indeed a notable limitation of current causal optimization methods and of our work. We acknowledge that this limitation was not highlighted sufficiently in the initial version of our paper. To better communicate this, we now added a limitation section.

Firstly, in many domains causal knowledge is readily available. For example, the effects of drugs are studied in experimental trials in the medical field. Knowledge about the effects, side-effects, and interactions is available and an example of causal relationships that can be used for CBO. Secondly, if causal information is not readily available, there exist methods to discover it, falling under the field Causal Discovery (Zanga et al., 2023). Here, the corresponding methods aim to identify causal structures from data.

Health Problem. We refer to the Health example as a real-world problem since the relations between the variables were derived from real-world analyses (Ferro et al., 2015). We agree that weight and BMI are not easy-to-treat variables. However, they are treatable under high costs. We increased the cost of intervention for the BMI and weight variables in our experiment and added a respective discussion. As a result, the distinction between MOBO and MO-CBO is even clearer, with MO-CBO demonstrating a significantly higher coverage of the causal Pareto front compared to MOBO. The IGD under MO-CBO is 0.02 whereas the IGD under MOBO is 0.05 in an output space ranging from values 0.05 to 0.4 for the Statin medication (thus the difference in IGD is not negligible) .

Economics Problem. We include one more real-world problem, where the SCM describes macro-economic relations based on real-world data (Höllig et al., 2023). Due to space limitations, we would like to point you to our response to reviewer 7sGg for preliminary results.

Graph Representation of Interventions

Regarding your question if intervening on the system would render a different causal graph: An intervention on variables $\mathbf{X}\_s \subseteq \mathbf{X}$ involves replacing the structural equations $f\_{X}$ with a constant $x$ for all $X \in \mathbf{X}\_s$, denoted $\text{do}(\mathbf{X}\_s = \mathbf{x}\_s)$. Thus, performing an intervention removes the influence of all variables on $\mathbf{X}\_s$. The graph $\mathcal{G}\_{\overline{\mathbf{X}}\_s}$ represents this intervention and is obtained by removing incoming edges into $\mathbf{X}\_s$. Therefore, $\mathcal{G}\_{\overline{\mathbf{X}}\_s}$ is used to denote the causal structure under intervention and to perform theoretical analyses regarding the effects of interventions. Importantly, the optimization problem itself remains unchanged.

Performance Metrics

We added the definitions of GD and IGD from Schulze et al. (2012). They are as follows: Let $\mathbf{A}$ be the set of points from an approximated Pareto front and let $\mathbf{Z}$ be the set of points on the true Pareto front.

GD. The GD is the average distance from any point $\mathbf{a}\_i \in \mathbf{A}$ to its closest point in the Pareto front $\mathbf{Z}$. Formally, $\text{GD}(\mathbf{A},\mathbf{Z}) = \biggl( \ \frac{1}{|\mathbf{A}|} \sum\_{i=1}^{|\mathbf{A}|} d\_i^p \ \biggr)^{1/p},$ where $d\_i$ is the Euclidian distance from $\mathbf{a}\_i$ to its nearest point in $\mathbf{Z}$; we set $p=2$ in our experiments.

IGD. The IGD measures the average distance from any point $\mathbf{z}\_i \in \mathbf{Z}$ to its closest point in $\mathbf{A}$. Formally, $\text{IGD}(\mathbf{A},\mathbf{Z}) = \biggl( \ \frac{1}{|\mathbf{Z}|} \sum\_{i=1}^{|\mathbf{Z}|} \hat{d\_i}^p \ \biggr)^{1/p},$ where $\hat{d\_i}$ is the Euclidian distance from $\mathbf{z}\_i$ to its nearest point in $\mathbf{A}$.

The GD evaluates the convergence of the approximated Pareto front to the true front, whereas the IGD measures the diversity of the solutions across the output space.

We also implemented your remaining suggestions and included the definition of the hypervolume in the Appendix. We hope our explanations fully address your concerns, and we look forward to your response!

References

Zanga et al. 2022. A survey on causal discovery: Theory and practice.Int. J. of Approximate Reasoning, 151 (2022), 101–129.

Schutze et al. (2012). Using the averaged hausdorff distance as a performance measure in evolutionary multiobjective optimization. IEEE T. on Evolutionary Computation, 16(4), 504–522.

Ferro et al. (2015). Use of statins and serum levels of prostate specific antigen. Acta Urológica Portuguesa, 32.

Höllig et al. Semantic meaningfulness: Evaluating counterfactual approaches for real-world plausibility and feasibility, in: xAI, 2023, pp. 636–659.

Dear Reviewer 7sGg,

Thank you for your thoughtful feedback and for recognizing our contributions! We have incorporated all of your suggestions to enhance readability, include additional baselines, and provide a clearer formulation of the causal Pareto front for the community.

Baselines

Thank you for your suggested improvements to our experimental analysis! We now include the MOBO algorithms DGEMO, TSEMO, ParEGO, MOEA/D-EGO, qParEGO, and qNEHVI as baselines. Additionally, we added another real-world problem, where the SCM describes macro-economic relations and is based on real-world data (Höllig et al., 2023). The average GDs are (qParEGO and qNEHVI are still being implemented):

Analyzing both GD and IGD, we find that our method performs better than all baselines across all tasks.

Experiments

IGD and GD values at zero interventions. Thank you for pointing this out - this is a x-ticks label error. Our algorithm is initialized with prior data that is used to compute a first estimate of the causal Pareto front before starting the optimization loop. We now label the start of the plots with len(prior data) = 5 to resolve this unclarity.

Similarity between MOBO and MO-CBO. It is theoretically grounded from Proposition 4.4 that both methods will converge to the same results when no $Y\_i$ is confounded with its ancestors via unobserved confounders, which is indeed the case for both Synthetic-1 and the Health example. In these scenarios, MOBO intervenes on all treatment variables $\mathbf{X}$, while MO-CBO intervenes only on the parents of $\mathbf{Y}$. Note that intervening on $\mathbf{X}$ yields the same results as restricting the intervention to the parents of $\mathbf{Y}$. As shown in Fig. 4, MO-CBO identifies a greater number of solutions by avoiding unnecessary interventions on elements outside the minimal intervention set defined by the parents of $\mathbf{Y}$. This approach reduces costs while achieving broader coverage of the causal Pareto front. However, when some $Y\_i$ is confounded with its ancestors via unobserved confounders, our method can establish superior results than traditional MOBO approaches as demonstrated in Fig. 6 for Synthetic-2.

Moreover, we added the definitions of our metrics as well as the runtime results to the Appendix. All methods, including ours, have an average runtime per iteration between 0.3 and 14 seconds. The fastest method is TSEMO with 0.2 - 4 seconds per iteration (depending on the problem) while ours ranges between 3 and 6 seconds.

General Discussion

Number of independent Gaussian Processes. The number of independent Gaussian processes (GP) $m$ is equal to the number of target variables, we describe this in the preliminaries of MO-CBO in Section 2. We train the GPs independently from each other, resulting in a linear increase of computational cost with $m$.

Proof Sketches. Thank you for this suggestion - we think this greatly improves the overall understanding of our methodology! We have now included a description of the main idea of each proof within 1-2 sentences in the main body of the paper. For instance, for Proposition 3.4 we write: “The rigorous proof of Proposition 3.4 is given in Appendix A, and it exploits the fact that the space of all intervention set-value pairs is the union of the input spaces of each local problem. This allows to match the Pareto optimal intervention set-value pairs with the Pareto-optimal solutions from the local problems where the intervention set is fixed.”

Causal Pareto front. In MO-CBO, the input space comprises all possible intervention set-value pairs. Based on proposition 3.4, the causal Pareto front is constructed by identifying the Pareto-optimal points across the solutions of local problems, each corresponding to a fixed intervention set. However, since Pareto front is a well-established concept in MOBO, we acknowledge that the phrase "causal Pareto front" can cause confusion. To clarify, the causal Pareto front simply represents the best attainable trade-offs within in a causal system, considering intervention set-value pairs as inputs. Based on your comment, we propose to rename the term to Pareto front in causal systems.

We have addressed all of your remaining comments as well as added the clarification on PSA (prostate specific antigen) and Statin (a type of medication) to the experiments section. Thanks again for your comments to enhance the clarity of our work! We look forward to your response.

Reference

Höllig et al. Semantic meaningfulness: Evaluating counterfactual approaches for real-world plausibility and feasibility, in: xAI, 2023, pp. 636–659.

Dear Reviewer fTDx,

We greatly appreciate your detailed feedback and your recognition of our contributions! We are happy to address your remaining questions below:

Proposition 4.2

Proposition 4.2 is based on Proposition 1 of Lee & Bareinboim (2018), which formalizes the concept of minimal intervention sets for graphs with a single target variable $Y$. We extend this result to causal graphs with multiple target variables $Y\_1,…,Y\_m$.

Your question refers to the “if” direction of the proof. Let us first state the definition of minimal intervention sets along with the proposition.

Definition 4.1 (Minimal intervention set). A set $\mathbf{X}\_s \in \mathbb{P}(\mathbf{X})$ is called a minimal intervention set if there exists no subset $\mathbf{X}\_{s}' \subset \mathbf{X}\_s$ such that for all $\mathbf{x}\_s \in \mathcal{D}(\mathbf{X}\_s)$ it holds $\mu(\mathbf{X\_s},\mathbf{x}\_s) = \mu(\mathbf{X}\_{s}',\mathbf{x}\_s[\mathbf{X}\_{s}'])$, $1 \leq i \leq m$, for every SCM conforming to $\mathcal{G}$.

Proposition 4.2 Given a causal graph $\mathcal{G}$, $\mathbf{X}\_s \in \mathbb{P}(\mathbf{X})$ is a minimal intervention set if and only if it holds $\mathbf{X}\_s \subseteq \textnormal{an}(\mathbf{Y})\_{\mathcal{G}\_{\overline{\mathbf{X}}\_s}}$.

Proof Strategy

The "if" direction of the proposition states: $\mathbf{X}\_s \subseteq \textnormal{an}(\mathbf{Y})\_{\mathcal{G}\_{\overline{\mathbf{X}}\_s}} \implies \mathbf{X}\_s$ is a minimal intervention set. To prove this, we use contraposition, meaning we will show the equivalent statement: $\mathbf{X}\_s$ is not a a minimal intervention set $\implies$ $\mathbf{X}\_s \not\subseteq \textnormal{an}(\mathbf{Y})\_{\mathcal{G}\_{\overline{\mathbf{X}}\_s}}$. We show this contrapositive via contradiction: Suppose $\mathbf{X}\_s$ is not a minimal intervention set, but assume for contradition $\mathbf{X}\_s \subseteq \textnormal{an}(\mathbf{Y})\_{\mathcal{G}\_{\overline{\mathbf{X}}\_s}}$. By definition 4.1, there exists a subset $\mathbf{X}\_{s}' \subset \mathbf{X}\_s$ such that for all SCMs conforming to $\mathcal{G}$, intervening on $\mathbf{X}\_{s}'$ yields the same outcome as intervening on $\mathbf{X}\_s$, i.e., $\mu(\mathbf{X}\_s,\mathbf{x}\_s) = \mu(\mathbf{X}\_{s}',\mathbf{x}\_s[\mathbf{X}\_{s}'])$, where $\mathbf{x}\_s [\mathbf{X}\_{s}']$ are the values of $\mathbf{x}\_s$ corresponding to $\mathbf{X}\_s \cap \mathbf{X}\_{s}'$. We construct a specific SCM that conforms to $\mathcal{G}$, meaning that we assign structural equations between the variables within $\mathcal{G}$. Then, we leverage these structural equations along with the assumption $\mathbf{X}\_s \subseteq \textnormal{an}(\mathbf{Y})\_{\mathcal{G}\_{\overline{\mathbf{X}}\_s}}$ to establish a contradiction, which is given by $\mu(\mathbf{X\_s},\mathbf{x}\_s) > \mu(\mathbf{X}\_{s}',\mathbf{x}\_s[\mathbf{X}\_{s}'])$, thereby breaking the equality. This invalidates our initial assumption, yielding $\mathbf{X}\_s \not\subseteq \textnormal{an}(\mathbf{Y})\_{\mathcal{G}\_{\overline{\mathbf{X}}\_s}}$. We have thus proven the contrapositive statement.

Generality. Since $\mathbf{X}_s \not\subseteq \textnormal{an}(\mathbf{Y})\_{\mathcal{G}\_{\overline{\mathbf{X}}\_s}}$ is a graph-topological property (i.e. it does not associate to any specific SCM), the above strategy completes the proof. Different SCMs cannot yield different graph-topological properties. The constructed SCM serves only as a tool to establish the desired result. We recognize that this aspect may not have been fully articulated in the original proof and have now revised it to improve clarity. While our proof required some technical modifications to the one by Lee & Bareinboim (2018), the core idea remains unchanged.

Thank you for your time and extensive review of all our proofs. We believe your questions and remarks greatly improve the understanding of our derivations for future readers! We hope this clarification fully addresses your concerns, and we look forward to your response.

References

Lee, S. and Bareinboim, E. Structural causal bandits: Where to intervene? In Advances in Neural Information Processing Systems, 2018.