Counterfactual Effect Decomposition in Multi-Agent Sequential Decision Making

Thank you for your valuable feedback and positive score. We are glad to see that you find our paper well-written and useful for multi-agent systems, and our discussion of related work good. Please find below our response to your comments and questions. For references, please see our general comment.

Response to Comments and Questions

Some of the notation could be better explained. For example ... in some of these expectations. Could you please clarify the notation I mentioned in Weakness 1) above? (Just to be sure I understood the paper as I think I did!)

Thank you for your feedback (from Weakness 1) and your question. Our intention is indeed to make the paper accessible to researchers who may not be fully immersed in the intersection of counterfactual reasoning and decision-making. While we did explain the meaning of the subscript $M$ in Section 2.3 of the original manuscript, we recognize that we did not explicitly clarify the notation $P(\cdot|\tau;M)$ as you pointed out. This was an oversight, and we have now addressed it in the revised manuscript. Thank you for bringing this to our attention.

Regarding your clarification question, $P(Y=y|\tau;M)\_{M^{do(I)}}$ denotes the counterfactual probability of $Y$ taking the value $y$ in the modified model $M^{do(I)}$, conditioned on $\tau$ having been realized in $M$. This is equivalent to $P(Y_{do(I)}=y|\tau)\_M$. We choose the former notation because we believe it allows for a more clear distinction between quantities such as $P(Y=y|\tau;M)\_{M^{do(I)}}$ (e.g., in Definition 3.1) and $P(Y_{a_{i,t}}=y|\tau;M)\_{M^{do(I)}}$ (e.g., in Definition 3.2).

Additionally, addressing a related comment from Reviewer N9FM, we have also included a table in Appendix B summarizing all the causal quantities and key notations introduced in our paper to further enhance its clarity and accessibility.

Both experiments considered only two interacting agents simulated over relatively short time horizons, and the approach isn't demonstrated on larger-scale systems. I appreciate that this is explicitly discussed as a limitation in the Discussion section, however.

Thank you for your feedback. We recognize that evaluating and improving the scalability of our approach would be important for its practical applicability in systems with larger numbers of agents and longer horizons. However, we view this as an independent research challenge that, while important, lies somewhat outside the main scope of this paper. Our primary focus here is to provide an interpretable solution to the problem of counterfactual effect decomposition in multi-agent MDPs.

Even though, there are plenty of interesting multi-agent settings with only a few agents, e.g., AI assistants or AI supervision, we definitely believe that empirically evaluating, within our framework, the mitigation strategies against computational complexity proposed in our Appendix, constitutes a meaningful future research direction.

Lastly, as a point of clarification, our Gridworld environment involves three agents: an LLM Planner and two RL actors.

To what extent does the complexity of the agents ... it is easier to scale up these scores to larger numbers of agents?

This is a very interesting point of discussion. The answer to your question is yes, if increased agents' capabilities implies increased inference time. The reasoning is the following: our approach to estimating counterfactual effects involves sampling trajectories from the posterior distribution and then averaging the values of the response variable across these trajectories. Sampling a trajectory from the posterior distribution generally requires to prompt each agent once for every counterfactual state in which they need to act.

For reference, in the Gridworld environment, more than $90$% of the time required to sample one counterfactual trajectory is spent on the inference of the LLM agent, while the remaining $\sim10$% is shared between the two RL agents (raw values and additional details on the time compute of our experiments are included on the README file of our code). Consequently, if we were to use an LLM agent with reduced cognitive capabilities, and hence less inference time, then the scalability of our approach in this experiment would significantly improve.

We have added this discussion to Appendix I of our revised manuscript. Thank you for the insightful observation.

Conclusion

We thank you again for your comments and questions. We would be happy to answer anything else in addition.

Thank you for your valuable feedback and positive score. We are glad to see that you find our paper well-written and well-structured, and our work clearly motivated and with significant relevance to causal attribution. Please find below our response to your comments and questions. For references, please see our general comment.

Response to Comments and Questions

The work appears to be a direct extension ... it would be beneficial to further emphasize how the authors handle these complexities.

Thank you for your comment, please let us address it step by step.

Total agent-specific effect. Our approach to decomposing the total ASE, termed ASE-SV, builds on the notion of agent-specific effects introduced by [5]. Fortunately, ASE is already well-defined in the MMDP setting, allowing us to leverage this concept directly. However, it is crucial to clarify that simply summing the agent-specific effects of individual agents does not yield the total ASE. In fact, our experiments in the Sepsis environment reveal discrepancies of up to 95% in certain scenarios. To derive an efficient attribution method, we formulated the problem as a cooperative game [7]. This formulation enabled us to define Shapley value in the setting of agent-specific effects. Furthermore, in Section 5 and Appendix E, we explain and formally define in this context a set of well-known desirable properties uniquely satisfied by Shapley value.

Reverse state-specific effect. Our approach to decomposing r-SSE, termed r-SSE-ICC, builds on the concept of intrinsic causal contributions (ICC) introduced by [8]. ICC is an information-theoretic measure quantifying how much an observed variable contributes to the uncertainty of another variable in a general SCM. We argue that extending ICC to address the attribution of uncertainty in a complex counterfactual effect, such as r-SSE, to the state variables of an MMDP-SCM is not a straight-forward process. A challenge specific to MMDP-SCMs, for instance, is that apart from state variables also action variables may directly or indirectly influence the variation in r-SSE.

General comment on contributions. We acknowledge that the theoretical results in this paper, i.e., Theorem 3.3 and Theorem 4.2, are not challenging to prove with the appropriate technical background. However, we view our work more as foundational, addressing a newly proposed and complex problem that is highly relevant to accountability in multi-agent AI systems. The primary contribution of this work lies in integrating concepts from a wide range of fields, including multi-agent sequential decision making (MMDPs), counterfactual reasoning (ASE), mediation analysis (Theorem 3.3), cooperative game theory (ASE-SV), information theory (r-SSE-ICC) and more, to tackle this novel problem. Finally, the empirical evaluation of our proposed solution demonstrates that it is interpretable and aligns with standard intuitions. This validates the potential of our approach to inform future research in this area.

The paper addresses the computational complexity ... time-accuracy tradeoff would strengthen the discussion.

First, we would like to thank you for taking the time to review the (optional) Appendix and for your feedback. We recognize that evaluating and improving the scalability of our approach would be important for its practical applicability in systems with larger numbers of agents and longer horizons. However, we view this as an independent research challenge that, while important, lies somewhat outside the main scope of this paper. Our primary focus here is to provide an interpretable solution to the problem of counterfactual effect decomposition in multi-agent MDPs.

To address the second part of your question, the mitigation strategies proposed in our Appendix are grounded in theoretical considerations and have not yet been tested empirically within our setting. Even though, there are plenty of interesting multi-agent settings with only a few agents, e.g., AI assistants or AI supervision, we definitely see testing these strategies empirically as a meaningful future research direction.

Lastly, as a point of clarification, our Gridworld environment does involve three agents: an LLM Planner and two RL actors.

Thank you for your valuable feedback and positive score. We are glad to see that you find our approach reasonable and technically sound, and our simulations comprehensive and supporting our algorithms. Please find below our response to your comments and questions. For references, please see our general comment.

Response to Comments and Questions

The identification of the proposed counterfactual measures relies on the additional parametric assumption of weak monotonicity. This assumption is restrictive and may limit the paper's application to many practical domains.

We acknowledge that causal assumptions, such as weak noise monotonicity, albeit necessary for counterfactual identifiability may indeed restrict the applicability of our effect decomposition approach in more practical domains. This is a common trade-off in counterfactual reasoning, as highlighted in the references from our Discussion section.

We have conducted additional experiments to evaluate the robustness of our results to the noise monotonicity assumption. A concise summary of these experiments is included in our General Comment (Additional Experiments Evaluating Robustness to Noise Monotonicity), with the complete set of results detailed in Appendix M of the revised manuscript.

The paper is quite notation dense. It would be best improved by including a table summarizing all the proposed measures and necessary notations.

Thank you for your suggestion. In response, we have added a table in Appendix B of the revised manuscript that summarizes all the introduced causal quantities along with the key notations used in the paper. Additionally, we have included a reference to this table in the main text to guide readers more effectively.

The introduction of each new counterfactual measure ... of the proposed counterfactual measure. Can the authors provide more detailed examples for each proposed measure? For instance, can the authors explain these measures and the decomposition in the context of Figure 1 or Figure 2a? What causal mechanisms are these metrics evaluating, and what are their implications?

Thank you for your suggestion. We would like to kindly point out that our (submitted) manuscript already explains in Section 6.1 all the introduced counterfactual measures, alongside the metrics they are evaluating, in the context of Figure 2a, particularly in the paragraph titled Counterfactual effects. Additionally, in the subsequent paragraphs of the same section we present the outputs of our decomposition formula for the considered simulated scenario.

In response to a related comment made by Reviewer NMck, we have included in our revised manuscript a new paragraph at the end of Section 3, where we revisit and highlight the subtleties of all causal quantities from that section.

Conclusion

We thank you again for your comments and questions. We would be happy to answer anything else in addition.

Thank you for your response and insightful suggestion.

Thank you for your valuable feedback and positive score. We are glad to see that you find the exposition of our paper clear and our development of mathematical tools principled. Please find below our response to your comments and questions. For references, please see our general comment.

Response to Comments and Questions

The mathematical notation are quite dense and at times a little hard to grasp. For example, Def 2.1, 3.1, 3.2 all look similar but clearly they are different. The authors included textual descriptions of these mathematical definitions (individually) which is great, but I would appreciate a paragraph comparing these definitions and highlighting the differences (for example, $Y$ vs $Y_{a_{i,t}}$).

Thank you for your feedback. In response, we have added a dedicated paragraph in Section 3 of the revised manuscript to incorporate your suggestion. We would also like to point out that Section 6.1 contains a similar discussion, which uses the Gridworld environment to provide a visual comparison of these definitions and better illustrate their subtleties.

Additionally, based on a related comment from Reviewer N9FM, we have also included a table in Appendix B summarizing all the introduced causal quantities and key notations to further enhance the clarity and accessibility of the paper.

Percentage conversion: Theorem 3.3 suggests the "total counterfactual effect" can be decomposed into "total agent-specific effect" minus "reverse state-specific effect". How are the two parts converted to a percentage contribution, for example in the pie chart in Fig 1b? It might be useful to show the raw effect values as well.

Thank you for your suggestion. In this simulated scenario, the total counterfactual effect (TCFE) is equal to 0.82. The total agent-specific effect (tot-ASE) is approximately 0.374, and the negative of the reverse state-specific effect ($-$r-SSE) is approximately 0.446. Consequently, our method attributes $\frac{0.374}{0.82} = 45.6$% of the TCFE to how the two agents would have responded to the intervention, while the remaining $\frac{0.446}{0.82} = 55.4$% is attributed to the intervention's influence on the patient state. Following your feedback, we have updated the manuscript to include the raw effect values as well.

L370: cells denoted with stars are delivery locations - where are these?

In Figure 2a, delivery locations, denoted with stars, are located in the leftmost column of the Gridworld.

Conclusion

We thank you again for your comments and questions. We would be happy to answer anything else in addition.

Thank you for your valuable feedback. We are glad to see that you find the problem addressed in our paper interesting and meaningful, and our proposed solution reasonable. We are also happy to see that you find our paper well-written and our experimental results promising. Please find below our response to your comments and questions. For references, please see our general comment.

Response to Comments and Questions

The decision-making process in the Gridworld experiment (Section 6.1) is not clearly conveyed. A simplified trajectory or causal graph—such as a plot similar to Figure 1(a) or Table 1 in Appendix I—would help clarify the process.

Thank you for your suggestion. Appendix K.1 of our paper provides a detailed textual depiction of both trajectories from Fig. 2a. This depiction includes the Planner's observations and instructions, the Reporter's messages to Planner, as well as the actions taken, rewards granted and penalties incurred by both actors throughout each trajectory. In response to your feedback, we have added an explicit reference to these depictions in the caption of Figure 2a in the revised manuscript to enhance clarity.

In Figure 1(b), what does the unannotated section of the pie chart represent?

It represents the contribution score assigned by our decomposition approach to the patient state at step 15 for the depicted simulated scenario. It was omitted to avoid overcrowding the figure.

In Figures 3(a) and 3(b), why is the trust parameter allowed to be 0 in (b) but not in (a)?

When the trust parameter is set to 0, all of the AI's actions are effectively overridden by the clinician. In this scenario, intervening on the AI's actions would not yield any counterfactual effect on the trajectory's outcome. For that reason, we omit this case from Figure 3(a).

While the identifiability of TCFE is discussed under the condition of noise monotonicity, what about the identifiability of r-SSE and ASE?

Assuming exogeneity alongside noise monotonicity guarantees counterfactual identifiability in general, encompassing also more complex counterfactual effects such as r-SSE and ASE. This is formally established in Lemma 4.4 and discussed in detail in Appendix H of [5]. To clarify this point and avoid potential confusion, we have revised the paragraph Assumptions and counterfactual identifiability in our manuscript to explicitly highlight this aspect. Thank you for pointing it out.

Adding a causal graph to illustrate ASE and r-SSE would improve clarity and enhance understanding of the counterfactual effect decomposition.

Thank you for your suggestion. In response, we have added in Appendix N of our revised manuscript a figure that graphically illustrates all counterfactual estimates appearing in Definitions 2.1 (TCFE), 3.1 (tot-ASE), 3.2 (SSE) and Equation 2 (r-SSE) using the Sepsis example from the introduction section of [5] and the causal graph defined therein. Does this sufficiently address your concern? If not, we would like to kindly ask you for additional clarifications so that we can better implement your feedback.

For the proposed decomposition of TCFE, is there a way to verify the validity of the results? Specifically, how reliable are the percentage contributions assigned to each agent or state transition? Could simulations with known ground-truth causal graphs be included to demonstrate their reliability? By comparing the estimated contributions of each agent and state with the ground truth, the causal explanation formula and subsequent algorithm would be more compelling.

Thank you for your feedback. In response, we have conducted additional experiments to support the reliability and validity of our empirical results. A concise summary of these experiments is included in our General Comment (Additional Experiments Evaluating Reliability), with the complete set of results detailed in Appendix L of the revised manuscript.

Conclusion

We thank you again for your comments and questions. We would be happy to answer anything else in addition.

Thank you for your valuable feedback. We are glad to see that you find our approach insightful. Please find below our response to your comments and questions. For references, please see our general comment.

Response to Comments and Questions

The paper examines the counterfactual effect of a single agent’s action. However, it would be more impactful and interesting to investigate the estimands for multiple agents' actions in a multi-agent MDP, especially under varying interaction structures.

This is an interesting point of discussion. We view the extension of our current methodology to decomposing the total counterfactual effect of multiple agents' actions as a meaningful and impactful direction for future work. Our current approach can be straightforwardly extended to explain the effect of multiple actions taken by different agents at the same time-step. However, the situation becomes more complex when the actions have not been taken simultaneously. It is not obvious how to define the potential response of an agent, across time, to a (natural) multi-step intervention that might also be forcing some of the agent's future actions. Addressing this would require extending the definitions of agent- and state-specific effects to explicitly incorporate this temporal dimension. We see our work as a foundation for addressing these challenges, providing a systematic framework for extending effect decomposition to more complex counterfactuals in multi-agent decision making systems.

Additionally, we agree that investigating the interpretability of effect decomposition approaches under varying interaction structures is a critical avenue for exploration. To this end, our experiments already include two environments featuring heterogeneous agents and diverse interaction protocols. In the Gridworld environment, two actors trained with reinforcement learning follow the instructions of an LLM planner, who observes the environment through a Reporter module. In contrast, the Sepsis environment (formulated as a turn-based MMDP [2,3]) features an AI agent primarily acting in the environment under the supervision of a human clinician, who can interrupt the process at any time and override the AI's decision.

Additionally, under the single-agent action setting, Theorems 3.3 and 4.2 are straightforward extensions of the causal mediation formula (Pearl, 2001) and intrinsic causal contributions (Janzing et al., 2024).

Let us address your comment step by step.

Causal explanation formula. Even though Theorem 3.3 from our paper can be viewed as an extension of Theorem 3 from [4], as discussed in the paragraph Connection to prior work from our paper, we respectfully disagree with the characterization of this extension as "straightforward". As highlighted in the introduction section of our paper, prior work in mediation analysis has largely focused on effect propagation through causal paths. In contrast, Theorem 3.3 goes beyond this by explaining the total counterfactual effect of an agent's action through its influence on the state transitions and subsequent agents’ actions in an MMDP.

Reverse state-specific effect. Our approach to decomposing r-SSE, termed r-SSE-ICC, builds on the concept of intrinsic causal contributions (ICC) introduced by [8]. ICC is an information-theoretic measure quantifying how much an observed variable contributes to the uncertainty of another variable in a general SCM. We argue that extending ICC to address the attribution of uncertainty in a complex counterfactual effect, such as r-SSE, to the state variables of an MMDP-SCM is not a straight-forward process. A challenge specific to MMDP-SCMs, for instance, is that apart from state variables also action variables may directly or indirectly influence the variation in r-SSE.

General comment on contributions. We acknowledge that the theoretical results in this paper, i.e., Theorem 3.3 and Theorem 4.2, are not challenging to prove with the appropriate technical background. However, we view our work more as foundational, addressing a newly proposed and complex problem that is highly relevant to accountability in multi-agent AI systems. The primary contribution of this work lies in integrating concepts from a wide range of fields, including multi-agent sequential decision making (MMDPs), counterfactual reasoning (ASE), mediation analysis (Theorem 3.3), cooperative game theory (ASE-SV), information theory (r-SSE-ICC) and more, to tackle this novel problem. Finally, the empirical evaluation of our proposed solution demonstrates that it is interpretable and aligns with standard intuitions. This validates the potential of our approach to inform future research in this area.

Thanks for the response. Considering the limited novelty in the contributions, particularly on the multi-agent aspect, I will maintain my weak reject score. However, I do not hold a strong opinion on acceptance or rejection.

We have conducted additional experiments to support the reliability of our empirical findings. First, for the Gridworld environment, we computed the ground-truth contribution scores for all agents and state variables as determined by our causal explanation formula. These ground-truth scores were then compared against the estimates we had obtained using posterior sampling in the original submission. Second, for the Sepsis environment, we repeated the experiment across 10 different seeds and analyzed the standard error distributions of all estimated counterfactual quantities.

The detailed description of the new experiments and the complete set of results are included in Appendix L of the revised manuscript. Below, we provide a concise overview of our findings:

Gridworld experiment. The relatively simple dynamics of the Gridworld environment and underlying causal model allowed us to compute the exact contribution scores as determined by the proposed causal explanation formula. In Appendix L.1, we present a side-by-side comparison of these ground-truth scores with the estimates we had obtained using posterior sampling in the original submission. Notably, the ground-truth values consistently fall within the standard error bounds of the estimated quantities.

Sepsis experiment. Exactly computing the values of all estimated counterfactual quantities in the Sepsis experiment is significantly more challenging than in the Gridworld setting. To assess the reliability of our findings in this experiment, we repeated it across 9 additional seeds. Appendix L.2 presents box plots for all estimated counterfactual quantities, illustrating their empirical standard error distributions across the 10 seeds. The standard errors are shown relative to the absolute means. These visualizations reveal minimal variability in the estimates of our causal explanation formula across seeds, with only a very small number of outliers. These results reinforce the reliability of our findings and support the robustness of our effect decomposition approach in the Sepsis experiment.