Thank you for your comments and questions. We are glad to see you consider the problem of justifiable decision-making to be of high practical importance, and that you consider our presentation to be clear and straight to the point. We are also thankful for the references you have provided, and are happy to discuss them further. In the following, we try to answer all of your questions and comments. The key points of our response can be summarized as follows:

• We discuss similarities and differences to the two works you kindly suggested;
• We answer your questions about optimality of deployed policies, definition of $R_J$ and choices of model architectures.

Discussion on General Comments

Bottom of page 1, “the human as a suboptimal decision maker”: do you mean that the RL agent is optimal? But if optimality refers to the actions deployed in practice, the RL agent will not be optimal either, no?

What we intended to say there is that a human may not act optimally or rationally in every situation, as it is inherently limited in capacity. To extend on this, our line of reasoning touched upon the fact that in certain situations, a human may not be able to comprehend the full state available to the acting agent, which may hinder its judgment. We also acknowledge your comment about optimality of the deployed agent. Indeed, the agent is learned, and will therefore also be suboptimal, although the referenced sentence does not imply any assumptions about this. Lastly, we have slightly expanded Sec. 3.1 to better clarify the assumptions made by our framework about optimality and choice of the baseline policy.

The authors in [A,B] also consider sequential decision-making (MDPs) influenced by a baseline policy and an agent to model partial adherence to an algorithmic recommendation. Is the setup in this submission close to the setup in [A,B]?

Thank you very much for sharing these works. We have also updated the related work section to include them. Regarding [A], we find a conceptual similarity in that both approaches aim to improve what is assumed to be a given baseline policy. However, [A] operates in a context of expert-in-loop setting, where a baseline policy is considered to represent a human decision-maker and a policy to be learned (in their terms, recommendation policy) aims to adapt its actions to it. Another prominent difference is found in a way the environment evolves: in [A] both the baseline policy and the learned policy can influence dynamics of the environment. Differently, in our setup, baseline policy never influences state transitions: the justifiable agent is a sole actor in the environment. Likewise, authors in [B] examine a setting similar to the one in [A], which also exhibits mentioned similarities and differences. As a general note, both [A] and [B] consider a different problem setup, where the primary goal is to improve and adapt the AI policy to the baseline policy representing a human, which is the main decision-maker in the environment. In contrast, we consider a problem of improving the baseline policy in a setup where a learned (i.e., justifiable) policy is the main actor in the environment, and a baseline policy is only used as a reference point to improve upon.

Definition of $R^J$: what is the expectation over? The transition probabilities (since policies are deterministic)? Is the decision to be optimized $a_t$ or $a_t^B$?

In the definition of the justifiability reward $R^J$, the expectation is over the trajectories, which are stochastic due to non-deterministic transitions. The decision to be optimized is the one coming from the justifiable policy, namely $a_t$, whereas the decision $a_t^B$ is coming from a (fixed) baseline policy. We have also slightly updated the paper to make this part more easily understandable to the reader.

p4: the definition of U introduces the notation J but the next sentence uses J. Please clarify.

Thank you for noticing inconsistencies in our presentation here. The definition of U introduces J as a model of a human judge. Later mentions also refer to the same entity. We have corrected the paper accordingly.

Thank you for your detailed review, perceptive comments and questions. We are glad to see you find the problem to be of importance and that you think our method has merits. In the following, we try to answer all of your questions and comments. The key points of our response can be summarized as follows:

• We address your comment about ethics concerns pertaining to our method;
• We address your questions and concerns about an additional benchmark comparing actions proposed by the justifiable and baseline agents, and confuser evaluation of the isolated agent with additional experiments;
• We address your questions about the complexity of the debate and evaluation of evidence, payoff function G, evaluation of the clinician and choice of the Bradley-Terry model.

Questions Involving Additional Experiments

A missing benchmark is how often the benchmark and justifying agents choose the same action. This would be highly informative.

Thank you for pointing this out. We acknowledge that there is an opportunity to provide richer information to the reader in this regard. To address this, in App. D.2 and Plot 6a, we show the percent of times an action from the justifiable policy was preferred to that of the baseline policy (JP), percent of times an action of the baseline policy was preferred to that of the justifiable policy (BP) and percent of times when the two were equally preferred (EP). In addition, we updated the Plot 2b to show the percent of times an action from the justifiable policy was preferred to that of the baseline policy, but only when the two differ. In our experiments, the judge deemed two actions equally justifiable only when they were the same.

Unless I misunderstand the methodology, the benchmark with the isolated agent does not seem entirely appropriate. The isolated agent is trained to provide L pieces of evidence and then tested in a setting where it provides L/2 and a confuser provides L/2. It is not surprising that it is vulnerable to a confuser that provides the evidence least-supportive of its L/2 pieces of information. A better benchmark would be to train the isolated in a setting where it only provides L/2 pieces of information, and then see how robust it is to the setting where L/2 adversarially selected pieces of information are added.

Thank you for raising this concern. The isolated argumentative agent was initially trained to propose L=6 evidence, whereas during evaluation against a confuser, it was proposing L=3 evidence in a debate-like setup. As a response to your comment, we introduce a new setup for the isolated agent (also described as part of the changes made to the paper in our general response). In the new version, the isolated agent is trained to propose L=3 evidence. When evaluating robustness, we do not perform debate, but rather let the agent first propose L=3 evidence, followed by a confuser proposing the remainder. We report the new results in the main text (Sec. 5.3, Plot 4b). We also include the previous setup and do a comparison to the updated one in App. C.2.1 and App. D.3. The conclusions we drew about robustness of the isolated agent in Sec. 5.4 remain the same in both cases.

Thank you for your insightful comments and questions. We are happy to see that you overall enjoyed the paper and that you consider the idea of using debate in this context an interesting research direction. In the following, we try to answer all of your questions and comments. The key points of our response can be summarized as follows:

• We address your questions pertaining to the use of judge’s reward difference in the utility function, choice of an alternative action in the preference dataset and confuser evaluation of the isolated agent with additional experiments;
• We clarify the definition of the preference dataset from Sec. 4.1, made assumption about the justifiable actions, sampling of evidence during training of the judge and approximate optimality of the baseline agent;

Questions Involving Additional Experiments

Debate Game (Section 3.2): I'm just curious: Did you consider defining U to be the difference between the judge's judgments instead of the binary +1, -1? If you did, why did you choose not to use it?

Thank you for the comment. We apologize for the confusion that may have occurred due to our mistake in writing of the utility function (Sec. 3.2). Throughout the paper, we assumed that when two actions are equally justifiable (e.g., as in a case when they are equal) the debate game draws, ending in both players obtaining a score of 0.

Initially, we have only considered defining the debate utility function using rewards +1, 0, -1. However, the idea you put forward sounds interesting, and we examine it in more details. In particular, we have retrained maxmin and self-play agents with the utility function defined using a difference between predicted justifiability reward, namely $\\mathbb{U}(a_t, a_t^B, \\{e\\}) = \\mathcal{J}(a_t, \\{e\\}) - \\mathcal{J}(a_t^B, \\{e\\})$. In addition, we also train new confuser agents, so we can compare both effectiveness and robustness of the two approaches. We include the results in App. D.4, Plot 6c. The two approaches perform similarly in situations which do not involve a confuser agent. However, it seems that defining the utility using a difference in judge’s rewards leads to slightly lower scores when debate agents are faced with an adversary.

Creating the synthetic dataset by comparing a random action with the clinician's action seems like it would result in a lot of pairs in the dataset where one action is clearly superior to the other. Is this true? Does this cause any issues?

Thank you for the question. We hypothesize that a situation where one action is clearly superior to the other would happen only in relatively rare cases (e.g., pairing an action of “no medication” with an action “highest IV/VC dose”, for a patient whose vitals are normal). In the context of our framework, the “difficulty” of the synthetic dataset primarily impacts the quality of a learned judge model. To empirically test this, we now examine a total of 3 different methods for pairing an alternative action:

• random: true action $a_t$ is paired with an alternative action sampled uniform-random (as described in the main text);
• exhaustive: true action $a_t$ is paired with every possible alternative action;
• offset: true action $a_t$ is paired with an action in the neighborhood of $a_t$. To define a neighborhood, we recall that there are 5 choices for both, vasopressors (VC) and intravenous fluids (VC). Therefore, we can write $a_t = 5 * IV + VC$, where $IV, VC \\in \\{0, 1, 2, 3, 4\\}$. To obtain an alternative action, we consider changing IV and VC by an offset sampled uniform-random from a set $\\{-1, 0, 1\\}$, for both IV and VC. For example, for $a_t=21$ we have $IV=4$ and $VC=1$, and we randomly select an action that has $IV \\in \\{3, 4, 5\\}$ and $VC \\in \\{0, 1, 2\\}$ (ensuring we don’t end up with the same action).

For each of the 3 variants, we train a new judge model using the same procedure described in the main paper, and show the accuracy on the test set in App D.5, Plot 6d. The exhaustive variant represents the most informative, but also unrealistically large, dataset. The random variant represents somewhat of a “middle ground” in terms of dataset difficulty. Lastly, the offset variant represents the most difficult case, as differences between two actions are more nuanced. However, while the achieved accuracies reflect the difficulty of the corresponding dataset (i.e., the highest accuracy for the exhaustive variant, followed by random and offset variants), the difference seems to be small enough that we can assume our choice of the random variant did not cause issues.

Thank you for your comment. We are pleased to see that you are positive about our work. We would be happy to respond to further questions, should you have any.

Thank you for your very thorough response and the impressive amount of additional experiments performed. I already recommended the paper for acceptance, and with these corrections and improvements, the recommendation is only stronger.

If it were possible, I would raise my rating from a 6 to a 7. Since I cannot, I will consider raising my rating to an 8 or keeping it at a 6.

Thank you for your very perceptive comments and insightful questions. We are very pleased to see that you overall enjoyed the work and are happy to engage in further discussion. In the following, we try to answer all of your questions and comments. The key points of our response can be summarized as follows:

• We address your concerns about requirement of the baseline policy and offline sampling during training;
• We clarify our choices of the Bradley-Terry model, multi-step debate, and elaborate on a case when two actions are equally justifiable;
• We discuss ways of interpreting proposed evidence, similarities to AC methods and connection to RLHF;

Discussion on Paper Weaknesses

The method appears to require explicit knowledge of the optimal deterministic policy (called the baseline policy as part of the debate framework (unless I have misunderstood something). This seems like a big limitation in terms of practically applying this method to larger-scale problems.

The method does require access to what we refer to as a baseline policy. However, this policy can be any well-performing policy, that is not necessarily optimal. The reason why we require a baseline policy to be well-performing is to avoid trivial solutions. In particular, if we consider a case involving a poor quality baseline policy, it might be easy, but useless, to be more justifiable than it. We have updated the paper to mention this.

Sec. 4.2 mentions that the judge proxy model follows a 'Bradley-Terry' model (and provides a citation). This seems like an important point in the design of your method, but I'm not familiar with this model or the background or history here. Can you elaborate on why this model is chosen?

The choice of a Bradley-Terry model was inspired by the seminal work that examined specification of the reward function for reinforcement learning agents via preference ratings over trajectories [1]. It is a standard model for estimating score functions from pairwise preferences. In our framework, it can be understood in the context of equating judge’s rewards with a justifiability ranking scale. In particular, the difference in judge’s reward of two actions estimates the probability that a human considers one more justified than the other, given a particular set of evidence.

The training of the justifiable agent (and the argumentative agent) appears to use offline i.i.d. sampling of (s, a, a) tuples (Sec 4.3). This seems like it could pose over/under-representation problems for some MDPs with deep trajectories that might be less or more common (i.e. problems where the i.i.d. assumption is more heavily violated by the dynamics). Can you comment about this limitation of the method?

Thank you for your comment. Regarding the argumentative agents (Sec. 4.3), it is important to acknowledge that in a general case of online RL there is a mismatch between a distribution of contexts used during their training (the i.i.d assumption), and a distribution of contexts that arises during the training of the justifiable agent. This distributional mismatch might affect the quality of the justifiable policy, since $\hat{r}^d$ might not approximate well $r^d$ in states that are much more frequently visited by the justifiable policy than by the behavior policy that generated the offline data. In our case, however, because we are operating in an offline reinforcement learning setup, the dataset used to train both argumentative and justifiable agents is the same, so we do not encounter such a distributional mismatch (i.e., states are sampled i.i.d during training of both agents). Regarding the justifiable agent, we believe that the limitations one could face in this setup are the same as those encountered in standard offline reinforcement learning problems [3]. Therefore, one could tackle out-of-distribution challenges using existing offline RL approaches, such as batch-constrained Q-learning [4] or Conservative Q-Learning [5].

Due to the additional structure of the debate game, the method introduces a number of additional hyper-parameters (e.g. evidence size, mixing coefficient, scaling coefficient, as well as architecture of the various policy models, to say nothing of the learning algorithm hyper-parameters. Because of this, evaluation on one or more additional MDP problems would strengthen the evidence for the efficacy of this method.

Thank you for your comment. In this work, we focused on an in-depth analysis within the domain of treating sepsis, with a goal of achieving a comprehensive understanding of the intricacies of our method in a setting where learning to justify is of great importance. We agree that additional evaluation in different domains will provide further insights into the efficacy of the method, and it represents a part of the future work.