High Probability Contextual Bandits for Optimal Dosage Selection

We are grateful for your feedback and for sharing your insightful questions and remarks. Below are our reflections.

Weaknesses:

1. We agree that the concept of making the toxicity threshold a function of $X_t$ is intriguing. This adjustment could likely be addressed within the second part of our paper, which focuses on the case of non-linear functions.

2. An upper bound on $L$ and $S$ is sufficient to determine the initial safe dosage. In practice, the vectors $X_t$, $\theta$, and $\mu$ are normalized, such that these bounds can be equal to one.

3. We agree with that commend, we work on that for the updated version.

Questions:

1. Unfortunately, the minimum dosage is equal to zero and in that case we do not receive any feedback.

2. One technical difference is that the way to use the structure of the constraint to bound the cost term in the regret, that is the Lemma 5.2. In general, the main techniques from the cost-reward literature apply to our setting with some changes.

3. Since there is no previous work regarding high-probability constraint satisfaction, we plotted in graph (c) the constraint violations of the proposed algorithm from Pacchiano et al. (Contextual Bandits with Stage-Wise Constraints). Since in their algorithm the learner, at every round, chooses a vector that is not provided as input as in our algorithm, we made the following modification and concluded that, in the first $T$ rounds, their algorithm violates the constraint a linear number of times. In the updated version, we can simulate more setups to study this trade-off.

We appreciate your feedback, as well as your insightful comments and questions. Please find our thoughts below.

1. We agree that the linear model is a simplified one. That is why we extended our results to any non-linear function that we can use to model the efficacy and the toxicity mechanisms.

2. Our differences with Abbasi-Yadkori et al., (2011) is the decomposition of the regret to express it as two terms, one induced by the reward and one by the cost signal. Some technical difficulties are the following:

   1. Deriving a result that holds with high probability instead of in expectation.
   2. Deriving an algorithm that learns efficiently the feasible set.
   3. Bounding $(\tilde{\alpha}_t - \alpha_t)\langle X_t, \theta^* \rangle$

3. Thank you for that comment, this is definitely a good idea. We are going to adopt that change in the updated version.

4. The term "stage-wise constraint" in the bandits literature is that in every round a reward and a cost signal, $R_t, C_t$ respectively, are generated and the constraints are regarding to some function of the cost signal.

Thank you for taking the time to provide such a thoughtful and clear response. I truly appreciate your effort. While your explanation is helpful, it does not substantially change my understanding of the contributions, especially when comparing with Abbasi-Yadkori et al., (2011). I believe that Abbasi-Yadkori et al., (2011) also provided high-probability bound. The design and the analysis are heavily depend on the inequality provided by Abbasi-Yadkori et al., (2011). Thus, I think it’s appropriate to maintain the current score.

Thank you for providing your valuable feedback and thoughtful questions and comments. Here is our response.

1. As we mention in our submission, in the paper the reviewer mentions ("Contextual Bandits with Stage-wise Constraints") the authors come up with an algorithm that ensures that with high probability the conditional expectations of the cost signals at every step are bounded. This is a much weaker notion of control than requiring the realizations of the cost signals to be bounded in all rounds. Controlling the expectation, even if with high probability means allowing for some realizations of the cost signal to be very large, and therefore dangerous for some patients. Thus our results provide stronger safety guarantees than theirs.

2. It is true that past data can often be used to estimate an initial safe dosage. However, there are instances where such data are unavailable. For example, consider the emergence of a new illness, such as COVID-19, or the appearance of a novel pathogen that has recently crossed the species barrier, as seen in zoonotic viral diseases.

3. In bandit theory, we focus on ensuring sub-linear regret behavior so that, asymptotically, the error between the optimal policy and the policy to which we converge approaches zero. Another reason for minimizing regret is its equivalence to maximizing total efficacy. In the absence of constraints, the maximum permitted dosage should be assigned only when $\langle X_t, \theta^* \rangle > 0$. This necessitates learning the unknown vector $\theta^*$.

4. Some technical difficulties are the following:

   1. Deriving a result that holds with high probability instead of in expectation.
   2. Deriving an algorithm that learns efficiently the feasible set.
   3. Bounding $(\tilde{\alpha}_t - \alpha_t)\langle X_t, \theta^* \rangle$

Thank you for your feedback and your insightful comments and questions. Here are our thoughts.

Weaknesses:

1. We adopted a linear model to formulate the reward and cost generating procedures for the following reasons. Firstly, linear models are widely used in the bandits literature, as mentioned in https://arxiv.org/pdf/2401.08016 and https://arxiv.org/pdf/2010.00081. The inner product in statistics can be interpreted as a measure of the correlation between the context $X_t$ and the unknown vector $\theta^*$. For example, consider the following scenario. $X_t$ may represent the results of medical tests of a patient before assigning a dose, as in chemotherapy. However, the doctor may not be able to precisely determine the significance of each characteristic (e.g. blood pressure, concentrations of specific substances in the blood, or specific pixels on an X-ray image) with respect to the efficacy of the drug for the patient. In this context, $\theta^*$ represents a probability distribution vector that captures the initially unknown importance of each feature. In this case, if the inner product $\langle X_t, \theta^* \rangle > 0$, it indicates that the patient's medical condition is positively correlated with the disease, suggesting that the patient suffers from that disease. When the patient suffers from the disease, increasing the dosage (up to the allowed threshold) enhances the therapeutic effect. However, increasing the dose also amplifies both the toxicity and the noise, meaning that higher doses intensify the randomness of the drug's effect on the patient. Therefore, we need to balance these two aspects. In the second section, we extend our results to any function of the context $X_t$. In this case, such a function could be a neural network trained to learn the importance of medical records for disease diagnosis, or any nonlinear function established in the medical literature.

2. We really appreciate all reviewers feedback and we are working on incorporating it to our manuscript.

3. Regarding $\gamma_{\alpha}$, do you mean $\gamma_c$, or $\gamma_r$? The positive real-valued constants $\gamma_r$ and $\gamma_c$ characterize the measure of sub-Gaussianity of a random variable. This property describes the decay rate of the tails of the random variable's distribution, analogous to how the standard deviation characterizes the spread in a Gaussian distribution. For a formal definition of sub-Gaussian random variables, we refer readers to classical texts in bandit theory and high-dimensional statistics such as:

i) Lattimore, T., Szepesvári, C. (2020). Bandit Algorithms. Cambridge University Press.

ii) Vershynin, R. (2018). High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press.

About $\mathcal{K}(x)$, we can equivalently write it as $\mathcal{K}(x) =$ { $\mu \in \mathcal{C}^t_\mu \mid \mu \\ \text{ belongs to the feasible solutions of (6) with input }x$ }.

Questions:

1. The assignment of any drug to a patient induces both positive and negative effects. The positive effect reflects the drug's efficacy, while the negative effect pertains to its toxicity. We employed a contextual approach, recognizing that the effects of a drug depend on the patient's phenotype—specific characteristics measurable through medical tests. A linear model was chosen to evaluate the correlation between the patient’s phenotype and the significance of each feature in the mechanisms generating reward (efficacy) and toxicity. Linear models are the simplest in statistical analysis, and simpler models tend to generalize better. However, when additional structural information about the mechanisms of efficacy and toxicity becomes available, the approach can be extended to non-linear functions. Any established non-linear function in the literature can then be employed to better capture the relationship between the patient's condition and the mechanisms determining the reward-to-cost ratio.

2. In bandit theory the regret can take negative values too, so it does not matter if $\alpha_t \leq \alpha_t^*$. We only care about the sub-linear behavior of the regret. Without the cost signal constraint, it is guaranteed with high probability over the past realizations of the noise that $\alpha_t \geq \alpha_t^*$ as we choose $\alpha_t$ optimistically over the possible values of $\theta^*$. Although, taking into account the cost-toxicity constraint, we are initially robust in our choices for $\alpha_t$ so, during the first rounds it happens that $\alpha_t \leq \alpha_t^*$.

Dear reviewer, we kindly request your attention to review our responses and share your valuable feedback. Your insights and guidance on these points are highly meaningful to us and will greatly contribute to advancing the discussion.