Maximizing Benefits under Harm Constraints: A Generalized Linear Contextual Bandit Approach

Thank you for your thoughtful comments! We address your concerns below.

• The paper considers just an instance of constrained bandits. Even the modeling is a special case of algorithms [9,18] in Table 1 (consider concat $(x_t, u_k) = x_{t,k})$.

  The safety-constrained bandit problems mentioned in part 4) of Section 2 are quite different from our approach. Amani et al (2020) assume we only observe one set of responses to the reward $Z_t$, and $Z_t = g(X_t^T\beta)+e_t$, where $g(\cdot)$ is the inverse of link function (known), $\beta$ is an unknown parameter and $e_t$ is noise. They use a safety constraint $h(BX_t^T\beta)\leq c$ with known function $h$ and matrix $B$. In practice, modeling harm based on reward as well as assuming a known linear transformation $B$ may be limited. The other method by Kazerouni et al(2017) assumed the existence of a known safe policy as a reference policy, which may also be limited. Our modeling is not a special case of the algorithm in [9,18], even though we may concat $x_t$ and $u_k$. Since [9, 18] assume a linear component $x_{t,k}^T\beta$. If we simply concat $x_t$ and $u_k$ under the model of [9, 18], we are assuming an additive relation of $x_t$ and $u_k$: $x_{t,k}^T\beta = x_t^T\beta_1 + u_k^T\beta_2$ for $\beta = (\beta_1, \beta_2)$. Our model (see equation 4,5,7,8) is more general and considers interaction between $x_t$ and $u_k$.

  Overall, our method cannot possibly solve all instances of constrained bandit problems but aims at fitting a gap in literature. The formulation to use varying coefficient generalized linear models also extends the modeling choices in literature.

• How can a practitioner come up with $\lambda$ and $\theta$? We do not consider $\theta$ as a hyperparameter. $\theta$ is the upper bound of harm that practitioners want to force in the experiment. A major example of this work is clinical trials where the typical $\theta$ value can be 0.166 or 0.33. $\lambda$ will affect the probability of the event "chosen arm has harm $p >\theta$". So intuitively, if practitioners want this probability to be low, they can specify an arbitrarily large $\lambda$.

• Depending on the problem instance, $\kappa$ could be arbitrarily small. Then the regret bounds in Thm. 1 are meaningless. How can you address this issue? $\kappa$ controls the rate of convergence of the maximum likelihood estimator in GLM. If $\kappa$ is small, the the parameter $\hat\beta_t$ and $\hat\gamma_t$ will converge to $\beta$ and $\gamma$ slowly. In this case, it's not surprising the regret bound will be large. If you make $\kappa$ arbitrarily small, then the problem at hand is arbitrarily difficult, thus the result is not ideal. We do not think this can be resolved by a particular bandit algorithm. So it is not fair to say it is a drawback of our work.

• No real data experiment. Since there is no real experiment conducted according to the algorithm, we cannot provide real data analysis.

Thank you for your detailed comments! We are happy that you find our parametrization of the harmful effect on the regret necessary and original. We address your questions below and will incorporate all feedback.

• The initialization rounds of the algorithm would still cause some rounds to cause high harm. As we admit in Section 4 above Algorithm 1, initialization and exploration should be more carefully designed if the risk of causing harm is intolerable. In clinical trials, there are designs like 3+3, interval 3+3, and Bayesian optimal interval. These methods do not consider context but are widely used in practice to guarantee safety. We may use those as a reasonable initialization step. We currently do not assume prior information (knowledge from earlier phases of clinical trials, or data from clinical trials concerning a similar compound). But if some prior information is available, it can be used to make the algorithm safer, and possibly shorten the initialization stage. However, the purpose of this paper is to propose a framework considering harm. Tailoring the algorithm into an applicable clinical trial method will be interesting for future work.
• If the Pareto MOMAB work of Turğay et. al. were to focus only on conflicting objectives, how would the authors compare the results obtained by the two works? Please see the response to Reviewer GyGV. Our goal is to control the harm effect under the tolerable level $\theta$. So a treatment with both high harm and high reward may be in the Pareto optimal front, however unacceptable in the harm aspect.
• Confidence intervals along with the mean in the plots. Thank you for the suggestion. We will add confidence bands in the plots.

Thank you for your insightful comments! We have incorporated all feedback.

• It is not so fair to make direct comparisons to previous MOMAB works. Thank you for pointing this out. Although scalarization is a plausible approach to solve MOMAB, it is not the current focus of MOMAB (Pareto regret). Since originally we have 2 objectives, our work is related to MOMAB, and we would like to keep the MOMAB part in the literature review. But we have removed the claim and emphasized we do not work on the Pareto regret. We also think more deeply about why the Pareto regret may be not a good choice for us, see the response to Reviewer GyGV.
• The assumptions made in this paper are implicitly strong but not sufficiently clear. Thank you for your careful thoughts. We have added assumption A4 for $\Sigma_1$ and $\Sigma_2$ to be positive definite.

Missing definitions We have added more explanation in the paper. $p$ and $q$ are the means of harm and reward, respectively. $g$ and $h$ are the inverses of link functions for the GLM, as explained below in equations (5) and (8). Some examples of link functions are given in 3 paragraphs below equation (5). $\zeta$ and $\xi$ denote the systematic component of the GLM and their forms are defined in equations (5) and (8), respectively.

Limited experiments We provide more experiment results in the supplement material. We did not carry out simulation for a larger time horizon $T$ since the algorithms are pretty stable at $T = 5,000$. Since there is no real experiment conducted according to the algorithm, we cannot provide real data analysis.

Thank you for your valuable and detailed comments! We are encouraged that you find our consideration about harm a very important question. We address your concerns below and will incorporate all feedback.

Modelling Choices

• The choice of $u_i$ for arms: $u_1\leq u_2\leq\cdots \leq u_K$ is just an assumption for simplicity. Otherwise one may re-order the arms to achieve this rank. It does not mean arm 1 to $K$ are best to worst, or vice versa. Since there is context information $X$, the best arm will be different for different values of $X$. So the order of $u_k$ does not imply the optimality of arms in any sense. The physical meaning of $u_k$ will depend on the problem at hand. For example, in clinical trials, $u_k$ may mean the dose level ($u_k$ mg/ml of medicine) at treatment $k$ (see 1st paragraph of section 3). This notion of $u_k$ is also used in [3, 20] as shown in table 1.
• Choice of optimization problem taken in equation (1): Since the regret in multi-armed bandit literature is typically defined as the linear gap between the best arm and the chosen arm ($q^*_t-q_{A_t,t}$ in our notation), when we introduce another objective, we still define regret as the linear gap(equation 2). We do not see a motivation to modify the regret definition. Thus, it makes sense to define the optimization problem as a linear function instead of a quadratic function or other kind of functions. We also discussed in detail alternative choices of best arm in equation (3) and the paragraph below equation (3).
• Why is equation (5) linear but equation (8) quadratic? Is it coming from some existing models? It stems from dose-repsonse models but we find it reasonable in general. We may assume the harm is increasing with the level of treatment (dose, for example), but benefit may not be monotone. So equation (5) for harm is chosen to be linear so that it is easy to make it monotonely increasing. Equation (8) is quadratic to allow increasing then decreasing behavior. The linear and quadratic functions are only examples of modeling choice, as explained below equation (8).

Missing lower bound We do not have a lower bound for the regret ready. However, the bound of rate $\sqrt{T\log T}$ matches the state-of-art upper bound for generalized linear contextual bandit problems in literature.

Algorithm ambiguity

• $\epsilon_t$ greedy. Thank you for your suggestion. We have changed the algorithm name to $\epsilon_t$ greedy. We did not name "varying coefficient" as "$\epsilon_t$-greedy" as all the methods under comparison are under the $\epsilon_t$-greedy framework, but different modeling choices.
• Why is forced exploration (through $\epsilon$-greedy methodology) required? As explained in Section 4 below Algorithm 1, since the scalarization makes the problem essentially nonlinear, an upper confidence bound based on the linear term(systematic component in GLM) cannot be constructed. Thompson sampling would be another approach, but we failed to get good results in some earlier simulation. Also, running MCMC for posterior with no closed form is expensive. It may be worthwhile to investigate how to finely tune MCMC and choose a good prior for this kind of models, but it's beyond the scope of this paper to propose a framework considering harm.
• There seems to be some mistake in the pseudocode line 3-5 (initialization phase). We double checked and did not find a mistake. To make it more clear: We have $K$ arms, and we'd like to gather $m$ independent samples for each arm. For example, for time $t = 1,\cdots, m$, we pull arm 1, and for $t = m+1, \cdots, 2m$, we pull arm 2,...,and for $t = (K-1)m+1, \cdots, mK$, we pull arm $K$. But overall, we just need to pull each arm $m$ times during $t=1,\cdots, mK$.
• What is the complexity of each inner loop of the $\epsilon$-greedy algorithm? We need to fit GLM in each inner loop for $t-1$ data points at round $t$. Using the notation of the paper, the dimension of covariate for harm model is $2d_1$ ($\Phi(X_t)$ is $d_1$-dimensional) and the dimension of the covariate for reward model is $3d_2$, then the complexity for fitting 2 GLM is $O(t(d_1^3 + d_2^3))$.

Connection with Multiobjective MAB Our scalarization formulation is from MOMAB, but with a specific focus on controlling harm, i.e., we penalize the harm probability when it exceeds the harm threshold. As for the popular Pareto formulation in MOMAB, it may not suit the needs of controlling harm very well. Consider a simple example, let $(p,q)$ denote a pair of mean harm $p$ and mean reward $q$, and there are totally two possible arms $(p,q) = (0.3, 0.7)$ and $(p,q) = (0.6, 0.8)$. Remember we prefer smaller $p$ and larger $q$. Then both arms $(0.3, 0.7)$ and $(0.6, 0.8)$ are in the Pareto optimal front, since they don't dominate each other. However, a harm of $p = 0.6$ may be unacceptable.

Safety-constrained MABs We would appreciate it if you can provide some examples of safety budget bandit works.