Batch Bayesian Optimization of Delayed Effects Corrections for Thompson Sampling Bandits: A Practical Tuning Algorithm for Adaptive Interventions

1. The novelty of the proposed method is limited. The proposed method is a straightforward application of conventional batch BO to optimize the delayed effects correction terms in TS. I didn't identify any novel technical issues or challenges for doing this. Also, the motivation for choosing batch BO is not clear. Since the relationship between the $v_a$ and $\beta_a$ has been formulated (line 6 in Algorithm 1), why is a black-box optimization algorithm (i.e., BO) selected? Why not learn $\beta_a$ directly using Bayesian inference, SGD, or any other optimization method?

2. I have serious concerns about the soundness of the experiments:

• Firstly, the proposed method is not fairly compared with the baselines. Even though the results of DQN and REINFORCE are shown in Appendix A.2, the figures only include the learning curve of the baseline and don't show that of the proposed method. In the current presentation, I don't know how to achieve the conclusion that "BOTS can achieve better performance than these methods ...". For example, in the first two graphs of Fig. 3, it seems that DQN can achieve a return larger than 2000 with only 100 episodes, which is better than many blue points in Fig. 1. I suggest the authors to re-organize the graphs such that the comparisons between different results can be straightforward.

• Secondly, one key contribution of the proposed method is to optimize the delayed effects correction terms using BO. To demonstrate the priority of the proposed method, some simple baselines (such as the conventional TS, TS+Bayesian inference, TS+random search, etc.) can be considered. Without comparing with suitable baselines, it's hard to measure the significance of the proposed method.

• Besides, from the results shown in Figs. 1&2, it seems that batch BO doesn't work in the proposed method. In all the experiments, the reward increases almost linearly in the number of BO rounds, which are negative observations for supporting the usage of batch BO in this scenario and contradict to the motivation (i.e., execute TS trials in parallel) of this work.

3. The writing of this paper needs to be improved. There are many undefined, inconsistent notations, and typos that make it hard to follow. For example, $x_s$ in Algorithm 1, $\sigma_y\$, and $\sigma_Y$ in Section 4.1 are not defined. Do $x_s$ and $s_t$ denote the same variable? The notation r is used to represent both the reward and the round index. Some examples of grammatical errors and typos are: "... approach addresses the problem warm-starting the ...", "by for each action", "correspond return", "suing", "As the in the ...", etc.

I think that the paper is not yet ready for publication. There are three main issues:

• there is no theoretical guarantee for the algorithm, therefore so far can be considered only as a heuristic
• the experimental results are not convincing, lacking a proper and fair comparison with the stat of the art
• the paper requires major rewriting since it lacks many details (both in the methodological and in the experimental part.)

Details: The paper lacks a proper formulation of the BO problem. In the background section, you mentioned some methods but you did not provide references for them. What do you mean with for a \in [0, A] It is hard to get all the details from the description you provide of the algorithm. You should be more specific in the description and add an explanation of the pseudocode line-by-line. Moreover, some of the quantities that are present in the different algorithms are not defined, nor they are present in the final notation table.

The experimental part of the paper lacks a proper baseline comparison. I think that comparing your algorithm only with other versions of your algorithm is weak proof that what you proposed is working. Some comparison has been proposed in the appendix, but they should be included in the main paper. Moreover, the comments on the results are not detailed at all. I suggest to try to go deeper and provide more comments on the results you obtained. You mention that your experiments were produced by repeating the run 5 times and you show some uncertainty regions in the figure. What did you provide there? Standard deviations? Confidence intervals?

I find that the paper did not do a good job motivating the solutions presented. Delayed feedback in experimental design has been extensively studied in previous work, but it seems the authors immediately settled on penalizing penalizing immediate reward to account for the delayed effects of an action (which I'm personally still not sure how that works). I believe a more thorough exposition of this idea and why it is more promising that other ways of accounting for delayed feedback would benefit the paper.

One of the main contributions of the paper is the adaptive refinement of the prior used by the bandit TS policy, but I couldn't find any relevant discussion on this prior. My understanding is that TS samples from the Bayesian linear regression model whose parameters we are learning each round. It is not clear what role this extra prior plays.