Thank you for the thoughtful review. We address the listed weaknesses and questions below.

Weaknesses

Yes, there are three primary challenges. We will include these descriptions below in the final revision.

The first is the methodological challenge of figuring out how to bring all of the pieces together. DML, for instance, requires independent subsets of data, so we had to develop a data-splitting technique that is appropriate for dynamic settings. As another example, NNR is typically applied to distinct users in stationary settings, so we had to adapt the technique to handle nonlinear time trends.

The second challenge is extending the standard proof techniques for bandit regret bounds to handle (a) the nonlinear baseline reward and Graph Laplacian regularization and (b) our method’s growing number of parameters. For (a), we need to adapt the standard proof bounding the regularized least squares (RLS) linear predictor error to handle the use of weighted least squares, the non-linear baseline and Laplacian regularization (our Lemma 6). This requires careful analysis of how the RLS estimator relates to the double robustness of the pseudo-reward and bounding the variance of the pseudo-reward. For (b), our final regret bound involves a sum of such regularized least squares linear predictor errors, which is bounded by a sum of weighted norms of feature vectors inverse scaled by stages. Bounding this requires a simple but non-obvious application of Cauchy-Schwarz to decouple the weighted feature norms from inverse stage scaling, which allows us to apply standard techniques to the sum of norms and bound the sum of inverse stages by the harmonic number.

The third challenge was creating an efficient implementation. Re-computing inverses and log determinants at every stage would have been prohibitively slow, so we had to experiment with efficient rank-one update procedures (implemented via the SuiteSparse package).

Question 1: Network

Our main rebuttal contains a discussion of these points, including some recently added sensitivity analyses. To summarize, the quality of the network and number of neighbors does influence the performance, but these differences are small relative to the differences between methods.

To form the network, we created a distance metric using four baseline covariates: gender, age, device, and baseline average daily steps. Then we connected users to their 5 closest neighbors. We had to move this information to the appendix due to space restrictions.

Question 2: Supervised Model Selection

Because the simulation required many repetitions over three settings and eight methods (including those in the Appendix), our goal was to find a nonlinear regression method that had an easy-to-use and computationally efficient online implementation. The River library contains several regression methods that meet these criteria. We chose an ensemble of decision trees (essentially a random forest) because we have found that random forests typically perform well with limited tuning.

We also used a random forest model for the case study, largely for the reason stated above. However, the computational demands were lower, so we were actually able to refit it at each stage using standard (not online) decision trees.

In general, we recommend using nonlinear regression methods. When computation time is not a large concern, we recommend choosing a method with high predictive accuracy for the problem, which could be assessed using previous studies or in an adaptive fashion via cross validation. When computation time is a major concern, we recommend selecting methods that have efficient online updates. In particular, linear models with flexible basis functions (e.g., splines) are particularly well suited because they have efficient rank-one updates.

Question 3: Assumption 1

This is somewhat of a subtle point. We sample actions in a hierarchical fashion. We first select a non-baseline action and then we randomize between that action and the baseline (“do-nothing”) action. The regret bound is based only on the second randomization. It bounds the regret of a policy that is restricted to two actions: the baseline action and the given non-baseline arm. Consequently, the $K$ appearing in the regret bound is really the number of stages in which the given non-baseline arm is selected. Our analysis mimics that of the action-centered (AC) paper in this regard (Greenewald et al., 2017).

To summarize, we do not list a positivity assumption for other actions because we define and bound the regret in a way that restricts the randomization to a single non-baseline arm. In practice, however, introducing more arms will generally lead to higher overall regret (with a fixed total study length) because less data will be available for each arm.

Question 4: Computation Time

Thank you for the good suggestion. We reported these timings in the main rebuttal.

Question 5: $N$ vs. $K$

Thank you for catching this. In our asymptotic regime, $N = K$, so either is correct. For consistency and simplicity, however, we have replaced the $N$’s with $K$’s in the main text. The algorithm could still be applied under a more realistic regime in which $N \neq K$, but the matrices would need to be resized based on the number of users ($N$) and time points ($T$).

Thank you for your insightful comments and questions. We address the latter below.

Q 1

We included a wide variety of competing algorithms (see appendix for the full set) to (1) assess how well DML-TS-NNR performs relative to simple baselines and (2) understand which aspects of DML-TS-NNR contribute most to its performance.

1. The comparison to Standard and AC shows that there is a large benefit to pooling data across users.
2. The comparison to NNR-Linear shows that the benefit of DML-TS-NNR is not entirely due to the additional regularization (though, it is helpful).
3. The comparison to DML-TS-SU shows that ignoring participant heterogeneity leads to suboptimal performance.
4. The comparison to DML-TS-NNR-BLM (which uses bagged linear models instead of an ensemble of online decision trees) shows that DML-TS-NNR is robust to the choice of supervised learning algorithm but that a more flexible algorithm can lead to better performance when the baseline rewards are nonlinear.
5. The comparison to IntelPooling, the most relevant previous method, shows that there is additional benefit to the new components of our method: NNR and DML (IntelPooling uses neither).
6. The comparison to Neural-Linear shows that the performance of DML-TS-NNR is not entirely (or even largely) due to the flexible baseline model; the time effects and NNR also play an important role.

Q 2

The appendix includes an additional off-policy evaluation study for the Intern Health Study (IHS). IHS is a mental health study for medical interns. Similar to the case study in the main paper, we again found that DML-TS-NNR outperformed competing algorithms.

Thus, to answer your question, we have applied DML-TS-NNR to two different areas already and DML-TS-NNR achieved state-of-the-art performance in each. We are involved in applied collaborations in multiple sub-areas of medical and behavioral science and plan to use DML-TS-NNR across these domains. While we do not foresee any particular difficulties for specific domains, we expect that these collaborations will lead to an improved understanding of when DML-TS-NNR performs particularly well (or poorly) and how to address practical issues, such as specification of the context and reward variables.

Q 3

Our main rebuttal addresses this point. In its current form, we think the algorithm could be used without modification/approximation up to about 1,000 stages—much more than most academic mobile health studies. At that point, some of the strategies suggested in the main rebuttal could be used to enable scaling to even larger data sets.

Real-time implementation of the algorithm across many mobile devices would require communication between each user’s device and a central server. While this carries numerous engineering difficulties, the methodological difficulties are fairly limited. The system could be set up such that devices send information to the server each time a decision is required, which would require essentially no methodological changes. Alternatively, the algorithm parameters ($V, b$ in particular) could be updated in batches, and the mobile devices would make decisions using the most up-to-date cached parameter values available to them. In this case, the mechanics of the algorithm would remain essentially unchanged; however, the regret bound would need to be adapted to account for the batched updates.

Q 4

One of the benefits of the DML-TS-NNR algorithm is that it remains agnostic to the choice of the specific supervised learning model and its corresponding optimization algorithm. The only theoretical requirement is the consistency condition of Assumption 3. In fact, as pointed out in our response to reviewer cFGH, we could still obtain a regret bound of the same asymptotic order as that given in Theorem 1 (albeit with larger constants) under violations of Algorithm 3, provided the supervised learning model converges to a bounded function.

That said, we generally expect highly predictive supervised learning models to produce the lowest regret. Consequently, it would be reasonable to select a supervised learning algorithm via standard cross validation techniques (though, adaptive strategies like this are not necessarily compatible with our regret bound).

We also note that our additional simulations in Appendix B include a comparison of two supervised learning algorithms: an ensemble of online decision trees (used in the main paper) and a bagged ensemble of linear models. The former performed slightly better in the Nonlinear Setting (beating DML-TS-NNR in 30/50 simulations), presumably because it is flexible enough to model the nonlinear baseline rewards.

Q 5

To address these privacy considerations, we would need to ensure that data is transmitted from mobile devices to the server in a secure fashion. We would also need to ensure that the server meets the relevant requirements (e.g., HIPAA in the United States) for collecting, storing, and processing that data. The data transmission from the server to the devices would be less of a concern because the data is aggregated (in fact, we would not need to send the parameters for user $i$ to user $j$ [with $i \neq j$]).

Regarding the treatment of different demographic groups, we believe the personalized, network-based nature of DML-TS-NNR has the potential to ameliorate these concerns. Allowing users to have their own parameters means that decisions will be directly adapted to users’ own actions—not the conglomerate of some larger population. If practitioners believe that treatment effects will vary substantially by demographic group, then demographic information could be used to design the network structure. The result would be that the algorithm would suggest treatments that are effective for “similar” (based on the network) users rather than treatments that are effective for the “average user,” particularly in the early stages for a given user when subject-specific data is minimal.

Thank you for your helpful comments. We address the outlined weaknesses and questions below.

Hyperparameter Tuning

Hyperparameter selection plays an important role in the performance of bandit algorithms. While virtually all such algorithms require specification of some hyperparameters (at a minimum, a “prior” mean and variance), DML-TS-NNR does require more than a basic parametric bandit algorithm (e.g., the Standard method in our simulations). As detailed in the main rebuttal above, we added 8 sensitivity analyses to assess the impact of our method’s hyperparameters on its performance. The main finding is that DML-TS-NNR consistently outperforms the other methods in the Nonlinear Setting (the most realistic) even when we specify poor hyperparameters. The reason for the superior performance is that DML-TS-NNR is the only one that accounts for all of the important problem structure, including heterogeneity across users and time, explainable variation in baseline rewards, and network information. These results suggest that correctly accounting for the problem structure is relatively more important than hyperparameter tuning. This is analogous to the fact that out-of-the-box ML models often outperform tuned linear models (e.g., ridge regression / LASSO) on nonlinear regression tasks.

We also re-emphasize that there are adaptive strategies that could be explored for setting many of the hyperparameters. The primary challenge with these strategies is that the regret bound is not guaranteed to apply when the hyperparameters are not specified in advance. This is an interesting and important avenue for future research.

Assumption 4

Assumption 4 is best understood by comparing it to corresponding assumptions made by competing approaches. We provide this context below, beginning with simple bandit algorithms:

• Standard / AC: Were we to apply Standard / AC to the mHealth setting, we would effectively be assuming that $\theta_{i,t} = \theta^{\text{shared}}$ for all $i, t$. This assumption is clearly very limiting and partially explains the poor performance of these methods in the simulation / case studies.
• NNR-Linear: Graph bandit approaches, such as the NNR-Linear method from our simulation, allow all users to have their own parameters, effectively assuming $\theta_{i,t} = \theta_{i}^{\text{user}}$. By regularizing these parameters toward each other and zero (i.e., including penalties analogous to $\lambda$ and $\gamma$, respectively), we are essentially expressing a prior belief that the $\theta_{i}^{\text{user}}$ values are small and similar between neighbors. A slight improvement to this approach is to set $\theta_{i,t} = \theta^{\text{shared}} + \theta_{i}^{\text{user}}$ so that estimates are regularized toward the (learned) shared parameter instead of zero. These graph bandit approaches are much more effective than the Standard / AC approach in heterogeneous settings; however, they still assume constant effects over time, which is why NNR-Linear is not competitive in our Nonlinear simulation setting.
• Mixed Model Approaches: A common solution to this problem in mixed modeling is to assume a random slopes model: $\theta_{i,t} = \theta_{i0}^{\text{user}} + t \cdot \theta_{i1}^{\text{user}}$; i.e., we assume a subject-specific linear trend over time. This assumption improves on the static assumption of graph bandit approaches, but it is still highly parametric and likely to be misspecified.

With this information as background, we can see that Assumption 4 improves on simpler (and commonly used) alternatives by allowing for variation across both users and time. Unlike the random slopes model, we allow for non-parametric trends over time by including a separate parameter for all members of the (infinitely increasing) set of time points. We also note that IntelPooling employs a version of Assumption 4 by including both user- and time-specific random effects.

We conducted an exploratory analysis of the Valentine Study to motivate the user-level and temporal effect heterogeneity. For this analysis, we formed pseudo-rewards according to Equation (3) and conducted an ANOVA test. The results (Table 2 of the rebuttal PDF) show strong evidence of both forms of heterogeneity. Figure 1 in the rebuttal PDF shows a Loess fit of the causal effects over time, indicating strong nonstationarity in the causal effects.

Network Structure Assumptions

See our main rebuttal for a discussion of this point. To the best of our knowledge, all work in this literature, including Cesa-Bianchi et al. 2013, Herbster et al 2021, Vaswani et al. 2017, Choi et al. 2023 and many other papers use the same assumption of a known network structure. In many cases where contextual bandits are useful, such as in social networks, there are known relationships between users.

That said, learning the network structure is a potentially important avenue for future work: thank you for mentioning this.

Double Robustness and DML

We see the point being raised. The regret bound does indeed require the ML model to be a consistent estimator of $f$ (Assumption 3); hence, a nonparametric estimator of $f$ is needed in general for the regret bound to apply.

In the proof of the regret bound, however, the only impact of consistently estimating $f$ is that it reduces the size of certain confidence sets, resulting in a tighter regret bound. As long as $f$ converges to a bounded function, DML-TS-NNR still produces a regret bound with the same asymptotic order, but the constants in the regret bound could be improved by consistently estimating $f$. Thus, the robustness to misspecification of $f$ in the pseudo-reward (lines 136 – 137) does have important theoretical implications when Assumption 3 fails. It is for this reason that we listed parametric models in lines 151-152. This is a subtle point that we could briefly explain in the final version of the paper.

Weaknesses & Hyperparameters

Thank you for raising these concerns. Other reviewers asked about them as well. Our main rebuttal contains a detailed discussion of all three. Below, we include a brief summary of the main points and a few additional details for your specific questions:

1. Computation: We made our implementation extremely efficient using sparse rank-one updates to Cholesky factors. In its current form, DML-TS-NNR is about three orders of magnitude faster than is required for a typical mHealth research study. Several optimizations and approximations could be used to speed up the algorithm even more for large data sets.
2. Known Network: This limitation is shared with all or nearly all of the related work in graph bandits. Our new sensitivity analysis shows that the algorithm is fairly robust to specification of the network. The algorithm can easily be extended to include edge weights; the only required change is scaling the corresponding entries in the Laplacian matrix.
3. Hyperparameters: We added eight sensitivity analyses showing that the simulation results are robust to variations in the hyperparameters. While hyperparameters do affect the performance of DML-TS-NNR, the differences between methods are much larger than the differences due to changing the hyperparameters. Empirical Bayes strategies can be used to set the hyperparameters. The penalty hyperparameters ($\lambda, \gamma$) can also be interpreted as prior precisions (inverse variances), which can provide some intuition on an appropriate order of magnitude.

Long-term Treatment Effects

We see two main approaches to address long-term effects.

The first approach is specifically designed to address treatment fatigue. It involves modifying the observed rewards by subtracting a penalty from rewards at treated time points (i.e., when $A_{i,t} > 0$). Or, equivalently, we could modify the algorithm such that individuals are treated with probability equal to the “posterior” probability that the differential reward is above a positive threshold as opposed to zero; see Equation (10). The threshold could be set based on a moderation analysis using the weighted, centered least squares method of Boruvka et al., 2018 [2]. Specifically, we would estimate treatment moderation according to a model including a linear term for the number of previous treatment occasions (perhaps over some moving window, such as the last 10 time points). Then we would set the threshold equal to the corresponding estimate for this linear moderation term, effectively anticipating the impact of current treatment on future outcomes.

The second approach would be to generalize some of the ideas from this paper to more flexible RL methods, such as Q-learning. These methods directly model long-term effects, but uncertainty quantification is generally more challenging relative to bandits and may require some form of Markovian assumption, which would likely be unrealistic in mobile health.

We do not have immediate plans to pursue these directions. However, we are actively involved in designing and analyzing mobile health studies, and we plan to use DML-TS-NNR to optimize online decision making in these studies. If we observe empirically that DML-TS-NNR consistently sacrifices long-term performance for small short-term improvements, then we will revisit this problem and the approaches outlined above.

[2] Boruvka, A., Almirall, D., Witkiewitz, K., & Murphy, S. A. (2018). Assessing time-varying causal effect moderation in mobile health. Journal of the American Statistical Association, 113(523), 1112-1121.