Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments

Comparison against standard A/B

Excellent comment! First, we would like to clarify that different estimators apply to different use cases. Taking ridesharing as an example, there are three different types of experiments: (i) temporal randomization (over time), (ii) spatial randomization (across geographic areas), and (iii) user-level randomization (across drivers/passengers). Our primary focus is (i), which applies to the evaluation of order dispatch policies that must be implemented city-wide, making (ii) and (iii) unsuitable. Spatial randomization (ii) is typically used for testing localized subsidy policies in different regions, while user-level randomization (iii) applies when assigning personalized subsidies to individual users. Second, when constraining to temporal randomization, the population corresponds to the entire time horizon, with each time interval representing an individual unit. When no carryover effects exist and residuals are uncorrelated (satisfying i.i.d. assumptions), similar to Theorem 1, we can show that standard uniform randomization over time is equivalent to both AD and SB. This is because temporal ordering becomes irrelevant under the uncorrelatedness assumption. Similarly, when randomizing is conducted at the daily level rather than per time unit, standard procedures are equivalent to AD designs.

We will be very happy to include these discussions in the paper shall it be accepted, to better connect AD, SB to vanilla A/B testing.

MSE & confidence intervals (CIs)

This is another excellent comment. We agree that CIs and p-values are equally important, as A/B testing is essentially a statistical inference problem. Our designs are tailored to minimize the MSE of the resulting ATE estimators. A closer look at the proof of Theorem 1 suggests that the three RL-based estimators are asymptotically normal, and their MSEs are dominated by their asymptotic variances. As such, optimal designs minimizing the variance of the resulting ATE estimator also minimize the length of confidence intervals and maximize the power of the resulting hypothesis test.

As we use RL-based estimators for A/B testing, existing methods developed in the RL literature are directly applicable for CI construction. These methods can be categorized into the following four types:

1. Concentration-inequality-based methods that construct CI based on concentration inequalities (Thomas et al., 2015, AAAI; Thomas & Brunskill, 2016, ICML; Feng et al., 2020, ICML)

2. Normal-approximation-based methods that utilize the asymptotic normality of the ATE estimator to construct Wald-type CIs (Luckett et al., 2020, JASA; Liao et al., 2021 JASA; Shi et al., 2021, ICML; Kallus and Uehara et al., 2022, OR, Liao et al., 2022, AoS; Shi et al., 2022, JRSSB; Wang et al., 2023, AoS)

3. Bootstrap methods that employ resampling for CI construction (Hanna et al., 2017, AAAI; Hao et al., 2021, ICML)

4. Empirical likelihood methods (Dai et al., 2020, NeurIPS).

During the rebuttal, we conducted additional simulation studies, employed a non-parametric bootstrapping method to construct CIs and reported the coverage probability (CP) & the average CI width in these plots. It can be seen that most CPs are over 92%, close to the nominal level. Meanwhile:

• For small values of $\lambda$, more frequent policy switch reduces the average CI width;
• When $\lambda$ is increased to 10%, AD produces the narrowest CI on average.

These results verify our claim that a reduction in MSE directly translates to a shorter CI.

Limitations.

Our framework relies on the Markov assumption. We have discussed the validity of this assumption in our motivating ridesharing example and outlined several potential approaches to handle settings when this assumption is violated. Refer to our response #2 to Reviewer yQjJ.

Transfer findings to real world scenarios.

In the discussion section, we proposed a three-step workflow for applying our theoretical framework to real-world experimental design:

(i) To make sure our findings are transferable, the first step is to properly determine the time interval to satisfy the Markov assumption. When historical dataset is available, this assumption can be verified based on existing tests (Chen & Hong, 2012, Econ Theory; Shi et al., 2020, ICML; Zhou, et al., 2023, JRSSB).

(ii) The second step is to examine the size of the carryover effects, which in practice can be estimated using company's simulator that evaluate new product's impact prior to deployment.

(iii) The last step is to determine the correlation structure, which can again be achieved using the historical data.

Presentations.

We will polish the paper to make it more accessible to general audience.

Subjective definitions

We did not define these terms to make the paper easy to follow. To address your comment, here are their precise mathematical definitions:

• ''Positively (negatively) correlated'' refers to the covariance $\sigma_e(t_1, t_2)$ being positive (resp. negative);
• A ''large carryover effect" occurs when $\delta \gg (R\_{\max} T)^{-2} \sum_{\substack{k_2-k_1=1,3,5, \ldots\\ 0\le k_1<k_2< T/m}} \sum_{l_1,l_2=1}^{m} \sigma_{e}(l_1+k_1m, l_2+k_2 m)$ so that Equation (5) will be dominated by the second term.

Markov assumption (MA)

We thank the reviewer for highlighting the MA, which is critical for our RL-based estimators. In collaboration with our ride-sharing industry partner, we've observed that intervals of 30 minutes or 1 hour typically satisfy MA, showing strong lag-1 correlations with rapidly decaying higher-order correlations. This justifies the use of RL in our application.

When applied to more general applications, we recommend to properly select the interval length to meet MA as an initial step in the design of experiments (see Fig. 6). If that's challenging, we further propose three approaches below, tailored to different degrees of violation of MA. Our current results directly extend to the first two cases, while the third case requires further investigation:

1. Mild violation: Future observations depend on the current observation-action pair and a few past observations. This mild violation can be easily addressed by redefining the state to include recent past observations. With this modified state, MA is satisfied. Our RL-based estimators and theoretical results remain valid.

2. Moderate violation: Future observations depend on a few past observation-action pairs. Here, the RL-based estimators remain applicable if the state includes these historical state-action pairs. However, our theoretical results on optimal designs must be adjusted. Preliminary analyses show that, under weak carryover effects and positively correlated residuals, the optimal switching interval extends to 1+k (where k is the number of included past actions) rather than switching at every time step. This is because each observed reward is affected by a k+1 consecutive actions, not just the most recent action. More frequent switching under these conditions causes considerable distributional shift, inflating the variance of the ATE estimator.

3. Severe violation: Data follows a POMDP. Although the existing literature provides doubly robust estimators and AD-like optimal designs (Li et al., 2023, NeurIPS) to handle such non-Markov MDPs, these estimators suffer from the "curse of horizon" (Liu et al., 2018, NeurIPS). Recent advances propose more efficient POMDP-based estimators (Liang & Recht, 2025, Biometrika) and designs (Sun et al., 2024, arXiv:2408.05342); however, these proposals are limited to linear models. Extending these methodologies to accommodate more general estimation procedures (e.g., Uehara et al., 2023, NeurIPS) represents an important direction for future research.

We also remark that in the first two cases, existing tests are available for testing the Markov assumption and for order selection (Chen & Hong, 2012, Econ Theory; Shi et al., 2020, ICML; Zhou, et al., 2023, JRSSB).

More real-world scenarios

Refer to our response #2 to Reviewer xXCu regarding other data sources.

One-step effect

We appreciate this insightful comment and would like to clarify that our analysis does not assume a one-step carryover effect. Under the MDP framework, each action potentially influences all subsequent rewards throughout the trajectory, and our RL-based estimators explicitly account for these long-term effects. As illustrated in Figure 2, an action (e.g., $A_{t-1}$) affects not only the immediate next reward ($R_t$) but also subsequent rewards. Nevertheless, the MDP structure is first-order Markovian, implying that these delayed effects propagate entirely through the state $S_{t+1}$.

Measuring $\delta$

$\delta$ can be measured using the size of delayed effects (e.g., $\lambda$), defined as the difference between the ATE and the direct effect. This is because under the MDP formulation, delayed effects are caused by differences in state transitions between different actions. Thus, these effects serve as natural measures for $\delta$. Practically, $\lambda$ can be estimated by computing the difference between our RL-based estimator for ATE and existing direct effect estimators (Dudik, 2014, Statistical Science).

Results when $\lambda=$5% ~15%

We conducted additional simulations using real datasets from the two cities and reported the results when $\lambda=$ (7.5%, 10%, 12.5%). Notably, as $\lambda$ increases, AD appears as the optimal design due to the increase in the carryover effect. This again aligns with our theory.

1. Realistic Covarince Structures : In Corollaries 1–3, we focused on autoregressive, moving average, and exchangeable covariance structures primarily to derive closed-form expressions for the first term (AC(m)) in Equation (5). While these structures might seem simple, they are widely used in practice (Williams, 1952, Biometrika; Berenblut et al., 1974, Biometrika; Zeger et al., 1988, Biometrika).

During the rebuttal, we have relaxed these assumptions and obtained the following corollaries that allow more realistic covariance structures:

Corollary 4 (Truncated covariances). Suppose $\sigma_e(i,j)=0$ whenever $|i-j|> K$ but remains positive when $|i-j|\le K$. Then AC(m) decreases monotonically in $m$ for any $m\ge K$.

Corollary 5 (Decaying covariances). Suppose $\sigma_e$ is nonnegative and satisfies $\sigma_e(i,j)=O(|i-j|^{-c})$ for any $c>2$. Then for any $m_1\le m_2$, we have $AC(m_1) \ge AC(m_2) + o(T^{-1}m_1^{-1})$.

The truncated covariance assumption in Corollary 4 is substantially weaker than the moving average assumption in Corollary 2. First, it does not require the covariance function to be stationary. Second, residuals are not required to be sums of white noise processes.

Similarly, the decaying covariance assumption in Corollary 5 is considerably weaker than the autoregressive assumption in Corollary 1. Like Corollary 4, it does not require stationarity. Additionally, it accommodates polynomial decay of covariance functions with respect to the time lag, thus allowing for stronger temporal dependencies compared to the exponential decay assumed in Corollary 1. As a trade-off, Corollary 5 provides a weaker guarantee: monotonicity holds asymptotically, only when $m_1$ and $m_2$ are sufficiently large.

2. Sensitivity Analysis and Other Data Sources:

Regarding Sensitivity Analysis:

(i) In synthetic environments, we conducted a sensitivity analysis to assess the robustness of our results under four additional covariance structures: moving average, exchangeable, uncorrelated, and autoregressive with a negative coefficient. The findings, summarized in Figures S1 & S2 (Appendix B), consistently align with our theories.

(ii) In the real-data analysis, we analyzed two datasets collected from different cities. We conducted a sensitivity analysis on the effect size, from zero to small, moderate, and large. Once again, these results support our theoretical conclusions.

Regarding Other Data Sources:

Publicly available experimental datasets suitable for our analysis are scarce. However, during the rebuttal, we found a publicly available simulator, which simulates a ridesharing market environment described in Xu et al. (2018, KDD). In this simulator, drivers and customers interact within a 9×9 spatial grid over 20 time steps per day. We used this simulator to conduct additional simulation studies, comparing the MDP-based order dispatch policy (Xu et al., 2018, KDD) against a distance-based dispatch method that minimizes the total distance between drivers and passengers. We varied the number of days $n=28,35,42,49,56$ and set $m=1,5,10,20.$ For each scenario, we tested 100 orders with the number of drivers generated from a uniform distribution $U(40,45).$

We report the RMSEs of ATE estimators computed under different designs, aggregated over 200 simulations, in this link, see the first plot. It can be seen that less frequent switching generally achieves smaller RMSE (m=10, 20). Further analysis shows that ATE is relatively small (~2%), suggesting weak carryover effects. Additionally, we observed negatively correlated reward residuals (see the second plot in the link). Under these conditions, our theory suggests that AD performs the best (see bullet point 3 on page 7), which aligns with the empirical results.

3. RL-based vs Conventional Estimators:

We ran extra simulations to compare the RL-based estimator with three non-RL estimators:

• Bojinov et al.'s (2023, Management Science) sequential importance sampling (IS) estimator, which handles carryover effects through time-dependent IS ratios;

• Hu et al.'s (2022, arXiv:2209.00197) difference-in-mean estimator, which employs burn-in to discard rewards during policy transitions to handle carryover effects;

• Xiong et al.'s (2024, arXiv:2406.06768) simple IS estimator that doesn't handle carryover effects.

As shown in our results, RL-based estimators consistently outperform methods (i) and (ii) in all cases. Moreover, DRL and LSTD perform much better than (iii) in most cases.

We deeply appreciate your thoughtful suggestions and will incorporate these analyses into our paper shall it be accepted.