Robust Sampling for Active Statistical Inference

We really appreciate your comprehensive feedback. We have responded to each point individually below, and hope our responses adequately resolve your concerns.

---

“Sensitivity to choice of budget-preserving path: In Figure 4, the linear and Hellinger budget-preserving paths appear to perform about as well as uniform sampling. In general, how sensitive is the method’s performance to the choice of budget-preserving path? In practice, how would a user select a good budget-preserving path?”

Our central claim is that our procedure yields an estimator that performs no worse than either the uniform or active baseline, and that the gains can in many cases be significant. We cannot guarantee the extent of performance gain for any specific problem.

However, to further evaluate the possible path choices, we added a new experiment comparing the geometric and linear paths against the active and uniform baselines in the setting of Figure 3. The findings presented below support our central claim above: the linear path consistently performs no worse than either of the two baselines, even if it does not achieve the same performance as the geometric path.

Burn-in size 606:

Burn-in size 1213:

Burn-in size 1819:

Burn-in size 2426:

Burn-in size 3639:

---

“Uncertainty set: Theorem 2 requires that $\hat{\rho} = \rho^*+o_p(1)$. Does the degree of misspecification of $\hat{e}$ affect whether this condition is violated or not? How does incorporation of the uncertainty set affect whether/when this condition is violated?”

Good question: the consistency of $\hat{\rho}$ is indeed affected by the degree of misspecification in $\hat{e}$. When $\hat{e}$ is consistent for $e$, the consistency of $\hat{\rho}$ can be shown under mild conditions (as detailed in the Appendix). However, a high degree of misspecification in $\hat{e}$ leads to $\hat{\rho}$ being inconsistent, which necessitates the incorporation of a robustness constraint.

The size of the constraint set influences $\rho_{\mathrm{robust}}$: a small set results in $\rho_{\mathrm{robust}}$ being close to 0 (active), while a large set yields $\rho_{\mathrm{robust}}$ close to 1 (uniform). If the constraint set is too large, we may lose the guarantee of convergence to $\rho^*$, but by appropriately choosing the constraint set $\mathcal{C}$ we still offer improved performance over the uniform baseline, as demonstrated empirically.

---

"In section 2.2, it is mentioned that a permissive uncertainty set results in a large value of $\hat{\rho}$, which is intuitive: A misspecified predictive model is less likely to produce a reliable active sampling policy. Are there advantages to solving the robust optimization problem instead of, for instance, adding a penalty for small values of \rho to the original optimization problem (line 131)?"

This is a very interesting suggestion. Both approaches are interpretable and computationally efficient. We feel that our approach is more interpretable, since it directly incorporates a robustness wrapper around the source of the misspecification, which are the error estimates. The regularization approach does this more implicitly. Tuning the regularization parameter also feels potentially more involved than determining the robustness constraint $c$, which is simply based on measuring the error of $\hat e$ on a small dataset. We will add a brief discussion of this alternative in the revised version of the manuscript. Thank you for suggesting it!

---

Thank you for your detailed review and helpful feedback. We have addressed each of your points below. Please let us know if these responses adequately address your concerns.

---

“The novel contributions of the paper need to more clearly delineated. As the paper is written, one might think that this paper is the first to consider the concept of finding the minimum variance interpolating sampling policy. However, this idea was already proposed in the original Zrnic and Candes active statistical inference paper, albeit only for the case of linear interpolants and without any robustness constraints. The contributions of this paper seem to be that it (a) expands the set of interpolating paths, (b) that it adds a robustness constraint to the variance estimator, and (c) that it gives a more thorough evaluation of uniform v.s. active v.s. robust active.”

Thank you for this observation. We acknowledge that we missed including a discussion of the robustness proposal from [1]. We will make sure to clarify the distinction in the revised version of the paper. The proposal from [1] assumes access to historical data to tune the linear interpolation coefficient. Otherwise, it selects a default value for the coefficient, such as 0.5, which does not have the guarantee of outperforming uniform and active sampling. Our analysis is far more thorough and systematic, expanding the set of interpolating paths, not requiring historical data but incorporating a burn-in period, and adding in a robustness constraint. These are all crucial for the practicality and reliability of the method. For example, the robustness constraint is critical: for natural choices of $\pi$ proposed in [1], tuning the interpolation coefficient only would result in trivial sampling rules (see paragraph starting in line 230).

---

“It is hard to tell how useful the robustness constraint is. There is one figure comparing the performance with and without the robustness constraint, confusingly shown twice (Figure 4 and Figure 10). However, because the x and y-axes are not the same between the two, I can't really eyeball how useful the robustness constraint is. There should be several figures comparing with and without robustness for the various path interpolants on ALL the datasets, and they should also be on the SAME axes (maybe use dotted and solid lines). It would also be helpful to see what happens as the robustness hyperparameter c is varied.”

This is a very good suggestion. (And thank you for pointing out the repeated figure; Figure 10 includes the coverage plots accompanying Figure 4 due to the space limit of the main paper.) The robustness constraint is indeed crucial. As discussed in Section 4.1 (line 230), if we set $\pi \propto \hat{e}$, without the additional robust optimization step, the method would trivially estimate $\hat{\rho} = 0$. This means that in all experiments that rely on this choice of $\pi$, the plain active baseline is equivalent to not including the robustness constraint. To give some numbers, here are some results from Figure 5:

Below we give an additional example when $\pi$ is not proportional to $\hat{e}$, specifically in the setting of Figure 4, where the burn-in size is 1819, and $\pi \propto \min\\{f(X), 1-f(X)\\}$:

We will include the above results in the revision. Thank you again for this valuable suggestion.

---

“There is no mention of multiple runs/random seeds in the paper nor are there any confidence intervals/error bars in the plots. Given how much randomness is involved in this paper (initial training set, random label selection according to the policy, etc), it would be good to see how confident we should be in the results of the paper.”

We appreciate and agree with this suggestion. We average all metrics over 500 trials. For the coverage plots, given that they are obtained by averaging 500 binary indicators, the standard deviation does not add any information, since it equals $\sqrt{\mathrm{coverage}(1-\mathrm{coverage})}$, and should not be plotted (as per standard convention). However, we will plot the standard deviation of the effective sample size everywhere. Below we include the numbers for the problem of collecting politeness annotations with LLMs.

---

“As the paper currently stands, there is no guidance on how to choose the interpolating path. There is one experiment showing that the geometric path empirically outperforms the other two proposed paths on the Pew survey dataset, but there's no reasoning given. Moreover, it's worth pointing out that one could actually consider many types of paths simultaneously and choose the best path/interpolating fraction based on minimizing the variance estimator.”

We have stress tested the geometric path on multiple datasets (with different modalities, uncertainty scores, etc) and found it to be the best choice among the considered paths. While we cannot guarantee that this will be the case everywhere, our main goal is to provide a method that consistently performs no worse than uniform or plain active sampling. Across all datasets, interpolating paths, uncertainty measures, we have found this core message to be true. Developing a deeper understanding around what properties of the problem determine the right interpolating path is a non-trivial and exciting future direction. We agree that it would be beneficial to simultaneously consider multiple paths and select the best one automatically. The risk is that this might be somewhat less practical, but it would be worthwhile to study the performance gains and see if they justify the extra effort. For now, given our current findings, we recommend the geometric path as a simple, practical default.

---

[1] Zrnic, Tijana, and Emmanuel Candes. "Active Statistical Inference." International Conference on Machine Learning. PMLR, 2024.

Thank you for your comprehensive review and your valuable feedback. We detail our response below, and please kindly let us know whether our response addresses your concerns.

---

“The process of obtaining $\rho_{\mathrm{robust}}$ is not fully specified in Algorithm 1. Do the authors perform grid search or use gradient-based methods to solve the minimax problem? If grid search is used, how is the grid step chosen? The paper should address the sensitivity of the solution to the grid step size or local minima, especially since small changes in \rho can lead to large changes in the objective function.”

We employed a grid search for $\rho$ in the outer loop, utilizing a step size of 0.01. The inner problem, being convex, is straightforward to solve and even possesses a closed-form solution when the constraint set is simple, such as the $\ell_2$-ball. If the objective function is Lipschitz, we can always locate an $\epsilon$-optimal point in terms of objective value, with the grid resolution determining $\epsilon$.The sensitivity of the results to the grid step size is a good question. We conducted new experiments to explore different step sizes and confirmed the robustness of our results to step-size selection, as shown below. Due to the restriction on uploading images/plots, the results are presented in a table format.

On the Pew election dataset, our results show that a step size of 0.01 yielded $\rho_{\mathrm{robust}} = 0.43$, while a step size of 0.1 resulted in $\rho_{\mathrm{robust}} = 0.4$. The ESS gap between these two estimators is minimal, and both significantly outperform uniform and active baselines.

---

“The choice of the misspecification set C is not well justified. The authors appear to assume an $\ell_2$-ball, but other choices (e.g., $\ell_{\infty}$, greedy perturbations, or adaptive constraints) could lead to different behaviors. There is no discussion or comparison of how different choices of $\mathcal{C}$ affect robustness or optimization difficulty. Given the known difficulty of minimax optimization (non-convexity, local optima), how do the authors ensure the optimal solution can be found?”

As we briefly discussed in Section 2.2, we found that $\ell_2$ misspecification performs better than alternatives empirically. Further intuition is provided in the Appendix. For example, the $\ell_1$ norm is ineffective since it concentrates perturbations on a single error entry, leaving the remaining entries unchanged. Conversely, the $\ell_{\infty}$ norm uniformly distributes perturbations across all error entries, which is also not ideal.

Regarding minimax optimization, as noted in the paper, the inner optimization becomes a convex problem—and thus easily solvable—when the misspecification set $\mathcal{C}$ is convex. This is crucial for the practicality of our proposal. The outer loop can then be solved via a simple, one-dimensional grid search over $\rho$.

---

“In lines 112–113, the paper introduces linear and geometric interpolation paths between $\pi$ and $\pi^{\mathrm{unif}}$. However, the choice of these two is not motivated. Are there other potential interpolation strategies (e.g., entropy-based, adaptive weightings)? A discussion of why only these two paths are chosen and their tradeoffs would strengthen the paper.”

We have also included the Hellinger path in our paper, with its definition and validity detailed in Appendix B. Figure 4 illustrates a performance comparison between all three paths. These specific paths were chosen due to their simplicity. More generally, these are all examples of geodesic paths: paths that are roughly defined as “shortest paths.” Any geodesic path can be used in our proposal. We elaborate and give more examples of geodesics in Appendix B.

---

“The paper does not clearly specify which active sampling method $\pi$ is used in the experiments. This is essential for reproducibility and clarity. Active learning encompasses many strategies (uncertainty sampling, margin sampling etc.). The current evaluation appears to be based on a single $\pi$. To convincingly demonstrate the robustness of the proposed method, it would be helpful to evaluate it on multiple active sampling baselines.”

We agree that stress testing the methodology with different sampling methods $\pi$ is important. We want to clarify that our paper does indeed test different $\pi$ and define these rules for each experiment. For instance, in Figure 2, $\pi(X) \propto \min \\{f(X), 1 - f(X)\\}$ (see line 222); in Figure 3, $\pi(X) \propto \hat{e}(X)$ (see line 233); in Figure 7, $\pi$ is based on verbalized LLM confidence (see line 273). We intentionally choose varying $\pi$ to demonstrate the robustness of the method.

It is important to distinguish our framework from active learning despite their shared "active" nature, as their objectives are fundamentally different. Active inference aims to perform statistical inference on typically low-dimensional parameters. Consequently, many sampling strategies from active learning are not directly applicable or meaningful within the active inference framework, as it would be unclear how to perform valid statistical inference with those strategies. We will clarify these differences in the revised version.

---

“Typos. line 76: "distribution of the unobserved labels" should be "observed labels".”

Thank you for raising this point. This is not a typo; rather, it highlights a crucial motivation behind our work. The true parameter $\theta^*$ is influenced by the entire data distribution of $(X,Y)$. However, we a priori only observe unlabeled $X_i$ (line 73), no labels $Y_i$ (line 74).

---

“The method relies on a well-estimated error $\hat{e}^2$. If this is poorly estimated (e.g., under small data regimes), the robust sampling policy might become unreliable.”

We want to emphasize that our robust sampling procedure remains effective even with a poor error estimation. A poorly estimated $\hat{e}$ would typically result in a deficient active estimator. However, with our additional robustness layer, a poorly estimated error leads to a larger constraint set $\mathcal{C}$, which in turn yields a larger $\rho_{\mathrm{robust}}$. This larger value pushes the sampling towards uniform sampling, ensuring the reliability of our method despite imprecise error estimation. Figure 6 illustrates the relationship between the interpolation coefficient and burn-in size: smaller burn-in sizes, leading to poorly estimated errors, result in larger values of $\rho_{\mathrm{robust}}$.

Relatedly, we see that $\mathcal{C}$ is large when the burn-in size is small; see below. The corresponding dataset is the Pew election dataset, and $c$ is the magnitude of the $\ell_2$ constraint, i.e. $\mathcal{C} = \\{\epsilon: \\|\epsilon\\|\_2 \leq c\\}$.

As expected, we observe a more conservative constraint when the errors are poorly estimated.

---

“The lack of theoretical guarantees about convergence or uniqueness of $\rho_{robust}$ is a concern.”

To reiterate our previous point about computational efficiency: when the constraint set $\mathcal{C}$ is convex, the inner problem of the minimax formulation becomes a convex problem. This simplifies optimization and prevents convergence to local minima. The effect of performing a one-dimensional grid search over [0,1] to search over $\rho$ is minimal, as illustrated above.

Furthermore, in line 767 in the Appendix, we offer a theoretical guarantee: $\hat{\rho}$ will converge to $\rho^*$ provided $\rho^*$ is unique and error consistency is achieved. If a consistent error estimator is available, the misspecification set will asymptotically vanish. Consequently, the consistency of $\rho_{\mathrm{robust}}$, the asymptotic normality of its corresponding estimator w.r.t $\rho_{\mathrm{robust}}$, and the property of its asymptotic variance can be derived by following the same steps used in the proofs of Theorem 1 and Theorem 2. However, formally working out these details is beyond the scope of the present work.

---

[1] Zrnic, Tijana, and Emmanuel Candes. "Active Statistical Inference." International Conference on Machine Learning. PMLR, 2024.

We appreciate the reviewer's comprehensive summary and helpful feedback. We have addressed each of your points below. Please let us know if these responses adequately address your suggestions.

---

“Since the algorithm randomly decides whether to observe each instance, it is possible that all instances are observed in extreme cases. Does this paper allow the budget to be exceeded? If so, can the same claims as in this paper be made while keeping the total number of observations within by adaptively changing the sampling distribution for each observation, similar to active testing (Kossen et al., 2021)?”

Like prior work [1], our algorithm enforces the budget in expectation. However, since the number of labeled data points is a sum of independent Bernoullis, a standard Hoeffding argument guarantees that the realized labeling rate will closely match the budget with high probability. Specifically, the labeling ratio will not exceed $\frac{n_b}{n} + \epsilon$ with a probability of $1 - \delta$, provided that $n > \frac{\log (1/\delta)}{2 \epsilon ^2}$ for any $\epsilon, \delta > 0$. (Alternatively, this can be interpreted as: if we run the algorithm with a slightly reduced budget $n_b - \epsilon$, then with high probability the budget $n_b$ will be met.)

Active testing [2] pursues a different objective (high‑precision risk estimation for a fixed model) and uses a distinct estimator so the program is very different. That said, the active data collection is closely related to the setting of active inference. Thank you for pointing this out–we will add a discussion of the connection between active inference and active testing in the revision. However, our primary contribution lies in robustifying the sampling process, rather than proposing a new active approach to statistical inference. We believe that this is a novel contribution.

---

“In lines 9, 101, and 297, is this claim asymptotic? Does it hold for finite samples?“

Yes, our theoretical claim is asymptotic, as supported by Theorems 1 and 2. We will make sure to clarify this. Our empirical findings in the experimental section show this claim holds true in finite samples.

---

“The explanation of the inconsistency in the axis scales in all figures, including Figure 1, and how to read the figures should be clarified.”

Thank you for flagging this issue. We acknowledge the presence of multiple axis scales within the paper, specifically for sample size, coverage, and interpolation coefficients.

For coverage and interpolation coefficients, the scale remains consistent across all figures, ranging from 0 to 1. The scale for the budget $n_b$ varies between figures due to several factors, including dataset size (for example, we set the budget as a percentage of the overall dataset size, e.g. 10%) and burn-in size (which is included in the budget). The y-axis, representing the effective sample size (ESS), is determined by how much the procedure gains over uniform sampling, which is in turn determined by the accuracy of the predictions and uncertainties.

To accurately interpret the figures, the reader should understand the concepts of effective sample size (ESS) and coverage.

ESS quantifies a method’s efficiency compared to uniform sampling. For instance, if our method utilizes a labeling budget of $n_b = 500$ and achieves an ESS of $n_{\mathrm{eff}} = 600$, this means that our method with 500 labels is as effective as uniform sampling with 600 labels. A higher ESS relative to the baseline signifies greater efficiency. (The uniform estimator by definition will always have an ESS equal to its budget ($n_{\text{eff}} = n_b$)). Readers can also refer to the definition of effective sample size provided in Lines 193-200 of Section 4.

Coverage captures how often the constructed confidence intervals cover the quantity of interest. Coverage should be at least 0.9 for the error to be controlled, and ideally it should be as close to 0.9 as possible.

---

“References to related research. The differences between active testing and the present setting should be mentioned. In addition, the issues with performing statistical inference after classical active learning should be mentioned. My understanding is that the bias caused by not randomly selecting instances is the problem. Is this correct? If so, related research on statistical inference after decision-making based on statistical models, such as post-selection inference (Lee et al., 2016), should be mentioned to see if the bias of active selection can be corrected. Perhaps this is difficult because the selection of instances by active learning does not fit into the framework of post-selection inference. Is that correct?”

Thank you for this suggestion. We will expand on the distinctions between active testing and active inference in the revised version, as discussed above. However, we want to reiterate: our primary contribution is to robustify the sampling of active inference. This means our sampling method performs well regardless of the uncertainty scores obtained, a benefit not offered by active testing. It is true that classical active learning's greedy label selection can disrupt inference by introducing bias. Our method, consistent with [1], effectively mitigates this bias using an inverse probability weighting method.

Post-selection inference (POSI) [3] addresses a fundamentally different problem: adaptively selecting hypotheses based on fixed data. Our framework, however, focuses on adaptively choosing data based on fixed hypotheses. These represent two different sources of bias, each with its own unique challenges. The bias mitigation techniques developed and leveraged in the two lines of work are entirely distinct.

---

“It is assumed that $\hat{e}$ converges in probability to $e$, but I am not sure if this assumption is strong. Are sufficient conditions given?”

Thank you for your question. The assumption that $\hat{e}$ converges to $e$ follows from classical arguments. As the size of the training or burn-in dataset increases, more information becomes available to estimate the conditional expectation. Flexible estimators, such as the multilayer perceptron (MLP) and XGBoost models used in the paper's experiments, are universal approximators. With sufficient data, they can learn complex relationships and provide a consistent estimate of the true conditional expectation, $e$, assuming the model is correctly specified and $e$ belongs to a sufficiently smooth or low-complexity class. While the rate of convergence depends on the specific estimator and the complexity of the true function, the convergence itself is a standard result from learning theory.

(More generally, this type of assumption is common in the field of semiparametric inference. For instance, the widely-used doubly robust estimator [4, 5], which is closely related to our estimator, relies on consistent estimation of nuisance functions.)

---

“Is $Y$ limited to 0 or 1? Can this be extended to general distributions?”

Our methodology does not assume $Y$ to be restricted to 0/1. Indeed, we consider general convex M-estimation problems, such as linear regression. For example, in the experiment on US Census data in Figure 5, $Y$ corresponds to income and is thus a continuous variable.

---

“If $\mathcal{C}$ is an $\ell_2$-ball and the radius is reduced to 0 (as $n \rightarrow \infty$), can the same theorem be derived?”

This is a good question. Roughly speaking, if the estimated error is consistent and the constraint set $\mathcal{C}$ vanishes as $n \rightarrow \infty$, the consistency of $\rho_{\mathrm{robust}}$, the asymptotic normality of its corresponding estimator w.r.t $\rho_{\mathrm{robust}}$, and the property of its asymptotic variance can be derived by following the same steps used in the proofs of Theorem 1 and Theorem 2. This supports the asymptotic validity of the robust sampling procedure. However, formalizing this precisely is beyond the scope of the present work.

---

“I think it would be better to clarify what the expected values refer to, such as line 85 and Equation (2). I think that expected values of $X$ and those of $(X,Y)$ are mixed together.”

The expectation is consistently defined throughout the paper: all expectations $\mathbb{E}$ average out all randomness in the argument, unless a conditional expectation is used, in which case the conditional expectation corresponds to a random variable that is measurable with respect to the variable being conditioned on.

---

[1] Zrnic, Tijana, and Emmanuel Candes. "Active Statistical Inference." International Conference on Machine Learning. PMLR, 2024.

[2] ​​Kossen, Jannik, et al. "Active testing: Sample-efficient model evaluation." International Conference on Machine Learning. PMLR, 2021.

[3] Lee, Jason D., et al. "Exact post-selection inference, with application to the lasso." (2016): 907-927.

[4] Glynn, Adam N., and Kevin M. Quinn. "An introduction to the augmented inverse propensity weighted estimator." Political analysis 18.1 (2010): 36-56.

[5] Robins, James M., Andrea Rotnitzky, and Lue Ping Zhao. "Estimation of regression coefficients when some regressors are not always observed." Journal of the American statistical Association 89.427 (1994): 846-866.

Thank you for your thorough and constructive feedback. See below for our responses to your points, and please let us know if these responses adequately address your suggestions.

---

“I am not entirely convinced how practical this method might be given its reliance on some hand-picked hyperparameters, namely the set of misspecification ($\mathcal{C}$) and the choice of path. While the choices made in the paper seem to work well, they do not sufficiently convince me how the results might generalize.”

We appreciate this comment. We have tried several options for both the path and the misspecification set, and across many datasets (including varying data modalities) and uncertainty measures. We have found that the method is reliably never worse than uniform sampling, and often outperforms uniform sampling and active inference by a substantial margin. We have done our best to stress test the methodology; if you have suggestions for additional experiments for validating these observations, we will gladly run them.

We would also argue that two parameters (misspecification set and choice of path) is not an impractically large number of user-chosen parameters. In our extensive evaluations we have found the $\ell_2$-constraint and geometric path to be simple and lead to good performance. This is why, to maximize practicality and ensure running the method is as streamlined as possible, we recommend them as a default.

---

“Relatedly, in Fig 3, I am not sure how much of the performance difference can be attributed to the overall scheme vs. just the choice of path. Just eyeballing Fig 4 against Fig 3, it appears the choice of path leads to a substantial difference in performance, with a similar magnitude to the performance difference in Fig 3. It would be useful to see where, say, a linear path lands in Fig 3. Essentially some more ablation studies to really understand the benefits of the proposal.”

Thank you for this insightful suggestion. Our core claim is that our procedure consistently yields an estimator that performs no worse than either the active or uniform estimator. However, we acknowledge that the degree to which it outperforms these baselines is highly variable, depending on the specific datasets, paths, and models used.

To address your question, we conducted new experiments comparing the geometric path, linear path, and the baselines (active and uniform) in the setting of Figure 3. The results, presented below, demonstrate that while the linear path does not achieve the same performance as the geometric path, it consistently performs no worse than either of the two baselines. This finding supports the above claim. We will add these new results to the revised version of the paper. Due to the restriction on uploading images/plots, the results are presented in a table format.

Burn-in size 606:

Burn-in size 1213:

Burn-in size 1819:

Burn-in size 2426:

Burn-in size 3639:

---

“For the choice of $\mathcal{C}$: the $\ell_2$ norm constraint doesn’t seem very intuitive. What if the error estimates are off systematically? We might suspect such systematic differences for the regions where the model is uncertain vs. where it is fairly certain. Would this still bound the overall variance? Would it likely just converge to uniform sampling in practice?”

We agree that a simple $\ell_2$ norm constraint may not always be the most powerful choice of $\mathcal{C}$. Zooming out, our method can in principle be combined with any choice of $\mathcal{C}$, including one where we learn regions of the space where scores are systematically overconfident or underconfident. At a high level, our method (1) learns $\mathcal{C}$ (in our experiment, the “learning” is a simple fitting of $c$ through cross-validation), and (2) solves a robust optimization problem with $\mathcal{C}$ in place. Your suggestion is an interesting choice of step (1).

We developed a dataset featuring a central "hard" region ($|X| \leq 2$) flanked by two "easy" regions ($2<|X|<5$). In the easy regions, error data was sampled from $\mathcal{N}(2, 0.05)$. In the hard region, error was drawn from $\mathcal{N}(1, 0.25)$. The estimator of error, $\hat{e}(X)$, is designed to underestimate the error in the hard region and overestimate the error in the easy region. Specifically, $\hat{e}(X) = 0.5$ for $|X| \leq 2$, and $\hat{e}(X)=2.5$ otherwise.

Subsequently, we trained a meta-classifier, a gradient boost classifier, $h(X)$, to identify these regions solely based on the performance of $\hat{e}(X)$, without prior knowledge of the region boundaries.

This approach proved highly effective, with the meta-classifier achieving over 99% accuracy in identifying the regions. This demonstrates our success in learning the error regions and enables us to separate the constraint set $\mathcal{C}$ based on these distinctions. For instance, $\mathcal{C}$ can be defined as $\\| \epsilon_{\mathrm{easy}} \\|\_2 \leq c_{\mathrm{easy}}$ for the easy region ($2<|X|<5$) and $\\|\epsilon_{\mathrm{hard}}\\|\_2 \leq c_{\mathrm{hard}}$ for the hard region ($|X| \leq 2$). Or even simpler, we can only optimize over hard regions, i.e. $c_{\mathrm{easy}}=0$. While these regions' dimensions are not fixed and depend on $X$, this presents no practical difficulties because we have complete information about $X$.

Next, we compared this structured constraint with the global constraint. Here, for the structured constraint, we only optimize over the hard region. The following table shows the result when $n=7000$, $n_b=1400$, and $\pi \propto \hat{e}$.

We found that incorporating the structured constraint provided a slight gain in ESS over the global constraint while reducing the constraint size ($c_{\mathrm{global}} = 85$ vs. $c_{\mathrm{hard}} = 75$). This suggests that a more focused perturbation can be beneficial when we have strong knowledge of confident regions. However, we note that the global constraint remains a simple and practical approach given the limited gain.

---

“Minor: in Fig 2, consider using the same scaling for both figures so it’s easy to quickly compare the two. You could even combine the two figures and just use different formatting for the curves (-- vs .-). In line 214, “we collect all labels $Y_i$”, I presume you mean just the burn-in labels. In Fig 3, curious why are we using such odd burn-in sizes?”

This is a helpful suggestion. We have revised Figure 2 as suggested, aligning the x-axes. Yes, "we collect all labels $Y\_i$" refers to the burn-in labels; we will rephrase this sentence for clarity. The burn-in sizes in Figure 3 are set by the proportion of the burn-in data to the whole data (e.g., 10% for a burn-in size of 606), which explains their seemingly unusual appearance. We will clarify this in the revised version.

---