Response to Reviewer ibeC

We sincerely thank you for recognizing the novelty of our methodological contribution, the thoroughness of our theoretical analysis, and the compelling empirical validation presented in our work, as well as for your thoughtful and constructive feedback, which is invaluable for refining the manuscript. Below, we address each of your concerns point by point:

Responding to Weaknesses

We appreciate your concern regarding the accessibility of our paper to non-experts in active inference. We acknowledge that while the mathematics in the paper is presented rigorously, some readers may find it difficult to distinguish the novel aspects of our method from existing techniques.

What is novel and what is standard?

• Standard components: Active inference in the sense of Zrnić & Candès (2024) uses independent probability-proportional-to-uncertainty sampling together with an AIPW estimator.

• Our new contributions:

1. We reinterpret model uncertainty as a continuous auxiliary variable and impose a population-level balancing constraint on it;
2. We are, to the best of our knowledge, the first to integrate the cube method into active inference.
3. We provide closed-form asymptotic variance expressions and prove that they are strictly smaller than those of independent sampling, yielding quantitative efficiency gains.

We will revise the manuscript in the camera-ready version to make the contributions and methodological innovations more clearly distinguishable from standard components.

Responding to Questions

Q1. Explanation of Remark 1

We appreciate the reviewer’s comment regarding Remark 1. To clarify, Remark 1 states that if $\\hat{f}$ is balanced, we have the balance condition: $\\sum_{i=1}^n \\frac{\\hat{f}\\left(X_i\\right) \\xi_i}{\\pi\\left(X_i\\right)}\\approx\\sum_{i=1}^n \\hat{f}\\left(X_i\\right).$ Under this condition, the estimator simplifies as follows:
$

\\begin{aligned}
    \\tilde{\\theta}&=\\frac{1}{n} \\sum_{i=1}^n\\left[\\hat{f}\\left(X_i\\right)+\\left(Y_i-\\hat{f}\\left(X_i\\right)\\right) \\frac{\\xi_i}{\\pi\\left(X_i\\right)}\\right]\\\\
    &=\\frac{1}{n}\\sum_{i=1}^nY_i\\frac{\\xi_i}{\\pi(X_i)} + \\frac{1}{n}\\left[\\sum_{i=1}^n \\hat{f}\\left(X_i\\right) - \\sum_{i=1}^n \\frac{\\hat{f}\\left(X_i\\right) \\xi_i}{\\pi\\left(X_i\\right)}\\right],
\\end{aligned}

$

where the second term vanishes due to the balance condition. This shows that the estimator reduces to the standard inverse propensity weighted (IPW) estimator when $\\hat{f}$ is sufficiently balanced.

Q2. Sensitivity Analysis on $\tau$

Thank you for this important suggestion. We acknowledge that the choice of $\tau = 0.5$ warrants explicit justification.

• Sensitivity analysis on $\tau$ across multiple datasets will be added in the revised version.
• The setup is: fix the sampling budget at 0.2 and compute the RMSE and interval width (analogous to Figure 1 in the main text, but with $\tau$ ranging from 0 to 1 on the x-axis) for methods.
• Below is a representative subset of our experimental results:

Table Q2.1 Bike Sharing, RMSE

Table Q2.2 Bike Sharing, Interval Width

Table Q2.3 Friedman, RMSE

Table Q2.4 Post-election survey research, RMSE

Table Q2.5 Credit Fraud Detection, RMSE

• All coverage rates in the above results closely approximate the target confidence level, confirming the validity of our findings.
• Across all datasets, the RMSE and interval width generally maintained the following ordering: cube_active < poisson_active < uniform < classical.
• At $\tau = 0$: The performance of poisson_active becomes nearly equivalent to uniform, as both rely solely on simple random sampling.
• As $\tau$ increases, both cube_active and poisson_active exhibit precision improvement (decreasing RMSE and interval width), reaching lower value around $\tau = 0.5$ for different datasets. This demonstrably quantifies the precision gains enabled by active sampling.
• When near $\tau = 1$, RMSE and interval width increase on some datasets. This occurs because the model for estimating prediction uncertainty ($\hat{u}$) is inherently imperfect. When the model erroneously assigns near-zero uncertainty ($\hat{u} \approx 0$) to specific data points, it can significantly inflate the estimator variance. These findings are well-aligned with conclusions established in Zrnic & Candès (2024).

References

• Zrnic, T., & Candès, E. J. (2024, July). Active statistical inference. In Proceedings of the 41st International Conference on Machine Learning (pp. 62993-63010).

Thanks for this. I believe that this is a great plan for improving clarity. Although it is very hard to increase score without seeing the resulting manuscript.

Response to Reviewer WQHA

We sincerely appreciate your recognition of the novel methodological contribution, compelling experimental validation, and significant practical implications of our work, as well as your valuable feedback and critiques, which are crucial for refining our paper. Below we provide point-by-point responses that incorporate your suggestions.

Responding to Weaknesses

Weakness 1. Scope of the Paper

Thanks for raising concerns about the scope of the paper. While the generation to M-estimation was briefly discussed in the main text, we will delve deeper by formally extending the Balanced Active Inference framework to the general M-estimation setting, covering both regression and classification, and present some experimental results.

(i) Settings of M-estimation

• Given a family of functions $f(X_i;\theta)$, the task is to solve $\theta^{\ast}= \arg\min_{\theta} \mathbb{E} \left [L\bigl(X_1, Y_1; \theta\bigr)\right]$.
• $L(X_1, Y_1; \theta)$ measures the discrepancy between the true label $Y_1$ and the prediction $f(X_1;\theta)$.
• Our goal is to estimate $\theta^{\ast}$ using a small labeled set $\mathcal{D}_l$ and a larger unlabeled set $\mathcal{D}_u$.
• $f(X_i;\hat{\theta})$ is an initial estimator trained on $\mathcal{D}_l$.

(ii) Uncertainty Measures

• Regression. We use the absolute residual $|Y_i - f(X_i;\hat{\theta})|$ as the uncertainty measure.
• Classification. Let $p(X_i) = (p_1(X_i),\ldots,p_K(X_i))$ denote the predicted class probabilities. We define $u(X_i)=\frac{K}{K-1}\Bigl(1-\max_{j\in[K]} p_j(X_i)\Bigr),$ which peaks when the model is maximally uncertain (uniform distribution) and drops to zero when the model is highly confident in one class.

(iii) M-Estimation

• A model $\hat{u}(\cdot)$ is trained on $\mathcal{D}_l$ to predict the uncertainty of points in $\mathcal{D}_u$.

• Given a labeling budget $n_b$, adopt the sampling scheme described in Equation (2) of the paper.

• Generate an assignment via the cube method, ensuring the balancing constraint (Equation (4)) is satisfied.

• The proposed estimator is

  $\tilde{\theta}=\underset{\theta}{\arg \min } \frac{1}{n} \sum_{i=1}^n\{L(X_i, f(X_i ; \hat{\theta}) ; \theta)+[L(X_i, Y_i ; \theta)-L(X_i, f(X_i ; \hat{\theta}) ; \theta)] \frac{\xi_i}{\\pi(X_i)}\},$

  where $\pi(X_i)$ is the inclusion probability.

(iv) Experiment Results

• Experiments are conducted on both synthetic data (Table W1.1) and the Bike Sharing dateset (Table W1.2).

Table W1.1 Synthetic Linear, RMSE

Table W1.2 Bike Sharing, RMSE (First Feature: temp)

• The results show that our balanced active inference approach has the best performance in terms of RMSE.

A section on M-estimation will be added in the revised version.

Weakness 2. Limited novelty in theoretical contribution

• The theoretical foundations of the cube method are still under development.
• As noted in the most recent work by Davezies et al. (2024), even in simplified settings such as standard survey sampling without model uncertainty, only aysmptotic results are avaliable.
• Non-asymptotic guarantees remain elusive due to (1) the complex dependence induced by the balancing step, and (2) the lack of general concentration tools for such designs.
• Given these challenges, our asymptotic analysis represents a meaningful and necessary step toward understanding the statistical properties of balanced active learning.
• We view non-asymptotic extensions as an important and ambitious direction for future research, but one that lies beyond the scope of this paper.

Weakness 3. Strength of Assumption 5

• We acknowledge that the assumption effectively requires an accurate uncertainty estimator, which may not always hold in practice.

• As noted in Remark 2 of our manuscript, Assumption 5 is primarily introduced to facilitate theoretical analysis, particularly to derive an explicit form of the asymptotic variance.

• This assumption is not a strict requirement for the validity of our method.

• Empirical results consistently show that the proposed estimator achieves substantial variance reduction even when the uncertainty estimators are imperfect, suggesting robustness to mild violations of this assumption.

Weakness 4. Sensitivity Analysis on $\tau$

Thank you for this important suggestion. We acknowledge that the choice of $\tau = 0.5$ warrants explicit justification.

• Sensitivity analysis on $\tau$ across multiple datasets will be added in the revised version.
• The setup is: fix the sampling budget at 0.2 and compute the RMSE and interval width (analogous to Figure 1 in the main text, but with $\tau$ ranging from 0 to 1 on the x-axis) for methods.
• Below is a representative subset of our experimental results:

Table W4.1 Bike Sharing, RMSE

Table W4.2 Bike Sharing, Interval Width

Table W4.3 Friedman, RMSE

Table W4.4 Post-election survey research, RMSE

• All coverage rates in the above results closely approximate the target confidence level, confirming the validity of our findings.
• Across all datasets, the RMSE and interval width generally maintained the following ordering: cube_active < poisson_active < uniform < classical.
• At $\tau = 0$: The performance of poisson_active becomes nearly equivalent to uniform, as both rely solely on simple random sampling.
• As $\tau$ increases, both cube_active and poisson_active exhibit precision improvement (decreasing RMSE and interval width), reaching lower value around $\tau = 0.5$ for different datasets. This demonstrably quantifies the precision gains enabled by active sampling.
• When near $\tau = 1$, RMSE and interval width increase on some datasets. This occurs because the model for estimating prediction uncertainty ($\hat{u}$) is inherently imperfect. When the model erroneously assigns near-zero uncertainty ($\hat{u} \approx 0$) to specific data points, it can significantly inflate the estimator variance. These findings are well-aligned with conclusions established in Zrnic & Candès (2024).

Weakness 5. Claims not well supported

Thanks for pointing out this issue. We will carefully check our claims, and add sufficient support or delete non-supported ones.

Responding to Question

Q1. $\pi$ in Theorem 1

Thank you for the insightful question.

• Theorem 1 holds for any $\\{\pi_i\\}_{i=1}^n$ as long as Assumption 3 in the paper is satisfied.
• The balanced sampling constraint (Equation (4)) is enforced by the Cube method after a $\\{\pi_i\\}_{i=1}^n$ is given.
• Therefore, the variance reduction benefits proven in Theorem 1 are not tightly coupled to a specific construction of $\\{\pi_i\\}_{i=1}^n$.
• We will clarify this point in the revision to avoid potential confusion.

References

• Davezies, L., Hollard, G., & Merino, P. V. (2024). Revisiting randomization with the cube method. arXiv:2407.13613.
• Zrnic, T., & Candès, E. J. (2024, July). Active statistical inference. ICML.

Response to Reviewer oZYC

We sincerely appreciate your recognition of the problem's significance, the practical relevance of our work, and the soundness of our methodological approach. We also find your constructive feedback and critique extremely valuable for optimizing our manuscript. Below, we provide point-by-point responses to your specific suggestions:

Responding to Weaknesses

Weakness 3 (i). Active Learning Strategies in Inference

Thanks for your constructive feedback.

• Our proposed "balanced active inference" method belongs to the emerging field of active statistical inference , which is fundamentally different from traditional active learning. Consequently, we respectfully argue that a direct comparison to AL strategies would be inappropriate.

• The primary goal of traditional AL is to improve a model's predictive performance (e.g., accuracy, AUC) with minimal labeling cost (Deng et al., 2023).

• In contrast, the goal of active inference is to improve the efficiency of statistical inference, i.e., to obtain narrower confidence intervals and more powerful p-values for population parameters, not to improve the model itself. Our work contributes to the latter.

• AL query strategies typically select samples to maximize the model's information gain to better define its decision boundary.

• Active inference sampling strategies are designed to optimally reduce the asymptotic variance of the statistical estimator. As formally derived by Zrnic & Candès (2024), this is a fundamentally different optimization problem tailored for inference, not learning.

• Critically, it has been shown that naively applying AL for statistical estimation introduces significant sampling bias, which invalidates classical statistical procedures (Zhang et al., 2021).

• In summary, comparing our inference-focused method against a learning-focused AL strategy would not be meaningful due to conflicting goals, methodologies, and evaluation metrics. The contribution of our work lies in achieving more efficient and robust statistical inference, and we believe it should be evaluated by those standards.

Weakness 3 (ii) (iii). Discussion of Uncertainty Quantifications

Thank you for this insightful comment.

• The proposed method belongs to the error-prediction paradigm in active inference. This strategy was established by Yoo & Kweon (2019) and continues to be an active and promising research direction, with recent work such as Sinha & Dongaonkar (2024).

• We specifically chose to predict the absolute error (an L1-loss objective) due to its robustness. Unlike the L2-loss (mean squared error), the L1-loss makes the resulting uncertainty estimation less sensitive to outlier predictions. As noted in Hastie et al. (2009), the robustness of L1-norm-based methods against outliers is a fundamental principle in statistics and machine learning.

• This approach is substantially more efficient than many widely-used UQ schemes. Methods like Deep Ensembles (Lakshminarayanan et al., 2017) require training multiple models, while MC Dropout (Gal & Ghahramani, 2016) necessitates numerous forward passes per sample. Our method requires only a single forward pass through a lightweight prediction head, making it highly scalable and practical.

Weakness 4. Classification Problem

Thanks for raising concerns about classification problem. We formally extend the Balanced Active Inference framework to the general M-estimation setting, which cover the classification problem.

(i) Settings of M-estimation

• Given a family of functions $f(X_i;\theta)$, the task is to solve $\theta^{\ast}= \arg\min_{\theta} \mathbb{E} \left [L\bigl(X_1, Y_1; \theta\bigr)\right]$.
• $L(X_1, Y_1; \theta)$ measures the discrepancy between the true label $Y_1$ and the prediction $f(X_1;\theta)$.
• Our goal is to estimate $\theta^{\ast}$ using a small labeled set $\mathcal{D}_l$ and a larger unlabeled set $\mathcal{D}_u$.
• $f(X_i;\hat{\theta})$ is an initial estimator trained on $\mathcal{D}_l$.

(ii) Uncertainty Measures

• Classification. Let $p(X_i) = (p_1(X_i),\ldots,p_K(X_i))$ denote the predicted class probabilities. We define $u(X_i)=\frac{K}{K-1}\Bigl(1-\max_{j\in[K]} p_j(X_i)\Bigr),$ which peaks when the model is maximally uncertain (uniform distribution) and drops to zero when the model is highly confident in one class.

(iii) M-Estimation

• A model $\hat{u}(\cdot)$ is trained on $\mathcal{D}_l$ to predict the uncertainty of points in $\mathcal{D}_u$.

• Given a labeling budget $n_b$, adopt the sampling scheme described in Equation (2) of the paper.

• Generate an assignment via the cube method, ensuring the balancing constraint (Equation (4)) is satisfied.

• The proposed estimator is

  $\tilde{\theta}=\underset{\theta}{\arg \min } \frac{1}{n} \sum_{i=1}^n\{L(X_i, f(X_i ; \hat{\theta}) ; \theta)+[L(X_i, Y_i ; \theta)-L(X_i, f(X_i ; \hat{\theta}) ; \theta)] \frac{\xi_i}{\\pi(X_i)}\},$

  where $\pi(X_i)$ is the inclusion probability.

Weakness 5. Figure 1

Thanks for pointing out this issue.

• The method traditional_active (referring to independent random sampling, also known as Poisson sampling) was inadvertently labeled as poisson_active in the original figure.
• We will revise the figure accordingly.

Weakness 6. Coverage Rates for Different Confidence Levels

• We have calculated the coverage rate under different confidence levels on multiple datasets.
• Below are some experimental results with a fixed sampling budget of 0.35:

• The results demonstrate that our method is close to the preset confidence level in all cases.
• The results will be included in the Supplement in the revised version.

References

• Deng, Y., Yuan, Y., Fu, H., & Qu, A. (2023). Query-augmented active metric learning. Journal of the American Statistical Association, 118(543), 1862-1875.

• Zrnic, T., & Candès, E. J. (2024, July). Active statistical inference. In Proceedings of the 41st International Conference on Machine Learning (pp. 62993-63010).

• Farquhar, S., Gal, Y., & Rainforth, T. (2021). On Statistical Bias In Active Learning: How and When to Fix It. In International Conference on Learning Representations.

• Yoo, D., & Kweon, I. S. (2019). Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 93-102).

• Haimovich, D., Karamshuk, D., Linder, F., Tax, N., & Vojnovic, M. (2024). On the convergence of loss and uncertainty-based active learning algorithms. Advances in Neural Information Processing Systems, 37, 122770-122810.

• Hastie, T., Tibshirani, R., Friedman, J. H., & Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction (Vol. 2, pp. 1-758). New York: springer.

• Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.

• Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48 (pp. 1050-1059).

Response to Reviewer f7YV

Thank you for your recognition of the novelty and methodological rigor of our research, as well as the thorough experimental validation presented. We greatly appreciate your constructive feedback and critiques, which are invaluable for refining our manuscript. Below are point-by-point responses incorporating your suggestions:

Responding to Weaknesses

Weakness 1. Intuition of Cube Method

Thanks for the valuable suggestion of providing a more intuitive explanation of the cube method. We will add the following explanation in the revised version:

• Covariate balancing is a commonly used statistical technique to improve statistical efficiency when designing a sampling scheme.

• The cube method (Deville & Tillé, 2004) achieves covariate balancing by consecutively updating selection probabilities to satisfy certain balancing constraints.

• Theoretically, the cube method achieves covariate balancing by sacrificing independence among sampled elements, making it challenging to investigate its asymptotic properties.

• Practically, the cube method is easy to implement and can be seamlessly integrated in active inference.

Responding to Questions

Q1. Computational Complexity of Cube Method

• As noted in Tillé (2011), the computational complexity of the cube method is $\mathcal{O}(N \times p^2)$, where $N$ is the population size and $p$ the number of balancing covariates.

• In our implementation, only one auxiliary covariate ($\hat{u}$, the uncertainty measure) is used ($p=1$). Thus, complexity reduces to $\mathcal{O}(N)$, meaning runtime scales linearly with the unlabeled size $n$.

• This linear scaling enables efficient handling of large datasets.

• A discussion on the computational complexity of the cube method will be added in the main text.

Q2. Stratified Sampling

• Thanks for the thoughtful suggestion of trying a stratified sampling approach.

• New experiments are implemented including the baseline of stratified sampling: stratifying by $\hat{f}(X)$ (predicted labels; Stratified_Y) and $\hat{u}$ (predicted uncertainty; Stratified_u) using 5 quantile bins (width=0.2) and sampling proportionally from each stratum.

• Below is a representative subset of our experimental results:

Table 1 Bike Sharing, RMSE

Table 2 Friedman, RMSE

Table 3 Credit Fraud Detection, RMSE

• Experiments on the datasets show that cube sampling achieves significantly lower RMSE due to its exact continuous balancing.

• Stratification balances auxiliary covariates discretely, whereas the cube method enforces exact population-level balance for continuous $\hat{u}$. This eliminates bias in uncertainty estimation, which is critical for active inference performance.

References

• Deville, J. C., & Tillé, Y. (2004). Efficient balanced sampling: the cube method. Biometrika, 91(4), 893-912.
• Tillé, Y. (2011). Ten years of balanced sampling with the cube method: an appraisal. Survey methodology, 37(2), 215-226.

We highly appreciate your thoughtful review and constructive feedback. Based on your comments, we have incorporated the following revisions:

Revisions addressing your specific comments:

• Provide an intuitive and comprehensive explanation of the Cube Method.
• Clarify the $\mathcal{O}(N)$ computational complexity of balanced sampling.
• Include stratified sampling as an additional baseline in our experiments.

Further enhancements to strengthen the paper:

In discussions with other reviewers, we have incorporated the following enhancements into the revised manuscript:

• Present a detailed formulation of our approach within the M-estimation framework, covering both regression and classification tasks.
• Provide additional experimental results under the M-estimation framework, demonstrating consistent improvements over baselines.
• Add more comparative experiments including:
  1. Sensitivity analysis of key parameters;
  2. Performance comparison on easy vs. hard problems;
  3. Coverage rate analysis under different confidence levels.

If you have any further suggestions or questions, we would be grateful for the opportunity to address them. Once again, thank you for your thoughtful review and valuable support.

Dear Reviewer f7YV,

Please note that the final justification is invisible to the authors and will only be revealed to them after the decision. Could you please leave them a comment to let them know whether you are satisfied with their response?

Best regards,

AC