Related Work Depth

“Related Work discussion is too brief to motivate/position the study clearly.”

As another reviewer also suggests, we will expand our related work section to include more discussion of counting and active learning in the literature.

Comparison with Active Learning Strategies

“Comparison with uncertainty-based active learning would be beneficial.”

We include a related comparison in Figure 6 to active testing, where the sampling strategy is proportional to the loss—a commonly used active learning heuristic. In this case, we do not observe much improvement in fine-tuning performance, and the mismatch with the estimation objective results in a noticeable drop in overall performance. We will also incorporate sampling based on other active learning criteria, such as uncertainty and expected model change, in the revision. Thank you for the suggestion.

Evaluation Metrics

“In Fig. 3, show # of labeled units, not only fraction.”

Thanks for this feedback, we will update the plot to make the # of labeled units more obvious in the final version.

Additional Baselines

“Encouraged to consider additional baselines beyond DISCount.”

For the counting problem, we tested two additional image-counting baselines: one motivated by prediction powered inference (PPI) and the other by active testing. Please see Figure 6 in the main text and Figure A8 in the supplemental material. As mentioned above, we plan to include comparisons with different active learning heuristics for selection in the revision.

Societal Impacts

“The paper does not explicitly discuss potential negative societal impacts”

Thank you for this observation. We briefly touch on this in Section 8, noting that our method is problem-agnostic and aims to improve the accuracy of existing measurement workflows. That said, we agree it would be helpful to more clearly address potential societal impacts, and we will ensure this is clear in the next version.

Limited Evaluation Scope

“Only two counting tasks; consider using Cryo-EM, satellite images.”

We thank the reviewers for their suggestions for additional domains to apply our work. In response, we performed experiments on two additional datasets: malaria infected cell counting and damaged building counting from satellite images. Specifically, we used image set BBBC041v1, available from the Broad Bioimage Benchmark Collection [1], and satellite images of the Palu Tsunami from the xBD dataset [2].

Fractional Error Rate On Malaria Cell Dataset

The Malaria Cell dataset comprises 1,364 images (~80,000 cells). For this evaluation, we focus on counting the number of infected cells, which are around 5% of the dataset. Due to time constraints we do not perform any hyperparameter tuning, and use the same settings as our Sky and Reeds image experiments. Results are averaged over 500 trials. For our initial model we finetune the default Faster R-CNN network on three randomly selected cell images.

Looking at the first table, we see similar trends to the image and radar experiments. AIS reduces error fairly early on as the model improves. WOR also has a consistent improvement, with a larger impact after around 10% of the data is labeled.

Fractional Error Rate On Building Dataset

For damaged building detection we focus on the Palu Tsunami subset of xBD which contains 113 satellite images of the shoreline affected by the tsunami. We count the number of damaged buildings, which corresponds to a label of “major-damage” or “destroyed”. Only the post disaster images are given to the model. We use the same hyperparameters as before, and results are averaged over 1,000 trials. The initial model is trained on five randomly selected images.

The results on this dataset follow similar trends: WOR provides the most benefit when a larger fraction is labeled, while AIS is more helpful early on. There is a bit more noise with this dataset, so we plan to average over more trials in the next version of the paper. We also note that our performance is lower than the numbers reported in DISCount [3], as they use a detector specifically trained for damaged building detection, whereas we use a simple ImageNet-pretrained backbone fine-tuned on just 5 satellite images.

[1] Ljosa et al., Nature Methods, 2012

[2] Gupta, R., et al, "Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.

[3] Perez, G., Maji, S., & Sheldon, D. (2024). DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20), 22294-22302.

Clarification on Error Bars (Fig. 1)

“Are the error bars ground truth or predicted variances? Can you provide for both domains?”

The error bars are 95% confidence intervals based on predicted variances. Since this figure is from a single trial (unlike the averaged results in the results section), ground truth variances are not available. Thank you for pointing this out, we will clarify it in the next version. More detailed results on predicted variances are included in Figure A7 of the supplemental.

Model Variance Incorporation

“Is there a reason not to incorporate a variance estimate from the AI model itself?”

We want to clarify that we focus on the variance of the estimation, which is different from the variance of the AI model. There are two challenges to using the uncertainty of the AI model. First, we sometimes need more advanced methods like ensembling and Bayesian neural networks to estimate the uncertainty of an AI model. Our scope of work is generic but specific improvements can be introduced as extensions of this work. Second, it is indirect to link the variances of the model to the variances of estimation. The predicted measurements are normalized and lie in the denominators in estimation, which by itself introduces interesting but non-trivial statistical problems.

Sample Correlation Concerns

“There is correlation within the sample set, e.g., between tiles in an image; could consider a GP model.”

These comments are very astute and touch on directions we are actively considering. A concrete example is the “reeds” image (Figure 1), where birds follow a clear spatial distribution.

First, we would like to point out that this correlation does not invalidate any of our analysis. This is because we work within a finite sample framework, where all images and ground truth counts $f(s_t)$ are fixed and non-random. This differs from a typical ML setup, which assumes data are drawn from an underlying distribution. Philosophically, this is because we are not interested in generalizing to future images from the same distribution—we only care about performance on the large but finite set of images. As a consequence, the only randomness in our estimator is sequence $s_t$ of sampled units and any randomness in the training of the AI model at each step. Our bias and variance analyses correctly handle these sources of randomness by considering the bias and variance at each step due to the sample $s_t$ conditional on all prior choices.

That said, considering correlation among images could be very useful. One promising opportunity is to better predict counts and thus improve proposal distributions to reduce the variance of active measurement. As an example, in the reeds image we have observed spatial auto-correlation in the residuals of the AI model. Fitting a GP can exploit this spatial structure to improve predictions for unlabeled tiles.

As you point out, a GP can also be used to model uncertainty in an active learning framework to help select samples that will improve the AI model as much as possible. This is an interesting direction for future work (we mentioned it very briefly in Lines 359–362.

Domain Generality

“Experimental results are good but fairly specific; would have been helpful to test on different data types.”

We thank the reviewers for their suggestions for additional domains to apply our work. In response, we performed experiments on two additional datasets: malaria infected cell counting and damaged building counting from satellite images. Specifically, we used image set BBBC041v1, available from the Broad Bioimage Benchmark Collection [1], and satellite images of the Palu Tsunami from the xBD dataset [2].

Fractional Error Rate On Malaria Cell Dataset

The Malaria Cell dataset comprises 1,364 images (~80,000 cells). For this evaluation, we focus on counting the number of infected cells, which are around 5% of the dataset. Due to time constraints we do not perform any hyperparameter tuning, and use the same settings as our Sky and Reeds image experiments. Results are averaged over 500 trials. For our initial model we finetune the default Faster R-CNN network on three randomly selected cell images.

Looking at the first table, we see similar trends to the image and radar experiments. AIS reduces error fairly early on as the model improves. WOR also has a consistent improvement, with a larger impact after around 10% of the data is labeled.

Fractional Error Rate On Building Dataset

For damaged building detection we focus on the Palu Tsunami subset of xBD which contains 113 satellite images of the shoreline affected by the tsunami. We count the number of damaged buildings, which corresponds to a label of “major-damage” or “destroyed”. Only the post disaster images are given to the model. We use the same hyperparameters as before, and results are averaged over 1,000 trials. The initial model is trained on five randomly selected images.

The results on this dataset follow similar trends: WOR provides the most benefit when a larger fraction is labeled, while AIS is more helpful early on. There is a bit more noise with this dataset, so we plan to average over more trials in the next version of the paper. We also note that our performance is lower than the numbers reported in DISCount [3], as they use a detector specifically trained for damaged building detection, whereas we use a simple ImageNet-pretrained backbone fine-tuned on just 5 satellite images.

[1] Ljosa et al., Nature Methods, 2012

[2] Gupta, R., et al, "Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.

[3] Perez, G., Maji, S., & Sheldon, D. (2024). DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20), 22294-22302.

Application to Active Learning

“Would like to see if Proposition 3 improvements help active learning frameworks.”

In the context of active learning, our theoretical analyses mirror two types of biases: (1) bias due to sampling without replacement and (2) bias due to a varying acquisition function. With the biases, the samples are not acquired uniformly and should be weighted. There can be similar results as ours by formalizing the variances against the acquisition functions over the sample spaces. We think they may lead to novel weighting strategies that help active learning frameworks.

Missing Reference

“Could they cite a reference to prior work that identifies the same measurement problem?”

Thanks for pointing out the additional references of counting problems above. We will discuss them in our next version. For this work, we mainly focus on counting problems that have biological and ecological implications. For example, there have been similar problems of counting penguins [1], bats [2], and other birds [3] in those domains.

Additional Datasets

“The paper would be strengthened if it included results on popular crowd counting datasets [ECCV 2018, CVPR 2013].”

We thank the reviewers for their suggestions for additional domains to apply our work. In response, we performed experiments on two additional datasets: malaria infected cell counting and damaged building counting from satellite images. Specifically, we used image set BBBC041v1, available from the Broad Bioimage Benchmark Collection [4], and satellite images of the Palu Tsunami from the xBD dataset [5].

Fractional Error Rate On Malaria Cell Dataset

The Malaria Cell dataset comprises 1,364 images (~80,000 cells). For this evaluation, we focus on counting the number of infected cells, which are around 5% of the dataset. Due to time constraints we do not perform any hyperparameter tuning, and use the same settings as our Sky and Reeds image experiments. Results are averaged over 500 trials. For our initial model we finetune the default Faster R-CNN network on three randomly selected cell images.

Looking at the first table, we see similar trends to the image and radar experiments. AIS reduces error fairly early on as the model improves. WOR also has a consistent improvement, with a larger impact after around 10% of the data is labeled.

Fractional Error Rate On Building Dataset

For damaged building detection we focus on the Palu Tsunami subset of xBD which contains 113 satellite images of the shoreline affected by the tsunami. We count the number of damaged buildings, which corresponds to a label of “major-damage” or “destroyed”. Only the post disaster images are given to the model. We use the same hyperparameters as before, and results are averaged over 1,000 trials. The initial model is trained on five randomly selected images.

The results on this dataset follow similar trends: WOR provides the most benefit when a larger fraction is labeled, while AIS is more helpful early on. There is a bit more noise with this dataset, so we plan to average over more trials in the next version of the paper. We also note that our performance is lower than the numbers reported in DISCount [6], as they use a detector specifically trained for damaged building detection, whereas we use a simple ImageNet-pretrained backbone fine-tuned on just 5 satellite images.

[1] Arteta, C., Lempitsky, V., & Zisserman, A. (2016, September). Counting in the wild. In European conference on computer vision (pp. 483-498). Cham: Springer International Publishing.

[2] Horn, J. W., & Kunz, T. H. (2008). Analyzing NEXRAD doppler radar images to assess nightly dispersal patterns and population trends in Brazilian free-tailed bats (Tadarida brasiliensis). Integrative and Comparative Biology, 48(1), 24-39.

[3] Buler, J. J., Randall, L. A., Fleskes, J. P., Barrow Jr, W. C., Bogart, T., & Kluver, D. (2012). Mapping wintering waterfowl distributions using weather surveillance radar. PloS one, 7(7), e41571.

[4] Ljosa et al., Nature Methods, 2012

[5] Gupta, R., et al, "Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.

[6] Perez, G., Maji, S., & Sheldon, D. (2024). DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20), 22294-22302.

WOR vs AIS Clarification

“It would be nice to understand when and why WOR vs AIS gives larger and smaller improvements in the dataset.”

In theory, WOR consistently reduces estimation error regardless of the detector, with improvements becoming more pronounced as more units are labeled. In contrast, AIS yields the greatest benefit when fine-tuning leads to substantial improvements in model predictions. Additional gains arise when our weighting strategy aligns with the (unknown) error rates. That said, our weighting scheme includes guarantees that bound the worst-case error, ensuring robustness even when this alignment is imperfect.

Domain Generality

“It would also be nice to demonstrate this on another domain (e.g., cell counting) instead of two similar domain applications.”

We thank the reviewers for their suggestions for additional domains to apply our work. In response, we performed experiments on two additional datasets: malaria infected cell counting and damaged building counting from satellite images. Specifically, we used image set BBBC041v1, available from the Broad Bioimage Benchmark Collection [1], and satellite images of the Palu Tsunami from the xBD dataset [2].

Fractional Error Rate On Malaria Cell Dataset

The Malaria Cell dataset comprises 1,364 images (~80,000 cells). For this evaluation, we focus on counting the number of infected cells, which are around 5% of the dataset. Due to time constraints we do not perform any hyperparameter tuning, and use the same settings as our Sky and Reeds image experiments. Results are averaged over 500 trials. For our initial model we finetune the default Faster R-CNN network on three randomly selected cell images.

Looking at the first table, we see similar trends to the image and radar experiments. AIS reduces error fairly early on as the model improves. WOR also has a consistent improvement, with a larger impact after around 10% of the data is labeled.

Fractional Error Rate On Building Dataset

For damaged building detection we focus on the Palu Tsunami subset of xBD which contains 113 satellite images of the shoreline affected by the tsunami. We count the number of damaged buildings, which corresponds to a label of “major-damage” or “destroyed”. Only the post disaster images are given to the model. We use the same hyperparameters as before, and results are averaged over 1,000 trials. The initial model is trained on five randomly selected images.

The results on this dataset follow similar trends: WOR provides the most benefit when a larger fraction is labeled, while AIS is more helpful early on. There is a bit more noise with this dataset, so we plan to average over more trials in the next version of the paper. We also note that our performance is lower than the numbers reported in DISCount [3], as they use a detector specifically trained for damaged building detection, whereas we use a simple ImageNet-pretrained backbone fine-tuned on just 5 satellite images.

[1] Ljosa et al., Nature Methods, 2012

[2] Gupta, R., et al, "Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.

[3] Perez, G., Maji, S., & Sheldon, D. (2024). DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20), 22294-22302.