Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environments

Q1. Limited operating range.

A1. Thanks. As mentioned in the fifth paragraph of Sec. 5.5, our method works very well when mAP is greater than 0.15. We believe this could cover sufficiently many scenarios. As for extreme cases, we recognized the potential failure of our system: we cannot even compute BoS score if there are too many missed detections. That said, existing methods also face the same problem. As we make an early attempt at accuracy prediction in object detection, we believe future endeavors can be made to address this problem. In the revised version, we also discuss another two limitations on the working scope of BoS score in the fifth paragraph of Sec. 5.5. Please refer to our reply to Q2 of Reviewer h1Z9.

Q2. Can BoS be used to improve the detection results?

A2. Great question. We have discussed this question in the third paragraph of Sec. 5.5 of our paper. Our answer is no. There are two reasons. (1) The BoS-mAP correlation might not hold when a model just starts training, whose performance on a test set is extremely poor. (2) BoS score is not differentiable, since its calculation involves the perturbed detection results which are constantly changing during training. However, we might improve detector performance from the insights of our paper. For example, we can apply a resistance module (like MC-DropBlock [1]) to improve detector generalization ability.

[1] Deepshikha, Kumari, et al. "Monte Carlo dropblock for modelling uncertainty in object detection." arXiv preprint arXiv:2108.03614 (2021).

Q3. Performance in generic object detection with multiple classes.

A3. As described in Sec. 5.2, BoS score can be easily extended to multi-class detectors theoretically because of its class-wise nature. To verify this, we should collect a large scale object detection datasets that have diverse appearances and the same label space. To answer this question, we implemented BoS on the 5 datasets we can find that contain pedestrians and vehicles simultaneously. The results in Table 3, where BoS is shown to be overall competitive compared with existing methods. In another aspect, we would like to emphasize there are very few real-world object detection datasets that satisfy the requirements of the same label space, so we primarily focused on single-class scenarios. We clarify this reason in Sec. 5.5 of our revised paper.

Q4. Page Limit.

A4. Thanks. We have corrected it in the revised paper.

Q5. Is the proposed method unstable in multi-class setting?

A5. When confronted with multi-category mAP predictions, the features between different categories may interfere with each other. Thus, it is expected that the performance on single-category datasets is better than multi-category for the proposed approach. Existing methods also face the same problem. For example, In Table 3, under the multi-class detection setting on the Kitti dataset, the method of prediction score (0.99% -> 2.26%), entropy score (2.23% -> 4.71%), average confidence (1.69% -> 3.2%) and Fr´echet distance (2.98% -> 3.35%) all perform much worse than their performance on single-class pedestrian detection setting in Table 2. We appreciate the reviewer’s comment and have added an explanation on this point to the paper.

Q6. Does the number beside the dataset or class name report the real mAP? Compared to Table 1, the mAP on Kitti and CityPersons datasets in Table 2 is low.

A6. Yes, the number beside the dataset or class name reports the real mAP. The condition of mAP > 15% is not a strict mandatory condition for mAP estimation. But when mAP < 15%, the performance of BoS including other existing metrics might be worse than when mAP > 15%. To compare the mAP estimation performance for pedestrian detection in Table 2, we train a detector on a CrowdHuman training set (mAP = 33.5%) and report its ground truth mAP (%) on different unseen test sets in the header (besides the dataset). Since the domain shift, the detector performance on Kitti and CityPersons datasets drops to 13.3% and 11.8%, respectively.

Many thanks for the further question. Application wise, BoS can be used to assess detection performance, where no test ground truths are given. By focusing on this 'last-mile' problem in model deployment, our system has the potential to identify scenarios where the detector deviates from its optimal states (in-distribution performance), and how much the deviation is. This will allow users to have a quick understanding of their system performance in the wild, which informs their decision making.

Admittedly, as an early attempt, our system is still not perfect: low-end accuracy (mAP < 15%) and multi-class detectors are non-trivial to estimate, due to difficulty in estimating recall and finding class-shared datasets (Sec. 5.5). But, through the opportunity of sharing our findings, we believe joint efforts in the community will allow this problem to be solved in the end.

Q1. Meaning of the numbers in Table 1, Table 2, Table 3, and Table 4.

A1. Thanks for the reviewer's careful review. We will clarify it in the revised paper. In Table 1, we report ground truth mAP (%) on different unseen test sets in the header and the RMSE (%) in the main table. The lower the numbers reported in the main table are, the worse the mAP estimation performance is. The best number in each column is the lowest one. The same for Table 2, Table 3, and Table 4.

Q2. Why there is a big gap between the dropout rate=0.1 and 0.15 in Figure 3 (b)?

A2. Hyperparameters of BoS Score like dropout rate are selected during training (see Sec. 4.1 for training meta-set details). The results of Figure 3 (b) are actually based on empirical experiments. On further analysis, we find that the gap between the dropout rate=0.1 and 0.15 in Figure 3 (b) might be caused by the Roboflow dataset. If we remove sample sets generated by the Roboflow dataset in the training meta set (See M8 in the table below), the big gap disappears and the trend curve has smoothed out. In Table 1, we also find that the Roboflow dataset is the most difficult dataset for BoS to predict mAP.

(The coefficients of determination $R^2$ between BoS and mAP are reported on different training meta-sets. M means "training meta-set" and the leave-one-out evaluation procedure is employed.)

Q3. Why there is a peak at around 5 in Figure 4 (a)?

A3. As described in Sec. 4.2, from the strong correlation of BoS and mAP, we train a linear regression model as the mAP estimator on the constructed training meta-set to fit this relationship. As shown in the table below, the performance of the mAP estimator (measured by RMSE) can suffer from some fluctuations due to higher prediction variance when the meta set size is small. The peak at around 5 also means the performance of the mAP estimator fluctuates when the meta set size is too small. Nevertheless, we observe that as the meta set size is increased, both the RMSE and the variance in RMSE are reduced which leads to improved system robustness.

Thank you for your response. I have raised my rating and still have some questions.

1. Why Roboflow is more difficult than others?
2. The performance looks nice in Table 1, but why it seems hard to perform well on pedestrian detection in Table 2?
3. Why BoS is combined with PS in Table 1, Table 2 and Table 3? What do you want to illustrate?
4. If I have understood correctly, you need to train the w0 and w1 for estimating BoS using training meta-set. Do you train a set of w0 and w1 for a specific task like vehicle detection using some samples from several datasets? So the method cannot be directly trained by various tasks at once and then tested on different tasks.
5. You use a pre-trained object detector without re-training for it?
6. Do different tasks result in different w0 and w1?

Besides, I have a suggestion for the paper presentation. I do think you should make the font be bold for the best performance in each column in the tables for easier understanding.

Strengths

• All the reviewers find that this work is the first to estimate the detector performance in an unlabeled test domain, and it discovers that good detectors tend to output bounding boxes whose locations do not change much under feature dropout.
• All the reviewers also recognize that the proposed box stability score is better than other alternatives generally for vehicle detection and pedestrian detection

Weaknesses

• I agree with h1Z9 that the proposed box stability score cannot estimate the reliability of individual bounding boxes well but performs better when the whole dataset is given.
• I agree with xRLp that the proposed box stability score is not impressive in multi-class scenarios, while the task of object detection is inherently multi-class.
• In the authors' response to me, I find that the proposed box stability score does not have a good generalization ability. Each type of object needs to train a specific set of parameters. If we want to work on 10 classes, then we need to collect additional data from different domains to train the parameters for these 10 classes. This is unrealistic in object detection.

Considering the task of assessing the detector is new and the performance on vehicle detection and pedestrain detection is generally better than other alternatives, I raised my rating to 6, marginally above the acceptance threshold, because I think this work is inspiring for future work. However, rejecting this paper is also fine for me due to its limitation in evaluating individual bounding boxes (h1Z9), multi-class scenarios (xRLp) and generalization ability to various object classes (me).

Q11. Limitation in evaluating individual bounding boxes.

A11. We would like to clarify that results for evaluating individual bounding boxes are presented in Fig. 8 of the supplementary material. The coefficient of determination is $R^2>0.72$ between BoS and GT IoU, the latter is a tentative way of evaluating the quality of individual bounding boxes. This $R^2$ is much higher than the traditionally used confidence score method ($R^2>0.49$). For more details, please refer to our reply A1 to reviewer h1Z9.

Q12. Limitation in multi-class scenarios

A12. We appreciate the reviewer’s comment aboult multi-class scenarios and have added explanations on this point to the paper ensuring a more informative presentation (please refer to our reply A3 to reviewer xRLp).

Q13. I find that the proposed box stability score does not have a good generalization ability. Each type of object needs to train a specific set of parameters. If we want to work on 10 classes, then we need to collect additional data from different domains to train the parameters for these 10 classes. This is unrealistic in object detection.

A13. Thanks for this comment. To address the lack of data, we use image transformations to generate various sample sets, following [2]. If we aim at more categories, we need to collect some datasets (e.g., 10 datasets) that have these categories and then use image transformations to generate more sample sets for regression training, which has reasonable feasibility. In future our focus will be collecting more datasets sharing more common categories and deepen our understanding of mAP prediction of various object classes and environments. We feel this will make useful contributions to the community.

Thanks for the summary and rating. We really appreciate your thoughts and discussions during this period, which allow us to think deeper into our method.

[2] Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15069–15078, June 2021.

Q1. The reason why the BoS is correlated with the accuracy on the dataset level but failed on the individual level.

A1. Thanks for the question. For individual bounding boxes, as shown in Fig. 8 of our supplementary material, we tried to correlate box stability score with ground truth IoU (GT IoU) on individual bounding boxes. Compared to confidence score (coefficients of determination $R^2>0.49$ ), BoS achieves a stronger correlation (coefficients of determination $R^2>0.72$) with GT IoU. The result of "coefficients of determination $R^2>0.72$" indicates BoS could predict detector performance on individual bounding boxes to a certain extent. However, it is weaker when compared to the correlation on the dataset level (coefficients of determination $R^2>0.93$). This is because, at the dataset level, the impact of the randomness of BoS score is averaged out over many bbox samples, leading to a more stable and statistically significant measure of box stability. In contrast, for the level of individual bounding boxes, the box stability is measured on a single bbox sample and is thus more susceptible to the inherent randomness.

Q2. A limitation section is needed for a comprehensive analysis.

A2. We agree with the reviewer that a limitation section is needed. In the original 'discussions' section, we have mentioned some limitations of BoS score about its performance on the individual level, indifferentiable character, and working scope. In the revised version, we renamed the 'discussions' section to the 'discussions and limitations' section for better understanding. Besides, we add two other limitations on the working scope of BoS score. (1) We might encounter images from unseen classes in the open world. To improve the mAP estimation performance under this scenario, a feasible solution is using out-of-distribution detection techniques to detect and reject such novel images. (2) The effectiveness of the BoS score in predicting the mAP for multi-class detectors requires further investigation. This requires a large scale collection of object detection datasets across diverse appearances and the same label space. However, real-world datasets meeting these criteria are scarce, which limits us from exploring more findings. Part of our future work is to collect such datasets to share with the community.

Thank you very much for reviewing our work. We appreciate that the reviewer found our discovery important, results comprehensive, and elucidation clear.

After looking carefully through the other reviewer comments and interactions with the authors, I would summarise my final outlook as follows:

• Regarding concerns from reviewers mbpe and xRLp over the training of a regression model from a 'meta-training-set' in order to evaluate a given detector's generalization to unseen datasets. The main issue appears to be the intricacy of the procedure needed to generate the estimate. However, I do not see this being a major issue: I find it highly analogous to N-fold cross-validation - sure it's a pain to split the data into N folds correctly stratified across classes, but if one wishes to properly evaluate a classifier then that's the gold standard. Similarly, in the case of BoS there is a fair amount of effort involved to build the meta-training set and fit the BoS regression model, but that's simply what needs to be done to take advantage of the BoS-to-mAP correlation that the authors have discovered. And it will spur new ideas on how to take advantage / evaluate in ways that are better and less troublesome.

• On that note, I don't see the core of this paper to be the regressor/generalization evaluator at all, but rather the discovery by the authors of a correlation with BoS and mAP. This is a phenomenon independent of the author's use of it in the evaluator, and all reviewers agree both that the phenomenon appears to exist as the authors find, and that its use in a generalization evaluator would be extremely useful. My point is that the phenomenon itself needs independent verification by other parties to ensure it really is a thing/strong correleation and not some artifact of the author's test cases - hence the need to publish this. If it turns out to be valid/verifiable then this is automatically a landmark paper due to the sheer significance of such a strong correlation between a test-measurable metric (BoS) and a ground-truth evaluation (mAP). From there, there will of course be a flurry of papers that build better/simpler evaluators that what the authors put together, but that needs widespread knowledge of the phenomenon.

• Thus one can argue over the details of how the authors have implemented their evaluator (ie: their use of the phenomenon), but the phenomenon itself is the thing that truly matters here, and everyone is in agreement that it if true it will have significant effects on the entire field (object detection) - whether the authors have made full use of the discovery or not is of secondary importance.

• Hence I maintain my score (10) - the paper is technically solid in all areas and, most importantly, the authors have taken great pains to show the BoS-mAP correlation is probably real - that is what makes the paper immediately worthy of publication in my view since dissemination to, and independent verification by, the wider community is essential.