Zero-shot Object-level Out-of-distribution Detection with Context-aware Inpainting

Thank you for your detailed and constructive feedback. Below, we respond to most, if not all, of your comments at this stage. We apologize in advance if the response is long.

W1: Limited in technical novelty

Thank you for recognizing the novelty of our framework for OOD detection based on image inpainting. We agree that, having a more sophisticated or dedicated integration of inpainting and classification models together would make our method appear even more novel and appealing. However, we kindly think that, by doing so, inadvertently degrades the generality of our framework. Overall, our goal is to develop a zero-shot approach where newly released conditional generative models and object detectors can be easily plug-and-play without any extra modification efforts. We respectfully believe keeping our framework simple is actually a strength, not a weakness.

W2: Potential weaknesses using inpainting

We would like to discuss the weaknesses as follows:

W2.1. Computation Time

Thank you for raising this concern. We would like to note that it is often possible to construct shorter diffusion schedules that maintain comparable performance. Also, RONIN does not need extremely high-quality inpaintings, but rather good enough ones that make ID and OOD separable. Below we compare RONIN's performance using the original 20-step inpainting schedule vs. an optimized 2-step schedule. The 2-step schedule preserves most of the performance while running in near real-time. Finally, we respectfully emphasize that object detection is crucial in both online and offline systems, and certain application tasks favor accuracy over time performances. We will further clarify and elaborate on the time and resource computational in the final version of our paper.

OOD Detection Performance

Runtime in Seconds per Object

W2.2. Near-OOD performance.

Thank you for pointing out this. We recognize that near-OOD is indeed a very challenging case for the standard OOD detection algorithms. We indeed conducted a study about near-OOD in the appendix of our original paper, in particular, Appendix F (Line 820 - 858) of the original manuscript. What we found out is that using off-the-shelf generative models can actually have the strength to do near-OOD detection. Let’s say we have an implicated input as a photo consisting of OOD objects “zebra”, incorrectly predicted as an ID label “horse”. Because the two classes are really similar, if we just inpaint the zebra to become a horse, it may result in something quite similar to the original zebra, making near-OOD detection ineffective. Rather than that, we can pre-anticipate several near-OOD classes of the ID label. So, even though the ID class is a horse, we have already roughly guessed what are the near-OOD classes, including zebra. Then, based on that, we deliberately condition the diffusion model to generate a horse, but it doesn't look like any near-OOD class-like objects, including zebra. This will make our inpainted results even more aligned with the predicted ID label, and by doing so enhance the way our framework is robust to the near-OOD.

Based on your suggestion, we are investigating further near-OOD detection by extending what we conducted in our original appendix. In particular, we specifically look into some image samples that contain certain objects strongly similar to the ID labels in the VOC-COCO setting, stimulating the near-OOD samples to the PascalVOC object detector. We recognize that across these samples, our framework RONIN is able to tackle most of these cases effectively for near-OOD detection. We ensure that we will clarify this in detail in the main paper.

W2.3. Noise initialization for inpainting.

Thank you for your comment. We perform RONIN by giving 5 random starting noises and computing overall OOD measurements FPR@95 and AuROC across all samples. The small standard derivation suggests that RONIN does not suffer considerably under the noising of the diffusion model.

W3: Comparison with additional baselines

Thank you for your suggestion. Below we provide additional comparison with the most recent CLIPN [1]. The current results suggest that our framework, RONIN, still achieves superior performance.

W4: Clarification on the ablation study of similarities (Table 4)

Thanks for the constructive comment. We acknowledge that Table 4 could be improved to be more organized and supportive of our algorithm design (Equation 4). We will revise it in the final version and here is our plan.

In this new table, we will intergrade two similar ablation studies to support our design for the triplet similarity. In the first study, we prove that having all three types of similarity is important and necessary for effective similarity measurements. In the second study, given the triplet score with all three similarities, we empirically prove that focusing more on the vision-language similarity rather than the visual similarity, i.e. $\alpha > \beta$, supports our method RONIN to perform at its peak.

Additionally, we recognized that the performance of our framework RONIN can be sensitive given different choices of both $\alpha$ and $\beta$, making them effective hyperparameters if we want to tune our methods to tackle difficult settings. Initially, increasing $\alpha$ and $\beta$ does improve the accuracy, as you pointed out. However, our experiments suggest that keeping $\beta = 1$ is actually the most stable choice, and if having $\alpha \geq 4$, the performances of our framework then do not make significant improvements anymore. For convenience and simplicity, we just simply select $\beta = 1$ and $\alpha = 2$ for our algorithm design as in Equation 4.

W5: The effectiveness of RONIN in open-vocabulary scenarios

Thank you for raising this very interesting question. To our understanding, open-vocabulary means the object detector have the ability to detect any kind of object if someone provides a set of which objects will be detected. In other words, it requires the user to provide a set of candidate ID classes to let it operate. So based on this understanding, we see no blockage of applying our methods for open-vocabulary object detection. Here we conducted a very simple study by taking several image data from the VOC-COCO setting, containing horse (in-vocabulary) and zebra (out-of-vocabulary), and we ran GroundingDINO [2], a state-of-the-art open-vocabulary object detection. The object detector easily detects horses, but we found out there are a few false-detected zebras as horses. These false-detected can be identified with our framework, RONIN, which is particularly discussed in our response to W.2.2.

W6: Minor errors

Thank you for pointing out that, and we will correct these errors in the final revision of our paper. We would like to explain that, the name of our framework RONIN is the abbreviation of "ZeRo-shot OOD CONtextual INpainting".

Again, we appreciate your valuable and constructive feedback, which will significantly improve our study. We kindly hope that you find these responses satisfying and give us a chance to clarify any remaining unclear points.

[1] Wang et. al. "Clipn for zero-shot ood detection: Teaching clip to say no." ICCV 2023.
[2] Liu et. al. "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection." ECCV 2024.

We thank your valuable feedback to improve our manuscript further. Below, we respond to most, if not all, of your comments at this stage.

W1: Inference time comparison

Thank you for your suggestion. We have just come up with a new diffusion inpainting schedule, allowing us to perform our framework RONIN significantly faster by 10 times with only 2 denoising steps, while still maintaining an approximate performance compared to our official results in the manuscript. Despite the diffusion model can be costly, this new schedule allows RONIN to perform in near real-time. Furthermore, as discussed in Lines 042-048 and 474-476 of the original manuscript, we believe RONIN can excel in both online and offline object detection tasks, especially in scenarios where accuracy performance is favored over speed. Finally, with the simple and generalized framework, we believe newly released one-step diffusion models with real-time performance can be easily plug-and-play into our framework to make it more effective and efficient. We believe eventually RONIN can perform on par with other baselines in terms of inference time.

Runetime in Seconds per Object

OOD Detection Performance

W2: Performance clarification on BDD vs OpenImages

Thank you for your pointing out that, and we would like to clarify the performance as follows. In the main Table 1, across the 4 OOD settings, our proposed framework RONIN is the only method that performs competitively and consistently, while the other baselines fluctuate quite a bit. On the BDD vs. OpenImages setting, RONIN achieves 30.00 FPR@95TPR, and 91.60 AuROC. Although this is not a state-of-the-art performance, these numbers are still strong enough to allow RONIN to be among the top methods. On the other hand, BDD and OpenImages datasets are two significantly distinct dataset distributions. Under very effective visual embedding, such as SimCLR, metric-based learning methods like KNN or Mahalanobis perform extremely robustly to achieve peak performance. Despite that, they still underperformed RONIN in other scenarios.

Again, we appreciate your valuable feedback on improving our study. Please also kindly give us a chance to clarify any remaining unclear points.

Thanks for your valuable feedback to improve our manuscript further. Below, we respond to most, if not all, of your comments at this stage.

W1: High reliance on the generative model

Thank you for your comment, and we agree that our method, RONIN, does rely on an effective off-the-shelf diffusion model with well-trained, generalized performances for doing OOD detection in a zero-shot manner. This reliance is actually similar to several existing methods, including [1], [2], or [3] that also develop their own methods based on effective pre-trained generative models. We deliberately tried to keep the inpainting model intact to ensure the generalizability of our framework, as newly released off-the-shelf generative models can be conveniently plug-and-play without additional effort. However, to demonstrate the effectiveness of our framework without heavy reliance on the performance of the generative model, we conducted an additional experiment where we compared the default StableDiffusion 2 with two other sub-optimal generative models, StableDiffusion 1 and StableDiffusionXL, for our framework. Additionally, we further blurred the generated outcomes to stimulate an “underperformed” diffusion model with poor inpainting results. The result suggests that our framework is still able to perform similarly without significant derivation. In fact, we found that RONIN can be effectively operated with an acceptable reconstruction, without the the need for high-quality or detailed generated outcomes.

[1] Zhao et. al. "Unleashing Text-to-Image Diffusion Models for Visual Perception." CVPR 2024.

[2] Luo et. al. "Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models." NeurIPS 2024.

[3] Mou et. al. "T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models." AAAI 2024

W2: Clarification on the high resource consumption

Thank you for raising the concern. We find out that operating our framework with both the image inpainting and similarity comparison is actually not costly, especially given the modern computational resources. As discussed in Session 4.2 (Line 191-211) of the original manuscript, we designed a “class-wise inpainting” strategy to significantly reduce the number of inpainting per image, making RONIN highly efficient. Additionally, for the rebuttal, we have just designed a new inpainting schedule and denoising process that supports the generative models to perform image inpainting with only two steps but still approximates maintaining performance compared with our reported performance in the original paper, which resulted from 20 steps of inpainting. This means the new schedule allows RONIN to perform 10 times faster, approaching near real-time performance. Below we provide the time comparison of our framework RONIN, both with the original schedule of 20 steps and the improved schedule of 2 steps across the 4 main settings. We kindly ensure we will elaborate on this finding in the final version of our paper.

Runtime in Seconds per Object

OOD Detection Performance

W3: Further validation

Thank you for your suggestion. We are actively developing new OOD settings and validations, as preparing the dataset is time-consuming. We will update you in the next few days.

Again, we appreciate your valuable opinions and reviews, which will significantly improve our study. Please also kindly give us a chance to clarify any remaining unclear points.

Thank you for your constructive feedback. Please find our responses below:

W1: Technical contribution

We respectfully believe that our method is actually not limited and insufficient. We would like to emphasize that our work proposes a novel OOD detection framework based on off-the-shelf generative models. Our framework is intentionally designed to be modular, keeping the object detector and the off-the-shelf generative model intact, while investigating how to combine them effectively for performing OOD detection in a zero-shot manner. By doing so, the framework can easily generalize to future object detectors and newly released generative models with minimal modification effort. We believe that prioritizing simplicity and generalizability enhances the impact and applicability of our work, rather than diminishing its contribution.

W2: Additional comparison with other baseline methods

Thank you for your suggestion. Below we present some additional experiments with a newer baseline CLIPN [1] across 4 settings, where our framework RONIN still outperforms the baseline.

W3: The sensitivity of $\alpha$ and $\beta$ in Table 4

Thank you for your comment. We would like to clarify that performance fluctuations mainly occur when certain similarity measurements are entirely excluded (row 1-2 of Table 4). However, when all similarity measurements are included, the performance is more consistent (rows 3-6 of Table 4). We recognize that varying the values of $\alpha$ and $\beta$ can lead to performance differences. This can in fact be beneficial for having the flexibility for hyperparameter tuning to meet specific practical needs. Our empirical results indicate that $\beta = 1$ and $\alpha = 2$ perform robustly across diverse datasets, making them reliable default choices.

Q1. The experiments on ImageNet-1K

Thank you for your suggestion. We would like to clarify that, ImageNet-1K is commonly used for image-level OOD detection (e.g. for image classification tasks), rather than object-level tasks, such as object detection. Our paper investigates the object-level OOD problem, and the four datasets we experiment on follow the existing object-level works [2,3].

Q2. The simple baseline of generating an image instead of inpainting an image

Thank you for your suggestion. In fact, our implementation of the KNN baseline is similar to the suggested simple baseline. For KNN, we first construct a dataset by directly generating images of the ID classes using stable diffusion, and at inference time, we compare the detected objects to images in the generated dataset. Results in Table 1 show that RONIN consistently outperforms this approach.

Q3. Clarification on the choice of threshold

Thank you for pointing out this. We would like to clarify that in our evaluation, AuROC is calculated as the area under the curve of TPR vs. FPR at all possible decision thresholds; FPR@95 TPR is calculated at the threshold where the in-domain achieves 95% true positive rate. In practical use, the threshold can be selected based on criteria such as achieving a 95% TPR.

[1] Wang et al. “Clipn for zero-shot ood detection: Teaching clip to say no.” ICCV 2023.

[2] Du et al. “VOS: Learning What You Don't Know by Virtual Outlier Synthesis.” ICLR 2022.

[3] Du et al. “SIREN: Shaping Representations for Detecting Out-of-Distribution Objects.” Neurips 2022.