Looking Beyond the Known: Towards a Data Discovery Guided Open-World Object Detection

[W1] “The CROWD-D module …” - We agree with the reviewer that the selection of $U^t$ in CROWD-D is dependent on the known $K^t$. This dependence on $K^t$ has been a common trend in OWOD since its induction through ORE (even in latest SoTA paper OrthogonalDet) since examples $K^t$ are the only source of reliable labeled examples (often used as anchors). We would also like to point the reviewer to Gupta et al., 2022 (OW-DETR) who introduced a super-class separated benchmark (S-OWODB in our paper) ensuring varying distributions across tasks (a scenario in question) where CROWD demonstrates superior performance over SoTA methods. As shown by the authors of PRISM (Kothawade et al., 2022) for auxiliary tasks, submodular maximization of SCG shows exemplary performance in mining rare/unseen instances where the size of the query set is very small compared to the ground set. This is because SCG relies on gain in information in contrast to simple distance/similarity measures used in other approaches. Our work extends this property to OWOD tasks with modifications discussed in Sec. 3.3.

[W2] “The formulation …” - We understand the concern of the reviewer. Due to lack of space in the main paper we provided the key property of SCG functions alone in Sec. 3.2 relevant to the formulation of CROWD in Sec. 3.3 and beyond. However, we would like to point the reviewer to Sec. A.2 of the Appendix alongside Sec. 3.2 (main paper) where we detail out the preliminaries of submodular functions and the mathematical modeling of SCG supporting the property of feature dissimilarity that SCG models upon submodular maximization (also detailed in Algorithm 2 of Appendix). This property is consistently used across CROWD-D and CROWD-L with differences only in the optimization strategy. In CROWD-D (a selection formulation) we maximize SCG using Greedy Optimization (Nemhauser etal., 1978) to select examples in $U^t$ dissimilar to $K^t$ while in CROWD-L (specifically in $L_{\text{CROWD}}^{\text{cross}}(\theta)$) SCG is maximized using SGD to model separation (dissimilarity) between $K^t \cup B^t$ and $U^t$ (mined in CROWD-D).

Additionally, for the notations used in the loss formulation of CROWD-L we would like to point the reviewer to Table 6 in the Appendix where we explicitly call out the definitions of each term used in our formulation. As suggested by the reviewer we would be happy to include this in the main paper to improve readability.

[W1] “The paper’s experiments …” - We understand the concern of the reviewer regarding the “replay buffer” used in our experiments. We would like to clarify that the use of replay buffer is not unique to CROWD and has been used in several benchmarks like ORE (Joseph et al., 2021) and OrthogonalDet (Sun et al., 2024). Since we built our work on top of OrthogonalDet’s framework we used the exact same replay buffer which was publicly released (a fixed image list which is also same in ORE) in their codebase, thus did not require ablation. The number of unknown instances mined in CROWD-D however is a contribution of CROWD and has been ablated upon in Table 8 of the Appendix.

[W2] “When comparing different …” - Since the end to end model training and inference takes roughly 32 hours in CROWD requiring a relatively large compute infrastructure we only conduct our ablation experiments on M-OWODB to limit compute and budgetary needs. This also follows the convention in SoTA approaches. However, we would like to point the reviewer to Table 2 (in main paper) where we demonstrate results on S-OWOD benchmark (as requested) for the exact same settings determined from the ablation study in Table 5 showing improvements in performance over SoTA approaches in ¾ tasks.

[Q1] “There seems …” - We apologize for the confusion caused and would like to clarify that Fig. 1 was intended to be a very high-level overview of CROWD so some minor technical details have been omitted. Here, we provide a step by step analogy between Fig. 1 and our method description and will update the figure to show all technical details mentioned below -

(1) At first we show that a model trained only on known classes has no knowledge of unknowns (discarded as low confidence BG by the NMS) (top subfigure in Fig.1).

(2) We then pull out the RoI proposals before application of NMS to include the BG proposals and discard proposals with the lowest confidence as true background. This is not currently shown in Fig. 1 but we highlight this in Fig. 3.

(3) Given the known class embeddings and the true backgrounds we apply CROWD-D to select the unknown class examples (middle subfigure in Fig. 1) and store them as $U^t$ for each task (also shown in Fig. 1).

(4) Finally, we apply CROWD-L to learn both known and unknown classes in incremental finetuning fashion (bottom subfigure in Fig. 1).

[Q2] “Whether using …” - We appreciate the question raised by the reviewer in this context. Using replay buffers (a common practice in literature both in OWOD and incremental learning) is a compute and memory efficient technique but does not completely solve catastrophic forgetting. According to our claims in CROWD we see that mining representative unknowns (CROWD-D) alongside a fixed buffer (this is predetermined from earlier research) and a better learning objective (CROWD-L) reduces the impact of forgetting. This is definitely a topic of continuing research and has been mentioned in our limitations as well.

[Q3] “Have any …” - Continuing from our response to a similar question in [W1] above we would like to emphasize that the replay buffer used in CROWD is not determined by us but is a fixed list list of images for both M-OWOD and S-OWOD benchmarks released publicly in ORE, OrthogonalDet etc. This fixed imagelist has been used in most existing SoTA approaches during finetuning to maintain a fair comparison.

[Q4] “The paper selects …” - We agree with the reviewer that we conduct our ablations only on the M-OWOD benchmark but use the exact same settings across tasks in both M-OWOD and S-OWOD benchmarks. From our literature review, this seems to be a common practice in literature adopted in SoTA approaches like PROB, OrthogonalDet, OW-DETR etc. Similar to our response in [W2] we would like to point the reviewer to Table 2 where we provide results on both M-OWOD and S-OWOD benchmarks keeping the exact same hyper-parameter settings as the ablation study.

[W1] “Missing study …” - We thank the reviewer for pointing this out. As stated in lines 230 - 231 $\eta$ controls the trade-off between known-unknown class separation and known class cluster compactness. A lower value of $\eta$ does not enforce separation between currently known and unknown exemplars but enforces intra-class compactness. This results in better retention of previously known objects but a drop in currently known objects due to increased confusion with unknown exemplars. On the other hand for a large value of $\eta$ the model enforces large separation between currently known and unknown objects boosting performance on the currently knowns but suffers from catastrophic forgetting of the previously known classes. This is shown in Tab. R2 below. We adopt the value of $\eta$ which provides the best overall performance (a middle ground between forgetting previously knowns and improving upon currently known classes).

Table R2: Ablation Experiments on the variation in $\eta$ in CROWD-L. Given the Graph-Cut based selection strategy in CROWD-D we vary $\eta$ between [0.5, 1.0, 1.5] and adopt the best performing value for our pipeline in CROWD-L. The selection budget $\mathtt{k}$ in CROWD-D was set to 10 for all experiments and a fixed seed value.

[W2] “The paper mentions …” - Both $\tau_e$ and $\tau_b$ are determined empirically. $\tau_e$ is an exclusion threshold which reduces the search space of CROWD-D by eliminating RoIs which have a low confidence threshold. As shown in Tab R3 below, increasing $\tau_e$ from 0 to 1 increases performance until $\tau_e = 0.2$ and then reduces. A lower value of $\tau_e$ allows for a large search space but includes a lot of noisy background objects leading to reduced selection performance. On the other hand a large value of $\tau_e$ can potentially earmark unknown foregrounds as unknowns resulting in reduced performance.

Next in Tab. R3 we keep $\tau_e$ fixed at 0.2 and ablate $\tau_b$ which controls the selection budget for backgrounds (higher the value more are the number of background RoIs identified). Increasing $\tau_b$ (percentage here) increases the fraction of RoIs treated as backgrounds. A small value of $\tau_b$ widens the search space for the combinatorial function when mining unknowns causing a small drop in performance due to confusions between true background and foreground unknowns. On the other hand, very large values of $\tau_b$ shrink the search space oftentimes considering unknown foregrounds as background objects showing a steep drop in performance.

Table R3: Ablation Experiments on the exclusion criterion $\tau_e$ and background budget $\tau_b$ in CROWD-D. The submodular function is kept constant as Graph-Cut in CROWD-D and a CROWD-FL based learning objective is chosen in CROWD-L under constant machine seed.

We appreciate the reviewer’s comment on the societal impact of CROWD. We agree that object detection is a largely applied technique in real-world scenarios and OWOD helps object detectors continually adapt to new settings. We will include this in the questionnaire in the immediate next revision.

[W1] “Lines 191-194 …” - We thank the reviewer for the detailed question. The reduction in computation cost referred to in lines 191-194 results from the exclusion of very low confidence RoI proposals potentially containing (line 2 in Algorithm 1). To put things numerically we contrast against existing SoTA approach OrthogonalDet (Sun et al., 2024) which models interactions between every possible pair of RoI features to calculate a relevance score (to select unknowns). For the Faster-RCNN family of object detectors the number of RoIs are 500 (as set in our work as well as OrthogonalDet) resulting in a 500 x 500 sized similarity kernel for every image in the dataset. On the other hand, excluding very low confidence proposals (thresholded on the objectness scores) in CROWD naturally reduces the number of RoIs to ~ 200-300 proposals reducing the compute overhead. Also, in terms of runtime we observe that on average OrthogonalDet requires about 36 hours to train, reweight and finetune on a particular task while our method CROWD can perform this in ~32 hours (using the same compute infrastructure and seed setting). This can be attributed to the joint impact of the combinatorial selection and faster convergence of the combinatorial objectives (similar results observed in SMILe (Majee et al., 2024) ) due to modeling of the learning task as a set problem (30k steps in OrthogonalDet per task vs. 24k steps in CROWD per task). We will release the log files from the best performing runs from our framework (built upon detectron2) upon acceptance showing this statistic.

[W2] “The framework …” - We appreciate the reviewer’s constructive criticism on the hyper-parameters in CROWD. We clarify that $\tau_e$ is an exclusion threshold which reduces the search space of CROWD-D by eliminating RoIs which have a low confidence threshold. As shown in Tab. R1 below, increasing $\tau_e$ from 0 to 1 increases performance until $\tau_e = 0.2$ and then reduces. A lower value of $\tau_e$ allows for a large search space but includes a lot of noisy background objects leading to reduced selection performance. On the other hand a large value of $\tau_e$ can potentially earmark unknown foregrounds as background resulting in reduced performance.

Keeping $\tau_e$ fixed at 0.2 we ablate $\tau_b$ which controls the selection budget for backgrounds (higher the value more are the number of background RoIs identified). Increasing $\tau_b$ (percentage here) increases the fraction of RoIs treated as backgrounds. A small value of $\tau_b$ widens the search space for the combinatorial function when mining unknowns causing a small drop in performance due to confusions between true background and foreground unknowns. On the other hand, very large values of $\tau_b$ shrink the search space oftentimes considering unknown foregrounds as background objects showing a steep drop in performance.

Table R1: Ablation Experiments on the exclusion criterion $\tau_e$ and background budget $\tau_b$ in CROWD-D. The submodular function is kept constant as Graph-Cut in CROWD-D and a CROWD-FL based learning objective is chosen in CROWD-L under constant machine seed.

[W3] “The paper positions …” - We thank the reviewer for the insightful question. As noted in lines 90–92 of the main paper, CAT employs a dual pseudo-labeling strategy: a model-driven approach leveraging attention scores and confidence to mine unknowns, and an input-driven approach using selective search to propose unknown candidate regions independent of model predictions. Similarly, OW-DETR uses unmatched object queries with high backbone activation scores as unknowns during training.

In contrast, CROWD-D takes a fundamentally different, principled approach by framing unknown mining as a greedy submodular maximization problem based on information-theoretic submodular conditional gain (SCG) functions (Sec. 3.3). Instead of relying solely on heuristic attention or objectness scores, CROWD-D first filters out clear background regions via low objectness scores (inspired by CAT) and then selects unknown candidates by maximizing the information gain relative to a fixed set of known RoIs. The optimization process also highlighted in Algorithm 2 (appendix) inherently encourages selection of examples that are maximally informative (high marginal gains) thereby dissimilar from known classes, reducing confusion with complex backgrounds. It is interesting to note that our selection strategy is further backed by strong theoretical guarantees (Khargonkar etal., 2021) and follows the laws of diminishing marginal return to achieve this behavior. Nonetheless, the effectiveness of CROWD-D does depend on the quality of underlying feature representations, motivating the joint use of CROWD-L to improve feature discriminability between known and unknown objects.

[W4] As shown in Table 5 …” - We agree with the reviewer that the standalone gain of CROWD-L is 0.4 to 1.2% over SoTA approaches while CROWD-D shows up to 2$\times$ gain in unknown class recall. The selection process in CROWD-D can potentially inject some noisy pseudo-labels in the learning process as shown by the drop in performance from 47.0 to 45.2 in Table 5 (row 2 vs. row 4). To maintain the performance on known classes CROWD-L complements CROWD-D and enforces sufficient separation between known and unknown RoI features without sacrificing on the boost in unknown recall resulting from CROWD-D (last row in Table 5).

[Q1] “Include runtime …” - We would like to request the reviewer to kindly refer to our comment [W1] in the same rebuttal where we describe the runtime costs incurred in CROWD over SoTA approach OrthogonalDet (Sun et al., 2024).

[Q2] “Provide a sensitivity …” - We would like to point the reviewer to our comment in [W2] of the rebuttal where we provide ablation experiments to show the impact of these hyperparameters.

[Q3] “Add qualitative …” - We understand the concern of the reviewer and we will definitely include some qualitative failure cases in the Appendix of our paper in the immediate next revision.

[Q4] “In Figure 2 …” - We understand the confusion of the reviewer regarding the symbols used in Fig. 2 (main paper). At first, we would like to point the reviewer to Table 6 of the Appendix where a glossary of symbols and their respective meanings in the context of CROWD has been provided. Particularly for Figure 2 we provide below a step by step explanation below -

(1) Fig. 2 is a detailed overview of a single task within OWOD. A task $T^t$ with task identifier $t$ accepts a model $h^t$ and a replay buffer $\hat{K}^{t-1}$ containing examples from previously known objects.

(2) At first we train $h^t$ on only currently known objects $K^t$ a replay buffer from which is stored as $\hat{K}^t$. For every example in $\hat{K}^t$ we also extract background RoIs $B^t$.

(3) CROWD-D mines a set of unknowns $\hat{U}^t$ from $\hat{K}^t$ and $B^t$ which is used by CROWD-L.

(4) CROWD-L learns an updated model $h^{t+1}$ by finetuning on $\hat{K}^{t-1}$ (previously known), $\hat{K}^t$ (currently known) and $\hat{U}^t$ (mined unknowns) using our novel learning objectives.

(5) A replay buffer of the currently learnt objects $\hat{K}^t$ and the model weights $h^{t+1}$ are passed on to the next task.

[W1] “The first part …” - We understand the concern of the reviewer on the title. We would like to clarify that the reasoning behind the use of “Looking Beyond the Known” was to highlight the importance of discovering high-quality unknown examples (which is achieved in CROWD-D). We would be happy to update the title with the technical details of our method in the immediate next revision.

[W2] “The method name …” - The choice of CROWD as an abbreviation highlights that our model is able to identify potential unknown foreground objects within a “crowd” of background RoIs. In our defence, the use of intermittent letters like “r” seems to be a common practise in literature, even within the domain of OWOD eg. PROB (Zohar et al., 2023) which is abbreviated form of “Probabilistic Objectness”.

[W3] “Line 27 …” - We thank the reviewer for pointing this out. The phrase pointed to refers to “known classes” in current task $T^t$. We will definitely update this in the paper for better readability. In line 29 “unknowns” do refer to unknown classes.

[W4] “Lines 30-31 … “ - We apologize for the oversight. Wilderness Impact and Absolute Open-Set Error (ASE) are demonstrated in Table 3 of the main paper. We will correct this in the immediate next revision.

[W5] “The symbols …” - We understand the concern of the reviewer and would be happy to replace the current symbols with the “t” in subscript. However, in our defence, these symbols of known and unknown classes are used as in OrthogonalDet (Sun et al., 2024) who use $t$ in superscript.

[W6] “The second part …” - We appreciate the criticism of the reviewer on this. Our goal here was to give a complete technical overview of our CROWD approach in a summarized form under the introduction with suitable references to details from the methodology section. We will be happy to dial down the amount of details and provide a high-level overview in the immediate next revision.

[W7] “Three contributions …” - We agree that the third claim in our paper is an empirical validation of the previous two claims. We would also like to emphasize that the empirical prowess of methods in OWOD has been enlisted as a core contribution in several previous SoTA methods like ORE (Joseph et al., 2021), PROB (Zohar et al., 2023), OW-DETR (Gupta et al., 2022) and OrthogonalDet (Sun et al., 2024). We follow a similar pattern in writing our contributions. These empirical gains (even though are results of the method used) show the applicability of CROWD to general Open-World scenarios which we deem to be a critical contribution.

Please find below our responses to the questions raised -

[Q1] “Why is the …” - We understand the concern of the reviewer and would like to emphasize that Figure 1 is a broad overview of the entire learning mechanism in CROWD and the instances shown are for illustration purposes only. The number of RoIs returned by the RPN for each image in our model is set to 500 (lines 263-264) with a large number of overlapping ones (since NMS is not applied in this stage) which we could not show in the figure (on the left). The figure on the right (with unknown instances plotted) highlights only the selected RoIs (3 / 10 selected) due to which the sizes appear different. We do acknowledge the difference in sizes and will ensure they are consistent in the immediate next revision.

Please find below our responses to the formatting issues raised in the review -

[F1] “In Figure 1 …” - We thank the reviewer for the suggestion and will update the figure as suggested in the next revision. We clarify here that Fig. 1a and 1b does not refer to two different sub-figures and points to the complete process of selection and learning in CROWD. The top two rows of Fig.1 resembles CROWD-D denoted as 1a and the bottom row points to CROWD-L, denoted as 1b.

[F2] “Put the …” - We thank the reviewer and will update the position of the subfigure titles.

[F3] “Split the …” - We understand the confusion of the reviewer. We will keep only the final formulation of $L_{\text{CROWD}^{\text{cross}}}$ here and split the equation to accommodate for space. The remaining would be referenced from the Appendix where we provide the detailed derivations.

[F4] “Put the …” - We thank the reviewer and will update the position of the subplot titles.

Dear Reviewer MYTT,

As the Author-Reviewer discussions will end, please read the author's response and engage in the discussion with the authors. Are the authors' answers satisfactory for your questions? If so, would you modify your score accordingly? If not, could you leave the reasons?

Thanks and regards,
Area Chair