Leaving No OOD Instance Behind: Instance-Level OOD Fine-Tuning for Anomaly Segmentation

We sincerely appreciate your encouraging feedback and helpful suggestions. We will give a detailed response to each of your insightful concerns below.

W1. Novelty is incremental. Instance-wise re-weighting and prototype separation are known ideas in few-shot segmentation and object re-ID; paper does not clearly situate itself w.r.t. those.

Thank you for your valuable comment! While the use of prototypes has been explored in few-shot segmentation (FSS) and object re-identification (re-ID), our method is fundamentally different from them: 1) In our work, OOD prototypes are constructed based on each individual OOD instance, rather than representing an ID semantic category (in FSS) or a specific ID object (in re-ID). This construction of OOD prototype is a fundamental innovation tailored to the unique nature of OOD detection. 2) We provide a novel theoretical foundation of why using OOD prototypes enhance small anomaly segmentation in the feature separation loss, which is unique in anomaly segmentation. Please see the response to W3. 3) In FSS and re-ID tasks, prototypes are directly used to match query image pixels for classification [1]. In contrast, our method formulates prototype usage as an optimization problem, where the key objective is to explicitly push the OOD prototypes away from the ID semantic prototypes. Although [2] in FSS aims to separate ID features, its goals and challenges are different from those in our OOD task. Our work is the first to explore the separation between OOD instance prototypes and ID category prototypes in anomaly segmentation. In summary, the prototype construction, the theoretical foundation, and the target of our feature separation strategy differ fundamentally from the work in FSS and re-ID.

As to the instance-wise re-weighting strategy, this idea naturally arises from the intuition that critical small anomalies deserve more focus. To our knowledge, this is the first work that explicitly targets small anomalies at instance-level in anomaly segmentation, supported by our rigorous theoretical foundation. We identify the limitations of previous global-level objectives and propose an extensive instance-level paradigm, not tied to any particular loss function. This is the first instance-based fine-tuning strategy designed for anomaly segmentation and it yields remarkable progress in segmenting small anomalies.

Following your helpful comment, we will include a new subsection in Related Work to more thoroughly compare our approach with prior FSS and re-ID methods, especially in terms of prototype construction, optimization objectives, and theoretical foundations of our instance-level design for both objectives.

W2. The experiments are reproduced by authors. Why do not follow the benchmark by original papers like M2A?

Thank you for pointing out this issue! As our core contribution lies in enhancing existing OOD fine-tuning methods with our instance-level objective, we believe it is reasonable to compare the results with and without this extension under the same experimental settings. Importantly, these settings strictly follow those adopted in prior works, rather than being designed by ourselves. We also note that some of the reproduced baseline results even outperform those originally reported, further confirming the robustness of our implementation.

W3. The bounds in the theoretical part ignore feature-separation loss.

Thank you for pointing out this issue! Below, we provide a theoretical analysis of the feature separation loss when globally separating each OOD pixel-wise representation from ID prototypes. This analysis shows that, without instance-level OOD prototypes, small anomalies are likely to be ignored in the feature separation loss.

$L_{gsem}=\frac{1}{K|\Omega_{out}|} \sum_{q_{c}\in Q} \sum_{x_{j}\in \Omega_{out}}cosSim(F_{x_{j}},q_{c})$

Lemma 2. The total loss of $L_{gsem}$ decomposes into instance-specific components:

$L_{gsem}=\frac{1}{K|\Omega_{out}|} \sum_{q_{c}\in Q} \sum_{o_{k}\in O} |\Omega_{o_{k}}|\cdot {\mathbb{E}}_{x_j \sim \Omega\_{o\_{k}}} cosSim(F\_{x_j},q_c) \triangleq \sum\_{k=1}^{N} r\_{k}L\_{gsem,o\_{k}}$

where $r_{k}=\frac{|\Omega_{o_{k}}|}{|\Omega_{out}|}$ is the normalized weight of instance, and $L\_{gsem,o\_{k}}=\frac{1}{K} \mathbb{E}\_{x\_j \sim \Omega\_{o\_{k}}} \sum\_{q\_{c} \in Q} cosSim(F\_{x_j},q\_c)$ is the mean ID semantic loss over $\Omega\_{o\_{k}}$.

Proof:

$L_{gsem}=\frac{1}{K|\Omega_{out}|} \sum_{q_{c}\in Q} \sum_{x_{j}\in \Omega_{out}}cosSim(F_{x_{j}},q_{c})=\frac{1}{K|\Omega_{out}|} \sum_{q_{c}\in Q} \sum_{o_{k}\in O}\sum_{x_{j}\in o_{k}}cosSim(F_{x_{j}},q_{c})=\frac{1}{K|\Omega_{out}|} \sum_{q_{c}\in Q} \sum_{o_{k}\in O} |\Omega_{o_{k}}|\cdot {\mathbb{E}}_{x_j \sim \Omega\_{o\_{k}}} cosSim(F\_{x_j},q_c)$

Based on the definition of $L_{gsem,o_{k}}$, we have:

$L_{gsem}=\frac{1}{|\Omega_{out}|} \sum_{o_{k}\in O} |\Omega_{o_k}| \cdot L_{gsem,o_k} \quad\square$

Theorem 3. If there exists $t\in\{1,2,...,N\}$, such that $|\Omega_{t}|\ge (1-\epsilon)|\Omega_{out}|$ for some $\epsilon \in(0,1)$, then $L\_{gsem}$ is bounded by $(1-\epsilon)L\_{gsem,o\_{t}}\le L\_{gsem} \le (1-\epsilon)L\_{gsem,o\_{t}}+\epsilon \cdot \max\_{k\ne t} L\_{gsem,o\_{k}}$.

Proof: The proof follows a reasoning analogous to that of Theorem 1.

Corollary 2. If $\frac{|\Omega_{t}|}{|\Omega_{out}|}\rightarrow 1$, then $L_{gsem}\rightarrow L_{gsem,o_{t}}$ in probability.

Proof: This proof mirrors the approach taken in Corollary 1.

Based on the theoretical analysis above, globally separating each OOD feature from ID prototypes also overlooks the optimization for small anomalies, thus our ID semantic objective $L_{sem}$ in Eq. (9) is more reasonable, which equally optimizes each anomaly. Moreover, the theory of using $L_{near}$ is the same.

W4&Q3. Authors run 3 seeds but only averages are shown; no std/CI or p-values for the headline tables.

Thank you for this great suggestion! We provide the mean and std of our results in Tables 1 and 2 below.

Results on component-level metrics:

Results on pixel-level metrics:

Based on the mean and std across three runs, we find that the std remains consistently low—typically within 1% of the mean—indicating that our method exhibits stable performance across different runs.

W5&Q2. Can LNOIB operate when anomaly masks are unavailable?

Thank you for raising this important point! LNOIB cannot function without anomaly masks. This is because the instance-level objective in Eq. (5) relies on the presence of anomaly masks to identify OOD instances. Similarly, the feature separation loss in Eq. (13) depends on OOD prototypes, which are constructed based on the anomaly masks in Eq. (7). Please note that, the copy-paste method is easy and common in previous OOD fine-tuning methods to generate mix-content images, while they treat all anomalies as the whole OOD region.

Q1. How sensitive is performance to imperfect instance masks?

Thank you for raising this interesting topic! To simulate imperfect instance masks, we replace the GT masks in COCO with the predicted masks from Mask R-CNN [3] (34.7 AP), which tend to have coarse boundaries. Using these masks, we apply the same copy-paste strategy to create mix-content images. Experimental results of M2A on FS L&F and Road Anomaly are presented below, showing that while using imperfect masks leads to a slight performance drop compared to GT masks, our OOD fine-tuning method remains effective. We attribute this to the fact that the fine-tuning mainly captures the overall OOD patterns. Imperfect masks may introduce noise into the OOD instance-level prototypes and thus slightly weaken the feature separation objective. But the impact is limited, showcasing the robustness of our method.

Results on FS L&F:

Results on Road Anomaly:

Reference

[1] Wang K, et al. Panet: Few-shot image semantic segmentation with prototype alignment. CVPR'19

[2] Okazawa A. Interclass prototype relation for few-shot segmentation. ECCV'22

[3] He K, et al. Mask r-cnn. ICCV'17

We sincerely appreciate your encouraging feedback and insightful suggestions. We will give a detailed response to each of your constructive comments below.

W1-1.The qualitative results are not convincing. For example, consider Figure 4. It is unclear what is ID or OOD in these images. In row 2, the small traffic cones are small anomalies. Is the large one also an anomaly?

Thank you for pointing out this issue! In the revised version, we will include the ground-truth annotations (ID vs. OOD) for each image shown in Figure 4 to enhance clarity. Specifically, in row 2, all the traffic cones—including the large one—are considered anomalies. This is because anomalies are defined as objects that fall outside the semantic categories of Cityscapes. Therefore, traffic cones are treated as anomalies in the evaluation datasets.

W1-2. In the top row, the LNOIB method has some yellow artifact to the right of the anomaly. Does this consistently appear in images segmented using this method?

Thank you for your sharp observation! After identifying this issue, we evaluate the fine-tuned M2A model with two additional random seeds (as reported in the main results) and observe no noisy information near the anomaly in this image. We also conduct OOD fine-tuning with our method using two other seeds, and the predictions for this anomaly remain clean. Additionally, we use the original M2A model from Figure 4 to visualize results on approximately 50 images across five datasets. Most anomalies are accurately segmented, with only about 2 anomalies (each covering less than 0.5% of the image) showing relatively noisy information around the boundaries. We suspect this is due to the downsampling and upsampling processes in the network, which may randomly affect a few small anomalies. While this has minimal impact on overall performance—since most small anomalies are well handled—it highlights an interesting direction for future work.

In the revised paper, we will provide an improved illustration generated using other seeds in Figure 4. Additionally, we will include these minority imperfect cases and give a discussion of the potential reasons in the Limitations and Future Work section to highlight potential avenues for further research.

W1-3. Maybe demonstrating the qualitative performance on different images from the other datasets could convince the readers that the approach practically works well.

Thank you for your insightful suggestion! We have used the M2A model to visualize results on approximately 50 images across five datasets. The results show that, with OOD fine-tuning using our method, most small anomalies are well segmented, significantly outperforming previous global-level strategies. As image uploads are not permitted during the rebuttal phase, we will include additional visualization results from FS L&F and Road Anomaly datasets in Figure 4 of the revised paper to better illustrate this improvement.

W2. The authors presented means of 3 random seeds as the results, but the variance or confidence bounds are not included.

Thank you for highlighting this important point! We present the mean and standard deviation of our main results across three runs below.

Results on component-level metrics:

Results on pixel-level metrics:

As shown in the tables above, our method demonstrates stability across different seeds, with standard deviations typically below 1%. We will add these results to Tables 1 and 2 in the revised paper.

Q1. Does $L_{pred}$ contain $L_{in}$ twice? It contains $L_{glob}$ which contains $L_{in}$ and $L_{ins}$ which also contains $L_{in}$. I am concerned about the relative weight of the $L_{in}$ ($\alpha$ becomes pretty meaningless) in the optimization. Is this a problem? Did you notice any increased focused on the in distribution pixels due to this if so?

Yes, although $L_{pred}$ contains $L_{in}$ twice, this does not pose a problem. When expanding $L_{pred}$ in Eq. (6), we find that the coefficient of $L_{in}$ is 1, while the coefficients of the second terms in $L_{glob}$ (Eq. (1)) and $L_{ins}$ (Eq. (5)) — denoted as $L_{out}$ and $L_{outins}$ — are $\alpha$ and $(1-\alpha)$, respectively.

Therefore, the ID loss $L_{in}$ remains independent of $\alpha$, which solely controls the balance between the global-level OOD loss $L_{out}$ and the instance-level OOD loss $L_{outins}$. Moreover, the weight of $L_{in}$ is 1, and the weight of the combined OOD losses ($L_{out}+L_{outins}$) is also equal to 1 ($\alpha+1-\alpha$), ensuring that no excessive emphasis is placed on the ID regions. This balanced formulation keeps the contrastive objective of ID region and OOD region within the same range, enabling effective optimization for both ID and OOD regions.

Q2. Being more explicit about "the selection of these metrics and datasets aligns with official guidelines".

Thank you for pointing out this issue! Under your kind reminder, we will use "official metrics" instead of "official guidelines" in the revised paper to better convey our intention. The datasets we use adopt official pixel-level metrics to evaluate overall performance on OOD regions, and component-level metrics to assess average accuracy across all anomalies. These official metrics have been commonly used in prior works. In contrast, the instance-level metric is a more recent addition, which we find helpful in validating our main claim on segmenting small anomalies.

Q3. Are the ablation results consistent on the other datasets? Why was the SMIYC-RA chosen for the ablation?

Thank you for your constructive concern! The ablation results are consistent on other datasets such as FS L&F and Road Anomaly. The reason we chose SMIYC-RA for the ablation is that we think one dataset is okay and we choose the commonly used one in prior works.

Motivated by your helpful concern, we conduct more ablation studies with M2A on FS L&F and Road Anomaly datasets, including the combination of $L_{glob}$ and $L_{ins}$ in prediction-based objective (please refer to the tables below), the combination of $L_{sem}$ and $L_{near}$ in feature separation objective (please refer to our response to Reviewer vbkr, Q3-3), as well as the hyperparameters of $\tau$ and $M$ in in feature separation objective (please refer to our response to Reviewer qk6n, W2).

Results on FS L&F:

Results on Road Anomaly:

Based on the results obtained on the FS L&F and Road Anomaly datasets, our approach demonstrates robust ablation performances across different datasets.

Q4. Does focusing on improving segmentation on small anomalies hurt segmentation on large anomalies? The quantative results don't seem to clearly distinguish between them.

Thank you for your insightful concern! We would like to clarify that the OOD fine-tuning with our proposed method does not compromise the segmentation performance on large anomalies. As evidenced in Table 2, the pixel-level metrics even improve when using LNOIB. Given that large anomalies contribute substantially to pixel-level scores, any degradation in their segmentation would have led to a noticeable drop in overall performance. The observed improvements thus indicate that segmentation on large anomalies is well preserved.

L1. The limitations are kept as a short paragraph in the appendix. Perhaps they should be moved to the main text of the paper.

Thank you for your great suggestion! We will move the limitations to the main text of our paper in the revised version.

L2. No potential negative societal impacts are discussed.

Thank you for your helpful comment! As to the potential negative societal impact, while our method shows significant improvements over prior approaches in autonomous driving scenarios, it still cannot guarantee 100% accuracy. Therefore, there remains a potential risk when deploying it in safety-critical applications. We will add this point to the revised paper.

We sincerely appreciate your encouraging feedback and valuable suggestion. We will give a detailed response to your insightful question below.

W1&Q1. What hyperparameter selection strategy or specific values should be considered for each of the loss weights? How likely is this to hinder adoption by the wider community?

Thank you for raising this interesting and insightful topic! We think that although these hyperparameters may increase the complexity in future adoption to other segmentation tasks, they also provide different selections to suit the corresponding tasks. We will give an analysis of the selection recommendation of each hyperparameter below.

1. Balancing Factor $\alpha$ (Global vs. Instance-Level Objective): The parameter $\alpha$ controls the balance between the global-level and instance-level objectives. In anomaly segmentation, both large and small anomalies are critical for ensuring safety, so we set $\alpha = 0.5$ by default. For tasks less sensitive to small objects, a larger value (e.g., $\alpha = 0.75$) may be more appropriate. Conversely, for applications that prioritize accurate segmentation of small objects, a smaller value (e.g., $\alpha = 0.25$) is preferable. Thus, the choice of $\alpha$ should be guided by the task's sensitivity to object scale.

2. Balancing Factor $\beta$ ($L_{sem}$ vs. $L_{near}$): The hyperparameter $\beta$ adjusts the trade-off between the ID semantic loss ($L_{sem}$) and the nearest-neighbor loss ($L_{near}$). Both losses serve complementary roles: $L_{sem}$ encourages global separation of OOD prototype, while $L_{near}$ ensures local separation of OOD features. We recommend starting with $\beta = 0.5$ as a balanced setting. If further tuning is needed, binary search around values like $\beta = 0.3$ or $\beta = 0.7$ can be used to identify task-specific optima.

3. Weight $\gamma_2$ (Feature Separation Objective): The parameter $\gamma_2$ controls the contribution of the feature separation objective (with $\gamma_1 = 1$ fixed). As the prediction-based objective directly adopts OOD masks for optimization, this objective serves as an auxiliary mechanism to enhance the separability of OOD features in the representation space. A very large $\gamma_2$ (e.g., $\geq 10$) may overly emphasize this component and is generally not recommended. Conversely, very small values (e.g., $\leq 0.01$) may render it ineffective. We suggest starting with $\gamma_2=0.1$ and $\gamma_2=1$, and using binary search for fine-grained explorations.

4. Threshold $\tau$ (Prototype Filtering): The threshold $\tau$ filters pixel-level features to build ID category prototypes. A larger $\tau$ yields purer but less informative prototypes, while a smaller $\tau$ retains more feature variation at the cost of increased noise. We recommend setting $\tau \in [0.5, 0.9]$ depending on task requirements. For tasks that value prototype richness and can tolerate some noise, $\tau = 0.5$ is a reasonable starting point. For applications demanding high purity, initial value like $\tau = 0.7$ is more suitable. Further refinement can be conducted using binary search.

5. Number of Neighbors $M$ (in Nearest Neighbor Loss): The parameter $M$ determines how many ID prototypes are considered when computing the nearest neighbor loss. This loss encourages OOD prototypes to stay distant from the closest ID prototypes. Setting $M$ too large may dilute this effect. We recommend starting with $M = 1$ and incrementally increasing it up to $M = 5$ to find the best value. This small range is generally sufficient for most tasks.

Once again, we sincerely appreciate your insightful suggestion! It has encouraged us to consider the broader applicability and adaptability of our method across diverse segmentation tasks.

We sincerely appreciate your encouraging feedback and valuable suggestions. We will give a detailed response to each of your insightful concerns below.

W1. Despite providing detailed theoretical analysis, the instance-level loss proposed in the paper is not entirely novel. Similar instance-level losses have been used in other detection tasks, which limits the originality of this contribution.

Thank you for your insightful comment! Emphasizing small instances is intuitive when they are critical to the task. However, our proposed instance-level objective is grounded in a rigorous theoretical foundation, which sets it apart. Specifically, our analysis reveals that existing global-level objectives tend to converge toward large anomalies. Building on this insight, our instance-level objective introduces instance-specific adaptation as a general extension of global-level objectives, rather than being tied to any particular loss function, which is unique in the field of anomaly segmentation. Moreover, incorporating this objective leads to significantly better performance on small anomalies, aligning well with our theoretical findings. Therefore, the theoretical foundation (unique to anomaly segmentation), the objective formulation (not tied to a specific loss), and the application scope are fundamentally different from those in other detection tasks.

Following your helpful comment, we will include an additional subsection in Related Work to more clearly highlight the originality of our approach.

W2. The specific impact of the feature separation objective in this work remains insufficiently demonstrated.

Thank you for your constructive comment! We have gained valuable insights from paper [1], which helped us improve the experimental design for evaluating the feature separation objective. Specifically:

1. We will provide t-sne visualization to demonstrate that using only $L_{sem}$ may lead to OOD prototypes aligning with the prototype of the "road" category (please refer to our response to Q3-1 for text description);

2. We include more detailed information about the datasets, such as the number of ID categories and the similarity between some different ID semantic prototypes (please refer to our response to Q3-2);

3. We report the individual performance of $L_{sem}$ and $L_{near}$ on the FS L&F and Road Anomaly datasets (please refer to our response to Q3-3 for experimental results).

4. Besides your valuable advice, we also provide extra theoretical analysis of why using the OOD prototype-based feature separation objective improves segmenting small anomalies (please refer to our response to Reviewer hxoy, W3).

These analyses will be added in the revised version of our paper, and we will cite the reference you kindly provided, which has greatly inspired our experimental design and presentation of the feature separation objective.

Q1-1. Provide the ground-truth annotations in Figure 4.

Thank you for your insightful suggestion! We strongly agree that including the ground-truth labels (ID vs. OOD) would greatly enhance the interpretability of the results shown in Figure 4. We will incorporate the corresponding ground-truth masks as an additional column appended to the end of each row, to provide a clearer visual comparison in the revised paper.

Q1-2. Whether LNOIB leads to an increase in false positives?

Thank you for raising this important and interesting point! As shown in Table 2, the application of LNOIB consistently reduces the FPR$_{95}$ metric in most cases. It is worth noting that this metric reflects the pixel-level false positive rate, rather than an instance-level evaluation.

Obtaining reliable instance-level metrics in our setting is particularly challenging, as our task focuses on anomaly semantic segmentation, which aims to distinguish between ID and OOD regions at the pixel level. Although some prior work (e.g., [2]) proposes approximating instance-level performance for anomaly detection, extracting ID instances remains difficult. This is because ID regions can include both discrete thing classes and continuous stuff classes, whereas OOD regions are typically more spatially separated and instance-like. The ambiguity in delineating ID instances—especially for amorphous or continuous regions—makes instance-level evaluation less reliable in this context.

Therefore, lower FPR$_{95}$ results in Table 2 indicate that incorporating LNOIB effectively reduces the number of false positive pixels.

Q2. Quantitative metrics—such as performance breakdown by anomaly scale—should be provided.

Thank you for this very insightful suggestion! Following COCO, we define small anomaly as smaller than 32*32 pixels. We compare the performance of M2A with global-level objective and LNOIB. The results on FS L&F and Road Anomaly are presented below.

Results on L&F of small anomaly:

Results on Road Anomaly of small anomaly:

Based on the results above, both pixel-level and component-level metrics show notable improvements on small anomalies when using LNOIB, further supporting our main claim.

Q3-1. The authors mention that solely adopting $L_{sem}$ may cause the OOD prototype to align with the prototype of a specific ID category. Could the authors provide specific visual examples to illustrate this point?

Thank you for your constructive suggestion! This is an excellent idea to further illustrate the limitations of using $L_{sem}$ alone. As we are unable to upload figures in the rebuttal phase, we try to describe our observations in detail and incorporate the analysis into the revised paper.

Our observations reveal that many road anomalies tend to appear directly on the road surface. When these small anomalies share similar color or texture with the road, using only $L_{sem}$ often causes their features to be pulled into the feature space of the “road” category, with a cosine similarity exceeding 0.6. In such cases, $L_{sem}$ remains relatively small—since it is averaged across $K$ categories—but fails to effectively separate OOD features from the ID “road” class, leading to misclassification.

In the revised version, we will include a t-sne figure that visualizes the pixel-wise representations of such OOD instances, the “road” category, and other ID semantic categories. This will help clearly illustrate the limitations of relying solely on $L_{sem}$ and motivate the necessity of incorporating our proposed $L_{near}$ objective.

Q3-2. Provide more information about the datasets used, such as the number of ID categories ($K$) and the similarity between different ID representations.

Thank you for your valuable comment! Our study involves a total of seven datasets: Cityscapes, MS COCO, SMIYC-RA, SMIYC-RO, FS-Static, FS L&F, and Road Anomaly.

In the first stage, we train a standard semantic segmentation model on Cityscapes, which contains 19 ID categories. This leads to 19×19=361 possible ID prototype pairings, making full visualization difficult. To provide a summary, we compute the average cosine similarity between category-wise prototypes using M2A with ResNet-50, which is 0.09. While a few category pairs show relatively high similarity—such as bus vs. train (0.31), car vs. truck (0.29), and traffic light vs. traffic sign (0.26)—the vast majority have low similarity (≤ 0.1), indicating well-separated feature representations.

In the second stage, we create synthetic OOD data by pasting MS COCO (with 80 semantic categories) instances onto Cityscapes images (excluding those overlapping with 19 Cityscapes categories). This forms the OOD fine-tuning phase, where the original Cityscapes content is treated as ID and the pasted MS COCO regions as OOD. After this phase, the average prototype similarity slightly increases to 0.10 when using the fine-tuned model to conduct close-set semantic segmentation on Cityscapes.

In the test phase, the remaining five datasets are binary-labeled (ID vs. OOD). Since they involve only a single ID class, computing prototype similarity among ID categories is no longer meaningful.

Q3-3. Provide detailed metrics on the performance drop when using $L_{sem}$ and $L_{near}$ alone on different datasets.

Thank you for this insightful suggestion! As shown in Table 3(b), we have already reported the results with and without both objectives on the SMIYC-RA. To further validate the effectiveness of combining both objectives, we additionally present results of M2A on the FS L&F and Road Anomaly datasets below.

Results on FS L&F:

Results on Road Anomaly:

The findings above are consistent across these datasets, confirming that the joint use of both objectives is indeed beneficial. We will add these findings in the revised paper to further support our paper’s arguments.

Reference

[1] Tang J, Lu H, Xu X, et al. An incremental unified framework for small defect inspection. ECCV'24.

[2] Nekrasov A, et al. Ugains: Uncertainty guided anomaly instance segmentation. GCPR'23.

We sincerely appreciate your time and valuable feedback on our paper. We will give a detailed response to each of your constructive comments below.

W1. The core idea (instance-level weighting) resembles techniques in long-tail learning (e.g., class-balanced loss). The feature separation objective borrows heavily from metric learning (e.g., prototype alignment). While positioned as a "paradigm," contributions feel incremental over prior OOD fine-tuning works.

Thank you for raising this insightful topic! We will clarify the key differences between our approach and the method you mentioned.

(1) Instance-level weighting resembles techniques in long-tail learning (e.g., class-balanced loss).

While the intuitive concept of emphasizing rare/small classes/instances may appear similar to that of long-tail learning, our core idea differs significantly in terms of theoretical foundation, objective formulation, underlying motivation, and application scope.

Firstly, our method is motivated by the theoretical findings (Lemma1, Theorem 1, and Corollary 1) that global-level objectives tend to focus more on large anomalies. These theoretical analyses are fundamentally different from long-tail learning.

Secondly, our instance-level objective is formulated by the variation of existing global-level objectives, rather than a specific loss function, guaranteeing its generality in anomaly segmentation. Moreover, our method is image-specific—operating at the instance level per image—rather than dataset-wide as in long-tailed learning methods [1].

Thirdly, as to underlying motivation, the methods in long-tail learning often emphasize improving performance on rare categories in multi-class classification tasks. In contrast, anomaly segmentation in our work is inherently a binary problem (ID vs. OOD).

At last, for application scope, while long-tailed learning aims to rebalance rare ID classes, our approach explicitly focuses on enhancing the segmentation of small anomalies, which is a different problem with distinct challenges and goals.

(2) The feature separation objective borrows heavily from metric learning (e.g., prototype alignment).

The use of prototypes is conceptually related to metric learning tasks, such as few-shot segmentation (FSS). However, our feature separation objective introduces significant novelties.

Firstly, we introduce instance-level prototypes for OOD regions, which is the first method to construct prototype at the instance level using anomaly masks. In contrast, FSS methods generate category-level prototypes for new ID classes.

Moreover, our method formulates prototype usage as an optimization problem, with the key objective of explicitly pushing OOD prototypes away from ID semantic prototypes. In contrast, prototype alignment methods [2] in FSS directly use prototypes to match query image pixels for classification, without involving any explicit optimization objective on the prototypes.

Additionally, we also provide a novel theoretical foundation of how these OOD instance-level prototypes enhances segmentation performance on small anomalies in the feature separation loss (Please see our response to Reviewer hxoy, W3), which is unique in anomaly segmentation.

In summary, the prototype construction (OOD instance vs. ID class), the objective of our prototype learning strategy (optimization vs. matching), and the theoretical basis differ fundamentally from the work in metric learning.

(3) While positioned as a "paradigm," contributions feel incremental over prior OOD fine-tuning works.

Firstly, we provide a novel theoretical analysis of why prior OOD fine-tuning works fail to segment small anomalies, which lays the foundation of our work. Moreover, our method acts as a general framework that can be effortlessly integrated into existing OOD fine-tuning strategies. Due to its broad compatibility, the design of LNOIB is intentionally simple, ensuring it fits seamlessly into a variety of pipelines. While the method may not appear complex at first glance, it effectively fills a widening gap—improving the segmentation of small anomalies, which is a known limitation of existing works. Furthermore, integrating our simple yet effective method into existing OOD fine-tuning strategies (e.g., EM, PEBAL, and M2A) leads to notable performance gains, especially on component-level and instance-level metrics. These improvements highlight the practical value of our method in accurately detecting small anomalies—an essential factor for safety-critical applications in anomaly segmentation.

Following your constructive suggestion, we will add a new subsection in Related Work to briefly introduce relevant methods you mentioned and clearly delineate the differences between our approach and existing strategies.

W2. More ablation results needed.

Thank you for pointing out this issue! In our original paper, we primarily conduct ablation studies on SMIYC-RA across multiple models. Following your helpful suggestion, we further investigate whether the ablation findings generalize to other datasets. Specifically, we use M2A to explore the optimal choices of $M$ and $\tau$ on two additional datasets: FS L&F and Road Anomaly.

Following the same setup as in Figure 7, we evaluate 25 combinations of $M$ and $\tau$ for each dataset. We present the results below.

sIoU Results of M2A on FS L&F (the first row represents the values of $M$, and the first column stands for the values of $\tau$):

sIoU Results of M2A on Road Anomaly (the first row represents the values of $M$, and the first column denotes the values of $\tau$):

As shown in the tables, the best performance on both FS L&F and Road Anomaly is achieved when $M=1$ and $\tau=0.7$, which is consistent with our findings on SMIYC-RA. This suggests that the selected hyperparameters generalize well across different datasets.

Moreover, we conduct further ablation studies to assess the effect of individual loss components, including $L_{sem}$ and $L_{near}$ (please see our response to Reviewer vbkr, Q3-3), as well as $L_{glob}$ and $L_{ins}$ (please see our response to Reviewer nMfG, Q3). These ablation experiments performed on FS L&F and Road Anomaly are consistent to the results on SMIYC-RA.

W3. Pixel-level metrics (AuPRC, FPR95) show marginal gains (Table 2), suggesting improvements primarily target small anomalies at the expense of broader robustness.

Thank you for your valuable comment! We think that the potential expense of broader robustness may lie in two aspects: 1) The decreased performance on large anomaly; 2) The decreased performance on close-set performance.

On the one hand, our method enhances the performance on small anomalies, and this improvement does not come at the expense of performance on large anomalies—both pixel-level and component-level metrics are consistently improved, as shown in Tables 1 and 2. As large anomalies account for a significant portion of the pixel-level metrics, any performance degradation on them would lead to a sharp drop in the overall pixel-level metrics. Thus, our method maintains strong performance on large anomalies.

On the other hand, as shown in Figure 3 and Appendix D, fine-tuning with our method results in only a modest drop in closed-set semantic segmentation accuracy, which is on par with previous OOD fine-tuning approaches.

W4. Theorem 2 assumes idealized inter-class separability—unrealistic for complex urban scenes. Proofs (Appendix A) are sound but lack empirical validation of assumptions.

Thank you for your insightful suggestion! Following your helpful advice, we conduct additional experiments to examine the similarity between category-wise prototypes on Cityscapes, which contains complex urban scene images. Specifically, we utilize EM (WideResNet38) and M2A (ResNet50) to compute the cosine similarity between prototypes, and the resulting average values are 0.13 and 0.09, respectively. While a few category pairs exhibit relatively high similarity—such as bus vs. train (0.31), car vs. truck (0.29), and traffic light vs. traffic sign (0.26)—the majority of category pairs show low similarity values (≤ 0.1). This indicates that, for well-trained semantic segmentation models on large-scale datasets like Cityscapes, most category-wise prototypes are fairly dissimilar. Although these category-wise prototypes are not rigorously orthogonal, this empirical observation suggests that the cosine similarity values are quite small in most cases.

Moreover, Theorem 2 reveals the inherent limitation of using only $L_{sem}$ and highlights the necessity of introducing $L_{near}$. To convey this insight, we adopt an idealized assumption for theoretical illustration. For empirical validation, although the prototypes of different categories are not strictly orthogonal, those of different ID categories remain well separated in the feature space. However, in cases where two ID prototypes are highly similar, relying solely on $L_{sem}$ may cause the OOD prototype to collapse into the joint feature space of these similar ID categories. In such scenarios, $L_{sem}$ still yields a small value, failing to distinguish the OOD region effectively. Therefore, incorporating $L_{near}$ becomes crucial to explicitly enforce separation between the OOD prototype and the closely aligned ID prototypes.

Reference

[1] Cui Y, et al. Class-balanced loss based on effective number of samples. CVPR'19

[2] Wang K, et al. Panet: Few-shot image semantic segmentation with prototype alignment. CVPR'19