Dual-Stream Adapters for Anomaly Segmentation

Q1, Q3, and W1 - task definition, metrics, and benchmarks: We agree with the reviewer that the concept of anomaly is very broad and open to different interpretations depending on the context and field of application. In this work, we consider specifically the task of anomaly segmentation in driving scenes, a task with a precise definition adopted by a plethora of works [A,B,C,D,E,F,G,H,I,J,K,L,M,N]. In autonomous driving applications, image segmentation is an important functionality that aims to precisely classify all the pixels according to a set of in-distribution categories that a model is exposed to during training. In this context, anomaly segmentation aims to detect and locate, with pixel-level granularity, objects belonging to previously unseen categories that are present in road scenes. This task definition is supported by several well-established benchmarks: Fishyscapes [K], Segment Me If You Can [J], Road Anomaly [M], StreetHazards [L], and BDD-Anomaly [L]. All these benchmarks assume a closed set of in-distribution categories, typically following the Cityscapes taxonomy (e.g., car, pedestrian, bicycle, etc.), and the anomalies are out-of-distribution categories that do not belong to this taxonomy (e.g., wildlife). The metrics used across these benchmarks to measure the model performance, as mentioned in Section 5 and in Appendix A.2, are $AuPRC$ and $FPR_{95}$. Using these two metrics is a common practice in this line of research [J,K,M,L], as they provide a good insight into different aspects: the AuPRC puts emphasis on detecting the minority class and it is generally considered to evaluate the separability of the pixel-wise anomaly scores, whereas the $FPR_{95}$ metric indicates how many false positive predictions must be made to reach the desired true positive rate. A well-performing anomaly segmentation will have high $AuPRC$ with lower $FPR_{95}$. We recognize that the terminology anomaly segmentation may be misleading, but we simply adhered to the terminology already in use in this field. To make the task definition and context more clear, we rephrased the lines between 028-030 in the introduction.

W2 and Q2 - multimodal large language models: It is true that multimodal large language models have shown impressive performance on general computer vision tasks, particularly tasks that require interactivity (e.g., task prompted segmentation) or reasoning. However, the task of anomaly segmentation for road scenes, as defined above, is a domain-specific task that requires a separation of the in-distribution categories (as provided by a training dataset) and out-of-distribution elements. It is also a task that does not have an interactive user component, which may warrant textual prompting. This is a task that can be addressed effectively by models that are specifically trained for the domain, and ours is particularly capable not only of segmenting anomalies but also retaining excellent in-distribution performance. Moreover, relying on an additional model, such as a multimodal LLM, besides the segmentation model deployed to segment the in-distribution classes is not a practical solution. Indeed, to the best of our knowledge, there are no works that address anomaly segmentation in driving scenes using a multimodal large language model, as suggested by the Reviewer. We are only aware of one arxiv preprint (currently listed as work in progress) titled Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Lead, which provides a first qualitative exploration on the use of GPT-4V for anomaly detection. However, that method does not provide pixel-level predictions of anomalies. The level of reasoning that can be achieved by multimodal LLM can be very useful when moving towards a general formulation of anomaly detection in driving scenes, as in the examples given by the Reviewer. Yet, that is not the aim of this paper and we are not aware of any benchmarks or datasets that currently support that kind of more general research in the driving setting.

W1 - experimental results: The Reviewer is correct in observing that the competitors' results in Table 3 are not the same as those in their original publications, and this is not by mistake. In fact, one of the biggest problems we have encountered in this line of research and in benchmarks such as SMIYC is that even though the testing protocol is well-defined, the training protocol used by different methods can vary a lot, resulting in unfair and not informative comparisons. For instance, if one checks carefully the methods in the SMIYC leaderboard would find that there are a lot of discrepancies among them: Additional inlier datasets: there are methods (e.g. UNO) that instead of training solely on Cityscapes, leverage additional inlier data from datasets such as Mapillary Vistas, which can significantly enhance anomaly segmentation performance. Variation in backbones: Different backbones are used in these methods—for instance, RbA uses Swin-B while EAM utilizes Swin-L. It was shown that using larger backbones often leads to improved performance [B, C], thus such unregulated comparisons are unfair. Outlier Exposure: It has been observed the choice of outlier exposure datasets like ADE20K or MS-COCO can significantly vary the model’s anomaly segmentation performance. These are just a few of the things we observed and we believe that this kind of apples to oranges comparison is unfair and may hide the real contribution of different methods. For this reason, for all most recent competitors using mask-based architectures (which are the current SOTA), we did not simply report the results from their papers, but we re-evaluated them following the same training protocol for all methods, using Cityscapes as inlier dataset, Swin-L as the backbone, and MS-COCO as the choice of outlier exposure datasets. We did not stress this as a contribution because it is not the goal of the paper to establish a fair benchmark, but we firmly believe that this kind of rigorous evaluation is necessary and valuable for this field. To make things more clear in the paper, we added a symbol in Table 3 and further details between lines 468-473 to indicate which methods have been re-trained.

W2 - Fig3 & Eq 6,7: We thank the reviewer for pointing out the discrepancy. We have changed the equation and figure accordingly.

W3 - training time: Our dual-stream adapters take ~65 Hours for the full training of the network. On the other hand, without the adapters, the full finetuning of the network requires ~77 hours of training time.

W4 - missing citations: Thank you for suggesting the citations. Image-Consistent Detection of Road Anomalies as Unpredictable Patches proposes DaCUP, a method that uses a novel embedding bottleneck and image-conditioned distance features to improve anomaly segmentation. Self-Supervised Likelihood Estimation with Energy Guidance for Anomaly Segmentation in Urban Scenes introduces two estimators to model anomaly likelihood where one is a task-agnostic binary estimator and the other gives the likelihood as residual of task-oriented joint energy. Differently from ours, these works are based on per-pixel architectures that are not well suited for anomaly segmentation and they require the entire network to be fine-tuned. We have added them to the related work of the draft.

Thanks to the author's effort in providing a rebuttal, as it has offered me additional information. I would like to finalize my scoring after checking the comments from other reviewers.

W1 - Mixed results: We thank the Reviewer for the attentive observation. We observed that there is a certain variability among different datasets, not just for our method but also for the competitors. Indeed, the results in Table 3 show that there is no single method that is the best in all datasets. We conjecture that this variation in performance is to be attributed to two primary factors: a) the size of the evaluation datasets and b) the domain shift between the training and evaluation datasets. The presence of such attributes among the anomaly evaluation datasets makes a method perform well on certain anomaly datasets while underperforming on others. For instance, methods like Mask2Anomaly perform exceptionally well on the SMIYC RA-21 dataset but fail to segment anomalies effectively on the SMIYC RO-21 dataset. Maskomaly is the best on SMIYC RA-21, but gives much lower results on FS-Static. A similar observation holds for the comparative analysis between various adapter types and anomaly segmentation methods presented in Table 1,2,3. We note that no method performs best in all datasets.

For these reasons, we believe that it is important to look at the average performance across all datasets. In average, the dual-stream adapter demonstrates the best performance across all anomaly segmentation methods and adapter types, being the most robust to the different types of benchmarks and anomalies.

W2 - Self-supervision: We appreciate the reviewer’s insightful observation. We used the term self-supervised to indicate that we train our model without requiring another dataset aside from the one used to train for the in-distribution categories, i.e., Cityscapes. However, this may indeed be inaccurate since we assume the presence of a void/background class to provide explicit supervision to the network. So, we will follow the advice and remove the term"self-supervised" in our revised draft. In the revised draft we write that our method uses void/background labels as supervision. We think it is best to not refer to the void/background class as auxiliary data to avoid confusion with respect to the existing literature, because in the context of anomaly segmentation "auxiliary data" typically refers to additional datasets, such as MS-COCO or ADE-20K, whose objects are used during the outlier-exposure phase. In contrast, the supervision in our case comes from the same dataset, i.e., Cityscapes. To avoid any confusion between these two kinds of supervision, in Table 3 we have introduced two symbols: one to indicate the methods that require void labels for training (our method), and another symbol to denote the methods that utilize extra data to perform outlier-exposure.

W3 - Missing method: Thank you for highlighting the overlooked method. We observe that UNO [a] uses Mapillary Vistas as additional inlier data to boost anomaly segmentation performance and ADE-20K as an outlier exposure dataset. To ensure fairness and consistency, we use the same training protocol as in our draft. Namely, we trained UNO using Cityscapes as the inlier dataset and incorporated MS-COCO image objects during the outlier-exposure phase. The table below presents the performance comparison across all validation datasets. We can observe that DSA-Large consistently outperforms UNO across all datasets. All the results are on the validation dataset due to the limited rebuttal time duration. We have included the results in the manuscript's appendix.