AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentation

We deeply appreciate your time and effort in providing us with such constructive comments. We would like to respond to them as follows:

Q1: In the tail-class memory bank module, how do you ensure that pasted tail pixels do not completely cover any category?

A1: The number of head class pixels in each image is 2.46 to 9.45 times greater than that of the tail classes (see the response to Reviewer WAeT's Q3), meaning that the number of tail class pixels is too small to completely cover the head classes.

We conduct 10000 random coverage experiments on the ADE20K, Cityscapes, and COCO-Stuff 164K. In the experiments, we track the number of instances where pixels of a certain category in the original image are completely covered during the random coverage process. The experimental results are as follows:

The experimental results indicate that the likelihood of complete coverage occurring is less than one in a thousand.

Q2: An improved version of stratified sampling can ensure coverage of all classes. So why use the Tail-Class Memory Bank?

A2: Please refer to C-A3 in General Response.

Q3: How does Tail-Class Memory Bank store data? Does it directly use ground truth or predicted results?

A3: The Tail-Class Memory Bank stores a certain number of pixels for each tail class, as detailed in Line 201-222 of the initial submission. We use ground truth to assist in storing the original image data. The storage branch is designed with three main purposes:

• The storage branch uses ground truth to extract corresponding pixels from the original image, which is what you referred to as "directly building a memory bank using ground truth."
• The storage branch records the original positional information of each pixel, which facilitates the restoration of their spatial positions by the Retrieve Branch during sampling.
• The storage branch also assists in the scheduling of the Memory Branch. When the number of stored items in the Memory Branch reaches the memory size, we use a replacement method as described in the paper (see the response to Reviewer WAeT's Q4).

Q4: Pasting tail-class pixels onto original image might occlude original image classes; would this behavior affect performance?

A4: As we mentioned in Q1, pasting tail class pixels almost never completely covers other classes. Below, we can explore the impact of occluding the original image on performance through ablation experiments.

The parameter "Resize Ratio" in the T-Memory Bank represents the scale of the image before pasting (see Lines 220-222 in the initial submission). Ablation experiments on this parameter allow us to investigate how the area of occlusion affects performance. In the initial submission Figure 6(c), we conducted this experiment on the ADE20K dataset. Below, we present additional experimental results on the Cityscapes and COCO-Stuff 164K datasets.

Cityscapes:

COCO-Stuff 164K:

It can be observed that when the Resize Ratio is too large, severe occlusion occurs, resulting in a decline in model performance. Due to the differences in image sizes across datasets, the Resize Ratio should be adjusted accordingly when switching to other datasets.

Q5: Why did your reproduction show significant performance gaps compared with the original work?

A5: We used the code and parameters provided by MMSegmentation. To ensure fair comparisons, we reproduced all the experiments using a batch size of 4 instead of the original 16. The performance difference is due to the different batch sizes. We have further supplemented the experimental results with different batch sizes. Please refer to C-A2 in the General Response.

Q6: Inconsistent performance gain.

A6: Please refer to C-A1 in General Response.

Q7: Comparison on large-scale pretrained models.

A7: We extend AUCSeg to fine-tune CLIP, using DenseCLIP[1] as the backbone (batch size=4, iterations=80000). The experimental results are as follows:

Our AUCSeg is also effective on large-scale pretrained models.

Q8: NeurIPS Paper Checklist lacks honest responses.

A8: There may have been some misunderstandings in our responses to the Checklist questions.

• In the initial submission, we provided reasons for our "yes" answers regarding code provision/new assets: "We will release the data and code once it is accepted." This might have caused some concerns. However, due to the rules during the rebuttal period, we have sent an anonymous link to the AC containing our core code (AUC Loss and T-Memory Bank), and you may need to request it from the AC.
• When calculating the time cost for AUCSeg in the initial submission, we reported error bars. You can see this in Lines 1178-1182 of the initial submission.

[1] Denseclip: Language-guided dense prediction with context-aware prompting, CVPR 2022.

We deeply appreciate your time and effort in providing us with such constructive comments. We would like to respond to them as follows:

Q1: Why did the authors choose the segmentation perspective? Would a contrastive classification approach achieve similar results?

A1: Thank you for the helpful suggestion! We will first explain why this paper focuses on the long-tailed semantic segmentation task. Then, we will discuss new findings by comparing AUC and contrastive learning from the perspective of loss functions.

Firstly, semantic segmentation, as a pixel-level classification task, also suffers from a severe long-tail problem, yet it has been largely overlooked or insufficiently explored by the long-tail community. We approach this issue from the perspective of loss functions, aiming to find a theoretically grounded loss. Noticing that AUC is popular and effective in image-level tasks, we extend it to address the pixel-level long-tail problem.

We chose to address this challenge from the loss function perspective because 1) The quality of the loss function directly impacts the model's learning quality and is directly related to performance. 2) Loss functions are more general and have the potential to be extended to other pixel-level tasks to address their long-tail issues. For example, in salient object detection, the salient objects often exhibit a long-tailed distribution. We apply AUCSeg to this task, using the latest SOTA method SI-SOD-EDN [1,2] as the backbone. See Table 5 in the PDF submitted in the General Response for the results.

Our AUCSeg achieves improvements across three commonly used evaluation metrics on three datasets. This demonstrates that our method is highly versatile and extensible.

Next, we will discuss new insights comparing AUC and contrastive learning from the perspective of loss functions, supported by theoretical analysis and experimental validation.

Theorem: Minimizing the weighted contrastive loss approximately corresponds to minimizing an upper bound of the logistic AUC loss:

$$\sum_{i}w_i\left[-\log\frac{e^{f(x^i)}}{e^{f(x^i)}+\sum_{j\neq i}w_je^{f(x^j)}}\right]\geq\sum_{i}\sum_{j \neq i}\frac{1}{n_in_j}\left[-\log\left(\frac{1}{1+e^{f(x^j)-f(x^i)}}\right)\right]$$

where, $w_j=\frac{1/n_j}{\sum_{k\neq i}1/n_k}$ and $w_i=\frac{\sum_{k\neq i}1/n_k}{n_i}$.

Due to space constraints, the proof will be provided in the next version. This theorem indicates that minimizing a weighted version of contrastive loss can implicitly optimize the OVO logistic AUC loss. The experiments in the table below also verify that Contrastive Loss + TMB and AUCSeg (Logistic) produce similar results. Our paper adopts a more general form of AUC loss, with the square surrogate loss showing the best performance.

Q2: The significance of Proposition 1 is not very clear, What additional information is this proposition supposed to convey?

A2: Thank you for your question. In Proposition 1, $x = \Omega(y)$ means $\exists c>0, x > c \cdot y$. It represents the minimum batch size required to include pixels from all classes with a high probability. For ease of reading, we will include a table of symbol definitions in the final version of the paper (See Table 1 in the PDF submitted in the General Response).

Due to the scarcity of tail classes and the coupling between pixels, random sampling becomes challenging. This proposition quantifies the minimal batch size to cover all classes with a high probability. As we discussed in Remark 2 of the initial submission, random sampling that meets the required conditions would necessitate sampling 759 images at once to form a mini-batch, an unbearable memory cost for model training.

Q3: The logic of using a memory bank in this paper seems weird. Why not use batch re-sampling? Rephrase this aspect of diversity.

A3: Thank you for your constructive suggestion!

• The reason for not directly using batch re-sampling: Please refer to C-A3 in General Response.
• Why is the primary function of the T-Memory Bank not aimed at enhancing the diversity of tail samples: As shown in Table 2 of our initial submission (we also provide this table below), using a memory bank does indeed enhance the diversity of tail classes as an implicit way of augmentation (comparing the rows for SegNeXt and SegNeXt+TMB in the table). However, we cannot rely on the bank to fully address the long-tail issue, since the bank capacity is always limited. That's why we also need to consider the problem from the loss perspective. We find that AUC loss focuses only on the ranking loss between positive and negative samples and is not sensitive to data distribution, fundamentally avoiding the risk of underfitting caused by insufficient training samples. We believe that the use of the TMB in our paper is intended to both facilitate the effectiveness of AUC loss and enhance tail sample diversity.

We acknowledge that, to some extent, the TMB also contributes to performance improvement by enhancing tail class diversity. Therefore, in the final version of the paper, we will discuss this point further in the Discussions section (Line 223 in the initial submission).

[1] EDN: Salient object detection via extremely-downsampled network, TIP 2022.

[2] Size-invariance Matters: Rethinking Metrics and Losses for Imbalanced Multi-object Salient Object Detection, ICML 2024.

Thanks for the response. I will consider slightly increasing the score because some of my concerns are being relieved. By the way, will the code be released after the acceptance?

We deeply appreciate your time and effort in providing us with such constructive comments. We would like to respond to them as follows:

Q1: The notations in this paper should be carefully defined. Some key symbols are used repeatedly, such as 𝑁 representing both batch size (In Alg.1) and sample size (In Thm.1).

A1: Thank you for the suggestion! We carefully reviewed the symbol definitions in the paper. The symbol $𝑁$ representing batch size in Algorithm 1 has been changed to $N_b$. Additionally, for ease of reading, we will include a table of symbol definitions in the final version of the paper (See Table 1 in the PDF submitted in the General Response).

Q2: This paper only examines the square AUC surrogate loss without considering two other popular surrogate losses for AUC optimizations (i.e., hinge and exponential losses). It is essential to provide empirical verification by applying the ignored losses to AUCSeg.

A2: Thank you for your constructive suggestion! In this version, we have explored two other popular surrogate losses (hinge loss and exponential loss) to AUCSeg. Additionally, we have included results for two AUC loss calculation methods (one-vs-one and one-vs-all) applied to AUCSeg.

The performance of these three surrogate losses (hinge loss, exponential loss, and square loss) is presented in the table below:

The performance of the two AUC calculation methods (ova and ovo) when using square loss is as follows:

The results indicate that AUCSeg shows improved performance with any of the surrogate functions. Among them, using square loss and the ovo calculation method delivers the best overall performance. We will include this discussion in the final version of the paper.

Q3: In Alg.1, the batch size scale will also impact the final performance of long-tailed classifications. The authors are recommended to conduct an ablation study on this aspect.

A3: Please refer to C-A2 in General Response.

Thanks for your constructive comments, and we would like to make the following response.

Q1: On COCO, the performance improvement of Tail Class seems to be limited. Could the author explain the reason?

A1: Please refer to C-A1 in General Response.

Q2: Fig. 6(a) shows the memory size reduction of the tail-class memory bank. The results show that as the memory size increases, the performance improvement decreases. What is the reason for this?

A2: This is a trade-off between diversity and learnability. When the memory size is too large, the probability of any single sample being effectively learned decreases. So the model fails to focus on important examples, and thus fails to capture their features, ultimately leading to underfitting. Conversely, if the memory size is too small, the diversity of samples is limited, which leads to overfitting. Hence, there is no free lunch for increasing the bank!

Therefore, we pursue a reasonable memory size. As shown in Figure 6(a) in the initial submission, we believe that a memory size of 5 is suitable in most cases. As the memory size increases/decreases, the performance slightly declines due to model overfitting/underfitting.

Q3: For Tail Class Memory Bank, what would happen if we use a pixel-level memory bank? For example, prototype or similar technology, would this lead to significant performance degradation?

A3: Thanks so much for your suggestion! There are two differences between the Pixel-level Memory Bank (PMB) and our Tail-class Memory Bank (TMB). First, the PMB stores pixels from all classes, whereas our TMB only stores pixels from tail classes. Second, in our TMB, the storing and retrieving processes are conducted on an entire object (we ensure that the pasted pixel forms a meaningful object). However the PMB typically focuses on a fixed number of pixels without structural information (regardless of whether these pixels can form a complete image).

We will first explain why we only store tail class pixels instead of all pixels.

The table below shows the average number of pixels from head and tail classes per image in the ADE20K, Cityscapes, and COCO-Stuff 164K datasets.

It can be observed that the number of head class pixels in each image is 2.46 to 9.45 times greater than that of the tail classes, meaning that storing head class pixels would require significantly more memory.

The table below compares the performance differences between storing all and tail class pixels. It shows that the PMB, which incurs additional memory costs, performs almost the same as the TMB, and even shows a noticeable decline in the Cityscapes dataset. This is because head classes appear in almost every image (for example, in urban road datasets, it is hard to find an image without head class pixels like 'road' or 'sky'), so they do not need additional supplementation. Even if some images require supplementation of head classes, their larger pixel counts might cause them to overwrite the original tail class pixels when pasted, leading to a decline in performance. Thus, we only store tail class pixels.

Next, we will explain why it is not feasible to focus on a fixed number of pixels.

We conduct tests on the ADE20K dataset by supplementing a fixed number of tail class pixels (10000/20000/30000/40000) in each image and find that compared to AUCSeg, the performance differences are -3.00%/-1.93%/-0.86%/-0.73%. This is because supplementing a fixed number of pixels can result in incomplete images, such as only adding the front wheel of a bicycle, therefore loss of the structural information. The model is then unable to learn complete and accurate features. Therefore, in our TMB, storing and retrieving are conducted on all pixels of an entire image.

We will include this part in the final version of the paper.

Q4: For the memory bank update problem, is there a more appropriate selection method instead of random replacement?

A4: Thank you for your constructive suggestion! Based on your suggestion, we have now tried three other different selection methods on the ADE20K dataset:

• First-In-First-Out (FIFO) replacement: Prioritizes replacing the images that are first stored in the Tail-class Memory Bank.
• Last-In-First-Out (LIFO) replacement: Prioritizes replacing the images that are last stored in the Tail-class Memory Bank.
• Priority Used (PU) replacement: Prioritizes replacing images that have previously been selected by the retrieve branch.

The experimental results are shown in the table below.

FIFO and PU both show better performance overall and on tail classes compared to random sampling. However, LIFO, by updating only the most recently added images in the T-Memory Bank, causes the earlier images to remain unchanged. This leads to overfitting and, consequently, a decline in performance.

While these complex strategies can improve performance, the gains are relatively limited. The random replacement method, on the other hand, is easy to implement. We will include this part in the final version of the paper. In the future, we will explore more complex and effective replacement methods, hoping to provide a direction for other researchers to explore as well.