Kill Two Birds with One Stone: Rethinking Data Augmentation for Deep Long-tailed Learning

Thanks so much for your time and positive feedback! We are encouraged and will continue to polish our work.

Thank you for your thorough review and positive feedback. Your insights on the strengths and the soundness of the theoretical and experimental work are valuable https://geometry-dashonline.com. Glad you found no weaknesses!

We appreciate your comments. To address your concerns, below we conduct further comparative experiments and provide point-to-point responses. We will carefully revise our paper by taking into account all your suggestions. Looking forward to discussing more with you.

Comment 1: Further Comparisons with UniformAugment. - "UniformAugment (LingChen et al., 2020) is a suitable baseline for comparison because of its straightforward application of a set of augmentations at random. It would be nice to see the UniformAugment baseline results."

Thank you for your insightful comments. Based on your suggestions, we have conducted comparative experiments on UniformAugment and DODA on CIFAR-100-LT (IR = 50 and 100). The experimental results shown in the table below indicate that under imbalanced distribution, DODA achieves better performance than UniformAugment. This improvement is comprehensive, especially in the tail classes, indicating that allowing each class to choose the augmentation method that is beneficial to itself can avoid significant sacrifices. This is consistent with what you mentioned.

Table: Accuracy (%) on CIFAR-100-LT dataset (IR = 50 and 100) wtih UniformAugment.

Comment 2: Results with Standard Deviations. - "While the main tables present average values, the standard deviations are missing; it would be better to provide complete results, including standard deviations, in the appendices."

Thank you for your comments. Here, we simply provide a demonstrative result. The table shows partial results of DODA on CIFAR-100-LT, including standard deviations. Furthermore, based on your suggestions, we will supplement other results in the appendix to ensure the comprehensiveness of our work.

Table: Accuracy (%) on CIFAR-100-LT dataset.

Comment 3: Explanation of the performance of RIDE. - "Could you provide some valuable insights regarding the decline in performance that we see in the RIDE column of Table 1?"

Thank you for your comments. As you mentioned, we find that the performance of DODA in balanced settings or settings close to balanced (IR = 10) is not as stable as in imbalanced settings, such as fluctuating performance on RIDE. We consider that its multi-expert setting may lead to inconsistencies in the tendency when choosing augmentations, i.e., conflicting tendencies. Nevertheless, by observing the experimental results, we find that DODA still shows superior performance in the medium classes.

Comment 4: More Experiments on balanced datasets. - "It seems that the proposed methodology could be effective even in scenarios handling balanced datasets. Have you experimentally verified this by any chance?"

Thank you for your comments. We have conducted exploratory experiments on balanced data distribution, e.g., original CIFAR-100, but it was not mentioned in our paper because our current focus is on the long-tailed setting. We strongly agree with your point of view, and the experimental results also confirm this. From the results in the table below, we can see that DODA still shows some effectiveness under balanced settings, although it is not as good as some methods for balanced distribution. Considering that DODA is aimed at the sacrifice problem under imbalanced distribution, we think this performance is acceptable.

Table: Accuracy (%) on CIFAR-100 dataset.

The original baseline uses Wide-ResNet as the backbone. To maintain fairness, we replace it with a standard ResNet-32.

Comment 5: Typos.

Thanks. We sincerely appreciate and promise to revise thoroughly.

I have an additional comment regarding Section 4.3. The paper said,

From the visualizations of the accuracy of each class in Figure 4, it can be found that compared with CUDA, DOA reduces the sacrifice rate by 31% and 24%, respectively, indicating that DODA makes fewer classes be `sacrificed.'

It seems that the authors' obtained 31% and 24% by computing 31% = 38% (of CE + CUDA in Figure 2) - 7% (of CE + DODA in Figure 4) and 24% = 29% (of CE + CUDA in Figure 2) - 5% (of CE + DODA in Figure 4), as they replied to Reviewer PqHc. However, when expressing the arithmetic difference between two percentages, %p should be used instead of %. Please make this correction; 31% to 31%p and 24% to 24%p.

We appreciate your comments. To address your concerns, below we provide the point-to-point responses to carefully illustrate the scalability of our method and provide additional explanations. We will carefully revise our paper by taking into account all your suggestions. Looking forward to discussing more with you.

Comment 1: Scalability of DODA — "It would be beneficial if the author could provide a brief explanation or discussion on the scalability of the proposed method, which could promote its application in other open scenarios."

Thanks for your comments. Indeed, the long-tailed distribution exists in various domains, not limited to the vision domain. We strongly agree that we need to analyze the scalability of DODA. Data augmentation is a common technique in different domains, e.g., graph learning and NLP, and there is a variety of DAs in these domains, although perhaps less abundant than in vision. Therefore, when faced with an imbalance challenge, using DODA to select the appropriate enhancement for each class can also help mitigate this challenge in these domains. In addition, the tasks and datasets that are well-established or mainstream in long-tailed learning are often confined to the vision domain, and we also try to establish standard datasets in other domains to verify the effectiveness of DODA. Considering the differences in data types and the specificity of data augmentation methods across different domains, it is unknown whether the performance of DODA would fluctuate, which motivates us to continue exploring.

Comment 2: Intuitive Explanation of Figure 4 — "Regarding Section 4.3, the authors mention that DODA can avoid more classes being sacrificed. However, it may not be intuitive to find this based on Figure 4. Therefore, it would be helpful if the author could provide a more detailed explanation of the model's effect, which would allow others to better understand the improvements brought about by this method."

Sorry for the confusion. By observing Figure 2 and Figure 4 simultaneously, we can observe that the sacrifice rate of DODA is much lower, indicating that it can significantly alleviate the sacrifice problem. We also display the sacrifice rates of different methods in the following table so that you can more intuitively confirm the effectiveness of DODA.

Table: Sacrifice rate (%) under diffirent DAs on CAFIR-100-LT dataset. $\delta$ is the relative improvement.

Comment 3: Further Comparisons with More Baselines. - "The effectiveness of DODA on state-of-the-art long-tailed methods PaCo, and NCL is not demonstrated in the paper. It would be great if the results are provided in the rebuttal."

Thanks so much for bringing these recent works to us. Following your suggestions, we have analyzed these two papers to determine the feasibility of combining them with DODA. Due to computational resource constraints, we have replicated PoCa and NCL and conducted tests under two imbalance settings on CIFAR-100-LT. We respectfully argue two points:

• For PoCa, based on the current experimental environment, we regret that we cannot directly run the official code, so we cannot provide quantitative results. However, we further analyzed the combinability of PoCa and DODA, and we found that through automatic augmentation of a single contrastive view, the two methods can achieve a good combination, and we continue to explore the subsequent results.

• For NCL, we conducted comparative experiments on CIFAR-100-LT to verify the superiority of DODA, and the experimental results are shown in the following table. We can find that DODA has achieved better performance under the two settings, indicating that DODA can effectively alleviate this intertwined imbalance problem.

  Table: Accuracy (%) on CIFAR-100-LT dataset (IR = 50 and 100) wtih NCL.

  Method	Head	Medium	Tail	All (50)	Head	Medium	Tail	All (100)
  NCL + CUDA	68.9	52.5	47.8	58.4	69.2	55.2	34.4	53.9
  NCL + DODA	68.1	53.2	49.2	58.6	69.4	55.6	35.7	54.4

Comment 4: Combineability in Contrastive Methods. - "For contrastive methods such as BCL, how is DODA being used to generate multiple views of the same image for different classes?"

Thank you for your comments. For contrastive learning-based methods, e.g., BCL, we define different views of the current input in the getitem function when we obtain the dataset. One view is obtained by sampling DA according to the augmentation weight distribution of each class in DODA, and the other is an augmented view obtained traditionally for all classes, i.e., according to the code provided by each contrastive learning baselines, a consistent augmentation is adopted.

We appreciate your comments. To address your concerns, below we conduct further comparative experiments and provide point-to-point responses. We will carefully revise our paper by taking into account all your suggestions. Looking forward to discussing more with you.

Comment 1: Further Comparisons with Remix. - "The proposed method for DODA is exclusively used where augmentations are solely for one single class and do not encourage inter class interaction. Augmentation methods such as mixup and its improved version Remix (Chou et al. 2020) have proven to be efficient in data-imbalance settings. However, comparison of the proposed method with Remix + LTL methods is lacking."

Thank you for your comments. Based on your suggestion, we have conducted comparative experiments on Remix, CUDA, and DODA on CIFAR-100-LT (IR = 50 and 100). The experimental results shown in the table below indicate:

• All three augmentation strategies can achieve performance improvements under imbalanced data distribution, demonstrating that reasonable use of data augmentation can alleviate the long-tailed problem.

• DODA achieved better performance than Remix, especially for tail classes, indicating that allowing each class to choose DAs beneficial to itself can avoid significant sacrifices.

Table: Accuracy (%) on CIFAR-100-LT dataset (IR = 50 and 100) wtih Remix.

Comment 2: Optimality of the Selection of DAs. - "The algorithm optimizes the augmentation strategy on 10 predefined augmentation functions with fixed strengths. The authors should discuss the optimality of the selection of these functions and provide ablation studies for the same."

Thank you for your valuable comments. To verify the effectiveness of DODA, we have chosen ten widely used augmentation methods. Essentially, we could also choose more augmentation methods. In this case, the augmentation preference (or preference list) for each class may change, as they find DAs that are more beneficial to them. That is to say, when we add other augmentation methods, each class still tends to choose the most beneficial augmentation for itself.

Based on your suggestion, on CIFAR-100-LT (IR = 100) and BS baseline, we have compared different candidate augmentation combinations, including 10, 15, and 20, and the corresponding experimental results are shown in the following table. We can find that DODA is still effective even though the predefined augmentation methods have changed. In addition, with the increase in the number of DAs, DODA may need more training overhead to obtain the optimal performance, i.e., it needs to iterate more on the weight matrix, and 200 epochs may not be enough to find the optimal weight.

Table: Accuracy (%) on CIFAR-100-LT dataset (IR = 100) wtih different candidate augmentation combinations.

We also provide specific augmentation methods.

l = [
    (Flip, 0, 1),
    (Mirror, 0, 1),
    (EdgeEnhance, 0, 1),
    (Detail, 0, 1),
    (Smooth, 0, 1),
    (AutoContrast, 0, 1),
    (Equalize, 0, 1),
    (Invert, 0, 1),
    (GaussianBlur, 0, 2),
    (Rotate, 0, 30), // 10

    (ShearX, 0., 0.3),
    (ShearY, 0., 0.3),
    (Color, 0.1, 1.9),
    (Brightness, 0.1, 1.9),
    (Sharpness, 0.1, 1.9), // 15

    (TranslateXabs, 0., 100),
    (TranslateYabs, 0., 100),
    (Posterize, 0, 4),
    (Solarize, 0, 256),
    (SolarizeAdd, 0, 110), //20
    ]

Dear Authors, As the paper received divergent reviews from other reviewers, I reviewed the response and paper again. I have the following concerns which are unaddressed:

1. PaCo code, which is claimed to be not reproducible, differs from my experience. The PaCo repository provides the exact commands that are needed to produce the result. Further, as the PaCo method is ICCV 2021, and the results produced by DODA are currently lower than that, it is hard to justify the SotA claim.

2. Unfair Baseline Results. I thank you for providing results on NCL. However, the CIFAR-100 results recorded are only 0.1 (imb = 100) and 0.4 (imb = 50) more than the results reported by the NCL paper, I am still determining the significance of the improvement claimed here.

Furthermore, there are a few other general concerns that I found while re-read the paper:

1. Sacrifice Rate Ambiguous: The definition of SR needs to be clarified. It seemed it determined with the percentage of classes where the model performance is lower than that of the baseline when the proposed DODA (/CUDA) method is combined with the baseline. However, in this case, even a small amount of lower accuracy (0.0001) can also lead to the counting of class as sacrificed. However, that is not true if we consider the statistical significance. Hence, the Sacrifice Rate should be determined in terms of the absolute accuracy difference. Furthermore, CUDA performance in absolute terms is similar to DODA; hence, it’s hard for me to see the difference.

2. Fair Comparison with CUDA: The authors propose that the role of augmentation should be studied in the literature. However, the CUDA paper studies it extensively. Hence, it would be good to have a section that lists reasons why DODA is better.

3. No experimental justification of theory in Sec. 2.2: The theory provided in Sec. 2.2 seems to be disjoint for the paper, as no experimental results are provided to support the theory. Hence, verifying its validity and significance is hard for me.

Due to the above concerns, I will maintain my rating of the paper in its current form. I hope authors will use the feedback to improve their submissions.