LLM-AutoDA: Large Language Model-Driven Automatic Data Augmentation for Long-tailed Problems

Thank you very much for your valuable comments and questions. Let me respond to them one by one:

W1: In our framework, different prompts play complementary roles. For example, some prompts are responsible for generating new augmentation algorithms, while others mutate existing algorithms. Through this collaboration, we can strike a balance between exploring new augmentation spaces and leveraging known effective algorithms.

The collaboration of multiple prompts helps the search strategy more comprehensively cover the design space of data augmentation. Different prompts can optimize the augmentation strategy from different perspectives, jointly promoting performance improvement. We will supplement more analysis in the paper to explain the benefits of this collaborative mechanism.

W2: You made a great suggestion. Comparing the performance of the two methods under different imbalance ratios can more comprehensively evaluate the ability of the searched strategies to adapt to different long-tailed scenarios.

In the revised version, we will add more experiments with different imbalance ratios to systematically analyze the strengths and weaknesses of SimpleLLM and LLM-AutoDA in dealing with different long-tail distributions. This will provide readers with richer experimental results and demonstrate the robustness of our method.

W3: Thank you for the reminder. We will carefully check the entire text again, correct the grammatical errors found, and improve the expression quality of the paper.

Q1: You raised an interesting direction for reflection. We believe that the paradigm of using LLM to assist data augmentation has the potential to be extended to other tasks and bring inspiration to more fields.

For example, in natural language processing, LLM can generate targeted augmentation strategies based on the characteristics of text data, such as synonym replacement and back-translation, to alleviate data sparsity problems. In the field of speech recognition, LLM may also help optimize audio augmentation and improve the generalization of models.

In the revised version, we will add some extensibility analysis in the discussion section to explore the possibilities and potential impacts of combining this paradigm with other fields.

Thank you again for your feedback, which is very helpful for improving our work. We will carefully revise the paper, hoping that the revised version can answer your doubts and meet the publication standards.

Thank you very much for your detailed comments and questions. These feedbacks are invaluable for improving our work. Let me respond to your questions one by one:

First, from your review comments, I noticed that our work may have caused some misunderstandings. For example, you thought our method "uses LLM to select augmentation parameters" and "trains a set of models". Your summary of our method as "carefully designing various prompts" "to improve validation performance on long-tail classification tasks" is also not as accurate as the summaries by Reviewer JafW, q3qy, and 2Q2t. Therefore, please allow me to reintroduce our research.

• Our research is not a data augmentation method that uses LLMs and prompt engineering to improve model performance, but a new process for discovering long-tailed data augmentation strategies. In the discovery process, the LLM continuously innovates and modifies existing data augmentation methods, while the long-tailed model evaluates these innovative methods through its image learning ability in long-tailed learning, guiding the evolution direction of the methods. Reviewers JafW, q3qy, and 2Q2t all emphasized the importance of this process, so we supplemented some of the obtained strategies and performance improvements during the process in Figure 1 in ONE-PAGE PDF.

• Although this discovery process uses LLMs and long-tailed model training, it is much less time-consuming and labor-intensive than manually designing new data augmentation methods.

• In this discovery process, we obtained brand-new data augmentation strategies that are universally effective on different long-tailed baselines and datasets, and have no higher cost in real-world usage, comparable to existing data augmentation strategies.

Next, I will reply to your questions one by one:

W1: Thank you for your opinion. First, LLM is indeed existing, but the current trend is to utilize or improve LLM in more practical domains to promote domain development. Therefore, as the other three reviewers mentioned, "this integration of LLM and long-tail learning is both creative and timely". Moreover, selecting better augmentation strategies itself is an ongoing research area, and our essential goal is to promote progress in this field.

Therefore, our method is innovative in the following aspects:

• We introduced LLM into the field of long-tailed data augmentation, leveraging its powerful language understanding and generation capabilities, combined with a dynamic data augmentation framework, successfully discovering data augmentation strategies suitable for long-tailed learning that surpass human-designed ones;
• We designed a complete LLM-driven data augmentation pipeline specifically for long-tailed problems, systematically studying how to utilize LLM to mitigate the challenges brought by long-tailed distributions. This is obviously different from improving model performance through prompts.

W2: My previous reply has addressed this question. In fact, we only run the model multiple times during the discovery process of new strategies. Once a data augmentation strategy is discovered (which takes about 6 hours), we can directly use it in testing and generalize it to datasets and long-tailed baselines unseen during the discovery process (as shown in the experimental results in this paper). In this process, the complexity of the discovered data augmentation strategies is consistent with manually designed strategies.

W3: Thank you very much for your valuable comments. We will improve and unify the definition and usage of symbols in the revised paper to enhance its readability and rigor. In our algorithm, each data augmentation method is assigned a weight $w \in [0, 1]$, representing the probability of selecting that method. We will clarify the meaning of weight $w$ in the symbol definition section to eliminate ambiguity. The selection matrix $A \in \{0, 1\}^{C \times M}$ is a binary matrix, where $C$ represents the number of categories and $M$ represents the number of augmentation methods. The matrix element $a_{ij} = 1$ indicates that the $j$-th augmentation method is selected for category $i$; $a_{ij} = 0$ means not selected.

Q1: We provided some updated performance comparisons of automatic data augmentation. In the appendix, we continue to add some comparisons with the latest baselines to demonstrate our performance. In the revised version, we will increase the comparative experiments with AutoAugment and RandAugment to comprehensively evaluate the performance of our method.

Q2: As answered at the beginning, we do not use LLM to do the same thing as auto, but focus on discovering new data augmentation strategies, not just searching for parameters.

Q3: In Figure 1 in ONE-PAGE PDF, we list some novel augmentation algorithms designed by LLM and show the changes in performance improvement. In Tabel 1, we will enhance more relevant analysis and examples.

Thank you again for your detailed review comments. We will conduct a comprehensive review and correction of the symbol definitions and usage in the paper accordingly. If you have any other suggestions, we will humbly listen and seriously improve.

Thank you for your valuable comments. These suggestions are very helpful for improving the quality of the paper. I will respond to your questions one by one:

W1: Your suggestion is very pertinent. In the revised version, we will supplement some examples of augmentation strategies generated by LLM and briefly analyze them.

For example, we can showcase a specific augmentation combination generated by LLM for a long-tailed class and discuss how this combination can effectively improve the learning effect of that class. Through example analysis, readers can have a more intuitive understanding of the working principle and advantages of LLM-generated strategies.

W2: I apologize for the lack of clarity in the description of Figure 5. Here, |gap| refers to the performance gap between the augmentation strategy generated by LLM and the optimal strategy. The larger 1/|gap| is, the closer the LLM-generated strategy is to the optimal, and the better the performance.

Different LLMs may have varying performances when dealing with long-tailed problems. Figure 5 aims to compare the performance curves of different LLMs and reveal their ability to narrow the gap with the optimal strategy. We will add clearer legend explanations in the revised version to facilitate readers' understanding.

W3: Thank you for providing the references. We indeed overlooked this. In the revised version, we will supplement the discussion of some representative methods in the field of resampling and data augmentation in the related work section, especially the paper you mentioned, to make our related work more comprehensive.

W4: You are right. We will carefully check and adjust the format in the appendix, especially Figures 9 and 11-13, to make their sizes and layouts more consistent.

Thank you again for your feedback. These comments are enlightening for us. We will carefully revise the paper accordingly to present higher-quality work. If you have any other suggestions, we will humbly accept them and actively improve.

Thank you very much for these pertinent and in-depth comments, which are very helpful in improving our work and clarifying our contributions. Let me respond to your comments one by one:

W1: Thank you for your advice. We have done our best to supplement the comparative experiments of our method and more data augmentation methods (including CMO) and present the results in Table 1 in ONE-PAGE PDF. We will update this section in the revised version to comprehensively review the different types of methods and their pros and cons.

As for the second point, I am sorry for causing your misunderstanding. We want to express that the current long-tailed learning methods based on data augmentation include data-level augmentation methods and feature-level augmentation methods, as highlighted in a recent long-tailed survey [1]. In addition, we strongly believe in augmenting the feature dimension, which is orthogonal to the traditional data dimension. However, as formulated by CUDA [2] and DODA [3], previous approaches can come with potential sacrificial effects if a class-independent augmentation strategy is imposed on all classes. Therefore, based on this motivation, we further propose new augmentation paradigms.

In general, we mention these works to make our survey more comprehensive, even though our work is not focused on feature dimension improvement. I hope you can understand our intention.

[1] Deep long-tailed learning: A survey. IEEE TPAMI 2023.
[2] CUDA: Curriculum of Data Augmentation for Long-tailed Recognition. ICLR 2023.
[3] Kill Two Birds with One Stone: Rethinking Data Augmentation for Deep Long-tailed Learning. ICLR 2024.

W2 and W3: You bring up a good point. With textual hints alone, LLMs may struggle to accurately grasp the characteristics and limitations of image data. Therefore, based on the same point of view as yours, our data augmentation discovery framework consists of an LLM-based evolution process and a long-tailed training process, where,

• The LLM-based evolution process will learn existing successful data augmentation methods and discover unseen data augmentation methods through mutation and crossover processes.
• The long-tail learning model evaluates the true effect of these newly discovered data augmentations through the real long-tail distribution environment and dynamic data augmentation during training.

That is, rather than letting LLMs devise class-specific data augmentation strategies based on text, we let LLMs learn from existing methods and invent new ones, and then evaluate the methods based on the ability of the long-tailed model to evaluate image data. By interacting with the LLM and the long-tailed model in this way, we can achieve a positive loop between evaluating the features of the image data and innovating the augmentation algorithm.

To give you a better understanding, we add an example in Figure 1 in ONE-PAGE PDF to demonstrate the discovery process of the algorithm. In addition, we compare the performance improvement curves of the augmentation strategies in the three evolution stages, to illustrate that our framework can generate corresponding data augmentation strategies towards methods that are beneficial for long-tailed learning.

W4: Your suggestion is to the point. Thank you for your experience in the field to help us improve our work. Limited by the training time, we tried our best to reproduce part of the baselines in the limited time and have added the corresponding comparison experiment in Table 1 in ONE-PAGE PDF. We'll add a more comprehensive comparison to cover more of the latest long-tailed data augmentation methods in the revised version.

Thank you again for your valuable advice. In the revised version, we will supplement the related work review and add the baseline as appropriate. At the same time, we are also happy to agree with you and apologize for the misunderstanding caused. We also look forward to further communication with you and listening to your suggestions.