Decoupled Data Augmentation for Improving Image Classification

Major:

1. Limited technical contribution. The paper utilizes existing techniques such as SAM, LayerDiffuse, and text inversion. While these technologies are well-utilized to produce seemingly credible results, there is a lack of novel technical contributions and design. Additionally, performance improvements gained by using SAM are unsurprising and come with increased computational costs. Would the method still work without SAM? Are there alternatives to SAM?
2. The writing needs further polish. There are confusing descriptions, particularly in the Method section's Conservative Generation of CDP. For instance, in Eq. 3, the variable $c$ seems to refer to a text prompt, but its definition is missing. Moreover, as a critical part of augmentation, there is no experimental discussion on it. Also, if Line 233’s $\lfloor T_{s}\rfloor$ indicates a timestep, what does $\lfloor S_{T_{s}}\rfloor$ in Eq. 1 mean? $S$ is also undefined—perhaps it means total steps? Numerous other issues are noted in the Minor weaknesses.
3. I recommend adding a Background section to provide a brief review of SDEdit and the diffusion model used, along with clear symbol definitions.
4. Experimental results are confusing. For example, the source of the experimental results is unclear—are they reproduced by the authors or cited from another paper? In Table 2, the authors label “training from scratch,” yet the vanilla results for DiffuseMix on ResNet-50@448 are 65.50, 80.29, and 85.52, compared to this paper’s 72.54, 71.53, and 91.32, which are significantly higher than those reported in DiffuseMix’s Table 14. The authors need to clarify the experimental setup and result sources. Additionally, in Table 5, results for Vanilla, CutMix, DA-Fusion, and Diff-Mix are cited from Diff-Mix, but the source for the remaining results is missing.

Minor:

1. Inconsistent use of \citet{} and \citep{}. Please verify correct usage in Line 058, Line 061, Line 068, and Line 346.
2. Incorrect citation. The citation for DiffuseMix in Line 315 seems wrong.
3. Incorrect Y-axis label. Why is the metric for multi-label classification in Figure 5(a) PSNR?
4. Line 731 seems incomplete. "For inpainting the missing part of For the training ..." What does this mean?????

Limited Novelty. The approach of using background changes to create augmented data has been studied before. For example, InSP[1] swaps the saliency part of two images from the same class and is tested on CUB, Stanford Car, and FGVC-Aircraft datasets. Copy-paste augmentation [2] is a low-cost augmentation method that copies and pastes a random object into another image, for instance segmentation. Applying textual inversion and SDEdit to transform objects was suggested in DA-Fusion. The SDEdit and SAM models are also off-the-shelf methods proposed in previous works. The proposed idea of transforming extracted objects and pasting them into the backgrounds from other images is somewhat incremental to existing works.

De-DA proposes to extract class-dependent parts of an image using SAM. If the SAM model is not pre-trained on the target domain, it may fail to capture the class-dependent part, such as medical images. In addition, the class-dependent parts of an image are not limited to the background. For example, in gesture classification, the gesture is also a class-dependent part; in scene classification, the whole image should be considered. It seems that the proposed method cannot capture non-background class-dependent features and is limited to fine-grained object classification tasks.

Limited evaluation of the tested datasets and tasks. It appears that all three tested datasets are fine-grained classification datasets. The backgrounds in fine-grained datasets are relatively easy to transfer as the objects are mostly the same. However, this may not be the case if the object classes are semantically different. For example, swapping the background of a marine animal and a desert animal may not be appropriate. Therefore, I suggest the authors test their method on large-scale datasets with more diverse objects, e.g., the ImageNet dataset.

[1] Zhang, Lianbo, Shaoli Huang, and Wei Liu. "Intra-class part swapping for fine-grained image classification." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.

[2] Ghiasi, Golnaz, et al. "Simple copy-paste is a strong data augmentation method for instance segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

• The paper only conducted experiments on datasets with relatively limited data. It did not illustrate the effectiveness on larger and more general datasets. Can this method work on larger datasets like ImageNet, similar to how mixup or cutmix do?
• In the experiments comparing with RandAugment, only the results of “DE-DA” and “DE-DA + RandAugment” are provided. I believe that further results for “RandAugment only” should be included. Because it would be better to demonstrate that the “DE-DA + RandAugment” method outperforms “RandAugment only”.

1.Although the Online Randomized Combination method is efficient, it may lead to semantically unnatural compositions. For instance, birds in the synthetic samples do not always appear naturally perched on branches (see the last row of birds in Figure 6, where proper positioning on branches is rare). While semantic naturalness may not always be critical for classification, this could reduce the generalizability of synthetic data.

2.The authors should include more visualizations of De-DA results, especially showcasing foreground variations in CIP and the inpainting results for CDP. Additionally, it would be valuable to discuss some lower-quality samples and include a discussion on the limitations of the current approach.

3.Some experimental settings are overly brief. For instance, details regarding the multi-label classification implementation are missing, and the authors provide insufficient explanation on how samples are constructed in this scenario.