Self-Paced Augmentations (SPAug) for Improving Model Robustness

Thank you for bringing our attention to these related works. However, we would like to clarify the differences between the proposed SPAug and two other related methods, Universal Adaptive Data Augmentation (UADA) [1] and Adaptive Data Augmentation [2].

When comparing our work to UADA, the main distinction lies in the fact that, although UADA adjusts the augmentation parameters adaptively over iterations, it maintains uniform parameters for all data samples in the dataset. In other words, UADA's goal is to change (or adapt) the augmentation parameters based on the model's gradient information during training. In contrast, our objective is to have instance-dependent augmentation parameters that change based on how each sample converges during training. We will discuss UADA in the related work section, highlighting this distinction between SPAug and UADA. However, it's worth noting that UADA does not provide a comparison of their performance on corrupted test sets (such as CIFAR-10-C and CIFAR-100-C), which currently prevents us from making a direct comparison between UADA and SPAug.

The main intuition behind the Adaptive Data Augmentation framework [2] is to find a small transformation that results in the maximum classification loss, leading to training with worst-case data augmentation. However, their proposed worst-case augmentation framework only works with translational augmentations and cannot be directly extended to more complex augmentation operations, such as color transformations or mixing. Furthermore, they have validated their results on very small datasets, such as MNIST-500 and Small-NORB, which prevents us from making a direct comparison with their work. Nevertheless, we will discuss this work in the Related Work section since it represents one of the early attempts to demonstrate the importance of adaptive data augmentations.

The proposed augmentation, as illustrated in Eq.1, shares a similar formulation as two widely used augmentation methods: CutMix and Mixup. However, there is no comparison between the proposed method and CutMix/Mixup.

Thank you for the comment. Although the proposed approach may appear similar to CutMix and MixUp at first glance, they are fundamentally different from our SPAug. The primary distinction lies in the fact that CutMix and MixUp fall under a uniform augmentation policy. In other words, the hyperparameters for CutMix and MixUp are fixed for all samples in the dataset, resulting in uniform augmentation intensity applied to all samples. In contrast, SPAug introduces sample-dependent augmentation intensity during training, tailored to how each sample converges. Samples that converge rapidly (referred to as "easy samples") are encouraged to undergo higher levels of augmentation intensity, while those converging slowly (referred to as "hard samples") are encouraged to train with less synthetic augmentation applied. Moreover, instead of employing computationally expensive optimization processes such as reinforcement learning or bi-level optimization, we draw inspiration from the curriculum learning literature and propose to use each sample's training loss (cross-entropy loss) as a proxy measure to determine their ease or difficulty of convergence.

However, in response to the reviewer's feedback, we have decided to compare our SPAug method with CutMix and MixUp to provide a more comprehensive analysis. Following your comment, we have conducted the comparison as shown in Table 1 and Table 2 (below).

Table 1: Comparison of corrupted Test Error (C-Err.) of SPAug with AugMix policy on CIFAR-10-C with WRN-40-2.

Table 2: Comparison of corrupted Test Error (C-Err.) of SPAug with AugMix policy on CIFAR-100-C with WRN-40-2.

As shown in Table 1 and Table 2, the augmentation policies CutOut, MixUp, and CutMix result in significantly higher corrupted test error rates (C-Err.) than Augmix+SPAugLearnable. This indicates that models trained with those augmentation policies have lower generalizability against common corruptions.

In regards to demonstrating results on ImageNet, we have done some preliminary comparisons in Table 3 of the main body. We will extend experiments on this dataset in future revisions.

In Section 4.3, you mentioned that the learnable SPAug has a significant improvement effect on the performance of AugMix in processing corrupted data. However, in the above Table 2, compared to the optimal model, your performance improvement is very limited or there is no improvement at all. The superiority of the model is not sufficiently reflected.

Thank you for your comment! After reviewing your feedback, we realized that the performance improvement of SPAug over AugMix may not have been adequately highlighted. The tables below provides a summary of the performance enhancements achieved by incorporating SPAug with the AugMix policy.

Table 1: Comparison of SPAug+AugMix vs. AugMix in terms of mean corruption error on CIFAR-10-C with WRN-40-2 backbone.

Table 2: Comparison of SPAug+AugMix vs. AugMix in terms of mean corruption error on CIFAR-100-C with WRN-40-2 backbone.

Table 3: Comparison of SPAug+AugMix vs. AugMix in terms of mean corruption error on CIFAR-10-C with WRN-28-10 backbone.

Table 4: Comparison of SPAug+AugMix vs. AugMix in terms of mean corruption error on CIFAR-100-C with WRN-28-10 backbone.

As demonstrated in the tables above, it is evident that coupling SPAug with AugMix consistently results in a significant improvement in performance on both CIFAR-10 and CIFAR-100, whether using the WRN-40-2 or WRN-28-10 backbone. This further underscores the benefit of employing instance-specific augmentation parameters over uniform augmentation parameters.

ntroducing an adaptive learning data augmentation strategy increases the complexity of the model and may lead to extended training times and computational costs. While you mentioned that this method introduces minimal overhead, there is no specific experimental data in the article to support this claim.

That is a valid question, and thank you for highlighting it. Since our method does not involve an expensive optimization process (such as the bi-level optimization procedure utilized in AugMax), we did not observe any significant slowdown compared to the baseline. To support our argument, we benchmarked the training time per epoch for standard training, AugMix, SPAug with AugMix policy, and AugMax on ImageNet with ResNet18, using a single NVIDIA A6000 GPU.

As shown in the table above, introducing SPAug into a AugMix slows down the code only by 3698-3622 = 76 sec/epoch which is very minimal computational overhead compared to augmentation policies that use complex optimization processes to determine instance-specific parameters, like AugMax, which can result in considerable computational overhead.

Due to the stochastic nature of the augmentations, the corresponding loss for a given sample can fluctuate and introduce a certain noise on the selection of the easy/hard samples. Did you find any instabilities in the training due to this noise?

Thank you for your comment. No, we did not observe any training instabilities during the training.

For example, Figure 7 in Supplementary Document shows CE loss when training WRN-40-2 backbone with AugMix policy w/ and w/o SPAug. As can be seen, there is no training instabilities observed.

How does the method perform on ImageNet with other existing augmentation techniques, and with other architectures (e.g. Transformers)?

Thank you for the comment. Following the previous works, we limited our backbones to ResNets and Wide-ResNet (WRN) architectures in the main body. We also included AllConvNet, DenseNet architectures as well in Table 2 of the supplementary material.) Similarly, following some of the literature, we have included ImageNet results in Table 3 on the ResNet-50 architecture. However, we see the value of your comment, and we will try compare the results of SPAug with ViT architectures on ImageNet in the next revision, as it requires longer training time and some time to set up the experiments.

In Tables 4 and 5, the results of models trained in the 100 and 200 epochs are given. However, the gained performance through more training epochs is not significant. For instance, in Table 4, the gained C-Err for SPAug-Learnable is on average 0.5, while this value for AA is 0.9. What is the reason for training more epochs to obtain trivial improvement using SPAug-Learnable? Besides, why is the baseline model WRN-28-10 having different C-Err in Table 2 and 4?

Thank you for the comment! The reason why we presented the results for 100 and 200 epochs is that we wanted to show that with SPAug, the network tends to converge pretty fast compared to having a uniform augmentation policy, as motivated in the introduction. You are absolutely correct in your assessment that with an additional 100 epochs of training, AA improves by +0.9%, whereas with SPAug, it improved by +0.5% because, with 100 epochs, it already converged to a better accuracy than AA. Another point we want to emphasize here is that usually when we incorporate an augmentation policy, it requires longer training time (like 200 epochs or more) to converge properly because with the uniform augmentation policy, some samples become much harder to fit, requiring longer training time. However, when we have instance-wise augmentation parameters, not all the samples undergo the same level of augmentation intensity, hence making the convergence easier and faster. We would like to thank the reviewer for this observation!

Regarding your second point about why the baseline numbers in Table 2 and 4 are different, it's due to the stochastic nature of the experiments, and these are average values of two independent sets of 3 trials conducted for AugMix and AA. As we can see, each average accuracy has a non-zero standard deviation, hence the average value can fluctuate slightly. We can see that both average C-Err. values are within their standard deviation (for example, on CIFAR-10 with WRN-28-10 backbone: $23.5\pm0.9$ vs. $23.3\pm0.3$).

In the experiments, how is the threshold $\tau$ that distinguishes easy samples from hard ones in the minibatch determined? Is it affected by the minibatch size? Is there any trade-off between them, considering needed computational resources?

We thank the reviewer for this interesting question. As noted in the supplimantary material, the threshold $\tau$ is determined like regular hyperparamter tuning. We find the best threshold $\tau$ for a given dataset by performing hyperparamter seach over $\tau$ values ranging from $0.0, 0.1, 0.25, 0.5, 0.75, 0.9, 1.0$. We did not change the batch-size values throughout the experiments and keep it same as the same batch size used in the standard training.

For example, Table 3 and 4 in Supplementary Document shows how the SPAug results varying for hyperparamter tuning on $\tau$ for CIFAR-10 and CIFAR-100 datasets with AugMix. As shown in the tables, we observed that $\tau = 0.75$ and $\tau = 0.9$ works well and can be considered as a good rule of thumb.

We thank the reviewer for pointing out these interesting related works [1-2]. While we recognize that the high-level idea of AugMax, MADAug, and SPAug is to have instance specific augmentation parameters, our approach of addressing this problem fundamentally differs from others.

Answer to Q1

How is the work different from [1,2] and what improvements have been made regarding them? This work is quite similar to AugMax[1] in the sense of combining augmented images with their original version using weights calculated by back-propagation.

When considering AugMax alongside SPAug, we would like to emphasize the following differences:

• In AugMax, these instance-specific parameters are determined in an adversarial manner. Hence, during training, each sample is trained with its maximum augmentation level. In contrast, SPAug determines these parameters based on each sample's Cross-Entropy (CE) loss at the previous epoch.
• Since AugMax determines the instance-specific parameters in an adversarial manner, for each sample, at each training iteration, they need to perform a potentially expensive adversarial attack, which adds computational overhead. In contrast, SPAug uses the sample's previous epoch loss as the proxy value, hence there is little to no computational overhead over standard training.
• In addition to the above two main differences, AugMax introduces changes to the network architecture to boost its performance. Since, in AugMax, samples are trained with their maximum augmentations, it leads to a deviation of distribution between augmented and input data, which makes the training more challenging, as noted in the paper. To address this issue, they use a different normalization technique termed DuBIN, to disentangle the instance-wise feature heterogeneity of AugMax samples. Since SPAug controls augmentation intensity based on its previous loss values, the augmented views won't deviate much from the original training distribution as in AugMax. In contrast, our method (SPAug) does not require additional modifications to network architecture or the training setup and can be easily integrated to existing training pipelines. When we use the same basic WRN40-2 architecture (with regular 2D batchnorm) that we use in our paper, we find that on CIFAR-100 AugMax has 36.6% corrupted error (which is 1.6% worst than that of SPAug) . In the next revision, we will include this comparison.

When considering the second work (MADAug), we want to draw the reviewer's attention to the fact that it was posted on arXiv on Sat, 9 Sep 2023, which is very close to the ICLR submission date and can be considered as a concurrent related work. Similar to SPAug, MADAug also proposes having instance-wise augmentation parameters, further demonstrating the recent trend in this direction. However, after carefully reviewing their approach, we have observed the following fundamental differences with our work:

• They also acknowledge that at the beginning of training, samples are not well-fitted; hence applying the same level of augmentations at that phase reduces the final accuracy. Therefore, they propose to use a manually designed probability schedule $p(t) = \tanh(t/\tau)$, where $t$ denotes the current training epoch. In addition, MADAug assigns an augmentation probability $p$ and magnitude $\lambda$ to each sample, determined by the policy network parameterized by $\theta$, which takes the image features extracted from the task model parameterized by $w$. The bi-level optimization process consists of three steps:
  1. One-step gradient descent on $w_t$ to achieve a closed-form surrogate $\hat{w}$ of the lower-level problem solution.
  2. Updating policy network parameters $\theta_t$ by minimizing the validation loss computed by the meta-task model $\hat{w}t$ on a mini-batch of the validation set.
  3. Updating the main model parameters $w$ based on the parameter $\theta_{t+1}$ of the policy model.

Due to this bi-level optimization, MADAug requires 2 forward passes for each training minibatch as well as forward passes on the validation minibatch, hence taking approximately $1.5\times$ longer training time per epoch compared to our method when training Wide-ResNet-28-10 on the CIFAR-10 dataset.