Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation

Thank you for your review! We clarify some misunderstandings below.

• "Figure 1 not support equal spacing along function g, and there is no proof."

By equal spacing, we meant that given the set of α values that fulfills Q, the set of g(α_i) are at equal intervals along the y-axis of the function g. We have included a proof for equal spacing here: https://drive.google.com/file/d/1oOqCZXiSmLV4k4T5YyJqZ2NDmS91U2RE/view?usp=sharing

We will be sure to include this proof and clarification in the final revision.

• "Authors explain that α1 < α2 < . . . < αL, without explaining why."

Thanks for pointing this out. The intuitive explanation behind solving for “adequate spacing” can be thought of as solving for “uniformly difficult augmentation levels” with respect to model sensitivity. For example, it could be that a model is robust to a wide range of intensity values for a particular perturbation, but robustness quickly degrades past a certain value.

• "Standard metrics like mAP should be used."

While mAP is standard for object recognition and instance segmentation, it isn’t commonly used with semantic segmentation, which classifies by-pixel.

• "Sensitivity analysis only considers augmentation intensity, without the type."

Our work considers only intra-augmentation sensitivity; currently, all augmentation types have an equal chance to be sampled at train-time, when models may be more sensitive to one type of augmentation over the other (e.g., geometric types and photometric types). Implementation wise, the change is simple: the augmentation distribution sampler can be modified to account for the absolute sensitivity of each augmentation all together. Note: we indeed considered such weighing in practice, but this distribution may skew too heavily towards certain types of augmentations, resulting in unintended overfitting.

• "Experiments only use one fixed random seed. There should be multiple random seeds used."

We agree that multiple iterations of each experiment under different random seeds is important to validate that improvements in performance are not attributed to randomness. While we conduct all experiments under one fixed seed in this work, previous work (AdvSteer) has validated the consistency for sensitivity analysis results across multiple random seeds that our work has compared against. Additionally, we reduce the role of randomness in experiments by initializing ALL models to the same initialization weights.

• "Authors miss the augmentation operations that are applied to the frequency domain of images."

The mentioned works in frequency space augmentation primarily deal with classification tasks, while our work focuses on segmentation. Direct translation of these works to segmentation is nontrivial, as segmentation is likely much more dependent on high frequency details. This is apparent in that some works explore frequency-based domain adaptation specifically for this task (https://openreview.net/pdf?id=b7hmPlOqr8). However, this idea is very interesting and we believe it valuable to explore in future work! Currently, the set of augmentation operations we use are consistent with other baselines we benchmark against. We choose to augment in the image space due to consistency with previous work, as well as interpretability/explainability and intuition w.r.t. sensitivity curves. Our framework can be directly applied in the frequency space, although the results in frequency space may not be a direct comparison anymore to the methods used in our current experiments.

• "The fine-tuning experiment results may be questionable since the baseline model may have already seen corrupted examples drawn from the same distribution as the evaluation set."

While the foundation models we initialize weights with are based on much larger datasets, which may have seen adverse weather samples at some point in their training, we still observe improvements over baseline fine-tuning experiments when applying augmentation at fine-tuning. Considering that downstream fine-tuning does NOT involve adverse weather samples, we still find that improvement on generalization on natural corruptions to be valuable, since they were not involved in the fine-tuning process whatsoever. Did not disclose exact augmentation operations used, and whether they use the same set of operations in other methods for comparison In Section 4 (Experiments), the “Experiment Setup” excerpt mentions that all methods use the same set of augmentation operations, with the exception of IDBH, which includes two additional augmentations. As explained in the main text, hyperparameter tuning details can be found in Appendix Section A5, which describes all the augmentation operations.

• "Figure 4: unclear meaning of different lines in one subplot"

We evaluate color channel sensitivity several times during training. This plot is meant to show how model sensitivity changes across color channels over the course of training.

Thank you! We appreciate the suggestions and will improve clarity in the revision, adding more references and notation. We address points below:

• “Efficiency improvement claims are missing benchmarks like inference runtime benchmarks and memory usage.”

We would like to clarify that our efficiency improvements are strictly compared to the previous works in sensitivity analysis like AdvSteer, as presented in Table 1. Previously, sensitivity analysis (SA) was NOT computationally feasible to compute during training; it required a huge amount of local storage and long computation times dependent on dataset sizes. Our work makes SA feasible for online training, and opens many possibilities of future work along this direction (see response above to mfiq). Our efficiency claims in the faster inference time or memory usage would not be a fair comparison against other augmentation techniques. Thus, our significant efficiency claims are benchmarked purely for sensitivity analysis. In terms of training complexity, our experiments re-compute SA only 4x during training, amounting to merely 38.4 more GPU minutes for one experiment. Dataset sizes do not influence the sensitivity analysis runtime due to the use of KID, whose data efficiency we verify against Frechet Inception Distance (FID) in Appendix Section A.15. Otherwise, the total train time and memory of our adaptive SA method is comparable to randomized augmentation approaches.

• “The results suggest that this approach is not SOTA, as shown in some results of Table 2 and 3.”

We respectfully disagree, and would like to clarify some interpretations of the results. In Table 2, our method occasionally performs worse on absolute accuracy (aAcc) metrics compared to other methods on rain and snow scenarios. As we mention in the corresponding Section 4.1 discussing the table results, higher numbers on aAcc but lower numbers on mIoU may be indicative of poor generalization to class imbalances, as aAcc measures the total # of correct pixels. In many of the Cityscapes data, there are disproportionately many pixels classified to “sky”. Our method achieves a higher mIoU than the other method (47.53 -> 49.36 compared to AugMix for Rain, and 45.35 -> 48.16 compared to IDBH for Snow). While more PIXELS are correctly classified, lower values on mIoU may suggest that underrepresented classes are poorly classified. We include BOTH metrics in table results to provide an interpretable metric (absolute pixel acc) as well as a class-balanced metric (mIoU) for a more complete picture comparison.

As for Table 3, while our method does not perform best on the AdvSteer benchmark compared to the next-best method, we note that it outperforms other methods in most other scenarios across multiple datasets. AdvSteer benchmark involves heavily altered synthetic data, as shown in the Appendix Section A.10; these alterations are not meant to reflect real-world corruptions, but should rather be interpreted as a synthetic limit test. In contrast, ImageNet-C benchmark reflects transformations meant to replicate real-world effects such as frost, snow, etc. We interpret the boost in performance of our results as contributing primarily to realistic corruptions, which aligns with our goals. Additionally, we include the results for AdvSteer for transparency.

• “Could the authors clarify what distinct advantages their augmentation approach offers, particularly in practical deployment scenarios?”

Our method estimates model sensitivity to various augmentation transformations and samples augmentations with uniform difficulty based on this sensitivity. Unlike randomized approaches (e.g., IDBH, TrivialAug, RandomAug, AugMix) that ignore model state or model-based methods (e.g., AutoAugment) requiring pre-trained policies, our approach is a middle ground by using a generic image classifier trained on ImageNet for KID computations. Although our method incurs additional computational cost compared to randomized approaches—adding ~38.4 GPU minutes to training (4 sensitivity evaluations at ~9.6 minutes each on an RTX A4000 GPU)—it scales efficiently across datasets since KID evaluation requires only a fixed number of samples. Sensitivity analysis currently runs on a single GPU, further optimization and acceleration possible. Also, our method affects only training time and has no impact on inference speed. Sensitivity analysis is also useful for interpretability of robustness, as shown in Figure 4.

In summary, our method 1) provides a middle ground between model-based and randomized augmentation, 2) adds a negligible amount of fixed overhead that is agnostic to dataset size due to use of KID, and 3) provides interpretability and explainability on failure cases for practical deployment.

To show inference comparison with competing methods, we include timing results on experiments with Segment Anything: https://docs.google.com/spreadsheets/d/1KF0lY8iyv7Uo8K53EJmvfZqVqGVfKbuItcu-6iXXjNM/edit?usp=sharing

Thank you for your comments, we are grateful to hear that you find our work impactful in real-world robotics applications and our analyses interesting for model sensitivity! We will be sure to add writing fixes in paper revision regarding Table 1 and increase visibility of results related to our claims, which many of were in the appendix.

Regarding the question, “why is uniform augmentation of computed sensitivity analysis values (alpha) almost as good as beta-binomial sampling?”, we believe that this may be due to that the most optimal sampling in the basis augmentation spaces is already close to uniform. In future revisions, we will include an experiment with such a scenario to emphasize the advantage of not needing corruption gradients.

Thank you! We appreciate your feedback, and address points below:

• "Could meta-learning offer a more direct optimization?"

Yes. NOTE: the difference is mostly wrt the choice between data augmentation vs meta-learning as the training approach, rather than an alternative for the sensitivity analysis (SA). In MAML, we can interpret each parameterized augmentation type as a synthetic task. Then, during bilevel optimization, inner loop updates would optimize performance on specific augmentations (support) while the outer loop maintains overall performance on all aug’s (query). Our SA modeling can still contribute additional info in this case when choosing the task data subsets; the way task data is selected or parameterized is largely open-ended in meta-learning. [1] strengthens model robustness by adding a learned adversarial noise to query (outer loop) data. The difference between this approach and our proposed technique is the paradigm in which sensitivity analysis is applied to add robustness to models (data augmentation sampling vs. noise sampling query attacks in meta-learning). We may also add adversarial noises in our framework.

• "How to extend to less parameter-friendly corruptions?"

One of our motivations is to utilize parameterized corruptions to improve generalization on natural corruptions that may not be easily parameterized. Prior art [1] has shown that many natural corruptions can be replicated with a composition of “basis augmentations”, from which this work is inspired by and generalized from. In cases where we don’t have access to a parameter ‘alpha’, the problem becomes similar with AdvSteer [1], which samples sensitive augmentations from a fixed set of augmentation values (instead of a continuous range). However, this approach has increased computational complexity, since we need to test all values from a selected subset of augmentations.

• "How does it scale to domain-specific augmentations?"

In InterAug’s case, the primary contribution appears to be a re-contextualization around the subjects in the image, s.t. spurious co-occurrences between subjects and background do not occur throughout training. Our is a direct complement to the context area extraction. Instead of considering entire images in our SA computation, we can consider the context bounding box only. The two concepts applied together may produce context-specific sensitivities. As for other domains, the adaptation may be case-by-case. For example, lesion augmentation in medical applications involves synthetically increasing the diversity of lesion shapes, locations, intensities, and load distributions [2] – all of these can be considered as aug- types within our SA framework.

• "How might parameter-efficient approaches like LoRA mitigate overfitting to specific corruptions?"

Training to increase robustness is often accompanied by degradation in clean accuracy, which may suggest either conflicting gradients or overfitting to corruptions. Using LoRA layers to mitigate this has been shown to work in AutoLoRA [3]. Since LoRA layers work very well when trained on small datasets, we may use our sensitivity analysis approach to select “uniformly difficult” augmentations for each class to generate the task dataset for LoRA training. Then, a routing approach similar to Polytropon [4] or MHR [5] can be used for inference on unseen data. This is an interesting extension of our work and a valuable future direction.

• "A benchmark for efficient adaptation would be nice, especially with a small dataset."

We show results on fine-tuning for the ACDC Snow dataset in the bottom of the following spreadsheet: https://docs.google.com/spreadsheets/d/1KF0lY8iyv7Uo8K53EJmvfZqVqGVfKbuItcu-6iXXjNM/edit?usp=sharing and plan to include some more small dataset finetuning experiments in future revisions.

• "Relying on short adaptation intervals might be problematic given grokking is common."

We observe in practice that increasing the number of training iterations (thus increasing the number of iterations per interval, since the number of intervals is fixed) has very little effect on performance outcome. This may suggest that the current interval values may be sufficient for generalization to sensitivity curves within intervals. We can include analysis on this in future revisions. How does this work perform relative to Segment Anything We show downstream fine-tuning results and inference time/memory benchmarks on SegmentAnything in the same spreadsheet as above. We’ll also include these in the updated revision.

• "$a_{max} \neq a_L$. $a_L$ is a level we solve for, but $a_{max}$ is the max intensity of the parameter range, which for our operations is 1."

[1] https://proceedings.neurips.cc/paper/2020/file/cfee398643cbc3dc5eefc89334cacdc1-Paper.pdf

[2] https://arxiv.org/abs/2308.09026

[3] https://openreview.net/forum?id=09xFexjhqE

[4] https://arxiv.org/abs/2202.13914

[5] https://arxiv.org/abs/2211.03831