Sensitivity-Adaptive Augmentation for Robust Segmentation

Thank you for your review! We improved our revisions based on your feedback, and address each point in specific, below.

1. Writing Quality. Thank you for your feedback on writing clarity. We have made substantial revisions to our writing, especially in the methodology section. We hope that the updated text provides more intuition and clarity regarding our approach.

2. Missing Comparisons. Our approach is a data augmentation scheme, similar to the works used in comparisons. Specifically, the AutoAugment used in our comparisons is the same policy trained on ImageNet in the original paper, which requires about 15,000 GPU hours and is shown to transfer strongly to other datasets. In our revisions, we have clarified that the ImageNet policy is used in comparisons and that we do not retrain the augmentation policy. While our approach models sensitivity based on the current training data, our augmentation routine is based purely on heuristics and is not a learnable policy. Thus, we cannot “transfer” one augmentation scheme from another dataset to ours, unlike AutoAugment. Our method does not require training an additional model for augmentation selection. Additionally, we evaluate models on corruption benchmarks with scenarios not seen in any model during training. In other words, evaluation benchmarks test generalization capabilities on a third distribution of data, which is neither the source dataset (ImageNet) nor the target dataset (dataset used in training). In our original submission, we included a small table of classification result comparisons on the CUB dataset for all methods. This demonstrates that our method still provides improvements over others in classification tasks. The CUB dataset, being much smaller than the ImageNet dataset, would have a much less robust augmentation policy under AutoAugment.
   Lastly, we considered reimplementing AutoAugment. While many public repositories describe existing augmentation policies, none reproduce the reinforcement learning process that trains the policy. As such, we decided to compare the official implementations of AutoAugment. Our method indeed has similarities with meta-learning approaches, but we would like to clarify that our approach is a data augmentation scheme explicitly based on “sensitivity modeling” as a function of input data corruption and model output performance. That is, to determine how the amount of variation in input data is related to the amount of variation in model output performance. Meta-learning algorithms describe a class of techniques for generalization across tasks. We emphasize this difference to highlight that our approach can be used orthogonally to meta-learning algorithms, not as a direct comparison. We have, however, implemented a more recent work in our revisions: IDBH. IDBH is an augmentation approach published in ICLR 2023 focused on boosting adversarial robustness through data augmentation. This method outperforms previous baselines we compared to in the original submission, and further strengthens our improvements.

3. Mathematical and Methodology Clarity. This feedback was helpful, and we have clarified these points in a simplified and rewritten methodology section. We emphasize that the original approach remains the same, but notation and explanations have been reworded for clarity to address reviewer’s suggestions.

4. Sensitivity Analysis. Per this feedback, similarly to that of other reviewers, we have included sensitivity analysis plots across training of Cityscapes, showing how sensitivity changes with respect to RGB, HSV, blur, and noise channels. Since the ablation study shows that photometric augmentations contribute most to generalization, this additional analysis (Figure 4) provides insight into which color channels contribute most to generalization. In the case of Cityscapes, we note that the Hue channel plays a large role in generalization, due to its volatility during training as the model generalizes. Exploring this avenue further would be interesting and beneficial for model interpretability.

Dear Reviewer RTLN: Similar to many, many published papers, this work offers multiple key contributions:

1. An efficient adaptive sensitivity analysis method for online model evaluation that iteratively approximates model sensitivity curves for speedup;
2. A comprehensive framework that leverages sensitivity analysis results to systematically improve the robustness of learning-based segmentation models;
3. Evaluation and analysis of our method on unseen synthetically perturbed samples, naturallycorrupted samples, and ablated contributing factors to robustification. Our newly derived theoretical framework on adaptive sensitivity analysis runs up to 10× faster and requires up to 200× less storage than previous approaches, enabling practical, on-the-fly estimation during training for a model-free augmentation policy. But, it's too mathy for most readers, in spite of sensitivity analysis is a very intuitive concept (how the input varies would impact the output of learning or black-box system). So demonstration on at least 1 application is expected by reviewers.

You're right that the contributions here are probably worthy of 2 papers and not incremental at all. The first contribution has tremendous impact on many other areas of learning-based visual processing, control, planning, synthesis and much more, but it's not possible to demonstrate this novel concept of on-the-fly sensitivity analysis without demonstration on at least ONE application and/or task. Just 1 application to segmentations requires so many comparisons and multiple man-years and the results in Appendices are already longer than the paper itself. We would appreciate your reconsideration of the potential impact of such work for the broader learning community. All learning algorithms can benefit from faster and leaner training. We will release the code upon acceptance of this work, so others will enjoy the same benefits of (nearly) free data-augmentation for robustification of learning-based algorithms beyond segmentation -- though immediately the results on segmentation already significantly advance the state-of-art.

To the concern: "the paper seems to sit somewhere between data augmentation for robust semantic segmentation and sensitivity analysis, without clearly committing to a specific focus", we want to highlight that our work makes online sensitivity analysis fast and viable for use in data augmentation schemes. Data augmentation is one of many ways to increase generalization capabilities of a neural network. While other works approach robustness from adversarial objectives or from the architecture side, our work focuses on the data side. Other methods for improving robustness, such as the adversarial techniques cited, can be implemented orthogonally to our method in the same framework. For our augmentation scheme, which is informed by a sensitivity heuristic, we choose to compare to other data augmentation approaches which boost generalization.

Our work has many contributions, but we believe that all contributions are related for one unified work.

Thank you!

Sincerely,

The Authors

Thank you for your feedback! From the comments, we understand that a major issue was the clarity of presentation and lack of analyses. We are happy to update the reviewer with new paper revisions incorporating their feedback. Specific points are addressed below.

• The description of sensitivity analysis could be more intuitive. Based on this feedback, we have updated the explanation of our methodology, along with simplified notation and intuitive explanation. Algorithm pseudocode for training has been added to the main text (Algorithm 1).

• The proposed fast sensitivity analysis lacks an intuitive explanation of why it works and how it significantly reduces the computational cost. Similar to the point above, we have revised our methodology section to address these concerns. For convenience, we reiterate here that the max-min objective prioritizes augmentation intensity values to be densely sampled in regions of high sensitivity (i.e., high change in accuracy, and low changes in KID). The computational cost reduction stems from an additional outer loop in previous work, which discretized the augmentation space uniformly along the augmentation value space, then optimized amongst the discretized values. Due to this offline sampling step, the storage necessity and computational complexity is much greater. Our adaptive sensitivity analysis focuses data augmentation near regions of high sensitivity, i.e. small variation in input would lead to high variation in the output performance. So, by sampling more around high-sensitivity regions and much less in low-sensitivity regions can lead to significant cost saving in the amount of augmented data, thus significantly reducing the computation and the storage.

• The iterative estimation process could be better illustrated with examples or visualizations. Our original submission contained a figure demonstrating this process, but we neglected to reference it in the main body. In our revisions, we have referenced Figure 12 in Section D.7 of the appendix, which describes the iterative estimation process involving PCHIP.

• The paper could explore more about the reasons why some methods perform better or worse in different scenarios. TrivialAugment is a simplistic augmentation approach that can be interpreted as uniformly and randomly sampling augmentation intensities. In our updated revisions, we included ablation study results showing several variations of our method and the contributions of each component. For our method, we believe that the largest boost in performance comes from sensitivity analysis itself; without it, the random augmentation scheme is equivalent to TrivialAugment. Additionally, we moved analyses from the appendix to the main body to highlight insights into how the sensitivity of each color channel changes as generalization improves during training. From the results, we find that Hue channel plays a significant role in generalization capabilities in the case of Cityscapes.

• Fig. 1 is confusing. Per this feedback, we have re-drawn Figure 1 to include all components of our method in more detail. We hope that this revision provides more clarity to the reviewer!

Thank you for your insight! We are happy to share that we incorporated most of the weakness feedback into our revisions.

• Lack of specific ablation experiments. We have added an ablation study in Section 4.5. Regarding pre-trained weights, all experiments are initialized with the same SegFormer weights pre-trained on ImageNet-1k, including baseline experiments. Thus, all performance improvements are fair with respect to model initializations. We also fix the random seed for all experiments to ensure reproducible results across runs.
• The comparison is limited. To improve on this feedback, we added an additional comparison to IDBH, proposed by Li et al., and published in ICLR 2023: https://openreview.net/forum?id=y4uc4NtTWaq. We have updated our results tables to include this comparison and maintain that our method produces better results in unseen corruption scenarios, both artificial and real-world.
• Consideration of low-light settings. We would like to clarify that our experiments evaluate models trained on clear, sunny weather on datasets of adverse weather, including that of nighttime data. Since nighttime data is not involved in training, we expect performance to be lower in these conditions. While augmentation for all comparisons include brightness augmentations, this alone is not sufficient to address the performance gap in night time data. We note this limitation of our work and consider it a promising area for future research, along with exploration of photometric perturbations and their impact on model generalization.
• The paper treats all types of augmentations as equivalent. To address this, we added an experiment in our ablation study to examine whether removing the assumption of equivalence among augmentations affects performance. While the performance is largely the same, we do observe slight improvements. We thank the reviewer for pointing out this assumption and have addressed it in the revision. For visualizations, we moved Figure 4 from the appendix of the original submission to the main body to illustrate how different color channel sensitivities change over training. Interestingly, we note that Hue may play a significant role in model generalization for segmentation.
• The explanation of hyperparameters is not detailed. We take reproducibility seriously and thank the reviewer for pointing this out. We have added a hyperparameter table (Table 7) to the appendix, detailing differences between augmentation techniques and training parameters for experiments. We also clarified in the main body that training parameters for all methods are the same, with the main differences being the data augmentation schemes. The set of augmentation functions across all methods is the same, with exception to IDBH, which involves two extra augmentations (RandomFlip, RandomErasing). We maintain these two additional augmentations to stay true to the authors’ original implementation.
• Minor writing issues. We have revised writing issues throughout the paper and hope that the improved clarity is reflected in the updated version!

Thank you for your feedback! We included revisions to our submission based on your suggestions.

• The contributions and insights of this work are somewhat limited. While we disagree with the assertion that our method relies on performance-boosting tricks, we acknowledge that we may have presumed ‘sensitivity analysis’ to be a well-known concept, which may have led to a lack of intuitive explanation and detailed insights in our approach. In fact, incorporating sensitivity analysis as an explicit step in data augmentation can offer more transparency to model robustness throughout training. To address weakness point #1, we added Figure 4 in our results to show how sensitivity analysis can be useful to draw insights on model generalization.

• The experiments lack alignment in terms of using the same semantic segmentation models. We have replaced all experiments using PSPNet in the main body with those using SegFormer, which is a recent state-of-the-art architecture for robust segmentation. This ensures consistency of architecture across experiments. Our original intention in showing results on both architectures was to demonstrate that our method produces performance boosts on different backbones—however, we understand that this may have caused more confusion than clarity. Results comparing PSPNet and SegFormer performance using our method can still be found in Section D.5 of the appendix.

• The augmentation benchmarking experiments are not particularly convincing. We would like to clarify a misunderstanding. Our method is not an ensembling technique of multiple augmentation algorithms; rather, it is a single routine that models sensitivity for several different augmentation types. It is as much as an ensemble as random augmentation (TrivialAug) would be considered an ensemble of uniform distributions. We also want to note that while our method determines an augmentation scheme which prioritizes sensitive augmentations, it differs from common policy optimization approaches in that it is neural-network free. Our approach is purely based on sensitivity-guided performance optimization and only the segmentation model itself is being optimized.