On the Use of Anchoring for Training Vision Models

We thank you for your positive feedback. We hope our responses address the questions you have raised.

1. Accuracy Remains Constant Regardless of Reference Set Size

We would like to clarify that this is precisely the problem with the original anchored training protocol, which we solve in this paper. As shown in Fig. 1 of the paper, the original anchoring protocol does not fully leverage the diversity of reference-residual pairs with increasing reference set size and maintains a relatively constant accuracy regardless of the reference set size. We hypothesize that this is because as the size of the reference set increases, the number of reference-residual pairs grows combinatorially. For e.g., when the reference set is the entire dataset D, there are |D| choose 2 pairs, making it impractical to explore all pairs within a fixed number of training iterations. This results in insufficient sampling of reference-residual pairs, increasing the risk that anchored training may overlook the reference and make predictions based solely on the residuals leading to non-generalizable shortcuts. This is problematic as a sample should not be identifiable without knowing the reference.

2. Alleviating the Reference Set Size Problem

One possible way of alleviating this problem is by reducing the reference set size. However, this reduces the diversity of the reference-residual pairs exposed during training and can lead to a poor solution. While the issue of diversity can be combated with large reference set sizes, increasing the number of epochs alone does not solve the problem as there exists a combinatorially large number of reference-residual pairs which cannot be practically explored, and the model will still be vulnerable to shortcuts. Moreover, modifying the number of training epochs results in non-trivial modifications in the training hyper-parameters (e.g., learning rate schedules) and can lead to poorly convergent models if the hyper-parameters are chosen incorrectly. Hence, we propose a reference masking regularizer for anchored training, that helps mitigate shortcut decision rules while also being computationally efficient.

3. Reference Set Selection Strategy

For all experiments in Section 4, we utilize the entire training dataset as the reference set and train both the original and the proposed anchored models. During inference, we randomly select a single reference from the reference set and perform evaluation on the different test datasets. We will better clarify this in the final version of the paper.

4. Impact/Novelty

Anchoring is a framework that is agnostic to any training strategy, domain or application and can be wrapped along with other strategies (data augmentation, loss functions, regularizers, ensembling) that help improve model performance. While this is attractive, our paper identifies the shortcomings of the existing protocol and deals with a fundamental problem of how to train and make predictions with anchored models in practice. We develop a novel reference masking protocol for training anchored models that can significantly improve overall model generalization. We demonstrate significant quantitative performance improvements over standard and original anchored training protocol across different datasets, tasks and architectures. Particularly, we find that our proposed algorithm leads to a wider and flatter optima corresponding to superior solutions (Fig. 4), and can be used on top of augmentation strategies (Fig. 5a) and better handle training label noise (Fig. 5b). We systematically establish the empirical efficacy of our approach on OOD generalization and model safety tasks ranging from calibration, anomaly rejection as well as task adaptation and domain generalization. We expect to foster interesting research directions with anchoring and even impact applications in different domains (e.g., text, graphs etc.) where generalization and model safety are paramount concerns.

5. Formatting Issues and Typos

We will make the changes to the tables and figures in the final version of the paper.

We thank the reviewer for taking the time to review our paper and providing useful feedback. As the discussion phase is ending soon, we will greatly appreciate if the reviewer can check our response and let us know if there are any additional questions.

We thank you for your positive feedback. Here are our responses to your questions. We plan to incorporate some of these clarifying comments to the manuscript as well.

1. Choice of $\alpha$

We want to clarify that at low reference set sizes, there is a high likelihood of exposing the model to all possible combinations of samples and references, and hence the risk of learning shortcuts is minimal. In this case, overemphasizing the reference masking probability (i.e., increasing $\alpha$) can significantly inhibit this exposure. Consequently, this leads to underfitting as the model is tasked with learning solely from the residuals which is undesirable in practice (Blue curve for reference set size $\leq$ 50 in Fig.1 of the paper). Reducing $\alpha$ can combat this behavior, as evidenced by the original anchored training (special case of reference masking with $\alpha = 0$, red curves for reference set size $\leq$ 50 in Fig.1).

Now, with larger reference sets (e.g., datasets in the scale of ImageNet 1K), the number of reference-residual pairs grows combinatorially, making it impractical to expose the model to all diverse pairs in a fixed number of training iterations. In such a scenario, reducing $\alpha$ can increase the risk of learning shortcuts and lead to suboptimal performance. Increasing $\alpha$ on the other hand can in fact aid training as it systematically avoids these shortcuts and improves generalization. In summary, the optimal $\alpha$ value depends both on the reference set size and the convergence behavior of model training.

2. Performance Improvements on ImageNet-R/S with VITb and SWINv2B

Anchored training with ImageNet-1K involves exposing the model to significantly large and diverse reference-residual combinations. Our results show that, in order to model such a large joint distribution and to leverage the diversity of the reference set, along with the proposed masking regularizer, higher capacity networks (VITb & SWINv2B) are also required. We hypothesize that this behavior helps such networks better handle challenging, far out-of-distribution datasets such as ImageNet-R/S. For instance, we observe an average of 1.5% and 1.7% improvements in ImageNet-R and S accuracies respectively when using such architectures over lower capacity models.

3. Formatting

We will make changes to the paragraph header, correct underline inconsistencies in the introduction and improve Fig. 4 in the final version of the paper.

We thank you for your positive feedback. We hope our responses address your questions.

1. Generic Applicability of Anchoring

Thank you for this question. We concur with you that anchoring is a protocol for training deep neural networks for use with any domain for any application (e.g., texts, graphs or vision tasks such as segmentation). In this paper, our goal was to identify the shortcomings of the original anchored training [1] and develop algorithms to improve the same, and vision was chosen only as a domain of convenience. Though we have performed initial experiments on using anchored training for other data domains (e.g., graphs, text), we did not include them to restrict the scope of the current submission. We find that our proposed anchoring approach produces performance improvements even in non-vision tasks. In the final version of the paper, we will include this discussion as part of the concluding remarks. In summary, anchoring is a domain-agnostic, architecture-agnostic, and task-agnostic training strategy.

2. Clarifying ‘Model Ignoring Reference Input’

It must be noted that the anchoring principle as demonstrated in [1] implicitly explores a large family of functions during training due to the lack of shift invariance of the underlying neural tangent kernel when the input is translated by a reference. We believe that this strategy produces a local optima similar in spirit to stochastic weight averaging [2] that averages multiple solutions along the trajectory of gradient descent. Unique to anchoring, the quality of the converged optima depends upon the diversity of the reference-residual pairs (induce a rich family of functions) exposed during training.

However, we observe that the original anchored training even with large reference sets does not fully leverage the reference-residual diversity and converges to a poor local optima (Fig. 4b). We attribute this to the anchored model relying on shortcuts to make predictions. Please note that, the usage of the term shortcut (ignoring the reference) in the context of anchoring is different from convention. Shortcuts manifest during anchored training when the model can predict well only with certain arbitrary references but on an average converges to a poor optima. Basically, the functions induced by ignoring the reference are entirely different from the ones obtained without ignoring them, making the model eventually converge to a sub-optimal (implicitly averaged) local optima. We will better clarify this in the final version of the paper.

3. Clarifying the Anchoring Inference Mechanism

We would like to emphasize that anchored training (i) enforces prediction consistency of a sample with any reference; (ii) produces a single model; and (iii) converges to a local optima in a manner akin to stochastic weight averaging [2]. Moreover, when the diversity of the reference-residual pairs is well leveraged during training, it allows the model to converge to a wider optima improving model generalization (Fig. 4 in the main paper). Therefore, the `quality’ of the optima governs performance during inference and as a result the inference strategy (e.g., choosing K random references for inference) does not alter the (mean) model performance. It must be noted that anchoring must not be viewed under the lens of model ensembles that train multiple models where each member explores different yet possibly diverse local optima. While [1] measures discrepancies in predictions of a sample with different references as a notion of epistemic uncertainty, we find that the mean performance does not change. Fig. 2 in the main manuscript compares different inferencing protocols (1 random anchor, K anchors and transduction) and finds no significant differences in accuracy.

4. Comparison with Existing OOD/Uncertainty Estimation Methods

Thank you for this very important question. While a systematic evaluation of OOD detection with state-of-the art methods is most essential, it is beyond the scope of this paper. Our paper aims to establish anchoring as a useful training protocol and demonstrate its efficacy across a spectrum of tasks and model architectures. Although [1] provided evidence of the efficacy of epistemic uncertainties from anchoring for OOD detection, we will be conducting a large scale study as a part of our immediate future work.

[1] Thiagarajan et al. Single model uncertainty estimation via stochastic data centering, Neurips 2022

[2] Izmailov et al. "Averaging weights leads to wider optima and better generalization”, UAI 2018

We thank you for your positive comments and feedback. We hope our response addresses your concern.

Domain Generalization Benchmarks

Thank you for this question. We would like to highlight that we performed experiments on DomainNet which is one of the benchmarks from DomainBed [1] (Line 293 of the main paper). Following the setup in [1], we trained on one of the domains (source) and evaluated the model on the remaining domains (target). While it is common in the domain generalization literature to train models end to end on the source dataset, we instead trained a linear probe on top of an anchored feature extractor pre-trained on ImageNet. This was motivated by the need to investigate the impact of anchored training in producing better generalizable feature extractor backbones. In particular, we trained a linear probe with ERM [1] using the ‘real’ and ‘sketch’ (source) splits respectively from DomainNet. We then evaluated performance on the other (target) domains and observed performance improvements over the non-anchored variant.

With respect to the domain generalization specific baselines and training strategies (e.g., DRO, IRM) used in [1], we would like to emphasize that our proposed anchored training protocol can be simply used as a wrapper on all such methods and we expect it to improve overall performance similar in spirit to our analysis on augmentation methods in Section 3.2 (Fig 5a of the main paper). We plan to perform an extensive analysis on domain generalization as a part of our future work.

[1] Gulrajani, Ishaan, and David Lopez-Paz. "In Search of Lost Domain Generalization." ICLR 2021