Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts

Q2 & Q3

Q2: Theorem 1 illustrates that the convergence of DISAM benefits from $\Gamma$. Can the authors explain more on the discussion of as DISAM enjoys a smaller $\Gamma$ than SAM, DISAM can permit the potential larger $\rho$ than that in SAM, thus yielding a better generalization". In particular, how does the convergence rate link with generalization? Q3. The last sentence in Sec 3 claims that ".. allowing larger $\rho$ for better generalization." Why does larger $\rho$ relate to better generalization?

Thank you for the advice on explanation about convergence and generalization, and we have added more discussion and analysis in the Appendix B.3 on page 21. In the following, we provide some clarification on these points.

• Generalization Theorem of SAM: In the SAM framework, the parameter $\rho$ plays a crucial role in determining generalizability. As established in [1], there exists an upper bound on the generalization error for SAM, suggesting that a larger $\rho$ could potentially enhance generalization, provided that convergence is not impeded. Here is the relevant generalization bound from [1]:

For any $\rho > 0$ and any distribution $\mathcal{D}$, with probability $1- \delta$ over the choice of the training set $S\sim \mathcal{D}$, $\mathcal{L} _{\mathcal{D}} (w) \leq \max _{\| \epsilon\| _2 \leq \rho} \mathcal{L} _{S}(w+\epsilon) + \sqrt{\frac{k \log \left( 1 + \frac{\| w \| _2^2}{\rho^2} (1+\sqrt{\frac{\log (n)}{k}})^2 \right) + 4 \log \frac{n}{\delta} + \tilde{O}(1)}{n-1}}$ where $n = |S|$, $k$ is the number of parameters and we assumed $\mathcal{L} _{\mathcal{D}}(w) \leq \mathbb{E} _{\epsilon_i \approx \mathcal{N}(0, \rho)} [\mathcal{L} _{\mathcal{D}}(w+\epsilon)]$. DISAM leverages a smaller $\Gamma$ than SAM, as shown in Theorem 1 in page 5. This allows DISAM to employ a potentially larger $\rho$, enhancing generalizability.

• Practical Implications: Combining the above theorem with the convergence theorem (Theorem 1 on page 5), there is a trade-off with respect to $\rho$. A larger $\rho$ might theoretically enhance generalization but poses greater challenges for convergence. This reflects the intuitive notion that searching for flatter minima across a broader range is inherently more complex, which can potentially affect training efficiency. However, if $\mathcal{L}_{S} (w + \epsilon)$ can be converged with a sufficiently small value, a larger $\rho$ corresponds to better generalization. DISAM, with a smaller $\Gamma$ compared to SAM, converges faster, which means that under the same convergence speed, a larger $\rho$ can be used to achieve better generalization.

• Empirical Validation: Our experiments, as illustrated in Figures 3(c) and (d) on page 6, demonstrate that DISAM effectively employs a larger $\rho$ compared to traditional SAM. DISAM's ability to handle a larger $\rho$ results in both consistent convergence and improved generalization compared to SAM, demonstrating its superiority in domain shift scenarios.

We appreciate the reviewer's comments on the theoretical parts and have adjusted the corresponding parts to make these points more clear.

Reference:

[1]. Sharpness-aware minimization for efficiently improving generalization, ICLR2021.

Q4

(minor) The notation in e.g., eq (5) can be improved, because the multiple subscripts $i$ in $\sum_{i}\frac{C_i}{\sum_{i}C_i}$ are confusing.

We appreciate your detailed feedback on the notation in Eq. (5) and have updated it on page 4 to avoid confusion.

W1

Stronger motivation needed. The authors motivates the domain difference using Fig. 1 (b). While the convergence behaviors among domains are indeed inconsistent at the early stage, the losses are similar after e.g., 30 epoch. The authors should also explain why the difference of convergence in early phase impact the generalization of SAM.

We would like to kindly point out that the y-axis of Figure.1 (b) represents the convergence degree instead of the loss value, which is computed by using the current loss value to divide the converged maximum loss. For the details, we follow the reviewer's advice and have added one section to strenghen the explanation about the motivation in the Appendix C.5 of the revised version.

Relationship of early-stage convergence and generalization: In Figure 7(b) of page 24, we present the curves of various domain losses during training, which can promote the understanding. Basically, the inconsistency of the convergence degree in the early phase in Figure 7(a) actually impairs the overall convergence of the model. As can be seen in Figure 7(b), SAM has converged at a higher loss value, which reflects that the model is optimized towards a poorer local minima, resulting in the worse generalization performance.

W2

More discussions on $\lambda$ in Eq. (7) are needed. This is a critical parameter that considers the variance/domain shifts in DISAM. However, this $\lambda$ does not appear in Theorem 1. Can the authors illustrate more on this point? And how does the choice of $\lambda$ influence convergence and generalization?

Thank you very much for the reviewer's advice. We have elaborated more discussion about the role of $\lambda$ in DISAM in Appendix B.2 (pages 18-21) in the revised submission. In the proof of Theorem 1, specifically Eq. (15) on page 20, $\lambda$ is integrated into $\beta$, serving as a hyperparameter that regulates the weight adjustment in DISAM. It functions by modulating the degree of correction for domain shifts:

$$\beta^i_t = \alpha_i - \frac{2\lambda}{M} \left (\mathcal{L}^i(w_t) - \frac{1}{M} \sum_{j=1}^M \mathcal{L}^j(w_t) \right)$$

The influence of $\lambda$: The choice of $\lambda$ influences how aggressively DISAM responds to variance or domain shifts, with a higher $\lambda$ leading to more pronounced adjustments in $\beta$. Our experimental analysis in Figures 5(c) and (d) on page 9, reveals that DISAM's performance remains relatively stable across a wide range of $\lambda$ values. However, choosing too large $\lambda$ can result in overly aggressive early training adjustments, yielding the negative impact on the convergence process and leading to the increased variance in repeated experiments. Consequently, we adopted a default $\lambda$ value of 0.1 in all experiments.

Q1

Relation with a recent work (https://arxiv.org/abs/2309.15639). The paper above also proposes approaches to reduce variance for finding perturbations, although not designed for the domain generalization setting. How does this work relate with the proposed DISAM?

Thank you for recommending this excellent work on variance suppression (VaSSO)[1]. We have added this method into the revised submission with the proper discussion. Generally, while DISAM and VaSSO both enhance the perturbation direction generation in SAM, they target different aspects:

• VaSSO Approach: VaSSO aims to reduce noise from mini-batch sampling by using averaged previous perturbation directions. Its primary focus is on stabilizing perturbation generation within the same domain.

• DISAM's Unique Focus: In contrast, DISAM incorporates domain information to specifically address domain-level convergence inconsistencies, a challenge prevalent in domain shift scenarios. DISAM's approach is to impose a variance minimization constraint on domain loss during the perturbation generation process, thereby enabling a more representative perturbation location and enhancing generalization.

We are running the experiments to compare/combine DISAM with VaSSO. Once the experiments are finished, we will report the results here and include the results in the submission.

Reference:

[1]. Enhancing Sharpness-Aware Optimization Through Variance Suppression, arXiv2023.

Dear Reviewer BeTH,

We sincerely appreciate the effort and time you have devoted to providing constructive reviews, as well as your positive evaluation of our submission. As the deadline for discussion and paper revision is approaching, we would like to offer a brief summary of our responses and updates:

• Supplementary explanation for Figure 1(b)
• The role and principles of $\lambda$ in the DISAM.
• Similarities and differences between DISAM and VaSSO.
• Mechanism of $\rho$'s impact on convergence and generalization

Would you mind checking our responses and confirming if you have any additional questions? We welcome any further comments and discussions!

Best Regards,

The authors of Submission 1547

W2

The improvement in out-of-distribution (OOD) performance using the DISAM methodology does not appear intuitive. In fact, when comparing its performance enhancements to those achieved with CLIPOOD, as reported, the difference seems marginal. This observation raises questions about the actual effectiveness of DISAM, particularly in the context of fine-tuning methodologies.

We would like to kindly clarify that in Table 2, the results of CLIPOOD in gray are from the orginal paper [1], but we cannot reproduce these results with the open source code of [1] (even after the considerable hyparameter searching). The best results with their open source code are reported as CLIPOOD* and on the basis of these results, DISAM's improvement is actually not marginal (see the following table for convenience).

Besides, it's crucial to highlight that DISAM's advantages become more evident in open-class scenarios. As can be seen in the following table (or Table 3 on page 8), DISAM notably outperforms CLIPOOD in these scenarios, achieving an average improvement of 2.8% on new classes and 1.0% on base classes. This is significant, considering that CoOp and CLIPOOD even underperform compared to zero-shot results in new classes.

For a comprehensive understanding of how DISAM enhances fine-tuning in open-class scenarios, we kindly refer the reviewer to the in-depth analysis provided in Appendix C.6 on page 25. This section substantiates DISAM's role in improving generalization in fine-tuning, especially in contexts that existing methods are challenging to achieve improvement.

Reference:

[1]. CLIPOOD: Generalizing CLIP to Out-of-Distributions, ICML2023.

Reproducible code

Can you provide the reproducible code during the rebuttal period?

We provide an anonymized version of the code repository, accessible through this 2-hop link: [https://openreview.net/forum?id=I4wB3HA3dJ&noteId=e1Uu30vHqy]. To promote the understanding of the code, the reviewers can also combine with Algorithm 1 and the "Pseudo Code of DISAM" in Appendix D on page 25-26.

W1 & Question

W1: The idea of minimizing the variance between losses, a core aspect of the presented methodology, is not entirely novel. Similar concepts have been previously explored in methods like vREX (Out-of-Distribution Generalization via Risk Extrapolation) and further extended to gradient computations in methodologies like Fishr (Invariant Gradient Variances for Out-of-Distribution Generalization). In this context, the proposed approach appears to be an incremental adaptation of vREX principles applied specifically to the challenges faced in Sharpness Aware Minimization (SAM) scenarios. Question: In summary, while the perspective that DISAM is an incremental version of existing methodologies is certainly tenable, a comprehensive evaluation would require a deeper exploration of how DISAM specifically adapts or augments the SAM framework to address its unique challenges. If such adaptations are significant, they could justify the novelty and utility of DISAM beyond a simple combination of existing techniques.

Second, DISAM is orthogonal to existing state-of-the-art methods including V-REx and Fishr and can improve their performance regarding generalization. The following tables present some results to prove DISAM’s superiority:

[Table 1. Comparison with existing methods.]

[Table 2. Comparison with existing SAM-based methods.]

These results clearly demonstrate DISAM’s distinctive approach and its effectiveness in enhancing generalization, supporting its novelty and utility beyond existing methodologies.

We appreciate the reviewer's constructive suggestion, and added a comparison table and more discussion for related works (see Appendix A.2.5 on page 17-18) to clarify the novelty of DISAM in the revision.

Reference:

[1]. Out-of-distribution generalization via risk extrapolation (rex), ICML2021.

[2]. Invariant risk minimization, arXiv2019.

[3]. Gradient matching for domain generalization, arXiv2021.

[4]. Fishr: Invariant gradient variances for out-of-distribution generalization, ICML2022.

W1 & Question

W1: The idea of minimizing the variance between losses, a core aspect of the presented methodology, is not entirely novel. Similar concepts have been previously explored in methods like vREX (Out-of-Distribution Generalization via Risk Extrapolation) and further extended to gradient computations in methodologies like Fishr (Invariant Gradient Variances for Out-of-Distribution Generalization). In this context, the proposed approach appears to be an incremental adaptation of vREX principles applied specifically to the challenges faced in Sharpness Aware Minimization (SAM) scenarios. Question: In summary, while the perspective that DISAM is an incremental version of existing methodologies is certainly tenable, a comprehensive evaluation would require a deeper exploration of how DISAM specifically adapts or augments the SAM framework to address its unique challenges. If such adaptations are significant, they could justify the novelty and utility of DISAM beyond a simple combination of existing techniques.

First, we are sorry about one typo issue in Eq.(7) that may mislead the understanding of the reviewer, which we has corrected with the proper description in the revised submission. Concretely, the $w$ (i.e., $\hat{w}$ in the revised version) in the variance term is actually without derivative taken, when optimzing the model parameter. That is to say, the $w$ (i.e., $\hat{w}$ in the revised version) only makes effect during the inner loop for the perturbation generation. This makes DISAM intrinsically different from V-REx [1] (an extension of IRM [2]). We use a table in the following the comprehensively compare DISAM and VRex to clarify our difference. Generally, V-REx focuses on achieving consistent loss values across domains, and Fish [3] and Fishr [4] emphasize gradient consistency across domains to foster out-of-domain generalization.

In comaprison, DISAM targets to address the challenge of effective perturbation directions for sharpness estimation in domain shift scenarios by introducing the guidance of the domain-level loss variance minimization, which actually does not affect the training objective (i.e., the first term of Eq.(7)). Unlike V-REx, which directly minimizes domain loss variance and can negatively impact convergence, or Fish and Fishr, which constrain gradient updates, DISAM adopts a distinct strategy, minimizing sharpness of multiple domains consistently, to enhance generalization.

Dear Reviewer d1EH,

We sincerely appreciate the effort and time you have devoted to providing constructive reviews, as well as your positive evaluation of our submission. As the deadline for discussion and paper revision is approaching, we would like to offer a brief summary of our responses and updates:

• Detailed explanation of DISAM's novelty and contributions.
• Description and analysis in an open-class setting for DISAM.
• Submission of reproducible code.

Would you mind checking our responses and confirming if you have any additional questions? We welcome any further comments and discussions!

Best Regards,

The authors of Submission 1547

W3 & Q1

W3: The value of $\rho$ in DISAM significantly influences both the convergence speed and generalizability. And it needs more discussion on how to effectively determine the value to maximize the benefits of proposed method. Q1: The article presents a theoretical analysis suggesting that larger values of parameter $\rho$ should lead to improved generalization, given that convergence is guaranteed. It is important to reflect this aspect in the experiments to provide stronger evidence and validation.

Thank you for the suggestion. Following the advice, we have enriched the discussion about the value of $\rho$ in the Appendix B.3 (page 21) of the revised version. Regarding the experiments, we actually have conducted the corresponding experiments to verify this aspect. We kindly refer the reviewer to Appendix B.3 for more details. We summarize the parts for the reviewer's questions as follows.

• Generalization Theorem of SAM: In the SAM framework, the parameter $\rho$ plays a crucial role in determining generalizability. As established in [1], there exists an upper bound on the generalization error for SAM, suggesting that a larger $\rho$ could potentially enhance generalization, provided that convergence is not impeded. Here is the relevant generalization bound from [1]:

For any $\rho > 0$ and any distribution $\mathcal{D}$, with probability $1- \delta$ over the choice of the training set $S\sim \mathcal{D}$, $\mathcal{L} _{\mathcal{D}} (w) \leq \max _{\| \epsilon\| _2 \leq \rho} \mathcal{L} _{S}(w+\epsilon) + \sqrt{\frac{k \log \left( 1 + \frac{\| w \| _2^2}{\rho^2} (1+\sqrt{\frac{\log (n)}{k}})^2 \right) + 4 \log \frac{n}{\delta} + \tilde{O}(1)}{n-1}}$ where $n = |S|$, $k$ is the number of parameters and we assumed $\mathcal{L} _{\mathcal{D}}(w) \leq \mathbb{E} _{\epsilon_i \approx \mathcal{N}(0, \rho)} [\mathcal{L} _{\mathcal{D}}(w+\epsilon)]$. This theorem's proof focuses solely on the magnitude of $\rho$, thus affirming the applicability of this theoretical framework to DISAM.

• Practical Implications: When considering the convergence Theorem 1 on page 5 alongside the above generalization theorem, a critical trade-off emerges with respect to $\rho$. A larger $\rho$ might theoretically enhance generalization but poses greater challenges for convergence. This reflects the intuitive notion that searching for flatter minima across a broader range is inherently more complex, which can potentially affect training efficiency.

• Empirical Validation: DISAM, with its accelerated convergence, can utilize a larger $\rho$ while still maintaining an acceptable convergence. This advantage is empirically showcased in Figures 3(c) and (d) on page 6, where we demonstrate that DISAM effectively employs a larger $\rho$ compared to traditional SAM. This ensures both convergence and enhanced generalization. Such a capability to balance between convergence efficiency and generalization is a distinguishing feature of DISAM over conventional SAM methods.

Reference:

[1]. Invariant risk minimization, arXiv2019.

Q2

Regarding the open class generalization of the clip-based model, further experimental analysis should be conducted to elucidate the reasons behind the superior performance of DISAM.

Thank you for the suggestion. To clarify the reasons, we complemented more analysis in Appendix C.6 on page 25 in the revised version. Generally, according to the results in Table 3 (we select some results in the following table for reference) and Figure 4, we can see:

• ERM tends to overfit to training data classes: As shown in the table below, although CoOp and CLIPOOD perform better on base classes than zero-shot, their performance on new classes is worse than zero-shot. This suggests that the fine-tuned parameters overfit to the existing training data distribution from both the domain and class perspectives. Figure 4 visualizes the change in performance trends during the training process, and we observe a trend where ERM initially performs well on base classes but then exhibits a decline on new classes, suggesting a collapse of the feature space onto the training data classes.

• DISAM improves generalization on new classes: Although SAM offers some relief from overfitting, its performance on new classes does not match zero-shot levels. In contrast, DISAM, by minimizing sharpness more effectively, shows improved performance on new classes, especially in domain shift scenarios.

W1

SAM-based optimization incurs twice the computational overhead and additional storage overhead in comparison to the commonly used SGD. While DISAM, the method proposed in this paper, demonstrates faster convergence under domain shift conditions when compared to SAM, it does not include a comparison with optimizers such as SGD or Adam.

Thank you for highlighting the point of the computational cost. We have to admit that although SAM-based methods help improve generalization, it is ususally at the expense of approximately double the computational effort per iteration compared to standard SGD, due to the sharpness-aware perturbation. DISAM shares the similar cost as other SAM-based methods, since we target to address the domain-level inconsistency during the sharpeness estimation, instead of the training accelaration. In the revision version, we highlight this point with the empirical verification in Figure 8 on page 25, pointing out that compared DISAM with ERM, revealing that although DISAM converges faster than SAM, it does not outpace ERM in terms of convergence speed. The potential extension combined with the acceleration methods like [1,2] can be explored in the future works.

Reference:

[1]. Efficient sharpness-aware minimization for improved training of neural networks, ICLR2022.

[2]. Sharpness-aware training for free, NeurIPS2022.

W2

This paper employs multiple benchmarks to evaluate the performance of multi-source domain generalization. The article highlights the need for advancements in the domain shift perspective of the SAM method and suggests conducting comparisons between DISAM and the state-of-the-art (SOTA) method to further validate the effectiveness of the proposed approach.

We would like to kindly clarify that in Table 1, we has conducted comparison with some SOTA methods [1-2] like the latest SAM-based method SAGM [1] in combination with CORAL [2]. To further address the reviewer's concern, we here add two new SOTA methods "Decompose, Adjust, Compose"(DAC-SC) [3] and Fishr [4] (following reviewer d1EH's recommendation). The table below presents the performance of different methods across multiple domains:

These results clearly illustrate the consistent improvement of DISAM on a range of methods in the field of multi-source domain generalization.

Reference:

[1]. Sharpness-aware gradient matching for domain generalization, CVPR2023.

[2]. Deep coral: Correlation alignment for deep domain adaptation, ECCV2016.

[3]. Decompose, Adjust, Compose: Effective Normalization by Playing with Frequency for Domain Generalization, CVPR2023.

[4]. Fishr: Invariant gradient variances for out-of-distribution generalization, ICML2022.

Dear Reviewer mma9,

We sincerely appreciate the effort and time you have devoted to providing constructive reviews, as well as your positive evaluation of our submission. As the deadline for discussion and paper revision is approaching, we would like to offer a brief summary of our responses and updates:

• Comparison of convergence speed on ERM.
• Expanded comparisons with more state-of-the-art methods.
• Discussion on $\rho$'s impact on convergence and generalization.
• Explanations for the open-class experiments.

Would you mind checking our responses and confirming if you have any additional questions? We welcome any further comments and discussions!

Best Regards,

The authors of Submission 1547

Q1

In the definition of the variance between different domain losses, the values of loss between different domains are restricted. Which one is more import? The value of losses in different domains, or the minimization speed of loss in different domains?

Firstly, we are sorry that one typo in Eq. (7) has mislead your understanding. In the revised manuscript, we have addressed such a typo issue in the description of Eq.(7) as follows.

$$\min _{w} \mathbb{E} _{\xi \in \mathcal{S}} [\mathcal{L} _{DISAM}(w;\xi)] \triangleq \min _{w} \max _{ \| \epsilon \| _2 \leq \rho} \left [ \sum _{i=1} ^M \alpha _i \mathcal{L} _i (w + \epsilon) - \lambda \text{Var}\{\mathcal{L} _i(\hat{w} + \epsilon)\} _{i=1} ^M \right]$$

Here $\hat{w}$ is $w$ without derivative taken during optimization, and it only makes effect in the $\max_{\| \epsilon\|_2 \leq \rho}$ loop without affecting the first term.

• Variance Minimization Focus: DISAM primarily focuses on minimizing variance during the generation of perturbation directions $\epsilon$. The outer optimization w.r.t. $w$ does not involve a trade-off between the empirical loss term and the variance term as we enforce $\hat{w}$ (assigned by $w$) without derivative taken.
• Crucial Role of Minimizing Variance: Minimizing domain-level variance in the perturbation generation loop is critical. Our experiments, illustrated in Figures 5(c) and 5(d) on page 9, show a marked decrease in generalization performance when $\lambda=0$, confirming its essential effectiveness in DISAM. Furthermore, DISAM exhibits a robust performance across a broad range of $\lambda$ values.

Q2

In the learning of multiple domains, there is Multi-Objective Optimization, so the domain-level convergence consistency is a general issue under domain shifts? Or the convergence consistency is a general issue in Multi-Objective Optimization?

Differences from multi-objective optimization:

• The research topic is intrinsically different from multi-objective optimization. Specially, the goal of DISAM has a single ultimate objective, improving the generalization performance of the model trained under multiple domains. This emphasizes both in-domain generalization and out-of-domain generalization, while multi-objective optimization usually refers to improving the multiple objectives of all collaborated tasks.
• Methodologically, we still use the SAM optimization framework, and do not involve multi-objective optimization process during training. Specifically, $\{\alpha_i=n_i/N\}_{i=1}^M$ are constant in Eq.(7) unlike the task-specific variables to be learnt in multi-objective optimization. We observed the negative impact of domain-level convergence inconsistency on SAM-based methods during the perturbation direction generation process. DISAM achieves better perturbation directions for out-of-domain generalization by minimizing the variance of the domain losses.

Weakness & Q3

Weakness: This paper considers the domain-level convergence consistency in SAM for multiple domains, and proposes to adopts the domain loss variance in training loss. The convergence consistency is a general issue, and the solution is normal, thus the novelty is not so clear for publication in ICLR. Q3: This paper considers the domain-level convergence consistency in SAM for multiple domains, and proposes to adopts the domain loss variance in training loss. The convergence consistency is a general issue, and the solution is normal, thus the novelty is not so clear.

Differences from general convergence consistency issue:

• Distinct focus: DISAM focuses on the issue where SAM-based methods are unable to accurately estimate sharpness in domain shift scenarios, leading to the ineffective sharpness minimization and reduction in generalization performance.
• Enhancing on top of general methods: While traditional solutions[1,2,3] aim at convergence consistency in parameter optimization, DISAM's methodology is distinct and orthogonal. It builds upon methods like V-REx[2] and Fishr[3], but goes further in enhancing out-of-domain generalization through better sharpness minimization. This is evident in our experiments, where combining DISAM with Fishr results in significant performance gains (as shown in the table below).

Novelty and contributions:

• We first identify that the straightforward application of SAM has a detrimental impact on training under domain shifts (as shown Figure 1 and Table 1). Specifically, we observed that the way SAM generates perturbation directions amplifies the inconsistency in convergence between domains, leading to inaccurate sharpness estimation and making sharpness minimization less effective.
• DISAM handle the above challenge by imposing a variance minimization constraint on domain loss during the sharpness estimation process, thereby enabling a more representative perturbation location and enhancing generalization.
• The significant improvements in extensive experimental results (as shown in Table 1-3) validate DISAM's novelty and practical relevance.

Reference:

[1]. Invariant risk minimization, arXiv2019.

[2]. Out-of-distribution generalization via risk extrapolation (rex), ICML2021.

[3]. Fishr: Invariant gradient variances for out-of-distribution generalization, ICML2022.

Dear Reviewer sVwJ,

We sincerely appreciate the effort and time you have devoted to providing constructive reviews, as well as your positive evaluation of our submission. As the deadline for discussion and paper revision is approaching, we would like to offer a brief summary of our responses and updates:

• Clarification on the novelty and contributions and expanded comparisons with more state-of-the-art methods.
• Clarification and discussion on the mechanism of the optimization function.
• discussion on the differences between DISAM and multi-opjective optimization.

Would you mind checking our responses and confirming if you have any additional questions? We welcome any further comments and discussions!

Best Regards,

The authors of Submission 1547

Most concerns were resolved by the author's rebuttal. I really appreciate the author's efforts. However, i have further questions about Q3. To validate the efficacy of DISAM over SAM on incremental application of existing domain generalization methods, the author should also conduct the experiments on

• V-REX+SAM vs V-REX+DISAM
• V-REX+SAGM vs V-REX+DISAM

and

• Fishr+SAM vs Fishr+DISAM
• Fishr+SAGM vs Fishr+DISAM

If these experiments (It is okay with simpler evaluation) also shows statistically significant results, i am willing to increase the score from 5 to 6.

Thank you again for the constructive comments. Now, we complement more experiments regarding different combinations to provide a comprehensive comparison. Please refer to the following table for the results. Note that, as DomainNet (600,000 images) are too time consuming for us to finish the training in the remaining time of this phase, we provide the results without DomainNet (we are still running on this dataset and can be provided in the final).

Two conclusions can be made:

• Under the backbone of V-REx, we can find that SAM, DISAM and SAGM improve the performance of the vanilla V-REx by 0.3, 1.2, 0.9 and DISAM performs the best. Besides, DISAM and SAGM both outperform the vanilla SAM from different designing perspectives in optimization, wherein DISAM's perspective seems to be more effective. Their combinations (i.e., V-REx+SAGM+DISAM) achieves the best performance compared to the either, which demonstrates their orthogonality and composability.
• Under the backbone of Fishr, we can find that SAM even brings the negative impact on the vanilla Fishr (70.5 v.s. 71.7), and SAGM cannot remedy the negative impact induced by SAM, yielding the overall lower performance than the vanilla Fishr (71.4 v.s. 71.7). DISAM (73.1) significantly outperforms SAM and SAGM, and exhibits the similar orthogonality and composability with SAGM.

Although DISAM shows the consistent superiority over SAM and SAGM under the backbones of V-REx and Fishr, we would like to specially summarize their differences as follows for clarity and for the proper claim.

• The vanilla SAM suffer from impairment under multiple domain discrepancy, while DISAM and SAGM both can alleviate this issue, as to some extent their improvements in design inherently consider the potential bias under the two-stage SAM optimization procedure.
• Differently, SAGM makes the change in the second stage of SAM, which considers the direction difference between the perturbed gradient by SAM and the original gradient by SGD during gradient updates, by narrowing the angle between these two gradient directions. However, DISAM focuses on the first stage of SAM, namely, the perturbation direction generation process, enhancing domain-level convergence consistency to achieve better sharpness estimation and, and consequently, improving generalization. DISAM is more direct to intervene the domain shift problem, while SAGM actually makes the implicit effect.

Overall, we do not intend to critize SAGM but to point out the difference under the scenario of our study. We hope the experiments and analysis can address the reviewer's remaining concerns. Any more comments and advices are welcomed.

Best,

The authors of submission 1547.