Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning

Response to Comment 1 (Lack of Theoretical Analysis or Empirical Evidence):

We sincerely thank the reviewer for emphasizing the importance of both empirical and theoretical validation of our method. To address the empirical aspect, we would like to highlight that the paper provides a broad set of experiments designed to demonstrate the effectiveness of M‑SAM in mitigating modality imbalance.

For example, Figure 1 presents a qualitative comparison of the loss landscapes of M‑SAM, traditional SAM, and a strong multimodal baseline (AGM) on CREMA‑D. This figure illustrates that M‑SAM consistently converges to flatter minima, a property that is widely associated with better generalization. Complementing this, Figure 2 reports results based on the generalization‑gap metric on AV‑MNIST, where M‑SAM demonstrates a clear and consistent advantage over all benchmarks. Beyond these visualizations, Tables 1 and 2 provide an extensive quantitative comparison across four diverse and challenging datasets (AV‑MNIST, CREMA‑D, UR‑Funny, and AVE) under both early‑fusion and late‑fusion architectures for single-modal and multi-modal accuracy.

These results, together with the training‑curve analyses presented in the Appendix, show that the benefits of M‑SAM hold across datasets and training regimes. Collectively, these results offer a comprehensive empirical evaluation spanning multiple settings and confirm the robustness of the proposed approach.

In addition to empirical validation, the paper offers a rigorous theoretical treatment. Specifically, we derive a closed‑form mathematical representation for the M‑SAM optimization target (Eq. 12), providing a principled foundation for the method.

Furthermore, our detailed analyses presented in the responses to Comments 1 and 2 of Reviewers oFnA and sAY6. Those responses address concerns about convergence proof, computation analysis of MSAM, and proof of the fact that the proposed update rule remains stable and effective even in the presence of oscillating dominant modality selection.

We would like to mention that these analyses complement the empirical findings and reinforce that the contributions of M‑SAM are both theoretically sound and practically impactful.

Response to Comment 2 (Validation-Training Accuracy Consistency and Generalization):

We would like to thank you for your thoughtful comment. Previous studies, including \cite{Ishida}, the paper that you mentioned in your comment, have firmly established that the gap between training and testing performance is an important indicator of the geometry of the solution found during training. For example, \cite{Keskar} demonstrates that when a training process results in a significant gap between training and testing accuracies, it signals that the network has converged to a sharp minimum, characterized by a significant number of large positive eigenvalues in $\\nabla^2 f$. Such a geometry indicates a higher sensitivity to perturbations and typically results in lower inference accuracy on unseen data.

Moreover, the relation between sharp minima and reduced generalization ability is a well-established topic that has been extensively investigated, from the pioneering work of Hochreiter and Schmidhuber \cite{Hochreiter_Schmidhuber} to more recent studies on robust optimization and Sharpness-Aware Minimization \cite{SAM_paper}.

We would like to highlight that in line 58, it clearly says "... huge gap shows it tends to overfit.." which is a highly compatible to what the paper you mentioned, \cite{Ishida}, elaborate on its Fig.1, when it discussed the generalization gap in phase A and phase B.

[Ishida]: "Do we need zero training loss after achieving zero training error?" Ishida, et al. (ICML2020)

[Keskar]: "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. " Keskar, et al. (ICLR 2017).

[Hochreiter_Schmidhuber]: "Flat minima.", Hochreiter, Sepp, and Jürgen Schmidhuber. Neural computation (1997).

[SAM_paper]: "Sharpness-aware minimization for efficiently improving generalization." , Foret, et al. (ICLR 2021).

Response to Comment 2 (Missing standard deviations or error bars in the results):

We thank the reviewer for highlighting the importance of reporting variability. We would like to mention that the values reported in the tables for M‑SAM correspond to the mean accuracy over five runs with seeds 1, 2, 3, 4, and 5. Also, we would like to clarify that our decision to report results based on just mean accuracy (without SD) was made to align with the result presentation approach established in widely adopted baseline methods such as Reacon-Boost (Classifier-guided Gradient Modulation for Enhanced Multimodal Learning- ICML2024), CGGM (Classifier-guided Gradient Modulation for Enhanced Multimodal Learning- NIPS2024), MM-Pareto (MMPareto: Innocent Uni-modal Assistance for Enhanced Multi-modal Learning -ICLR 2024), AGM (Boosting Multi-modal Model Performance with Adaptive Gradient Modulation-CVPR2023).

To address your concern, in the following tables we have included SD in our results (in Table.1, and Table.2) to more comprehensively demonstrate the reliability of our method.

standard deviations for Table.1 (late fusion):

standard deviations for Table.2 (early fusion):

Response to Comment 4 (Originality and Similarity to ImbSAM):

We would like to thank the reviewer for this thoughtful comment. As the authors transparently acknowledged in Sec. 3.5, the idea of this paper was partly inspired by (Zhou et al., 2023), i.e., the Imb-SAM framework. This, by no means, signifies that we forcibly applied Imb-SAM to our approach. The high-level differences between our approach and Imb-SAM are two-fold: (i) loss computation and decomposition (explained in Sec. 3.2), (ii) considering the dynamics of the learning process, i.e., where and how to apply the SAM-based approach (explained in Sec. 3.3). Using the intuition from Sec 3.2 and Sec. 3.3, we arrive at Eq. (12), whose final mathematical form happens to resemble that of Imb‑SAM, but the derivation, underlying objectives, and dynamics that lead to it are fundamentally different, as explained above.

Response to Comment 1 (computational efficiency):

We thank the reviewer for raising this point. Unlike Imb‑SAM and CCSAM (which address class imbalance), M‑SAM targets modality imbalance. Its complexity depends only on the number of input modalities $k$, not on the number of classes.

In each iteration, M‑SAM computes Shapley weights using all non‑empty subsets of modalities, excluding only the trivial all‑zero subset. This introduces $2^k-2$ additional forward passes over SAM, with no additional backward passes or memory overhead, since these extra passes are used solely for weight estimation and do not store activations and gradients.

In the paper we evaluated M-SAM using $4$ famous and widely used datasets in the field:

$k=2$ (e.g., AV‑MNIST, CREMA‑D): extra forward passes = $2^2-2=2$

$k=3$ (e.g., UR‑Funny): extra forward passes = $2^3-2=6$

M‑SAM introduces only a modest additional cost: for $k=2$ modalities, the forward‑to‑backward ratio changes from $(2+2)/(2+2)=1$ in SAM to $(4+2)/(2+2)=1.5$, and for $k=3$ modalities it becomes $(8+2)/(2+2)=2.5$, while the number of backward passes and memory requirements remain exactly the same as SAM.

I would like to emphasize that, this overhead is entirely independent of the number of dataset classes, and in practice the number of modalities rarely grows beyond $2$ or $3$, so the additional cost remains small.

Response to Comment 2 (Convergence and Theoretical Guarantees):

We thank the reviewer for this valuable comment highlighting the need for a deeper theoretical analysis.
To address this concern, we provide a convergence proof for the proposed M‑SAM method.
Under standard smoothness and bounded gradient assumptions, we show that the average gradient norm decreases at a rate of $\mathcal{O}\big(\tfrac{\log T}{\sqrt{T}}\big)$, which matches the classical convergence behavior of SGD and SAM.
This result formally guarantees the stability and convergence of M‑SAM even under varying modality dominance patterns.

Under the assumptions of:

1. $\\mathcal{L}(\\theta_t) = \\sum_{m=1}^{M} \\nu_m \\mathcal{L}_m(\\theta_t)$ and its gradient is bounded $K$-smooth ($K$-Lipschitz), i.e., $\\| \\nabla \\mathcal{L}(\\theta_t) - \\nabla\\mathcal{L} (\\hat \\theta_t) \\| \\leq K \\| \\theta_t - \\hat \\theta_t\\|$,

$\\forall (\\theta_t, \\hat \\theta_{t})$, that $\\|\\nabla \\mathcal{L}(\\theta_t)\\| \\leq G_{\\text{max}}.$

2. Learning rate $\\eta_t = \\frac{\\eta_0}{\\sqrt{t}}$ and perturbation $\\rho_t = \\frac{\\rho_0}{\\sqrt{t}}.$

Then when $T$ is large enough the M-SAM updates satisfy:

$

\frac{1}{T} \sum_{t=1}^{T} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \mathcal{O}\left(\frac{\log T}{\sqrt{T}}\right).
$

Before going through the proof, we would like to mention that, for tractability in the theoretical analysis, we assumed a smooth decay schedule $\eta_t = \eta_0/\sqrt{t}$. In practice, our implementation uses a step-wise decay (multiplying the learning rate by $0.1$ every $70$ iterations), which decreases $\\eta_t$ even faster. This practical schedule still satisfies the conditions required for convergence and does not weaken the theoretical guarantee.

considering $d_t = \\theta_{t+1} - \\theta_t = -\\eta_t \\nabla \\mathcal{L} (\\hat{\\theta_t})$ and \\hat{\\theta_t} = \\theta_t + \\rho_t \\frac{\\nabla_{m^\\ast} \\mathcal{L}(\\theta_t)}{\\|\\nabla_{m^\\ast} \\mathcal{L}(\\theta_t)\\|}; where \\nabla_{m^\\ast} denotes the gradient with respect to the dominant modality that generated by backpropagation of loss value corresponding to the dominant modality (Eq. 4 in the paper).

Using descent Lemma for a $k$-smooth function we can write:
$

 \\begin{aligned}

\mathcal{L}(\theta_{t+1}) - \mathcal{L}(\theta_t) \leq & -\eta_t \langle \nabla \mathcal{L}(\theta_t), \theta_{t+1} - \theta_t \rangle + \frac{K\eta_t^2}{2} \|\theta_{t+1} - \theta_t\|^2 = \\ & -\eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\hat \theta_t) \rangle + \frac{K\eta_t^2}{2} \|\nabla \mathcal{L} (\hat \theta_{t})\|^2 = \\ &-\eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\theta_{t}) -\nabla \mathcal{L}(\theta_t) + \nabla \mathcal{L} (\hat \theta_t) \rangle + \frac{K \eta_t^2}{2} \|\nabla \mathcal{L} (\hat \theta_t)\|^2 = \\ & - \eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 - \eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\hat \theta_{t}) - \nabla \mathcal{L}(\theta_t) \rangle + \frac{K \eta_t^2}{2} (\|\nabla \mathcal{L} (\hat\theta_{t})\|^2) \leq \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + \eta_t \| \nabla \mathcal{L}(\theta_t)\| \|\nabla \mathcal{L} (\theta_{t}) - \nabla \mathcal{L}(\hat\theta_t) \| + \frac{K \eta_t^2}{2} (\|\nabla \mathcal{L} (\hat\theta_{t})\|^2) \leq \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + K\eta_t \| \nabla \mathcal{L}(\theta_t)\| \|\hat\theta_{t} - \theta_t \| + \frac{K \eta_t^2}{2} (\|\nabla \mathcal{L} (\hat\theta_{t})\|^2) = \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + K\eta_t \| \nabla \mathcal{L}(\theta_t)\| \| \rho_t \frac{\nabla_{m^\ast} \mathcal{L}(\theta_t)}{\|\nabla_{m^\ast} \mathcal{L}(\theta_t)\|}\|^2) + \frac{K \eta_t^2}{2} (\|\nabla \mathcal{L} (\hat\theta_{t})\|^2) = \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + K\eta_t \rho_t^2 \| \nabla \mathcal{L}(\theta_t)\| + \frac{K \eta_t^2}{2} \|\nabla \mathcal{L} (\hat\theta_{t})\|^2 \leq \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + K\eta_t \rho_t^2 G_{max} + \frac{K \eta_t^2}{2} G_{max}^2 \end{aligned}
$

by rearranging the inequality, it will be as:

$

\\begin{aligned}
    &\\eta_t \\|\\nabla \\mathcal{L}(\\theta_t)\\|^2 \\leq \\mathcal{L}(\\theta_{t}) - \\mathcal{L}(\\theta_{t+1})+K\\eta_t \\rho_t^2 G_{max}  + \\frac{K \\eta_t^2}{2}  G_{max}^2
\\end{aligned}

$

considering definition of $\\eta_{t} = \\frac{\\eta_{0}}{\\sqrt{t}}$ and taking summation over all iterations on the left side of the above inequality, we can define a lower bound as follows:

$

\frac{\eta_{0}}{\sqrt{T}}\sum_{t=1}^{T} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \sum_{t=1}^{T} \eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \sum_{t=1}^{T} \big(\mathcal{L}(\theta_{t}) - \mathcal{L}(\theta_{t+1}) \big) + K \eta_0 \rho_0^2 G_{max} \sum_{t=1}^{T} \frac{1}{t\sqrt{t}} + \frac{k \eta_0^2 G_{max}^2}{2} \sum_{t=1}^{T} \frac{1}{t}

$

using telescope sum properties and considering $0 \\leq \\mathcal{L}(\\theta_t) \\forall t :$

$

    \sum_{t=1}^{T} \big(\mathcal{L}(\theta_{t}) - \mathcal{L}(\theta_{t+1}) \big) = \mathcal{L}(\\theta_1) - \mathcal{L}(\\theta_{T+1})
    \leq \mathcal{L}(\\theta_1)

$

So we can rewrite the main inequality as follows:

$

\frac{1}{T}\sum_{t=1}^{T} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \frac{\mathcal{L}(\theta_1)}{\sqrt{T}} + \frac{K \rho_0^2 G_{max}}{\sqrt{T}} \sum_{t=1}^{T} \frac{1}{t\sqrt{t}} + \frac{K \eta_0 G_{max}^2}{2\sqrt{T}} \sum_{t=1}^{T} \frac{1}{t}

$

Considering $\sum_{t=1}^{T} \frac{1}{t} \leq 1+\log T$ and the fact that $\sum_{t=1}^{T} \frac{1}{t^{3/2}}$ is a $p$‑series with $p=\tfrac{3}{2}(>1)$, its partial sum remains bounded, so the only term on the right-hand side that confine the asymptotic convergence rate is the harmonic term, which results in an overall rate of $\mathcal{O}\big(\tfrac{\log T}{\sqrt{T}}\big)$.

$

\frac{1}{T} \sum_{t=1}^{T} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \mathcal{O}\left(\frac{\log T}{\sqrt{T}}\right).
$

Since the average gradient norm $\frac{1}{T}\sum_{t=1}^{T}\\|\nabla \mathcal{L}(\theta_t)\\|^2$ decreases at a rate of $\tfrac{\log T}{\sqrt{T}}$ and tends to $0$ as $T$ grows, the updates approach a stationary point, which by definition establishes convergence (as in standard SGD and traditional SAM).

This bound, $\mathcal{O}\big(\tfrac{\log T}{\sqrt{T}}\big)$, matches the standard non‑convex convergence rate of SGD with a decaying learning rate schedule (where the logarithmic factor comes from $\sum 1/t$), so the proposed M‑SAM method has the same order of convergence as classical SGD and traditional SAM.

Response to Comment 1 (Theoretical Convergence Guarantees under the M-SAM optimization dynamics):

We would like to sincerely thank the reviewer for raising the concern about the convergence behavior of M‑SAM under its dynamic updates.

Through the following proof we will show that the convergence wont be affected by the choice of the dominant modality gradient at each step. Even if there is approximation error in Shapley-based modality selection, the updates remain stable and converge.

Under the assumptions of:

1. $\\mathcal{L}(\\theta_t) = \\sum_{m=1}^{M} \\nu_m \\mathcal{L}_m(\\theta_t)$ and its gradient is bounded $K$-smooth ($K$-Lipschitz), i.e., $\\| \\nabla \\mathcal{L}(\\theta_t) - \\nabla\\mathcal{L} (\\hat \\theta_t) \\| \\leq K \\| \\theta_t - \\hat \\theta_t\\|$,

$\\forall (\\theta_t, \\hat \\theta_{t})$, that gradient is bounded: $\\|\\nabla \\mathcal{L}(\\theta_t)\\| \\leq G_{\\text{max}}.$

2. Learning rate $\\eta_t = \\frac{\\eta_0}{\\sqrt{t}}$ and perturbation $\\rho_t = \\frac{\\rho_0}{\\sqrt{t}}.$

Then when $T$ is large enough the M-SAM updates satisfy:

$

\frac{1}{T} \sum_{t=1}^{T} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \mathcal{O}\left(\frac{\log T}{\sqrt{T}}\right).
$

(Before going through the proof, we would like to mention that, for tractability in the theoretical analysis, we assumed a smooth decay schedule $\\eta_t = \\eta_0/ \\sqrt{t}$. In practice, our implementation uses a step-wise decay (multiplying the learning rate by $0.1$ every $70$ iterations), which decreases $\\eta_t$ even faster. This practical schedule still satisfies the conditions required for convergence and does not weaken the theoretical guarantee.)

Considering $d_t = \\theta_{t+1} - \\theta_t = -\\eta_t \\nabla \\mathcal{L} (\\hat{\\theta_t})$ and \\hat{\\theta_t} = \\theta_t + \\rho_t \\frac{\\nabla_{m^\\ast} \\mathcal{L}(\\theta_t)}{\\|\\nabla_{m^\\ast} \\mathcal{L}(\\theta_t)\\|}; where \\nabla_{m^\\ast} denotes the gradient with respect to the dominant modality, generated by backpropagation of loss value corresponding to the dominant modality, m^\\ast (Eq. 4 in the paper).

Using descent Lemma for a $k$-smooth function we can write:
$

 \\begin{aligned}

\mathcal{L}(\theta_{t+1}) - \mathcal{L}(\theta_t) \leq & -\eta_t \langle \nabla \mathcal{L}(\theta_t), \theta_{t+1} - \theta_t \rangle + \frac{K\eta_t^2}{2} \|\theta_{t+1} - \theta_t\|^2 = \\ & -\eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\hat \theta_t) \rangle + \frac{K\eta_t^2}{2} \|\nabla \mathcal{L} (\hat \theta_{t})\|^2 = \\ &-\eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\theta_{t}) -\nabla \mathcal{L}(\theta_t) + \nabla \mathcal{L} (\hat \theta_t) \rangle + \frac{K \eta_t^2}{2} \|\nabla \mathcal{L} (\hat \theta_t)\|^2 = \\ & - \eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 - \eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\hat \theta_{t}) - \nabla \mathcal{L}(\theta_t) \rangle + \frac{K \eta_t^2}{2} (\|\nabla \mathcal{L} (\hat\theta_{t})\|^2) \leq \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + \eta_t \| \nabla \mathcal{L}(\theta_t)\| \|\nabla \mathcal{L} (\theta_{t}) - \nabla \mathcal{L}(\hat\theta_t) \| + \frac{K \eta_t^2}{2} (\|\nabla \mathcal{L} (\hat\theta_{t})\|^2) \leq \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + K\eta_t \| \nabla \mathcal{L}(\theta_t)\| \|\hat\theta_{t} - \theta_t \| + \frac{K \eta_t^2}{2} (\|\nabla \mathcal{L} (\hat\theta_{t})\|^2) = \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + K\eta_t \| \nabla \mathcal{L}(\theta_t)\| \| \rho_t \frac{\nabla_{m^\ast} \mathcal{L}(\theta_t)}{\|\nabla_{m^\ast} \mathcal{L}(\theta_t)\|}\|^2) + \frac{K \eta_t^2}{2} (\|\nabla \mathcal{L} (\hat\theta_{t})\|^2) = \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + K\eta_t \rho_t^2 \| \nabla \mathcal{L}(\theta_t)\| + \frac{K \eta_t^2}{2} \|\nabla \mathcal{L} (\hat\theta_{t})\|^2 \leq \\ &-\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + K\eta_t \rho_t^2 G_{max} + \frac{K \eta_t^2}{2} G_{max}^2 \end{aligned}
$

Here we would like to highlight that the step of finding the upper bound on the right-hand side of the inequality is the only place where the dynamic nature of modality selection could potentially affect convergence. However, it is clearly shown that the convergence is independent of \\nabla_{m^\\ast} \\mathcal{L}(\\theta_t). This is because the upper bound, \\| \\rho_t \\frac{\\nabla_{m^\\ast} \\mathcal{L}(\\theta_t)}{\\|\\nabla_{m^\\ast} \\mathcal{L}(\\theta_t)\\|}\\|^2, depends only on $\\rho_t$_, and not on the direction of the selected modality’s gradient. Hence, the convergence is independent of modality selection at each iteration.

By rearranging the inequality, it will be as:

$

\\begin{aligned}
    &\\eta_t \\|\\nabla \\mathcal{L}(\\theta_t)\\|^2 \\leq \\mathcal{L}(\\theta_{t}) - \\mathcal{L}(\\theta_{t+1})+K\\eta_t \\rho_t^2 G_{max}  + \\frac{K \\eta_t^2}{2}  G_{max}^2
\\end{aligned}

$

considering definition of $\\eta_{t} = \\frac{\\eta_{0}}{\\sqrt{t}}$ and taking summation over all iterations on the left side of the above inequality, we can define a lower bound as follows:

$

\frac{\eta_{0}}{\sqrt{T}}\sum_{t=1}^{T} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \sum_{t=1}^{T} \eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \sum_{t=1}^{T} \big(\mathcal{L}(\theta_{t}) - \mathcal{L}(\theta_{t+1}) \big) + K \eta_0 \rho_0^2 G_{max} \sum_{t=1}^{T} \frac{1}{t\sqrt{t}} + \frac{k \eta_0^2 G_{max}^2}{2} \sum_{t=1}^{T} \frac{1}{t}

$

using telescope sum properties and considering $0 \\leq \\mathcal{L}(\\theta_t) \\forall t :$

$

    \sum_{t=1}^{T} \big(\mathcal{L}(\theta_{t}) - \mathcal{L}(\theta_{t+1}) \big) = \mathcal{L}(\\theta_1) - \mathcal{L}(\\theta_{T+1})
    \leq \mathcal{L}(\\theta_1)

$

So we can rewrite the main inequality as follows:

$

\frac{1}{T}\sum_{t=1}^{T} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \frac{\mathcal{L}(\theta_1)}{\sqrt{T}} + \frac{K \rho_0^2 G_{max}}{\sqrt{T}} \sum_{t=1}^{T} \frac{1}{t\sqrt{t}} + \frac{K \eta_0 G_{max}^2}{2\sqrt{T}} \sum_{t=1}^{T} \frac{1}{t}

$

Considering $\sum_{t=1}^{T} \frac{1}{t} \leq 1+\log T$ and the fact that $\sum_{t=1}^{T} \frac{1}{t^{3/2}}$ is a $p$‑series with $p=\tfrac{3}{2}(>1)$, its partial sum remains bounded, so the only term on the right-hand side that confine the asymptotic convergence rate is the harmonic term, which results in an overall rate of $\mathcal{O}\big(\tfrac{\log T}{\sqrt{T}}\big)$.

$

\frac{1}{T} \sum_{t=1}^{T} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \mathcal{O}\left(\frac{\log T}{\sqrt{T}}\right).
$

Since the average gradient norm $\frac{1}{T}\sum_{t=1}^{T}\\|\nabla \mathcal{L}(\theta_t)\\|^2$ decreases at a rate of $\tfrac{\log T}{\sqrt{T}}$ and tends to $0$ as $T$ grows, the updates approach a stationary point, which by definition establishes convergence (as in standard SGD and traditional SAM).

This bound, $\mathcal{O}\big(\tfrac{\log T}{\sqrt{T}}\big)$, matches the standard non‑convex convergence rate of SGD with a decaying learning rate schedule (where the logarithmic factor comes from $\sum 1/t$). Thus, the proposed M-SAM method preserves the same theoretical convergence guarantees as classical SGD and traditional SAM while offering dynamic modality-based perturbations.

Response to Comment 2 (Implications of Dominant Modality Shifts across Training Phases and Mini‑Batches):

We thank the reviewer for raising this thoughtful point about shifts in the dominant modality during training and their effect on the learning dynamics of M‑SAM.

M‑SAM is designed to benefit from these shifts rather than suppress them. By re‑evaluating the dominant modality at every iteration using Shapley‑based contributions, the method dynamically allocates attention to the modality that is most informative at that stage of training.

As discussed in our response to Comment 1, such alternations do not compromise stability or convergence. Instead, they guide the currently dominant modality toward flatter minima, giving other modalities freedom to contribute. When dominance oscillates between modalities, this competition prevents any single modality from monopolizing learning, leading to a more balanced and robust multimodal representation.

Response to Comment 3 (Writing Issues):

We appreciate the reviewer’s detailed feedback on clarity and presentation. We acknowledge these issues and will address them in the camera‑ready version by correcting table references, subsection numbering, figure order, and improving the formatting of Algorithm 1 for better readability.