Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification

Q1: Please include standard errors over multiple runs for numerical results tables. The performance of Focal-SAM is often quite close to baselines, so establishing this statistical significance will be important.

A1: Thanks for your valuable suggestion!

In the original paper, we report the results using the same fixed random seed for all methods to ensure a fair and consistent comparison. Following your advice, we have now conducted additional experiments using three different random seeds to further assess the effectiveness of our method. Specifically, we report the average accuracy along with the standard error across these runs. All other experimental settings remain the same as those described in the paper. These additional experiments are conducted on the CIFAR-100 LT datasets.

Furthermore, we perform a statistical significance analysis using the Mann-Whitney U test to compare the performance of Focal-SAM against baseline methods:

• Baselines marked with $^{**}$ indicate that Focal-SAM outperforms the corresponding baseline with a p-value $\le 0.05$.
• Baselines marked with $^*$ indicate that Focal-SAM outperforms the corresponding baseline with a p-value $\le 0.10$.

The results shows that the average performance of Focal-SAM consistently surpasses other SAM-based methods across multiple runs. This further supports the effectiveness of the proposed Focal-SAM approach.

---

CIFAR-100 LT

Q1: Why don't use $\tilde{L}_S^{FS} (w)$ (eq 5) as the main objective? Specifically, why manage $\tilde{L}_S^{FS} (w)$ as a penalty of standard loss as in eq 6, which will add an additional hyperparameter $\lambda$?

Thanks for your questions!

The motivation behind SAM-based methods is that the geometry of the loss landscape is closely related to a model's generalization ability. Specifically, flatter minima in the loss landscape generally leads to better generalization compared to sharper ones. Therefore, SAM-based methods aim to simultaneously minimize both the loss value and the sharpness to seek out flatter minima.

In our Focal-SAM method, $\tilde{L}_S^{FS}(w)$ is a term which reflects the sharpness of loss landscape at $w$. If we only minimize $\tilde{L}_S^{FS}(w)$ , we can generally obtain a model parameter $w$ with a flat loss landscape. However, this does not necessarily minimize the actual loss value $L_S(w)$, which may remain large and degrade model performance.

Therefore, we incorporate the sharpness term $\tilde{L}_S^{FS}(w)$ as a penalty term to the standard loss value $L_S(w)$, weighted by a hyperparameter $\lambda$. This enables simultaneously minimization of both sharpness and the loss value, where $\lambda$ controls the importance of the sharpness term.

• If $\lambda$ is very large, the objective function primarily minimizes $\tilde{L}_S^{FS}(w)$, neglecting $L_S(w)$.
• If $\lambda=0$, the method reduces to minimizing only the standard loss $L_S(w)$.

In Fig.5 of our paper, we conduct an ablation study on $\lambda$ to explore these scenarios. The result show that both excessively small and large values of $\lambda$ lead to suboptimal performance, while a moderate $\gamma \approx 0.8$ achieves the best results. This highlights the importance of balancing loss minimization and sharpness control.

Q2: Why perturbation is computed only on $\tilde{L}_S^{FS} (w)$ rather than $L_S^{FS}(w)$ in eq 7?

Thanks for your questions!

To clarify, let's first recall the definition of the objective function $L_S^{FS} (w)$:

$$L_S^{FS} (w) = L_S(w) + \lambda \cdot \tilde{L}_S^{FS} (w)$$

where

$$\tilde{L}_S^{FS} (w) = \max _{|| \epsilon ||_2 \le \rho} \sum _{i=1}^C (1 - \pi_i)^\gamma \tilde{L}_S^i(w, \epsilon)$$

As shown, the perturbation $\epsilon$ appears only in the second term $\tilde{L}_S^{FS}(w)$ and is independent of the first term $L_S(w)$.

Therefore, when optimizing $L_S^{FS}(w)$, the perturbation is computed exclusively over the inner maximization problem in $\hat{\epsilon}(w)$. Specifically, we solve for the optimal perturbation $\hat{\epsilon}(w)$ as:

$$\hat{\epsilon}(w) = \arg\max _{|| \epsilon ||_2 \le \rho} \sum _{i=1}^C (1 - \pi_i)^\gamma \tilde{L}_S^i(w, \epsilon)$$

Q3: What is the computational cost of baselines such as LA, FFT, and LIFT? If their cost is approximately half that of SAM variants, a fairer comparison would be to evaluate them with twice the number of training steps.

A3: Thank you for your valuable suggestion!

Indeed, the computational cost of baselines such as LA, FFT and LIFT is approximately half that of SAM-based variants. To provide a fairer comparison, we conduct additional experiments in which we extend the training epochs of these baseline methods to roughly match the total computational cost of the SAM-based methods. Specifically, we double the training epochs to 400 or 40 (2$\times$ compared to the original 200 or 20) for these baselines, while keeping the SAM-based methods at 200 or 20 epochs. All other experimental settings remain consistent with those reported in the paper. These experiments are conducted on both CIFAR-100 LT and ImageNet-LT datasets. The results are summarized below.

The results clearly show that, under comparable computational budgets, Focal-SAM consistently outperforms the baseline methods across different datasets. This further demonstrates that Focal-SAM effectively enhances model generalization in long-tailed learning scenarios.

Below are the results:

---

CIFAR-100 LT

---

ImageNet-LT

Q1: (1) From Table. 2, I observe a significant performance enhancement with the original SAM, but the improvements from SAM to Focal-SAM are not obvious. (2) I notice that the running time of Focal-SAM is 50% more than the original SAM. This dilates the necessity of employing Focal SAM.

A1: Thanks for your question!

(1) Performance Gains: It is true that the improvement from SAM to Focal-SAM is generally smaller than that from the baseline to SAM. This is expected, as SAM itself is already an effective method, and Focal-SAM is proposed as a refinement of SAM. Therefore, the additional improvement is naturally smaller but still consistent and meaningful. We also conduct multiple runs and statistical significance analysis to further confirm this. Please see A1 for Reviewer bvws for details.

(2) Computational Cost: While Focal-SAM increases training time by around 50% compared to SAM, this does not mean it is unnecessary. To fairly assess its benefit, we conduct experiments where we extend the training epochs of SAM to match Focal-SAM's total computational cost. Specifically, we increase the training epochs to 300 or 30 (1.5$\times$ the original 200 or 20) for SAM, while keeping Focal-SAM at 200 or 20 epochs. In this setting, the total computational cost of SAM and Focal-SAM becomes comparable. All other experimental settings remain the same as in the main paper. We conduct these experiments on CIFAR-100 LT, ImageNet-LT, and iNaturalist datasets. The results are summarized below, from which we can observe consistent improvement.

---

CIFAR-100 LT

---

ImageNet-LT

---

iNaturalist

Q2: I think that a quite amount of theoretical contributions are provided in the appendix. This makes the conclusion of your theorem a little abrupt. I suggest the authors move several lemmas to the main paper.

A2: Thanks for your valuable suggestion!

Due to space constraints in the anonymous submission version, we included the proof sketch and the lemmas supporting Theorem 3.1 in the appendix. We acknowledge that this may make the conclusion of Theorem 3.1 appear abrupt. In the future version, we will revise the manuscript and relocate the proof sketch along with several key lemmas, such as Lemma B.2 and Lemma B.4, into the main text.

Q3: Meanings that Figure 1.b and Table. 1 try to convey seems repeated.

A3: Thanks for your question!

We acknowledge that there is some overlap between Figure 1(b) and Table 1 regarding the part of computational cost. However, the key messages we intend to convey through them are different. In Figure 1(b), our primary goal is to directly compare Focal-SAM and CC-SAM to demonstrate the effectiveness of Focal-SAM, showing that it achieves better performance with higher efficiency. In contrast, Table 1 is used to highlight the limitations of CC-SAM and to motivate the development of our method. Specifically, we compare CC-SAM other SAM-based methods—including SAM, ImbSAM, and Focal-SAM—and show that CC-SAM incurs higher computational costs than the others. This observation motivates our proposal of Focal-SAM as an efficient alternative.

Q1: On computational overhead, how different are the perturbations between the two methods?

A1: Thanks for your question!

The perturbation in Focal-SAM is computed as:

$$\hat{\epsilon}(w) = \rho \frac{\nabla_w L_S^\gamma(w)}{|| \nabla_w L_S^\gamma(w) ||_2}$$

where $L_S^\gamma(w) = \sum_{i=1}^C (1 - \pi_i)^\gamma L_S^i(w)$.

In contrast, CC-SAM computes a class-specific perturbation for each class $i$ as:

$$\hat{\epsilon}_i(w) = \rho_i^* \frac{\nabla_w L_S^i(w)}{|| \nabla_w L_S^i(w) ||_2}$$

In terms of computational overhead:

Each perturbation requires one backpropagation (BP) to compute the corresponding gradient:

• Focal-SAM requires only one BP to compute perturbation to compute $\nabla_w L_S^\gamma(w)$.
• CC-SAM requires total $C$ BPs, one for each class, to compute $\nabla_w L_S^i(w)$.

The difference between the two perturbation are twofold:

• CC-SAM employs class-specific radii $\rho_i^*$ to finely control the sharpness penalty for each class. In contrast, Focal-SAM uses a unified perturbation radius $\rho$.
• CC-SAM computes the gradient of class-wise loss $L_S^i(w)$ individually, while Focal-SAM computes the gradient of the weighted loss $L_S^\gamma(w)$.

If we disregard the difference in scaling across perturbations, i.e., assume:

$$\frac{\rho}{|| \nabla_w L_S^\gamma(w) ||_2} = \frac{\rho_i^*}{|| \nabla_w L_S^i(w) ||_2}, \forall i$$

then the Focal-SAM perturbation can be rewritten as:

$$\hat{\epsilon}(w) = \sum_{i=1}^C (1 - \pi_i)^\gamma \hat{\epsilon}_i(w)$$

This suggests that the perturbation in Focal-SAM can be viewed as a weighted aggregation of the class-wise perturbations in CC-SAM. Therefore, Focal-SAM implicitly considers the contribution of all classes and adjusts the emphasis across classes by tuning $\gamma$.

Q2: The second term in Eq.4 does not seem to be in ImbSAM or CC-SAM.

A2: Thank you for your questions!

In fact, we can reformulate the objective functions of ImbSAM and CC-SAM to make the second term, $L(w)$, explicitly appear. Furthermore, we can rewrite the objectives using the class-wise sharpness term (Eq.4).

For ImbSAM:

$$L_S^{IS}(w) = L_S^H(w) + L_S^T(w) + \max _{|| \epsilon ||_2 \le \rho} [L_S^T(w+\epsilon) - L_S^T(w)] = L_S(w) + \max _{|| \epsilon ||_2 \le \rho} \sum _{i \in T} \tilde{L}_S^i(w, \epsilon) \quad (1)$$

For CC-SAM:

$$L_S^{CS}(w) = \sum_{i=1}^C \frac{1}{\pi_i} \cdot L_S^i(w) + \sum_{i=1}^C \max_{|| \epsilon || \le \rho_i^*} \frac{1}{\pi_i} \cdot [L_S^i(w + \epsilon) - L_S^i(w)] = \sum_{i=1}^C \frac{1}{\pi_i} \cdot L_S^i(w) + \sum_{i=1}^C \max_{|| \epsilon ||_2 \le \rho_i^*} \frac{1}{\pi_i} \cdot \tilde{L}_S^i(w, \epsilon) \quad (2)$$

Eq.(1) shows that ImbSAM's objective includes the class-wise sharpness only for tail classes. This design specifically focuses on flattening the loss landscape for tail classes but may lead to poor generalization for head classes.

Eq.(2) shows that CC-SAM considers the class-wise sharpness for all classes, using class-specific perturbation radii $\rho_i^*$. This allows it to more effectively flatten the loss landscape for both head and tail classes.

Q3: Fig.4: Gamma seems to be quite small (close to zero) for the highest accuracy.

A3: Thanks for your questions! The reason is that even a small $\gamma$ is sufficient to create a relatively skewed $(1 - \pi_i)^\gamma$ from head to tail classes.

As stated in the paper, we assume that $\pi_1 \ge \pi_2 \ge \cdots \ge \pi_C$. For example, in the CIFAR-10 LT dataset ($C = 10$ classes), when $\gamma = 0.8$ (the value at which the LA method achieves the best performance, as shown in Fig.4), we calculate that $(1 - \pi_{10})^\gamma$ is approximately $0.99$, whereas $(1 - \pi_1)^\gamma$ is around $0.66$. This indicates that even a relatively small $\gamma$ can result in a highly skewed $(1 - \pi_i)^\gamma$.

Therefore, despite $\gamma$ being small, the focal component still plays a contributing role.

Q4: Figure 3: A lower $\pi$ has a higher $(1 - \pi_i)^\gamma$. However, this is not the case in the figure?

A4: Thanks for your question! We apologize if our description of Fig.3 is unclear and potentially misleading.

To clarify, Fig.3 presents the distribution of $(1 - \pi_i)^\gamma$ across various $\gamma$ and datasets. Specifically, for a given $\gamma$ and dataset, we plot the distribution of $\mathcal{P} = \\{ (1 - \pi_i)^\gamma \\} _ {i = 1} ^C$, where $C$ is the number of classes. Since the datasets contain many tail classes with similarly small sample sizes (i.e., small $\pi_i$), the distribution tends to peak at relatively large values of $(1 - \pi_i)^\gamma$. Hence, Fig.3 differs from the scenario you expected, where it should plotting $\mathcal{P}' = \\{(1 - \pi_{y_n})^\gamma \\}_{n = 1}^N$, with $N$ being the number of samples.