Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data

• Inconsistency is a statstical measure while local inconsistency is a optimizable generalization measure.
• IAM doesn't require additional backward pass, but one more forward pass compared to SAM.

W1 A similar inconsistency measure has already been studied previously in [1], and it is unclear whether the proposed metric has advantages in practice. It would be good to make an empirical comparison to this metric in addition to the current high-level justification.

Our proposed "local inconsistency" and the Inconsistency-Aware Minimization (IAM) framework are fundamentally novel in their (i) practical computability (sigle model, few gradient steps), and (ii) ultimate objective (acts as regularizer). The term "inconsistency" is used by both papers, but it refers to two different concepts.

• Inconsistency in Johnson & Zhang (J&Z): This is a procedural, statistical measure of a stochastic training procedure, denoted as $P$. It is defined as $\mathcal{C}\_{P}=\mathbb{E}\_{Z_n,Θ,Θ'\sim Θ_{P|Z_{n}}}\mathbb{E}_{x}KL(f(x, Θ)||f(x, Θ')).$ This metric quantifies the expected output disagreement between different models ($θ$ and $θ'$) that result from the same stochastic training process. It is a property of the model-generating distribution, not of a single model.

• Our Local Inconsistency: This is a local, geometric property of a single, specific model parameter vector, $θ$. It is defined as $S_{\rho}({\color{blue}θ})={\color{red}\max_{||δ||\le\rho}}\mathbb{E}_{x}[KL(f(x,{\color{blue}θ})||f(x,{\color{blue}θ}+δ))].$ This metric quantifies the worst-case sensitivity of a specific model's output to perturbations in its parameter space. It measures the local geometry of a given solution.

In essence, J&Z's inconsistency analyzes the randomness of the training process, while our local inconsistency analyzes the local geometry of the solution.

2. Innovation in Practical Computability

This conceptual difference leads to a critical innovation in practicality and computational feasibility, as decribed in line 140-143 on the paper.

• J&Z's Inconsistency is an analytical tool. To estimate it, one must train multiple models (e.g., 8) on multiple disjoint training sets (e.g., 4 sets), making it computationally prohibitive to use as a direct regularizer within a single training run.
• Our Local Inconsistency is designed to be a optimizable measure. It can be efficiently approximated for a single model using only a few gradient steps (often K=1). This single-model computability is the key that unlocks its use as a regularizer.

3. Algorithmic Novelty: From Analysis to Optimization

The most significant novelty of our work is the transformation of an analytical concept into a practical optimization framework.

Due to computational demands, "inconsistency" can't be directly minimized during single model training.

Our work bridges this critical gap. By defining a computable, single-model metric, we introduce the IAM framework. IAM-D and IAM-S are novel algorithms that actively seek solutions in regions of low output sensitivity.

W2 The authors have mentioned several drawbacks of using sharpness as the metric for generalization. While Figure 1 is a nice illustration for showing the difference, it would be good to provide some intuition to understand what makes the two metrics different in a geometric perspective, and to what extent it can solve the drawbacks of sharpness in the literature.

The fundamental geometric difference between sharpness and local inconsistency lies in what they measure and, critically, what data they use for that measurement. While both assess the geometry around a solution in the parameter space, sharpness evaluates the curvature of the loss landscape defined by the training data, whereas local inconsistency measures the sensitivity of the model's output distribution using unlabeled, held-out data. This distinction is key to understanding why local inconsistency can overcome some of the documented drawbacks of sharpness.

Geometric view of Sharpness

From a geometric standpoint, sharpness quantifies the local curvature of the training loss surface at a given parameter vector, $\theta$. It is commonly measured by the trace or the maximum eigenvalue of the loss Hessian matrix. A high sharpness value indicates a steep and narrow minimum, where small perturbations to the model's parameters cause a large increase in the training loss.

The primary drawback, as illustrated in the document, is that this geometric view is entirely contingent on the training data.

Geometric view of local inconsistency

In contrast, local inconsistency, $S_ρ(θ)$, offers a different geometric perspective. It is defined as the maximum expected KL divergence between the model's output distributions. $f(x,θ)$ and $f(x,θ+δ)$, under a parameter perturbation $δ$.

Crucially, this metric is computed using a disjoint, unlabeled dataset. Geometrically, this means local inconsistency is not measuring the curvature of the loss function, but rather the sensitivity of the function that the model itself computes.

Q1 The IAM algorithm seems to be computationally more expensive than SAM in [1], it is good to include the computation comparison, given that the performance improvement is rather marginal. In particular, it seems that Algorithm 1 in [1] also requires one trained model only. Can you explain more about the claim in lines 137-138?

• Computational expensivity is totally not large compared to performance gain.
• Local inconsistency can be calulated in single model unlike “inconsistency”, and can be estimated using unlabeled data unlike sharpness-based measures.

1. Concerns about computational burden

IAM requires three forward passes and two backward pass. Therefore, total computational burden is approximately 2.5 times that of SGD (SAM is 2.0x SGD). We attach a time anlaysis of a forward and backward pass for each methods, average of 100 iterations.

On top of that, there are additional empirical investigations that prove IAM’s ability to generalize better. It is comparable or even surpass ASAM’s performance.

Table: Test Error (mean ± stderr) of different optimization methods on various datasets

2. Additional explanation about single-model computability.

“Inconsistency” is a powerful measure, but it is computed over different models. Sharpness based measures are based on loss funciton, which requires explicit labels. On the other hand, local inconsistency enjoys the ability to be computed on a single model, while keeping the label-agnostic nature of “inconsistency”.

Q2 Is there a reason why KL divergence is preferred to measure the inconsistency instead of other metrics, such as L2 distance?

Kullback-Leibler (KL) divergence is favored over other metrics, such as the L2 distance, for measuring inconsistency due to its fundamental connection to information theory and the geometry of the model's parameter space.

While KL divergence is a more general measure for the "distance" between probability distributions, it relates to the L2 distance under specific assumptions. For instance, if a model's output is assumed to be the mean of a Gaussian distribution with constant variance (i.e. $p(x;\theta) =\mathcal{N} (NN_{\theta}, \sigma^2I)$), the KL divergence simplifies to a function of the L2 norm between the means of the original and perturbed outputs: $KL(p(x;\theta)||p(x;\theta+\delta)) = c||NN_{\theta}-NN_{\theta+\delta}||^2 + C.$

In this specific case, minimizing the KL divergence is equivalent to minimizing the squared L2 distance between the model's output vectors. However, KL divergence provides a more principled and general approach that is applicable to any probabilistic output, not just those that fit a simple Gaussian model, and it maintains a direct link to the underlying information geometry of the learning problem.

Q3 How many benefits can we gain from increasing the number of iterations K in Algorithm 1? It would be nice to provide some experimental evidence on this.

Mentioned in line 249-250, our experiments were conducted with K=1 for Algorithm 1. Empirically local inconsistency almost converges when $K=3$ (see figure in markdown file or figures folder in supplemental metrials). Furthermore, estimated $S_\rho(\theta)$ values with both $K=3$ and $K=1$ are highly correlated with Pearson correlation coefficient, being above 0.9 in trained models for figure 1.

W1. Regarding the novelty of IAM in fully supervised setting compared to “inconsistency”

Our proposed "local inconsistency" and Inconsistency-Aware Minimization (IAM) framework are fundamentally novel in their (i) conceptual definition (locality, worst-case sensitivity), (ii) practical computability (sigle model, few gradient steps), and (iii) ultimate objective (acts as regularizer).

1. Fundamental Conceptual Distinction

The term "inconsistency" is used in both papers, but refers to two different concepts.

• Inconsistency in Johnson & Zhang (J&Z): This is a procedural, statistical measure of a stochastic training procedure. It is defined as $\mathcal{C}\_{P}=\mathbb{E}\_{Z\_n,Θ,Θ'\sim Θ\_{P|Z\_{n}}}\mathbb{E}\_{x}KL(f(x, Θ)||f(x, Θ')).$ This metric quantifies the expected output disagreement between different models ($Θ$ and $Θ'$) that result from the same stochastic training process. It is a property of the model-generating distribution, not of a single model.

• Our Local Inconsistency: This is a local, geometric property of a single, specific model parameter vector, $θ$. It is defined as $S_{ρ}({\color{blue}θ})={\color{red}\max_{||δ||\le ρ}}\mathbb{E}_{x}[KL(f(x,{\color{blue}θ})||f(x,{\color{blue}θ}+δ))].$ This metric quantifies the worst-case sensitivity of a specific model's output to a perturbation in its parameter space. It measures a local geometry in a given solution.

In essence, J&Z's inconsistency analyzes randomness of the training process, while our local inconsistency analyzes local geometry of the solution.

2. Innovation in Practical Computability

This conceptual difference leads to a critical innovation in computational feasibility, as decribed in paper line 140-143.

• J&Z's Inconsistency is an analytical tool. To estimate it, one must train multiple models (e.g., 8) on multiple disjoint training sets (e.g., 4 sets), making it computationally prohibitive as a direct regularizer.
• Local Inconsistency is designed to be a optimizable measure. It can be efficiently approximated for a single model using only one or few gradient steps. This single-model computability is the key that unlocks usage as a regularizer.

3. Algorithmic Novelty: From Analysis to Optimization

The most significant novelty of our work is transformation of an analytical concept into a practical optimization framework.

Due to computational demands, "inconsistency" can't be directly minimized during single model training.

Our work bridges this critical gap. By defining a computable, single-model metric, we introduce the IAM framework. IAM-D, -S are novel algorithms that actively seek solutions in low output sensitivity regions.

W2. Q5. Regrading concern about absent of comparisons and discussion with FisherSAM

1. Difference in Maximization Objective

The most critical distinction lies in what is being maximized within the local neighborhood.

• FisherSAM aims to find a solution robust to the worst-case loss. The objective is $\max_{ϵ^{\top}F(θ)ϵ \le γ^{2}}\mathbb{E}_{x, y} [CE(f(x,θ+ϵ), y)]$, which requires labeled data.

• IAM, in contrast, seeks for a solution robust to worst-case output inconsistency. It directly maximizes the KL divergence between the model's outputs before and after perturbation: $\max_{||δ|| \le ρ} \mathbb{E}_{x}[KL(f(x,θ)||f(x,θ+δ))]$. The goal is to find regions in the parameter space where model has stable predictive distribution. This makes our measure label-agnostic and directly applicable to unsupervised or semi-supervised settings.

2. Difference in Neighborhood Definition and Perturbation Direction

Differing objectives naturally lead to different definitions of the "neighborhood" and an adversarial perturbation directions.

• FisherSAM defines neighborhood as an ellipsoid shaped by the FIM, $F(θ)$. The resulting perturbation is $F(θ)^{-1}\nabla l(θ)$. To caculate inverse of FIM, FisherSAM approximates FIM only with digonal term.

• IAM defines neighborhood as a standard Euclidean ball. Because the objective is KL divergence itself, the gradient ascent is taken with respect to the KL divergence. Using the second-order approximation $KL(\dots) \approx \frac{1}{2}δ^T F(θ) δ$, the gradient direction is $F(θ)δ$. This direction maximizes output divergence, which is fundamentally different from those of SAM or FisherSAM.

As the table illustrates, IAM is not a variation of FisherSAM. Introducing a novel objective directly targeting stability of model's output distribution, IAM can be computed without labels and results in a unique perturbation direction distinct from both SAM's gradient ascent and FisherSAM's preconditioned ascent.

W3. Regarding seeming similarity to VAT

We clarify that they are different in their objective.

The core distinction lies in what is being perturbed:

• VAT perturbs input data to find an adversarial direction in the input space, to smooth model's output with respect to local changes in the data manifold.
• IAM perturbs model parameters to find an adversarial direction in the parameter space, to find solutions in "flat" regions of the parameter landscape where the model's function is stable.

W4. Regarding concerns about comparison with strong competitors in semi-supervised learning

IAM can be readily applied to SSL scenarios by regularizing local inconsistency of unlabeled data. This allows leveraging second-order information from abundant unlabeled samples.

We have validated this by integrating IAM-D into FixMatch, a powerful and widely-used SSL method. Simply adding local inconsistency term as an additional regularizer to original FixMatch objective is all.

The results show that integration of IAM-D significantly improves the performance of FixMatch.

Table: Test Error (mean ± stderr) on CIFAR-10 with 250 labels using WRN-28-2 model.

This result demonstrates that IAM can serve as an effective "plug-and-play" regularizer enhancing existing SOTA SSL frameworks.

W5. Theoretical superiority of IAM with convergence rate.

Convergence rate is not our main focus. IAM is focused on performance enhancement and generalization.

Time analysis below shows time analysis compared to other optimization methods.

Q1. Settings of custom semi-supervised experiment.

To show it's applicability, we calculated $S_ρ(\theta)$ with same size of (un)labeled data and add it to CE on labeled batch ($\beta=1$ as described in Appendix E.2).

Q2. Rescent optimizers for self-supervised learning

We thank for the insightful suggestion. In the revised manuscript we will include additional experiments using AdamW (or related modern SSL optimizers) to evaluate their impact on generalization performance.

Q3. Q4. Regarding experiments with diverse datasets and SAM variants

We have conducted additional experiments to compare our method with ASAM (Adaptive Sharpness-Aware Minimization) [Kwon et al. 2021]. To demostrate the scalability, We train SGD and IAM-S on ImageNet with ResNet-50 and use $ρ=0.2$ for IAM-S. We train the model with SGD, IAM-S up to 400, and 200 respectively. We reduce batch size and learning rate to half (2048, 0.5) from [Foret et al., 2021] and used basic augmentaion. Other settings are the same with [Foret et al., 2021].

Table: Test Error (mean ± stderr) of different optimization methods on various datasets

Table: Top-{1, 5} error (mean ± stderr) of IAM-S and SGD trained with ImageNet.

The experiments demonstrate the efficacy of IAM across diverse and large-scale dataset, despite being conducted without hyperparameter tuning under limited computational resources. IAM shows comparable performance to ASAM in more complex dataset (CIFAR-100).

Q6. Regarding entropy minimization

IAM has empirically shows implicit entropy maximization effect with a higher entropy (0.10) than SGD (0.059). But SGD with explicit Label Smoothing ($\epsilon =0.1$) shows much higher entropy (0.52).

Q7. Regarding concern about lack of sensitivity analysis

Please see the "figures" folder in supplementary materials. We have carried out extensive grid‑search over the hyper‑parameters ρ and β, and showed IAM is less sensitive to hyper‑parameter choice.

We thank for your thorough review, and we will incorporate all points discussed here into the revision of the paper.

Thank you to the authors for their additional experiments with FishSAM and for providing clarifications that help distinguish the theoretical and practical differences between the IAM and J&Z papers.

1. Loss function of IAM-D: Could you elaborate on the similarity between the IAM-D update and the TRADES [1][2] methods, particularly in terms of the loss function, excluding the perturbation space? (TRADES perturbs in the input space, while IAM-D perturbs in the model space). Below is the TRADES loss function:

$L\_{\text{TRADES}}(\mathbf{w}) = \frac{1}{n} \sum\_{i=1}^{n} \left( \text{CE}(f\_{\mathbf{w}}(\mathbf{x}\_i), y\_i) + \beta \cdot \max \text{KL}(f\_{\mathbf{w}}(\mathbf{x}\_i) \parallel f\_{\mathbf{w}}(\mathbf{x}\_i') ) \right)$

The update form here is taken from paper [2]. While it’s clear that TRADES addresses robustness in the sample space (via adversarial attacks), and IAM focuses on robustness in the model space, the two methods share the same update form. In fact, the second KL term in TRADES also serves as a regularization technique for robustness.

[1] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019. [2] Wu, Dongxian, Shu-Tao Xia, and Yisen Wang. "Adversarial weight perturbation helps robust generalization." Advances in neural information processing systems 33 (2020): 2958-2969.

2. Generalization and robustness: The authors claim that IAM is designed to improve the generalization and robustness of models in both supervised learning (SL) and semi-supervised learning (SSL) settings. The lack of experiments and comparisons to other competitive methods in these areas significantly undermines the strength of this claim. Without such comparisons, it is difficult to assess the true effectiveness of IAM in achieving its stated objectives, particularly when compared to other state-of-the-art approaches in improving model robustness and generalization. Even when compared to FisherSAM, which is not the current state-of-the-art, IAM fails to demonstrate a clear advantage, especially given the additional computational costs involved.

Core Contribution: Efficacy and Versatility of Local Inconsistency

First and foremost, we would like to reiterate the central objective of our paper to provide context for our response. Our primary goal is not to achieve a new SOTA result on every individual benchmark, but rather to introduce, analyze, and validate local inconsistency as a novel and powerful principle for improving generalization. We focus on demonstrating its efficacy and, crucially, its versatility across a wide spectrum of learning settings: fully-supervised, semi-supervised, and self-supervised.

We believe the value of a new method lies not only in surpassing a specific SOTA result in a single, highly-tuned setting but also in its conceptual novelty and breadth of applicability. By establishing this theoretically-grounded and broadly applicable method, our work opens new, orthogonal avenues for future research, especially in leveraging second-order information in label-scarce environments.

IAM's Clear Advantage: Versatility in Label-Scarce Settings

Regarding the comparison with FisherSAM, the reviewer noted that IAM does not demonstrate a clear advantage.

We would like to clarify where IAM's distinct advantage lies. IAM's primary advantage is its versatility, particularly its effectiveness iqn label-scarce environments—a domain where label-dependent methods like SAM and FisherSAM cannot readily operate. This unique strength is clearly demonstrated by our semi-supervised learning (SSL) experiments as metioned in rebuttal.

As shown below, simply integrating IAM into a strong SSL baseline like FixMatch yields significant performance gains. This is a direct result of IAM's ability to leverage geometric information from unlabeled data, a task for which purely supervised sharpness measures are ill-suited.

Table: Test Error (mean ± stderr) on CIFAR-10 with 250 labels using WRN-28-2 model.

This result, we argue, is a clear demonstration of IAM's unique strength: it extends the benefits of second-order geometric regularization to semi-supervised learning, where other methods cannot. This wide applicability is its clear advantage.

Competitiveness and Untapped Potential in Supervised Learning

Furthermore, even in the fully-supervised setting, IAM is highly competitive. The table below shows that IAM outperforms the reported FisherSAM result on the more complex CIFAR-100 dataset and achieves comparable performance on CIFAR-10.

Table: test error of different optimization methodswith WRN28-10 on CIFAR datasets. * As reported in the original FisherSAM paper Kim et al., [2022]

It is also crucial to note that these strong results were achieved without extensive, setting-specific hyperparameter search for IAM. This is in contrast to the FisherSAM results, which represent a benchmark that has been highly tuned for that specific setting.

Moreover, our results hint at significant untapped potential. The performance boost seen in the IAM-S ($m$-sharpness = 32) variant explores a principle analogous to the $m$-sharpness concept introduced in Foret et al., [2021]. Our result (13.97% on CIFAR-100) suggests that IAM benefits from this same effect, indicating a clear path toward even greater performance in large-scale, parallelized training environments.

As we stated, our primary goal was not to narrowly pursue a SOTA result on a single benchmark. The fact that our method performs so competitively without extensive tuning—and shows clear potential for further enhancement—is a strong indicator of the robustness and fundamental soundness of the 'local inconsistency' principle.

1. On the Comparison with TRADES: A Fundamental Difference in Mechanism While the high-level loss structures of TRADES and IAM-D appear similar, the underlying algorithms for finding the "worst-case" perturbation are fundamentally different, leading to distinct optimization dynamics. This is not merely a change in the perturbation space but a change in the core mechanism.

• TRADES's Approach: TRADES typically employs Projected Gradient Descent (PGD) on the loss function to find an input perturbation $\delta_x$. This involves multiple iterative steps of gradient ascent for each individual sample, with the explicit goal of finding an adversarial example that maximally increases the classification loss.
• IAM's Approach (Algorithm 1): Our method, as detailed in Algorithm 1, does not use PGD on the loss. Instead, it finds the weight perturbation \delta via a completely different procedure:
  • Power Iteration, Not PGD: The maximization is approximated using an iterative method that is equivalent to Power Iteration. This algorithm is designed to efficiently find the dominant eigenvector of the Fisher Information Matrix (FIM), not to maximize the loss directly.
  • Batch-wise Maximization: Unlike TRADES’s per-sample attack, IAM finds a single parameter perturbation, $\delta$, that maximizes the expected KL divergence over an entire mini-batch ($\mathbb{E}_{x\sim p(x)}[KL(f(x,\theta)||f(x,\theta+\delta))]$). The gradient used in our Power Iteration step is computed over this batch-level expectation.

In summary, the algorithmic distinction is critical: TRADES uses multi-step PGD to maximize per-sample loss, while IAM uses a highly efficient, often single-step, Power Iteration to approximate the direction of maximum expected output inconsistency over a batch.

2. On Contributions: Orthogonality, Efficiency, and Scope

• SSL Setting & Orthogonal Approach: IAM is not a standalone optimizer but a regularization principle that is orthogonal to the choice of the base optimizer. Therefore, it can be readily integrated with modern optimizers like AdamW or LARS.
• SL Setting & Training Time: The computational overhead is less than might be assumed. The additional forward pass required by IAM to compute the perturbation does not require a corresponding backward pass, making its cost significantly less than a full additional training step, as mentioned in rebuttal.

Concluding on the Value and Scope of Our Research Our primary goal was to introduce and validate local inconsistency as a novel, versatile, and theoretically-grounded principle for improving generalization across diverse settings (fully-supervised, semi-supervised, and self-supervised). Its value lies in this conceptual novelty and broad applicability, which opens new research avenues, particularly for leveraging geometric information in label-scarce domains.

While achieving a new state-of-the-art (SOTA) is a significant accomplishment, we believe it is not the sole metric for a method's value. Contributions such as introducing a novel, theoretically-grounded principle, or demonstrating remarkable versatility across diverse problem settings, are equally, if not more, crucial for advancing the field.

Our work on 'local inconsistency' was conceived with this spirit in mind. Our primary aim was to propose and validate a new, broadly applicable principle, rather than to narrowly focus on outperforming a specific SOTA benchmark in a single, highly-tuned scenario. We believe our paper's main contribution lies in its introduction of a versatile tool that opens new research directions, particularly for leveraging geometric insights in label-scarce domains.

We kindly ask that our work and our rebuttal be considered through this lens, focusing on the fundamental contribution and its potential for future research.

Regarding the motivation related to FIM

We must state that the review is based on a misunderstanding of our paper's core argument. The critique incorrectly conflates two distinct, yet causally linked, roles of the Fisher Information Matrix (FIM) in deep neural networks (higher uncertainty in parameter estimation = high parameter sensitivity = better generalization), leading to a flawed assessment of our work's motivation and contribution.

The Core Misunderstanding: Two Roles of the Fisher Information Matrix

The central issue in the review is a misinterpretation of how we employ the FIM. These two concepts are not in opposition; they are causally linked. A model that is highly (i) certain about its estimated parameter's value (i.e., has a sharp peak in its likelihood function, corresponding to high Fisher Information) will, by necessity, be highly (ii) sensitive to perturbations of that parameter [Karakida, et al. 2019], which is link to generalization of the model. The review stems from a failure to recognize this causal link.

• Our Perspective: As decribed in line 133-136, our paper uses the $S_ρ(θ)$ to measure the sensitivity of the model's predictions to parameter perturbations. $S_ρ(θ)$ is defined as the maximum KL divergence between the model's output distribution and the output from a perturbed model within a parameter neighborhood (Eq. 3). As we demonstrate (line 149), this quantity is directly approximated by the FIM's maximum eigenvalue: $S_ρ(θ)≈\frac{ρ^2}{2} λ_{\max}(F(θ))$. Therefore, in the context of our work, a high FIM unequivocally implies high output instability under parameter perturbation.

• The Reviewer's Perspective: Measuring Parameter Certainty. The reviewer's view that a high FIM signifies greater certainty in parameter estimation is a correct interpretation from classical statistics (e.g., via the Cramér-Rao bound).

This distinction is especially critical for overparameterized models, where countless minimizers can achieve zero training loss but exhibit vastly different sensitivities to perturbation. Our framework provides a theoretically sound and empirically verifiable method to distinguish between these solutions.

Regrading the motivation for unlabeled data.

On Motivation and Relation to Other Methods

We believe the following clarifications on the relationship between the Hessian and the FIM, alongside new experimental results in the semi-supervised setting, will fully address reviewer's concerns.

• local inconsistency can be estimated without labels, while Hessian can not.
• Hessian $\approx$ GN=FIM, when the model is sufficiently trained [Fort, S., & Ganguli, S. 2019; Sagun, Levent, et al. 2019].

Justification for Using the FIM as a Practical Proxy for the Hessian

The Hessian of the per-sample cross-entropy loss, $∇_θ^2 l_i$, can be decomposed as follows:

• The second term is problematic for unlabeled data because the gradient with respect to the logits, $∇_z l_i$, explicitly depends on the ground-truth label $y_i$.
• However, the first term, known as the Gauss-Newton (GN) matrix, is perfectly computable with unlabeled data. This is because $∇_z^2 l_i = \text{diag}(f(x_i, θ)) - f(x_i, θ)f(x_i, θ)^⊤$, which depends only on the model's softmax output probabilities.

Our approach is justified by two key facts:

1. For a softmax output layer with cross-entropy loss, the GN matrix is mathematically equivalent to the empirical FIM. This is not an approximation but a direct identity.
2. Approximation Quality: In modern overparameterized deep learning, models are often trained to a near-zero training error (interpolation). At such a minimum, the gradient term $∇_z l_i$ approaches zero, causing the label-dependent second term of the Hessian to vanish.

Therefore, the empirical FIM serves as a high-quality, principled, and computationally feasible proxy for the full Hessian, especially near a training minimum, with the crucial advantage of being computable on unlabeled data.

Regarding experiments with state-of-the-art approaches.

• IAM shows its efficacy on FixMatch, a consistency regularization, and pseudo-labeling approach.

Efficacy of IAM in State-of-the-Art Semi-Supervised Learning Methods

Our original experiments aimed to demonstrate IAM's fundamental ability to leverage unlabeled data. Demonstrating its compatibility with state-of-the-art methods strengthens our contribution.

To this end, we have conducted a new experiment integrating IAM-D with FixMatch, a powerful and widely adopted SSL framework. We simply added the $S_ρ(\theta)$ term from IAM-D as an additional regularizer to the standard FixMatch objective.

Table: The results on CIFAR-10, using a WRN-28-10 model with only 250 labels (0.5%), are presented below:

This new result clearly shows that IAM-D provides a significant performance improvement when combined with a strong SSL baseline. It demonstrates that IAM is not merely an alternative to older SSL techniques like entropy minimization but serves as an effective and complementary "plug-and-play" regularizer. It enhances modern SSL frameworks by incorporating a principled geometric constraint, allowing the model to leverage second-order information from abundant unlabeled data in a way that existing methods do not.

The theoretical connection between the proposed measure and the generalization gap is based on PAC-Bayes bounds and leverages the known relationship between the loss Hessian and PAC-Bayes generalization guarantees.

To establish a direct theoretical link between the FIM which can serve as a metric to measure the distance between the parametric probability density models in information geometrand the generalization gap, we derive a PAC-Bayes bound that explicitly depends on the FIM's spectrum, moving beyond Hessian-based measures. Our approach is inspired by recent efforts to obtain non-vacuous generalization bounds [Dziugaite & Roy, 2017; Yang et al., 2022].

Formally, we define $Q=\mathcal N\bigl(θ^*_{\mathrm{Im}},\frac{C}{r}F^{-1}\bigr)$ as the distribution that maximizes the entropy subject to the constraint that the expected KL divergence between output distributions is bounded: $E_{θ,θ' ∼Q}[KL(p(y|x;θ)||p(y|x;θ'))]≤C$. This result aligns with the Cramér-Rao bound, as our method derives the posterior covariance from a finite-sample stability constraint, which the theorem identifies as the correct asymptotic form (Bernstein-von Mises Theorem).

Proof Sketch

1. Orthogonal split.
   For rank‑$r$ Fisher $F=VΛ V^⊤$, $\mathbb R^{m}= \operatorname{Im}(F)\oplus\ker(F),\qquad\mathrm{KL}(Q||P)=\mathrm{KL}(Q_{\mathrm{Im}}||P_{\mathrm{Im}})$

2. Prior / posterior.
   Prior (data‑independent) $P=\mathcal N(0,σ_p^{2}I_r)$.
   Posterior on $\operatorname{Im}(F)$: $Q=\mathcal N\bigl(θ^*_{\mathrm{Im}} ,\tfrac{C}{r}F^{-1}\bigr)$

3. Closed form.
   $\mathrm{KL}(Q||P)=\tfrac12\Bigl[r\log\tfrac{σ\_p^{2}r}{C}+\textstyle∑\_i^r\logλ\_i-r+\tfrac{C}{σ\_p^{2}r}∑\_i^rλ\_i^{-1}+\tfrac{\\|θ^*\_{\mathrm{Im}}\\|^{2}}{σ_p^{2}}\Bigr].$

4. bounds ($λ_1,λ_r$).

   (\text{Complexity term})\le\frac{1}{2\sqrt{n-1}} \sqrt{r(\logλ_1+\tfrac{C}{σ_p^{2}rλ_r}+\log\tfrac{σ_p^{2}}{c}-1)+\tfrac{\\|θ^*_{\mathrm{Im}}\\|^{2}}{σ_p^{2}}+2\log\frac{n}{\xi}}$$

Interpretation. Complexity decrease as the landscape becomes flat (small $λ_1$) and isotropic (small $\frac{1}{λ_r}$). Empirically, $C=σ_p^{2}r$ yields non‑vacuous bounds [Yang et al., ICML 2022].

Regarding empirical comparison with more sophisticated meaure.

Adaptive sharpness show higher higher correlation with generalization ($\tau = 0.608$) than our local inconsistency in our setting. But main distiction with other measure is computability with unlabeled data enabling IAM applicable to label scarce senario.

IAM shows competable efficacy to ASAM on supervised setting.

Table: Test Error (mean ± stderr) of different optimization methods on various datasets

Q1. How the representation of softmax allows to use it as distribution in FIM, while it is a pseudo-distribution?

The soft‑max output represents the model’s assumed conditional distribution $𝑝_𝜃(𝑦∣𝑥)$. Because it meets the two formal requirements of a probability distribution—non‑negativity and summing to one—we can define the Fisher Information Matrix on the statistical manifold as $F(θ)=E[∇_θ \log p_θ ∇_θ \log p_θ^⊤]$.

Q2. How the Hessian is computed in the experiments comparing correlation with generalization gap?

Build the full Hessian is computationly almost impossible. However, it is possible to calculate Hessian vector product (HVP) in $O(Km)$. To calculate $v_{\max}(H), Tr[H]$ on training set, we use pyhessian library that calculate eigenvalue and trace of hessian without explicit build of hessian.

References:

Karakida et al. (2019) Universal statistics of fisher information in deep neural networks: Mean field approach.

Sagun et al. (2019) Empirical analysis of the hessian of over-parametrized neural networks.

Fort & Ganguli (2019) Emergent properties of the local geometry of neural loss landscapes.

W1, Q1. Informal version of Theorem 1.

The Hessian can be decomposed by GN metrix and residual part, $H = \sum_{i=1}^N(\nabla_θ z_i^\top (\mathrm{diag}(f(x_i,θ))- f(x_i,θ)f(x_i,θ)^\top)\nabla_θ z_i + \sum_{k=0}^{C-1}(f(x_i,θ)-e^{y_i})_k\nabla_θ^2z_k).$

When the neural network is sufficiently trained, the residual term become very small (i.e., $f(x_i, θ)-e^{y_i}\approx0$). So the FIM, coressponding the first term, is not significantly different from Hessian. Therefore we can say $H \approx F$ holds on near zero train error [Fort, S., & Ganguli, S. 2019; Sagun, Levent, et al. 2019]. Under this condition, we can provide the more clear version of Theorem 1 that upper bounds maximum loss increase with maximun eigenvalue of FIM (from version with Hessian derived in [Luo et al., 2024]), $L_D(θ) \leq L_S(θ)+\frac{m\rho^2}{2}\lambda_{\max}(F)+(\text{Complexity term}).$

It directly shows our motivation to regularize maximum eigenvalue of FIM.

W2. Datasets in the experiments may be small and easy (CIFAR10 and 100), large datasets such as ImageNet can be considered.

We also evaluate IAM on Fashion-MNIST, SVHN (without extra data), and ImageNet (IAM-S). We apply SGD and IAM-S to ResNet-50 and use $\rho=0.2$ for IAM-S. We used batch size 2048, initial learning rate 0.5, cosine learning rate schedule, SGD optimizer with momentum 0.9, label smoothing of 0.1, and weight decay 0.0001. We train the model with SGD, IAM-S up to 400, and 200 respectively. Reduced batch size and learning rate to half of those in experiments from [Foret et al., 2021] and used basic augmentaion. Models are trained on three A100 GPUs and other setting is the same with [Foret et al., 2021].

Table: Top-{1, 5} error (mean ± stderr) of IAM-S and SGD trained with ImageNet.

Table: Test Error (mean ± stderr) of different optimization methods on various datasets

These experiments demonstrate the efficacy of IAM across diverse and large-scale dataset, despite being conducted without hyper-parameter tuning under limited computational resources.

W3, Q2. Regarding the computational cost for the improvement of IAM.

A clear discussion of the training cost is valuable to be added in the paper. We propose following analysis to address this point.

Computational Cost and Efficiency

As mentioned in line 249-250, our experiments are conducted with K=1 for Algorithm 1 for efficiency. This results in IAM requiring three forward passes and two backward passes per update step. Therefore, IAM's theoretical cost is approximately 2.5x that of standard SGD, while that of SAM is 2.0x.

Table: Overhead comparison between optimization algorithm.

It's important to note the nature of the additional forward pass in IAM(-S). This pass is used to compute the reference model output, $f(θ)$, for the KL divergence term. Since it is not directly part of the backpropagation path for the main parameter update, it can be executed without storing intermediate activations. This makes the practical runtime overhead of IAM less substantial than the raw pass count might suggest, bringing its efficiency closer to that of SAM.

Although IAM has a slightly higher theoretical cost than SAM, this is justified by its significant performance gain, particularly in challenging scenarios.

On CIFAR-100 dataset, which is much more complex, IAM-S achieves a notable 0.75% reduction in test error over SAM, demonstrating its enhanced efficacy on difficult tasks.

Moreover, the primary strength of IAM lies in its broad applicability to label-scarce tasks. The core advantage of our proposed local inconsistency measure is its ability to be computed only on unlabeled data. This is a fundamental difference from sharpness measures like that of SAM, which require labels. Therefore, the additional computational cost is a worthwhile trade-off for achieving robust performance in these practical, data-limited scenarios.

Q3. In semi-supervised learning (Table 2), why does SAM outperform IAM-D on CIFAR-10 at the 20% label rate?

Our semi-SL experiments were designed to demonstrate the ability and unique advantage of IAM: its ability to leverage unlabeled data through its inherent label-free structure. Because the second-order geometry could be similar when the label data is sufficient, the benefit using second-order information of unlabeled data, could be smaller.

To address concern about efficacy of IAM in semi-supervised setting, we have conducted a new experiment integrating our IAM-D with FixMatch, a powerful and widely adopted SSL framework that utilizes consistency regularization and pseudo-labeling. We simply added the local inconsistency term from IAM-D as an additional regularizer to standard FixMatch objective.

Table: The results on CIFAR-10, using a WRN-28-10 model with only 250 labels (0.5%), are presented below:

This new result clearly shows that IAM-D provides a significant performance improvement when combined with a strong SSL baseline. It demonstrates that IAM serves as an effective and complementary "plug-and-play" regularizer. It enhances modern SSL frameworks by incorporating a principled geometric constraint, allowing the model to leverage second-order information from abundant unlabeled data in a way that existing methods do not.

Q4. Algorithm 1 initializes randomly, how does the choice of $\sigma^2$ affect the performance of IAM?

$\sigma^2$ barely affects the performance of IAM. The estimation result of algorithm 1 hardly changes within sufficiently small $\sigma^2 \in [0.01, 10]$. As long as the quadratic approximation holds, what matters is the direction of the initial vector, not its magnitude.

References:

Sagun et al. (2017) Empirical analysis of the hessian of over-parametrized neural networks.

Fort & Ganguli (2019) Emergent properties of the local geometry of neural loss landscapes.

Luo et al. (2024) Explicit eigenvalue regularization improves sharpness-aware minimization.

W1, W3, W4. Regarding concern about computational cost for IAM and evaluation.

Despite the multiple levels of approximation, estimating local inconsistency—and hence each iteration of the IAM-D and IAM-S algorithms—still requires $K$ additional gradient evaluations, which is computationally costly.

The empirical improvements by IAM-D and IAM-S compared to baselines such as SAM or SGD are very minor. Given that IAM-D and IAM-S require additional gradient evaluations per iteration for estimating local inconsistency, while SAM and SGD require two and one respectively, it seems that if we allocate the same budget (in terms of gradient evaluations) and train baselines such as SGD and SAM longer, the improvement would be negligible. I recommend that the authors account for gradient evaluations in their experiments and report results using the same number of evaluations for the different methods.

Tables demonstrating the additional runtime overhead of IAM-D and IAM-S are also quite helpful.

Computational Cost and Efficiency

• IAM is computationally efficient since we used a single ascent step $K=1$ in practice.
• IAM doesn't need additional gradient evaluation (backward pass) compared to SAM. So we can allocate the same budget.
• IAM shows better generalization than SGD with the same budget (trained 2.5x longer).

Our experiments are conducted with $K=1$ for Algorithm 1 for efficiency, as mentioned in line 249-250 It is true that Algorithm 1 needs $K$ additional gradient evaluation. However, empirically local inconsistency almost converges when $K=3$ (please see figure in md file or figures folder in supplementary metrial). Moreover, $S_\rho(\theta)$ estimated with $K=3$ and $K=1$ are highly correlated with Pearson correlation coefficient above 0.9 in trained models for figure 1.

Using $K=1$, IAM requires three forward passes and two backward passes (gradient evaluation) per update step. Therefore, IAM's theoretical cost is approximately 2.5x that of standard SGD, while SAM's is 2x of the SGD.

Table: Overhead comparison between optimization algorithms.

It's important to note a nature of the additional forward pass in IAM(-S). This pass is used to compute the reference model output, $f(θ)$ for the KL divergence term. Since it is not directly a part of the backpropagation path for the main parameter update, it can be executed without storing intermediate activations. This makes the practical runtime overhead of IAM less substantial than the raw pass count might suggest, bringing its efficiency closer to that of SAM.

SGD does not show better generalization than IAM when trained twice longer in our ImageNet experiment with SGD and IAM-S. In many other papers such as Foret et al. [2021], Kwon et al. [2021], SGD shows no better generalization than SAM when trained twice longer.

Table: Top-{1, 5} error (mean ± stderr) (ImageNet) of IAM-S and SGD trained with the same budget.

W2. Since local inconsistency and effectiveness of IAM-D and IAM-S are supported mostly experimentally, I would recommend a broader comparison to more recent SAM methods.

We compare our method to ASAM (Adaptive Sharpness-Aware Minimization) [Kwon et al. 2021] and IAM demonstrates performance comparable to, and often exceeding, that of ASAM.

Table: Test Error (mean ± stderr) of different optimization methods on various datasets

Table: Test Error (mean ± stderr) of IAM-D, SAM, and SGD on WRN-16-8 trained with CIFAR-100 with varying label rates.

Although the primary strength of IAM lies in its broad applicability to label-scarce tasks, The results show that our proposed IAM performs comparably to ASAM in the standard supervised learning setting. On the more complex CIFAR-100 dataset, IAM-S achieves superior generalization performance over ASAM.

Q1. What is $m$ in equation 5?

$m$ is the number of parameters $\theta \in \mathbb{R}^{m}$.

We thank the reviewer for the thorough feedback, and we will incorporate all points discussed here into the forthcoming revision of the paper.

Thank you for the additional clarification on the number of gradient evaluations. I will raise my score, provided that the authors include comparisons with more recent SAM methods in the camera‐ready version of the paper. SAM is a vibrant, fast-paced field, and claiming competitive or state-of-the-art performance requires fair, up-to-date baselines. Although I’m not a fan of this narrative, when the main contributions are primarily experimental without significantly novel ideas, such comparisons are necessary to support the claims.