Learning from Loss Landscape: Generalizable Mixed-Precision Quantization via Adaptive Sharpness-Aware Gradient Aligning

Q1 The claims of ASGA advantages and search cost.

A1: Many thanks. We think the ambiguity of claim's presentation may cause your confusion of this work. Our experiments (Table 1 in Section 3 of our paper) show that using ASGA with ResNet50 saves 16 GPU hours with better Top-1 compared to HAWQ alone, which validates our motivation. We explain this as below. HAWQ's second-order information of the Hessian matrix is data sensitive[1]. It works well if the proxy subset distribution matches target domain; otherwise, performance degradation occurs since distribution shift may disrupt Hessian estimation. In fact, proxy dataset is often distribution-discrepant, especially for privacy-sensitive scenarios where directly getting subset is not allowed. Conversely, ASGA’s generalizable policy—searched on proxy dataset—can obtain satisfactory performance with large reduction of search time on target datasets (Table 1). We will improve related presentations (related work). As for search cost, we have added comparison experiments with GPU hours evaluation (see A2).

---

[1]: Hawq: Hessian aware quantization of neural networks with mixed-precision, ICCV 2019.

---

Q2 Is the evaluation criterion fair?

A2: We have added experimental results of real training time as below (GH means GPU hours). However, we still believe con-epoch can be used for our scenarios. Rethink our two-step workflow: “X+A” means that ASGA first finds generalizable MPQ policy with small proxy data, and then baseline X finetunes this policy on target dataset with several epochs, where ASGA is not involved, and thus the search cost of each baseline at each epoch is not increased significantly by ASGA. Note that, the generalizable policy of model found by ASGA can be suitable to the baselines that require the same model structure, which largely reduces search burden. Without ASGA, baselines should train from scratch on large dataset, requiring much search time (Table 1).

Q3 Compare with HAWQ as baseline in other settings.

A3: We need to clarify that "H+A" refers to using HAWQ to finetune the MPQ policy searched by ASGA, as presented in A2, while "con-epoch" denotes the number of iterations required for convergence. In this sense, both DNAS and HAWQ can be used as baseline to utilize the policy found by ASGA. Then, we have added experimental results under ResNet18 model, which show that by fine-tuning the policy of ASGA, HAWQ saves 2.7 GH (GH means GPU hours). Due to time limitation, the experiments on MobileNet-V2 will be provided in the future.

Q4 The significance of ASGA.

A4: Notice that both DNAS- and ILP-based MPQs can employ ASGA for reducing search cost without sacrificing performance, which demonstrate the significance of the proposed method. In fact, our ASGA is suitable to the enhancement of ILP-based MPQs, for instance, HAWQ-V3 can quickly finetune ASGA’s policies for stable performance across different datasets and hardware. Besides of DNAS-based methods, ASGA offers a flexible and efficient solution for enhancing various MPQs, especially for resource-limited, cross-dataset scenarios.

Q5 The scalability of ASGA on Transformer.

A5: ASGA is applicable to various models including Transformer, since its core design—minimizing loss landscape sharpness — is not CNN-specific. In fact, [2] have demonstrated the scalability of SAM, which is applied to Transformers (BERT and ViT). Note that current MPQ researches (including SOTA works like EdMIPS and HAQ) primarily focus on CNNs, and thus our experiments also central on CNN models rather than Transformers. We will extend our work to Transformers in future research—a direction that is theoretically justified and critical for advancing practical MPQ deployments.

---

[2]: Towards efficient and scalable sharpness-aware minimization, CVPR 2022.

---

Q1 The novelty of the work.

A1: Many thanks for your comment! This is a valuable question. The method SAQ [1] you mentioned appears similar to our work, but they are actually different. First, the goal of our work is fundamentally different from such work. SAQ quantizes and trains target model on the same dataset, aiming to enhance model's anti-interference ability by smoothing the loss function to prevent the occurrence of overfitting. In contrast, our ASGA aims to significantly boost the search efficiency on large dataset via searching for a generalizable MPQ policy on a small-scale proxy dataset, enabling it to achieve SOTA accuracy with fewer computational resources on a large-scale target dataset (Table 1 in Section 3 of our paper). The advantage of ASGA lies in the improvement of cross-dataset generalization ability. Second, SAQ focuses on non-differentiable fixed-precision quantization, particularly uniform quantization scenarios, whereas ASGA is designed for mixed-precision quantization. Comparative studies[2][3] demonstrate that MPQ achieves better complexity-accuracy balance in practical applications. Moreover, we emphasize that ASGA is not merely an "A+B" combination. Beyond separating search and deployment datasets, ASGA's primary contribution lies in optimizing the existing MPQ training paradigm, offering novel solutions for cross-dataset generalization, convergence efficiency, and computational resource utilization.

---

[1]: Sharpness-aware quantization for deep neural networks[J]. arXiv preprint arXiv:2111.12273, 2021.

[2]: Haq: Hardware-aware automated quantization with mixed precision, CVPR 2019.

[3]: Towards mixed-precision quantization of neural networks via constrained optimization, ICCV 2021.

---

Q2 Literature supplementation and old baselines.

A2: Thanks for the advice. We'll cite these papers in revision. Regarding outdated baselines, according to the available source code in the papers you provided, we add comparative experiments on ResNet18 (GH means GPU hours). The results show that with ASGA-searched MPQ, RFQuant obtains up to 1.11X searching efficiency improvement, further validating our method's effectiveness. Due to time limitation, the comparison experiments with more new baselines will be conducted in our subsequent research.

---

[4]: Retraining-free model quantization via one-shot weight-coupling learning, CVPR 2024.

---

Q3 The differences between ASGA, SEAM and GMPQ.

A3: Thanks for your comment! In fact, both ASGA and GMPQ have similar goal, i.e., find generalizable policy across datasets, and thus share similar quantization process like Fig.7. Regarding Figure. 2 and Eqs.4-5, both SEAM and ASGA build upon the differentiable MPQ framework, and thus share several differentiable search formulations. However, our method is fundamentally distinct from SEAM and GMPQ: SEAM seeks generalizable MPQ policy by minimizing intra-class compactness and maximizing inter-class separation, and GMPQ exploits attribution consistency and it relies on a pre-trained full-precision model for quantized model alignment, requiring extra training costs; Different from them, ASGA focuses on loss sharpness optimization with no need pre-trained model, which is more efficient. For sake of clarity, we will modify the related figures and equations located by reviewers in the revision.

Q4 Performance improvement of ASGA compared with GMPQ and SEAM.

A4: Many thanks. First, many studies[5][6] have theoretically proven that minimizing loss landscape sharpness can well enhance model’s generalization ability. Then, we conducted extensive experiments to validate the superiority of ASGA over SEAM and GMPQ. Experimental results in Table 1 show the quantized model derived from MPQ policy found by ASGA (for ResNet18) achieves a Top-1 improvement of 0.7% and 0.1% on ImageNet over those of SEAM and GMPQ, respectively, and save training costs by 17% and 11.5%, respectively. Moreover, the comparative experiments on a set of baselines in Table 1 show that by using a small dataset of only 0.5% the size of the large dataset to search for quantization policies, ASGA achieves the same accuracy on the large dataset and improves the search efficiency by up to 150%. These validate the superiority of ASGA in improving generalization of quantization policies over both SEAM and GMPQ.

---

[5]: Fisher sam: Information geometry and sharpness aware minimisation, ICML 2022.

[6]: Surrogate Gap Minimization Improves Sharpness-Aware Training, ICLR 2022.

---

Q1 The rationality of the theoretical assumptions in the article.

A1: Many thanks for your insightful comments! As you mentioned, our theoretical analysis employs certain assumptions, including the PAC-Bayesian framework and convergence analysis under stochastic optimization. We adopt PAC-Bayesian theory primarily because it provides a rigorous theoretical foundation for model generalization capability. Building upon this framework, we theoretically demonstrate that reduced loss sharpness in quantized models leads to tighter upper bounds on generalization error, thereby enhancing the generalization of MPQ policies. Our experimental results validate this theoretical insight: when applying ASGA for MPQ search, we observe a 1.5% improvement in Top-1 accuracy on ImageNet compared to conventional methods. This empirically confirms that PAC-Bayesian error bounds effectively measure MPQ generalization capability in practice. Furthermore, Figure 6 in our paper illustrates the sharpness reduction during MPQ search, demonstrating ASGA's effectiveness in minimizing $\sigma_{max}$. These experimental findings align precisely with our theoretical predictions, reinforcing confidence in the validity of our analysis. Additionally, we analyze ASGA's convergence properties and derive its convergence upper bound under non-convex optimization conditions. As evidenced in Figure 4, our method demonstrates stable convergence, where the adaptive $\rho$ parameter enhances training stability for MPQ.

Q2 Is the dataset used representative?

A2: Many thanks. We selected multiple proxy and target datasets based on their widespread use and representativeness in image classification tasks. These datasets cover diverse image categories, styles, and difficulty levels, providing preliminary validation of ASGA's effectiveness and generalization capability. Experimental results demonstrate stable improvements in MPQ transfer performance across all scenarios. Although these rigorously selected datasets can, to a certain extent, already reflect the performance of ASGA quite well, it is undeniable that they still have certain limitations. For example, the data coverage in some specific fields may not be comprehensive enough, and so on, we believe ASGA's theoretical foundation and design approach possess universal applicability. To further evaluate ASGA's broad applicability, we plan to test it on larger-scale datasets, non-vision tasks (speech classification, autonomous driving detection), and extremely low-bit quantization tasks to ensure its effectiveness in more complex quantization environments.

Q3 The adaptability and scalability of the proposed method to models?

A3: Thanks for your comment! ASGA requires no substantial modifications when applied to different models. The core idea of the ASGA method lies in smoothing and optimizing the loss landscape. Through this approach, it effectively reduces the undesirable characteristics of the loss landscape, such as sharp local minima, and thereby significantly enhances the model's generalization ability, enabling it to better handle different input data and task scenarios. Our experimental validation in Table 1 in Section 3 of our paper across diverse model architectures, including ResNet18 and others, has consistently demonstrated improved performance, confirming ASGA's applicability to various structural designs. While minor parameter adjustments might be necessary when applying ASGA to more complex or specialized model architectures, such potential modifications would not compromise the method's general applicability. The architectural independence of our approach stems from its operation at the fundamental level of loss landscape optimization rather than specific network topologies.

Q4 The scalability of the proposed method in other deep learning fields.

A4: Thanks. As previously mentioned, ASGA demonstrates considerable versatility and can be applied to other deep learning domains without requiring substantial modifications. We plan to not only further investigate ASGA's performance in areas such as natural language processing, where its ability to handle sparse gradients and non-convex landscapes could offer distinct advantages, but also to deploy ASGA across diverse hardware scenarios, including both low-power devices and high-throughput systems, to explore its broader potential applications. These efforts will be complemented by rigorous benchmarking against state-of-the-art optimizers to obtain trade-offs in convergence speed, memory efficiency, and generalization.

Q1 The authors should briefly explain why σmax can measure the sharpness of the loss landscape.

A1: Many thanks for your comments! First, reference [1] has proven that $\sigma_{max}$ is positively correlated with the sharpness of the loss landscape. Moreover, $\sigma_{max}$ is the eigenvalue with the largest absolute value of Hessian, which reflects the curvature information of the loss landscape near a local minimum. In other words, a larger curvature indicates a sharper loss surface, leading to poorer model generalization. Conversely, smaller $\sigma_{max}$ corresponds to flatter regions where the loss changes gradually, leading to better generalization. Therefore, using $\sigma_{max}$ as a measure of the sharpness of the loss surface is appropriate.

---

[1]: Surrogate Gap Minimization Improves Sharpness-Aware Training, ICLR 2022.

---

Q2 Authors enhance model generalization by minimizing the loss sharpness. However, during actual training, the model may get stuck in locally flat regions, leading to convergence to suboptimal solutions. How can we ensure that the ASGA method, while pursuing a flat loss landscape, does not excessively get trapped in locally flat regions and miss the global optimum solution?

A2: Thanks for the insightful comment! Indeed, avoiding convergence to flat regions while pursuing a flat loss landscape is a critical challenge. We recognize this issue, and thus incorporates both empirical loss and perturbed loss into the design of the ASGA loss function. This aims to maintain low loss value while seeking a flat landscape. In addition, the implicit gradient alignment mechanism helps minimize sharpness while preventing convergence stagnation caused by gradient conflict. This ensures that the model converges along a flat yet globally optimal direction guided by the loss landscape. From an experimental perspective, we validate our strategy across multiple datasets, demonstrating that our method can obtain superior quantization policies to direct optimization on the target dataset while simultaneously improving generalization performance.

Q3 Has the author considered extending the ASGA method to other fields? For example, applying the ASGA method to other types of quantization methods, such as binary quantization?

A3: Many thanks. We are indeed considering applying ASGA to other quantization methods. The core idea of ASGA - improving model generalization by optimizing the sharpness of the loss landscape - is fundamentally generalizable. For instance, the binary quantization you mentioned represents an extreme form of low-bit quantization. However, this method has very strong constraints, which may lead to a drastic change in the loss landscape. As a result, the model optimization process becomes extremely difficult, and it is challenging to find the optimal solution. ASGA's sharpness optimization and adaptive gradient alignment mechanism can effectively mitigate this issue. Therefore, we plan to further investigate and validate the effectiveness of ASGA in other quantization approaches in our future research. We are confident that ASGA can also demonstrate excellent performance in other quantization methods.

Q4 In the paper, authors claim that the ASGA method can be applied to edge intelligence applications. A natural question is that whether the ASGA can be applied to other kinds of hardware platforms. For example, deploying this method on a Raspberry Pi?

A4: Thanks for the comment! In edge intelligence applications, the primary challenges are to reduce computational overhead and enhance model generalization—objectives that align closely with ASGA’s design goals. While this paper focuses on ASGA’s application to conventional edge devices, we believe our method can be applicable to resource-constrained platforms such as Raspberry Pi. As pointed out in Section 3 of our paper, unlike the traditional model training process, ASGA only needs to conduct policy search on a small-scale proxy dataset and then finetune it on the large-scale target dataset. This approach can significantly reduce a large amount of computational overhead for resource-constrained devices. Thus, even for low-power devices like the Raspberry Pi, ASGA can be practically deployed by first performing offline quantization policy searches before on-device deployment, making it feasible for energy-efficient scenarios.