QT-DoG: Quantization-Aware Training for Domain Generalization

We appreciate the reviewer’s positive feedback and recognition of the novelty of our approach, as well as the various visual and theoretical analyses presented. We also value the reviewer’s constructive comments and thoughtful questions, which we address in detail below.

Quantization vs. Noise — what's the difference?

We appreciate the reviewer’s insightful comment. While quantization and noise injection share the general idea of introducing perturbations during training to improve generalization, they differ in several key ways. Noise injection typically involves temporary, stochastic perturbations (e.g., Gaussian noise) applied to weights, activations, or gradients. In contrast, quantization imposes a persistent, structured constraint by discretizing weights and activations throughout training and inference. This constraint acts as a form of implicit regularization, consistently guiding the optimizer toward flatter minima — as supported by our loss landscape visualizations and flatness analyses. Unlike random noise, quantization changes the geometry of the optimization space in a deterministic way, further enhancing stability. Moreover, while many domain generalization methods rely on noise or data augmentation to promote robustness, our work is the first to show that quantization achieves similar regularization effects, with the added benefit of computational efficiency. We will revise the paper to make this distinction clearer.

Q1. Can you separate the positive effect? How much is the performance gain from DG, and how much is the performance loss from quantization?

We address this in Table 8 through ablation studies on quantization bitwidths, showing that, whereas aggressive quantization (e.g., 2-bit) may degrade accuracy, moderate quantization (e.g., 7-bit) enhances generalization while maintaining efficient inference. Additionally, Table 7 demonstrates that moderate quantization incurs little to no in-domain accuracy loss while significantly improving out-of-domain performance. We will clarify this point and emphasize that the net generalization gain often outweighs precision loss, with certain configurations offering both performance improvements and resource savings.

Q2. In Figure 4, what is different between the two networks except for quantization? Why are the validation accuracies of two networks before the quantization step?

The two models shown in Figure 4 are trained with the same architecture, training setup, optimizer, and random seed. In principle, their in-domain accuracies before the quantization step should be the same, but they nonetheless vary slightly because of slight GPU-induced randomness in the computation process, on which even the random seed has no impact. After quantization, however, we can see that the quantized model yields much more stable curves in the test domain (bottom plots). To mitigate this randomness, we followed the same process as SWAD and EoA, ensuring fixed random seeds and conducting multiple runs.

We appreciate the reviewer’s thoughtful feedback and recognition of our work’s contributions, especially in terms of its novelty and the various analyses provided in the appendix. Below, we provide detailed responses to each question and concern.

Performance Without Ensemble:

Thank you for pointing out that SWAD slightly outperforms QT-DoG without the ensemble. However, a key distinction is that QT-DoG is 4.6× smaller than SWAD, making it significantly more efficient. Additionally, when comparing models of equal size (EoQ vs. SWAD), our EoQ method achieves significantly better performance. We will clarify this important trade-off in the revised version.

Q1: What model is used for the NLP experiments in the appendix?

We used the same BERT model as in WILDS [1]. Different architectures and different modalities further demonstrate the effectiveness of our method. Thank you for appreciating this!

[1] WILDS: A Benchmark of in-the-Wild Distribution Shifts

Q2: How would the approach perform for additional one or two models of different architecture? E.g. simple CNN, different transformer model

As suggested by the reviewer, we conducted additional experiments using ViT-B/16 with a CLIP-based backbone, and our findings support the original results, demonstrating that QT-DoG effectively improves generalization across different architectures with different datasets. These results, shown in the table below, will be included in the revised version.

Thank you for your valuable suggestion!

Q3: How well would SWAD work if used in a similar ensemble?

SWAD [1] and SMA [2] are both weight averaging methods, yielding similar accuracies to QT-DoG but for larger model sizes. An ensemble of SMA corresponds to EoA [2], and our results show that EoA performs slightly below our proposed EoQ. Moreover, our EoQ is 6× smaller than EoA, making it significantly more efficient. Even if an ensemble of SWAD were to achieve similar or slightly better accuracy, it would also result in a much larger model compared to EoQ, reducing its practical advantages in terms of efficiency and deployment.

[1] SWAD: Domain generalization by seeking flat minima. Cha et al., NeurIPS 2021.
[2] Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization, NeurIPS 2022.

Would be better to explain how QAT is used in more depth, even if it may be a rather direct application..

Thank you for your suggestion. We will incorporate additional explanations of QAT in Section 3.1.

We appreciate your recognition of our results' competitiveness and your acknowledgment of the broader relevance of our work to the scientific community. We address your major concerns below.

Theoretical proofs combined with the theory of model compression and domain adaptation should be offered.

We appreciate the reviewer’s suggestion. We agree that building a rigorous theoretical bridge between quantization, model compression, and domain adaptation is a valuable direction. However, doing so is nontrivial, and thus our work focuses on highlighting an intuitive but underexplored connection, which we believe is of interest to the community: Quantization introduces structured noise that leads to flatter minima, which correlates with better generalization. While our current analysis based on second-order Taylor expansion does not constitute a full theoretical proof, we hope that it will provide the foundation for further investigation. We will clarify this in the revised manuscript.

More figure and analysis for EoQ should be offered. It also should be included in abstract.

We appreciate the reviewer’s positive feedback on our proposed EoQ. Our goal with EoQ is to demonstrate that quantization can be effectively integrated into domain generalization frameworks, particularly ensemble models, offering significant benefits in terms of inference efficiency and model storage reduction. If the reviewer envisions specific analyses that could further illustrate the impact of EoQ, we would be happy to provide them. We indeed have included EoQ in the abstract.

Novelty is a problem since only a visual experiment and main results for generalization.

It is unclear to us how novelty is related to the visual experiment and the results. Our novel contribution is to demonstrate that quantization-aware training, traditionally used for model compression, can serve as an implicit regularizer, with quantization noise enhancing DG. This novel insight has been positively acknowledged by all reviewers.

More details of the baseline choices? Can the authors compare with more SOTA methods

We choose the baselines following prior works in the domain generalization literature, as also noted by the reviewer. Following the reviewer's suggestion, we have included a comparison with the state-of-the-art CLIP-based method, CLIPooD. For a fair evaluation, we use the same architecture (ViT-B/16). The results below demonstrate that QT-DoG not only improves domain generalization performance by 0.3% in average accuracy but also yields a 4.6x model compression.

Why DG needs quantization methods; instead of a copy of the existing strategy to a new setting?.

Our work goes beyond merely applying an existing strategy to a new setting. It establishes a fundamental link between quantization and domain generalization, demonstrating how quantization can inherently improve generalization at minimal cost. This insight not only provides a practical and effective solution for DG but also introduces a new perspective and application for quantization methods, benefiting both research communities.

Moreover, as shown in Table 2, QT-DoG can be combined with existing DG methods (e.g., CORAL, MixStyle) to further enhance performance, demonstrating that quantization is not just an independent technique but a complementary tool that can strengthen domain generalization approaches.

Computational costs are important but for most DG tasks, the benchmarks are only of limited-scale

The main goal of our work is not to reduce computational cost via quantization but demonstrate a connection between quantization and domain generalization. The fact that quantization further reduces model size in an added benefit. Note that, while we agree that current benchmarks are of limited scale, this is likely to change, as we see an increasing trend in the deployment of large models that often must operate under resource constraints. Our approach that jointly reduces size and improves generalization will thus be highly relevant in this setting.

Q1. Do you plan to test the QT-DoG on a larger Transformer model (e. g., ViT-Base, BERT) to verify that it also applies?

Note that we already include BERT results on the Amazon-WILDS dataset in Table 11 in the appendix. We will explicitly mention this in the main paper. Moreover, in the final version, we will include the results discussed above for ViT-Base with a CLIP pretrained backbone.

Q2. How about the model generalization ability under extremely low bit quantification?

We have studied this in Table 8. After a certain point, there is a trade-off between compression and generalization; very aggressive quantization can lead to a performance drop.

Thank you for your thoughtful and constructive feedback. We greatly appreciate your recognition of our key contributions, including the theoretical insights connecting quantization and flat minima, and the practical benefits of model compression. We address your major questions and concerns below.

Smoother transitions and proper introduction of abbreviations:

In the revised version, we will improve the overall clarity and readability of the paper by ensuring smoother transitions and clearer exposition of key concepts. Specifically, at Line 213, we will add "Here, we argue that quantization inherently induces noise, which aids in finding flatter minima. To support this, we use a second-order Taylor series expansion to analyze how quantization-induced perturbations affect the curvature of the loss landscape, leading to flatter minima.". We will also make sure that all abbreviations are properly defined on their first appearance.

Comparison with recent CLIP-based approaches:

We thank the reviewer for the suggestion to include CLIP-based approaches to strengthen the evaluation and will do so in the revised version based. Specifically, we have now compared QT-DoG against CLIPooD and ERM with a CLIP backbone. For fairness, we have adopted the same architecture (ViT-B/16) as used in CLIPooD. The results evidence that QT-DoG continues to improve domain generalization performance even when applied to CLIP-based models, yielding a 0.3 improvement in the average accuracy while compressing the model by 4.6 times, as shown below.