Minimal Semantic Sufficiency Meets Unsupervised Domain Generalization

We thank the reviewer for the detailed and insightful feedback on our manuscript. We reply on each point raised and address them as follows:

W1,W6,Q1: Theoretical novelty

W1,W6,Q1-Ans. First, we clarify that minimal sufficient representation builds on the established statistical concept of minimal sufficient statistics. Information bottleneck (IB) previously applied these principles to supervised learning, addressing the research problems of that time. Some methods extend this to the SSL scenario to discuss learning a minimal sufficient representation in unsupervised learning.

Our framework adapts minimal sufficient statistics for learning domain-invariant representations in unsupervised domain generalization (UDG). Crucially, UDG and supervised DG face fundamentally different theoretical challenges. PAC learning risk bounds for supervised domain generalization don't apply to self-supervised models where labels aren't available during training. This theoretical gap requires new frameworks tailored to the unsupervised setting. Our work provides the first information-theoretic approach developed specifically for UDG. Two learning objectives are induced from our theoretical framework, with empirical results validating this theoretical foundation across multiple benchmarks. In addition, our framework first relates the representation in SSL model to the downstream generalization bounds in UDG.

In the end, we appreciate the reviewer thinking about the theoretical improvement. Our theoretical discovery on UDG parallels how IB-based methods [7,8] use the same statistical foundation for domain generalization—these works appeared at top conferences and demonstrated how classical concepts can tackle modern problems. In addition, the analogy can extend to methodological relationships: ERM(empirical risk minimization,i.e., CrossEntropy for classification) provides the baseline for domain generalization methods, while SSL serves as the foundation for UDG approaches like ours. We will clarify and emphasize the unique contributions in the revised version.

W2: Technical improvements

W2-Ans. We thank the reviewer for the detailed technical discussion. Our paper's core contributions span two interconnected areas: 1) information-theoretically grounded optimization objectives for UDG; 2) practical algorithms that implement these theoretical insights.

Our approach differs fundamentally from existing UDG methods. The UDG baseline [5] employs adversarial training with domain labels to achieve domain invariance. This strategy operates independently of information-theoretic principles and targets different learning objectives than our framework. Meanwhile, H. Wang et al. [6] focus on capturing non-shared task-relevant information across domains to avoid overfitting, which contrasts with our goal of disentangling semantic content within the shared representation space for domain generalization.

These methodological differences reflect deeper theoretical distinctions. While previous work addresses domain invariance through adversarial objectives or captures domain-specific information, our framework tackles the problem by formalizing what constitutes an optimal representation for generalization: one that preserves semantic sufficiency while achieving minimality. This information-theoretic perspective leads to novel optimization strategies that haven't been explored in prior UDG, DG, and SSL literatures.

W3,W4: 1. Misannotation and 2. table explaination

W3,W4-Ans.

Misannotation: We thank the reviewer for identifying the annotation errors. These have been corrected in the revised manuscript.

Table explaination: The tabulated results demonstrate our method's effectiveness across domains. Our loss design aims to improve generalization consistently across diverse domains rather than optimizing for individual domain performance. While the sketch domain shows a marginal decrease when using the complete loss formulation versus $\mathcal{L}_{max}$ alone, the remaining domains (photo, art, cartoon) all demonstrate substantial improvements. This trade-off yields the highest overall average accuracy (45.26), confirming that our unified framework successfully balances performance across the full domain spectrum rather than overfitting to specific domain characteristics.

W5: Explanation of the t-SNE visualization

W5-Ans. The t-SNE plots examine domain invariance properties by showing how our semantic representations $\mathcal{S}$ exhibit reduced domain clustering compared to baseline methods. The representation $v$ and vanilla MAE representations display clear domain separability, which validates our disentanglement approach, domain-specific information concentrates in the variation subspace, while semantic content becomes more domain-agnostic. This analysis targets the fundamental question of whether our method successfully separates domain-related variations from semantic content, rather than evaluating downstream classification performance. The visualization demonstrates that semantic representations avoid the domain-specific clustering patterns that typically hinder generalization, which aligns with our theoretical framework's emphasis on learning domain-invariant semantic features.

[1] INSURE: an Information theory iNspired diSentanglement and pURification modEl for domain generalization[J]. IEEE Transactions on Image Processing, 2024.

[2] Rethinking domain generalization: Discriminability and generalizability[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024.

[3] How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysis[J]. IEEE Transactions on Information Theory, 2025.

[4] Invariant information bottleneck for domain generalization[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022

[5] Towards unsupervised domain generalization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022

[6] Rethinking minimal sufficient representation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16041–16050, 2022

[7] Ahuja, Kartik, et al. "Invariance principle meets information bottleneck for out-of-distribution generalization." Advances in Neural Information Processing Systems 34 (2021): 3438-3450.

[8] Du, Yingjun, et al. "Learning to learn with variational information bottleneck for domain generalization." European conference on computer vision. Cham: Springer International Publishing, 2020.

We thank the reviewer for the positive recognition of the theoretical justification and the effectiveness of our proposed algorithm based on the experimental results.

Regarding the suggestion to include newer SSL methods such as MoCo-v3 [1], SMoG [2], ReLICv2 [3], and BAM [4] as additional baselines, we appreciate this valuable recommendation. We have conducted additional experiments incorporating SSL methods MoCo-v3, SMoG, and BAM within UDG settings, as outlined below. Our method demonstrates competitive performance when compared to these SSL approaches.

Regarding ReLICv2, the publicly available implementation provides only linear evaluation components without the complete SSL pretraining pipeline. Despite our efforts to contact the original authors for implementation details, time constraints prevented us from developing a full reimplementation from scratch. We acknowledge this limitation in our current evaluation and commit to including ReLICv2 in future benchmark extensions once the implementation becomes available.

We plan to publicly release our UDG benchmark implementations for these SSL methods to facilitate future research and ensure reproducible comparisons within the community.

All reported experiments utilize the official codebases of the respective SSL methods, adapted to conform with standard UDG evaluation protocols while maintaining their core algorithmic components.

Table 1. The results on PACS under 10% label fraction. We report the accuracy for every domain and the average accuracy for all domains.

Table 2. The results on DomainNet under 10% label fraction. We report the accuracy for every domain and the average accuracy for all domains.

We thank the reviewer for the thoughtful and positive feedback, including sound motivation, effectiveness of the method ,and promising experimental results. Below, we address each comment and question in detail.

W1: Explanation of semantic factor $\mathcal{S}$

W1-Ans. While self-supervised models cannot anticipate specific downstream tasks during pretraining, we can systematically identify categories of information that consistently impede cross-domain generalization, such as task-irrelevant information. In UDG settings, task-irrelevant information typically encompasses domain-specific characteristics such as imaging style, background patterns, lighting conditions, and acquisition artifacts, collectively represented as $\mathcal{V}$ in our framework. Given our decomposition $\mathcal{Z}=\mathcal{S}\oplus\mathcal{V}$, the semantic component $\mathcal{S}$ captures the complementary information: fundamental visual structures, object relationships, and content-based features that remain consistent across domains and prove valuable for downstream recognition tasks.

This principled separation between semantic content and domain variations builds upon established theoretical foundations in UDG literature [1,2]. The practical significance becomes evident in real-world scenarios where training data inevitably spans multiple domains. For instance, medical datasets may aggregate scans from different institutions, each using varied imaging protocols, scanner manufacturers, and acquisition parameters. The resulting domain shifts introduce spurious correlations that can mislead models away from clinically relevant features. Our framework addresses these challenges by explicitly optimizing representations to retain semantic sufficiency while systematically reducing dependence on domain-specific information.

W2: Clarify the finetuning data

W2-Ans. Both the SSL pretraining data $X_{ssl}$ and supervised finetuning data $X_{sup}$ originate from source domains, following the standard UDG protocol where no target domain data is accessible during training. This setup mirrors classical domain generalization but operates without category labels during the representation learning phase. The key challenge lies in learning generalizable features from source domain variations that transfer effectively to unseen target distributions at test time. Our experimental setup strictly adheres to the benchmark protocols established in [1,2], ensuring fair comparison with existing methods. Specifically, we use the same train/validation/test splits and domain partitions as prior UDG works, where source domains provide both unlabeled data for SSL pretraining and labeled data for downstream task finetuning, while target domains remain completely held out until evaluation.

[1] Zhang, An, et al. "Disentangling Masked Autoencoders for Unsupervised Domain Generalization." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[2] Zhang, Xingxuan, et al. "Towards unsupervised domain generalization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

We appreciate the reviewer for the positive recognition of our work, including the writing quality, comprehensive evaluation, complete ablation studies, and theoretical foundation. Below, we provide a detailed analysis of the failure cases and discuss the insights gained from competing methods that sometimes outperform MS-UDG in specific scenarios.

Missing Failure Analysis We conducted a comparative analysis with two advanced UDG methods, BSS and CycleMAE. The results highlight the strengths and limitations of each approach.

• BSS (Fourier Augmentation Method): BSS benefits from mixing domain information through augmentation techniques, which sometimes enables it to outperform our method in domains with simpler and more uniform textures, such as sketches, quickdraw, and clipart. This improvement may not be suitable for rich-style domains, like art and real, since these domains contain richer low-frequency information compared with other domains. The Fourier augmentation in BSS may not significantly alter the style, which limits the effectiveness of data augmentation. Without disentanglement or other generalization techniques, the performance of BSS on rich-style domains is often inferior to that on other domains.

• CycleMAE (Generation-Based Method): Generation-based methods, such as CycleMAE, learn informative representations through reconstruction. However, domains like sketches often result in trivially low values of reconstruction losses because of simple backgrounds and domain styles. As a result, CycleMAE tends to focus on learning representations from richer-style domains, such as real images, art, and infographics. This bias toward more complex domains can cause CycleMAE to outperform MS-UDG in those specific areas, while in simpler-style domains, it underperforms compared to MS-UDG..

Even though these two methods occasionally perform better than our method through their own advantages, our MS-UDG method strikes a balance between these two approaches, which can enhance the generalization ability. $L_{max}$ aims to learn the domain-related representation through reconstruction and $L_{min}$ helps disentangle the domain-related information from sufficient semantic representations. As a result, our method achieves state-of-the-art performance on average, while still maintaining robustness across a diverse set of domains.

We will add these discussions to the revised manuscript. Thank you for the advice for better insights into those methods.