Meta Domain Reweighting for Partially Known Out-of-Distribution Generalization

1. Although authors propose a new DG setting, called PKOOD, its significance in practice is not quite clear to me. Especially, the known prior knowledge of test domain in the paper seems to be in the form of data proportions of certain concepts, while this may be a somewhat strict prior knowledge that needs to know all test data beforehand.
2. The batch size used in the experiments is 1024. Does the proposed method require large batch size to work well?
3. The proposed method is relatively complex. It is better to provide the algorithm and complexity analysis of the method.
4. Authors claim that the bilevel meta domain reweighting method can simulate the known distribution shift and achieve invariant learning. Is there any qualitative experiment to verify these, e.g., the distance between learned $\alpha$ and known $\gamma$ and the t-SNE visualization of training domain and testing domain?
5. Why does Fig. 2(c) show completely different trend of $\delta$ from Fig. 2(a) and (b)? Could you please explain this?

W1 The theoretical justification of the approach.

Eq. 1 bounds the gap between target risk and the empirical risk $\hat{\epsilon} {\alpha}(h)$, computed after an $\alpha$ re-weighting of the source data. It remains unclear why the ideal $\alpha$-weighted error was included ($\epsilon_\alpha(h)$) as we are unable to estimate this in practice. The triangle inequality could have been reduced to include on the right-hand side only $bias_{unknown}$ and a $bias_{known}$ equal to $| \hat{\epsilon} {\alpha}(h)- \overline{\epsilon} t(h)|$ . This alteration would lead to the conclusion that the optimal $\alpha$ corresponds to $\gamma$, which makes sense if the target is the optimization of $\epsilon_t(h)$. This ideal error is present both in $variance$ and $bias_{known}$ terms, but further when they are employed for regularizing the model, the $variance$ term is not included, while from the bound of the $bias_{known}$ we only consider terms that are not connected to this ideal estimation.

W2 The proposed approach is not presented fairly in comparison to MAPLE [1] and MLDG [2], the methods on which it is built.

W3 The experimental analysis should be improved - see questions Q1-Q5

[1] Zhou et al. Model Agnostic Sample Reweighting for Out-of-Distribution Learning - ICML 2022

[2] Li et al. Learning to Generalize: Meta-Learning for Domain Generalization - AAAI 2018

• The method is evaluated on a small number of datasets: two. Using more scenarios would make the paper more persuasive in showing that the method really helps. Ideally at least one further dataset would be added, especially from the area of computer vision or NLP as these have been of primary interest in deep learning. A further tabular dataset would help strengthen the paper even further. Swapping the known and unknown domains in the two already considered datasets could also be interesting.
• It would be valuable to consider stronger DG baselines: e.g. on NICO++ the best approaches seem to be SWAD, EoA or CORAL but neither is included in this paper. This would be important in confirming if the proposed method is state-of-the-art for the identified setting.
• A measure of variability across runs would be good to add (e.g. stdevs or SEMs), to see how significant the improvements from the method are.
• The loss that is optimized includes a large number of components so it makes the method quite complex.
• The presentation could be improved: e.g. adding explanation to Figure 1 caption that only the proportions of the domains are known and not also the allocation of examples, explaining the practical implications of the theoretical analysis more and connecting it more strongly to the design of the method, fixing minor typos (e.g. “The regularizer of bias known is definded as follows:”, “Domain weights \alpha is updated based on the query loss.”).
• The ablation study would be more interesting if conducted on NICO++ as the Adult dataset can be seen as only a toy dataset with potentially limited more general implications.

• First, I hope the author could provide more justification on why such a partially known OOD generalization is of great significance. This paper regards the group distribution as an additional information known for target domains, are there any other factors that could be helpful from the quote ‘some facets of the distribution shift can be predicted’?
• The methodology mainly borrows meta-learning and IRM to reduce variance and bias, respectively. Although it is a reasonable design, the contributions are quite limited.
• The experiments are not sufficiently conducted Only two datasets are used, many popular datasets such as the datasets in DomainBed and Wilds are not considered.
• The compared baseline methods are insufficient, most of the comparisons are not recent advances. Please considers some SOTA methods such as SWAD, Cha et al, Domain Generalization by Seeking Flat Minima; Sun et al., CORAL, Correlation Alignment for Unsupervised Domain Adaptation, etc.