InCo: Enhance Domain Generalization in Noisy Environments

1. My biggest concern is that the motivation for the concept of environment noise is unclear because its difference from environmental features is very subtle. For the dog example, the snow and water can also be understood as generated by the environment. The mentioned environmental noise should already be present in any real-world datasets, even without manually injected noise. I hope the authors can provide a more detailed discussion about the motivation to treat them differently. If the authors can address this point clearly, I am willing to raise the score.

2. Only from section 2.2, it is difficult to see how the algorithm is actually implemented. It will be clearer to put the computation steps in pseudo code, as the computation is not straightforward.

3. The experimental result has limitation as the type of noise is limited to Gaussian noise. It will be more convincing if more realistic noise is added, but it all comes down to the authors’ definition of environment noise, which is not well defined.

• [Clarification on Loss Design] The paper would benefit from a more detailed explanation of InCo's loss design in Section 2.2. While it's evident that InCo extends VREx by modifying its variance term—from calculating the variance of loss to the variance of correlation between prediction and label—the rationale behind this change could be made explicit. Additionally, it may be worth exploring the variance term in InCo as equivalent to the variance of a specific loss function, such as the unbounded hinge loss, $L = 1 - y \cdot f(x)$). Providing this perspective could offer valuable insights into why the unbounded hinge loss is a superior choice for the loss function in this context.

• [Extensibility to Multi-class Labels] The current version of the InCo method seems to be limited to binary classification. Clarification is needed on whether the method can be extended to handle multi-class scenarios. Given that the DomainNet dataset involves multi-class classification, further details on how InCo is implemented in such a setting is needed.

• [Noise Implementation for Image Experiments] In the ColorMNIST experiments, the paper states, "There are three training groups in our experiments: {0,0}, {0, N(0, 0.5)}, and {0, N(0, 1)}." I don't understand how the noise is specifically implemented, given that the input is an image rather than a scalar. This question also applies to the DomainNet experiment.

Emphasizing the noise data, this paper seems to focus on the situation of adding perturbations or noises to images. However, noisy data also include those beyond computer vision. The case studies only used toy datasets simulated by simple processes. I am curious about how the method will perform on tabular datasets which may show high ``noise-to-signal'' ratios. The first time I saw the term noise in the paper, I thought it was about tabular datasets, which have inherent noises and need no artificial perturbations.

• Analysis: The analysis part is weak and the setting is simple from my perspective. The paper only considers linear models and linear regression under square loss. Thus, (1) it will make the classifier weights $v$ and representation weights $\Phi$ be merged to $w$ and somehow there is no classifier layer; (2) the analysis under the regression setting may not general enough to cover the problem defined in the classification setting. It would be good to consider logistic loss with some non-linearity, e.g., a 2-layer neural network.
• Experiment: The experiment's part is weak. (1) In Table 3 about Noisy PACS and Noisy VLCS, the author defines the noise by Gaussian perturbations. This kind of noise is too artificial. If the authors would like to show the method’s effectiveness, InCo should be evaluated on some real noisy domain generalization datasets rather than PACS with Gaussian perturbations. Table 3 can only show that InCo is a good regularization against Gaussian perturbations, but we do not know whether InCo works well in practical noisy environments. (2) Even for Table 3, it would be good to show the performance under different noise power levels. (3) Many of the latest SOTA works are missing in Table 3, e.g., [1,2] and many others.
• $\lambda$: The method needs a hyper-parameter $\lambda$. The paper only studies it in the simulation experiments. It should be studied in real datasets as well. Also, how to select $\lambda$ efficiently in practice is unknown.

[1] Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual-information regularization with pre-trained models. ECCV 2022.

[2] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. NeurIPS 2021.

1. In the considered setting, $\widehat{x_{inv}}$ and $\hat{y}$ are assumed to have invariant distributions, which is very restrictive. The idea of invariant correlation should focus on the invariance of the linear relation between $\widehat{x_{inv}}$ and $\hat{y}$ (i.e., $\gamma$), rather than the statistical correlation. In the IRM framework, the only parameter that is required to be invariant is $\gamma$. I think a simple adjustment to InCo is to divide the correlation $\rho_{f,y}^{e}(w)$ by the standard deviation of $f(x^{e};w)$

2. Fig 4. (a) is not what IRM concerns. Check Fig.3 from (Arjovskyetal., 2019).

3. Theorem 3.1 shows that the regularization term is zero when $f$ is a linear function of the invariant features, which is a straightforward result. The more important problem is whether a zero regularization term implies that $f$ is a linear function of the invariant features, as shown in Theorem 9 (Arjovskyetal., 2019). Without this result, it is not clear whether solving InCo helps to identify the invariant features.

4. Regarding the presentation, I think section 2.3 should be shortened. A simple example does not show the general behavior of the proposed method and the baselines. Two pages are quite redundant for such an example.