Out-of-Variable Generalisation for Discriminative Models

Also it seems like OOV generalization can be thought of just a case of OOD generalization--if we think about all the variables together as the input, the OOV generalization is just a case of OOD generalization (e.g. covariate shift) where some inputs have clear signal from some features and other inputs have clear signal from other features.

We are sorry for the misunderstanding. We would like to clarify OOV and OOD are quite distinct generalization notions. Covariate shift, for example, considers the input space $X$ remains the same but the distribution on input space shifts from source to target $P_s(X)$ to $P_t(X)$. However, OOV considers the input space between source and target contains different sets of variables, even though they could come from the same data generating process, as illustrated in Fig. 1a where X_s = [X1, X2] and X_t = [X2, X3], but X1, X2, X3 all come from the same distribution P(X1, X2, X3). As a concrete example, each country records own population’s vaccine data, e.g., [Age, Is_vaccined]. Such dataset have different distributions for each country (OOD) but share the same set of variables (In variable). However, if hospital A in the same country records their patients vaccine data as [Age, city, is_vaccined], and hospital B in the same country records data as [Age, health_history, is_vaccined], then though they share the same distribution as coming from the same country (in-distribution) they have different variable sets (OOV). We refer to Appendix E.3 for a more detailed discussion on OOD and OOV.

The paper does not demonstrate the practical applicability of the concept of OOV generalization

We thank you for your comment, though we hope to demonstrate the practical applicability of OOV through examples in AI in medicine (re introduction) and provide a concrete example in section 3.1. As the other reviewers also acknowledges our efforts in explicitly grounding the concept, for example, reviewer xwQY acknowledges that the paper provides “simple, clear real-world examples to elucidate the problem”, reviewer woi7 comments that “examples also made things concrete and easy to follow” and reviewer s9Ga thinks we “provide an extensive study … including…examples”.

We think your comment is maybe more directed on the lack of real world dataset. We carefully take your feedback into consideration and following your suggestion, we include an additional real world experiment with details in Appendix D.4.

the setting feels a bit contrived.

We are sorry for this impression, though this paper, unlike many mainstream papers, aims to navigate a new research direction and inspire the community on a surprising overlooked but important research area.

We studied how common transfer learning approaches would fail in OOV setting and show under what circumstance identifiability is achievable for dependent and independent covariates. We exposes a problem that, in the stated form, may appear unsolvable at first sight, and then goes on to provide a mathematically valid and experimentally corroborated solution. In addition, we openly discuss our limitations and point the readers to potential avenues of future research for the area (re conclusion).

We feel that the paper is already useful to the community at this point and sometimes, asking an important question, and giving an incomplete (but non-trivial) answer is more interesting and impactful in the long term. We realize this is highly subjective, and hope that you share our taste in research and our work could as reviewer woi7 says, “with the rigor and simplicity … can act as a foundation to build OOV research”.

It would be helpful to include more intuitive discussion throughout the paper providing more analysis on the sections. For example, more discussion on the assumptions of the settings/theorems would be helpful, and it's not clear exactly under what assumptions the proposed predictor is appropriate.

We thank the reviewer for the suggestion and have included

• a bullet list of assumptions on theorems and identifiability of the proposed predictors (Appendix E.2)
• a discussion on violations to each of the assumptions from causal graph, to function class and to noise distributions (Appendix E.2.1).
• On experimental analysis side, in addition to experiments on real world dataset (Appendix D.4), we conducted additional experiments on the robustness of our approaches when noise level changes (Appendix D.5), and when the distribution of noise is heavy tailed (Appendix D.6)

Overall, we thank you for taking time, which we carefully take into account your feedback and included both experimental results and theoretical discussions in the updated version. We hope that our response has addressed all your questions and concerns. We kindly ask you to let us know if you have any remaining criticism, and - if we have answered your questions - to consider reevaluating your score.

For future work, is there a more complicated/realistic dataset to validate the algorithm?

We thank the reviewer for the suggestion and included results on real-world dataset in Table 2 (Appendix D.4).

Is it possible to compare the algorithm to state-of-the-art marginal or causal methods such as Mejia et al. (2021) or Janzing (2018)? To validate if Vapnik’s principle holds and whether the proposed approach indeed improves results due to solving a less general problem.

We thank the reviewer for the suggestion, though we consider Janzing (2018)’s work is more on the theoretical fronts and Mejia et al. (2021)’s elegant analysis considers a merging dataset problem which does not directly fit to our particular setting.

Theorem 3 connects all moments of the residual distribution to the partial derivatives with respect to the unique variable of the target environment. If additional moments were to be calculated as part of the proposed algorithm, would it improve results (for the general function case)?

Thank you for the question, yes incorporating higher-order moments would improve the result as it allows us to estimate general function beyond first-order Taylor approximations. The higher moments provide a space of system of equations that can help to identify the first order partial derivatives and higher order partial derivatives, which assist in estimating a more complex function class.

In general, since the paper's main claim is that in the real world, it is likely to encounter both aspects of OOD and OOV - How simple is it to combine state-of-the-art OOD methods with the proposed approach? I cannot imagine at the moment a straightforward way to do that.

The reviewer asks re insights on combining both OOD and OOV approaches: we think sparse mechanism shift (SMS) might be a useful notion to characterize distributions shifts as SMS and our current approach both uses causal framework to assist analysis. It assumes the distribution shifts are caused by sparse causal mechanism shifts. We think depending on different variables to which the distribution shift happens would call for a different approach. For example, say, a functional relationship on $Y$ is $\alpha X_1 + X_2 + X_2X_3$. Changing $\alpha$ between the source and target environment does not affect our proposed approach on OOV generalization. However, if $\alpha$ is placed on $X_2X_3$ term, a more sophisticated method need to be developed. We do not have broader results yet, and we feel that the paper is already useful to the community as a starting point to tackle real-world problems that exhibit both OOD and OOV.

I thank the authors for answering my questions and appreciate their effort in providing additional experiments. My main concern regarding the applicability of the proposed OOV setting to real-world scenarios was partially addressed by the additional experiment presented in Table 2 of the rebuttal. The experiment clearly validates the superiority of the proposed algorithms. However, a more convincing argument would be to validate the proposed approach on more than one real-world dataset (and perhaps a more challenging one). Furthermore, since most real-world scenarios are susceptible to aspects of OOV and OOD, I think that the paper would benefit from validating the feasibility of combining the proposed approach with a leading OOD algorithm.

With the above in mind, the paper is well written and I recognize its potential value for the community. For these reasons, I’m updating my score.

It seems OOV fits very well the frame of missing-not-at-random and covariate-dependent missingness. Could the authors comment on that?

We thank the reviewer for the question. Missing-not-at-random and covariate-dependent missingness refer to a scenario where whether a variable is observed or not contains information about other covariates, or about certain other properties of the data point. One can thus hope to exploit this assumption to recover more information about the dataset. We agree that such problems are related to OOV, though think OOV addresses a larger set of problems. For example, the setting of our paper (Fig. 1a) studies when covariates are independent. Such assumption is even more challenging as it provides less information on the remaining dataset. We surprisingly find that even in such challenging scenario, one can enable OOV generalization for certain cases.

Theorem 2 is slightly confusing for me at first glance because I thought PA_Y by definition includes all parents of Y (so x1,x2, x3 in the example) and not just those in the target environment (x2, x3). It may be helpful to clarify.

Thank you for your feedback on improving the paper. We have incorporated an explanation on the difference of Theorem 2 setting with the original setup in the paragraph between section 3.3 and section 3.3.1 in the updated version.

How does the method fair with the oracle as the magnitude of the noise increases?

Thank you for the question, we performed an additional systematic analysis similar in Table 1 with increase in noise standard deviation from 0.01 to 1 in an interval of 0.2. We averaged results over 5 repeated runs and observe our method are robust to increasing noise level and consistently outperforms remaining benchmarks despite noise increases. Table 3 (Appendix D.5) in the updated pdf shows the results.

What if the noise is not gaussian but more heavy tailed?

Thank you for your question, we included an additional systemic noise when noise is heavy tailed. Specifically, when noise follows a log-normal distribution with mean 0 and sigma 0.5. We repeated the experiment over 5 runs and averaged over a hyperparameter sweep. Table 4 (Appendix D.6) shows the results. We observe as expected by our theorems (Theorem 3), the skew of the noise pollutes the skew estimation of the residual distribution, which deteriorates performance.

Does the performance degrade or improve with increasing number of variables?

Thank you for your question. When the number of variables increases in the source environment, we expect the performance remains the same as the base network $f_s$’s accuracy is indifferent to the number of variables given sufficient sample size; when the number of variables increases in the space of missing variables, we would expect our method would still be able to transfer under OOV setting. This is due to one can leverage multiple moments from the residual distribution to create a system of equations and solve multiple unknowns. We provide an additional theoretical analysis on a toy example facing two unobserved variables from the source environment in Appendix C.6.

I assume Theorem 3 does not apply to discrete variables because of the violation of differentiability; is that right?

Yes that is correct.

Can include missing-not-at-random imputation and covariate-missing imputation as two more baseline models (a search in Google scholar using the two key phrases yields some methods).

Thank for the suggestion, though (re discussion above) missing not at random and covariate dependent imputation do not apply in our setting as our assumptions are different. We assume our covariates to be independent whereas the proposed methods assume covariates are dependent.

Figure 3: I don’t quite understand the figure. It would be helpful to define OOV loss, be explicit about the number of samples on the y-axis being (x2,x3,y) or (x1,x2,y) or something else. I also don’t understand why relative loss is zero means the method is on par with the oracle predictor. Why not just show how the fine-tuning error compares with oracle training, which seems easier to interpret? Anyway, I am overall a bit confused about the figure, so my questions may not make sense.

We apologize for the confusion, the number of samples on the x-axis refer to (x2, x3, y), and the relative loss is calculated as $log(loss_{pred}/loss_{oracle})$, the log ratio of predictor’s loss divided by the oracle’s loss. If the predictor achieves the same loss as the oracle loss then the ratio would be 1 and $log(1) = 0$. The purpose of the experiment is to show what is the number of samples needed in the target environment, at which we would prefer the MomentLearn over the joint predictor. Figure 3 shows that up to ~100 samples in the target environment, MomentLearn outperforms the joint predictor.

We thank the reviewer for the suggestions to help us clarify Figure 1. We have incorporated the proposed changes in the updated version, by explaining the claim further and changing the border of the orange box. We also thank the reviewer for their detailed observations on typos and have corrected them in the updated version.

The abstract states "merely considering differences in data distributions is inadequate for fully capturing differences between learning environments." Doesn't out-of-variable technically fall under out-of-distribution, so shouldn't this be adequate? Perhaps more specificity is needed here.

Here we refer to settings exhibit in-distribution if the environments share the same data-generating process. For example, in Fig 1a, though the environments share the same data generating process (in-distribution) but observed different sets of variables (out-of-variable). To give an example on OOD but not OOV, consider vaccine data from different countries: they have the same variables [Age, is_vaccined], but coming from different distributions due to country differences. Thank you for your question and we will make this point clearer in the next version.

Why is the joint predictor considered an oracle predictor if MomentLearn outperforms it?

We apologize for the confusion. In Table 1, we refer to the joint predictor as oracle predictor in cases where there are enough data samples in the target environment such that regressing on the joint variable will lead to a near optimal predictor. In Figure 3, we showed that when the sample size is small (~100 samples) in the target environment, joint predictor can lead to estimation error. MomentLearn predictor thus is able to outperform it.

Could you explain why MomentLearn is reliably more sample efficient than the oracle predictor for "few"-shot prediction?

MomentLearn is more sample efficient than the joint predictor in “few”-shot prediction because with small sample size, the joint predictor would lead to estimation error, whereas MomentLearn leverages the information observed in the source environment (due to large sample size in source) are able to mitigate the problem of small sample size in the target environment.

Our method's (“MomentLearn”) OOV prediction performance in the target environment, compared to the "Marginal" baseline and the predictor that imputes missing variable with its mean ("Mean Imputed"). Shown are mean and standard deviations of the MSE loss between the predicted and observed target values on augmented 'Mtcars' dataset.