Class Distribution Shifts in Zero-Shot Learning: Learning Robust Representations

1. Soft-AUC: The use of soft-AUC instead of the standard AUC score is intended to make the complete loss (including the penalty) differentiable, thus enabling gradient-based optimization. As mentioned in lines 218-219, the soft-AUC pointwise converges to the standard AUC score as $\beta$ approaches infinity.
   For a fixed $\beta$, define $f(t_{1},t_{2})=I[t_{1}<t_{2}]$ and $g_{\beta}(t_{1},t_{2})=\frac{1}{1+e^{-\beta(t_{2}-t_{1})}}$. Denoting $t=t_{2}-t_{1}$ we have $\left\Vert f(t_{1},t_{2})-g_{\beta}(t_{1},t_{2})\right\Vert ^{2}=\int_{0}^{\infty}\left(1-\frac{1}{1+e^{-\beta t}}\right)^{2}dt+\intop_{-\infty}^{0}\left(0-\frac{1}{1+e^{-\beta t}}\right)^{2}dt=\frac{1}{\beta}\left[2\log2-1\right].$ We will include this calculation in the appendix, along with a graph illustrating the differences, which we have added as Figure 2 in the rebuttal figures file.
2. Gender attribute for the CelebA dataset:
   In principle, the male/female attribute could be used instead. However, on the CelebA dataset, representations that distinguish well between male individuals also distinguish well between female ones, and vice versa, for a reasonable representation network. Thus, shifts in gender do not call for interventions on the CelebA dataset, as all methods would be equivalent to ERM.
   This is not the case for the attribute of being blond. Our experiments show that the representation learned via ERM on mainly non-blond people (including hairless individuals, as this binary indication distinguishes blond people from all others) fails to separate blond people effectively.

We thank the reviewer for pointing out the typos in 3 and 4.

We thank the reviewer for their review and the provided references.

We did not use the datasets mentioned in your references since they either (i) do not have labeled attributes (e.g., the SUN dataset), (ii) the provided attributes correlate with the data-point label such that shifts in them do not affect ERM, making interventions unnecessary (e.g., the CUBS dataset), or (iii) include too few classes (e.g., the AWA2 dataset with only 50 animal classes).

Therefore, we performed the real-data experiments on the CelebA dataset (which is one of the most popular zero-shor benchmarks), and the ETHEC dataset, since both include a large number of classes and labeled attributes in addition to the primary labels (e.g., butterfly family in ETHEC and hair color in CelebA, in addition to species and person identity, respectively).

We thank the reviewer for their time and comments. Below we address the raised weaknesses and questions:

Weaknesses:

1. Definition of parameters:
   $\rho_{tr}$ and $\rho_{te}$ correspond to the the proportion of type $a_1$ classes in train and test sets correspondingly and are defined in lines 139-140, but we will make their definition more clear. $y_{uv}$ is simply the label for the pair of datapoints indexed by u and v: we need two sets of indices since Eq. 4 treats pairs from the same class ($y_{ij}=1$) and those from different classes ($y_{uv}=0$) separately.
2. Complexity:
   All OOD approaches from the considered family (see Section 2) involve optimization across multiple environments. Therefore, the complexity of each step includes: (i) the complexity of generating the environments, (ii) the original complexity of a single iteration over the representation neural network multiplied by the number of environments, and (iii) the computation of the aggregated penalty.
   Component (ii) is shared among all approaches from the considered family, while (iii) is negligible compared to training networks, usually involving simple operations like calculating the mean or variance of scalars. Thus, differences between the methods may arise due to (i). However, the additional training time due to the proposed hierarchical sampling, even when compared to training a simple linear representation, is also negligible since hierarchical sampling can be performed offline before training, as shown in the code provided in the supplementary material. We will include a note about this in the manuscript.
   For example, in the species recognition task, standard sampling takes 1.03 seconds, a naive implementation of hierarchical sampling takes 16.6 seconds, and training the representation takes 1 hour and 6 minutes. Thus, the additional time due to hierarchical sampling is less than 0.4% and similar results (all less than 0.5%) are achieved in all our experiments.
3. Exact performances: Simulation performance is shown in Figure 4, with exact performances reported in Table 1 in the appendix. For real-data experiments, exact performances (and corresponding p-values) are provided in Tables 3 and 4 in the appendix.
   Figures 5 and 6 highlight other important aspects: the learned feature importance in the simulations, demonstrating that our method relies on the intended features, and the performance of our method compared to ERM (i.e., the improvement over ERM).

Question: Unlike the standard OOD setting, our environments are synthetically generated via sampling, resulting in many more environments (Nc over k) than specified in Equation 10. Therefore, there is no reason to use fewer environments, and using more can only enhance the performance of our method. However, we will include an analysis in the appendix that examines performance as a function of the number of environments used for both ERM and our method.

We thank the reviewer for their review and questions. Below we address the raised weaknesses and questions.

Weaknesses:

1. Comparison with Creager et al. (2021):
   Thank you for referring us to Creager et al. (2021). We found their work very interesting and will cite it in our related work.
   The main difference between our construction of environments and theirs is that Creager et al. (2021) infer the worst-case environments for a fixed classifier (e.g., trained via ERM). In contrast, we consider a shift in an unknown class attribute, which is not necessarily the worst-case and may be uncorrelated or misaligned with the worst-case scenario.
   However, in our synthetic data simulations (but not real-data experiments), we demonstrated our method on the worst-case shift. Therefore, following your suggestion, we compared the performance of our method using our environments versus those of Creager et al. (2021).
   The results showed no statistically significant improvement over ERM when using Creager et al.'s environments. We included the results in Figure 1 in the rebuttal figures file.
   Analyzing the results, we discovered that in the context of contrastive learning, Creager et al.'s assigned environments were almost random, as the optimized soft-assignment q barely changed from the random initialization. We attribute this to Creager et al.'s method being based on the IRM objective, which directly applies to gradients that are known to be noisy in contrastive learning (as discussed at the end of Chapter 5.1). This finding aligns with our broader results, which show that applying the IRM penalty, even on our environments where other penalties provide improvement, does not yield significant improvement over ERM.

2. Performance on unshifted distributions:
   The performance of the proposed approach on the unshifted train and test sets of CelebA and ETHEC you mentioned is indeed included in figure 4 for the simulations and in table 3 in the Appendix and indeed shows no negative effect on unshifted distribution performance. We agree it might be preferable to move it to the main article.

Question: In both experiments, we set the minimal $\rho$ to 0.15. Therefore, the number of synthetic environments in Experiment 1 (CelebA, with 450 classes after filtering) and Experiment 2 (ETHEC, with 117 classes after filtering) corresponds to $\alpha=0.05$ and $\alpha=0.09$, respectively. We will specify this in the manuscript as well.

Minor:

1. Figure 2: Thank you for pointing this out, we will add an explanation in the manuscript. Figure 2 shows that the optimal weights of dimensions corresponding to a given type are larger for higher proportions of that type, and that the discrepancy between the weights increases as the ratio of the type variances grows.
2. Location of Section Section 4.4: We accept the recommendation to move Section 4.1 immediately after Section 4.4 and thank the reviewer for the suggestion.