Intermediate Layer Classifiers for OOD generalization

Thank you for insighftul comments. We address them below.

W1: Gap between transformers and CNNs

Thank you for raising this point. We currently lack a definitive explanation but hypothesize it may relate to differences in receptive fields or architectural biases between CNNs and transformers. We will add a discussion on this in the paper

W2: Influence of Feature Dimensionality

We acknowledge that the feature extraction protocol was insufficiently clarified in the current draft and will be addressed this in the revised manuscript.

Specifically, in this work, we did not apply any form of feature pooling before probing. While we understand the reviewer’s concern regarding the potential impact of feature dimensionality on performance, we believe this factor is secondary to the informativeness of features themselves. Our perspective is twofold:

1. For ILCs to succeed, the features in intermediate layers must be inherently informative of the task. If such features exist at these layers, dimensionality alone should not significantly influence performance (either the useful features are there or they are not).
2. Feature dimensionality does not explain the observed pattern in in-distribution (ID) performance, where the highest performance is almost always achieved using last-layer features. As shown in Figure 18 of the Appendix, ID performance peaks, or is near the peak, when using penultimate layer representations in the classifier $ILC_{L-1}$.

To ensure dimensionality is not a confounding factor, we conducted an additional experiment on CIFAR-10C using ResNet18. Initial output dimensions of layers were as follows: Layer 4: $16,384$, Layer 5: $8,192$, Layer 6: $4,096$, Layer 7: $2,048$, and Layer 8: $512$.

Representations $\mathbf{R}_i$ were extracted from training samples at each layer $i$, centered as $\bar{\mathbf{R}}_i = \mathbf{R}_i - \text{mean}(\mathbf{R}_i)$, and the top $512$ PCA directions $\mathbf{V}_i \in \mathbb{R}^{d_i \times 512}$ were computed. For a test $\mathbf{x}$, its representation $\mathbf{r}_i(\mathbf{x})$ were centered using the training mean and projected as $(\mathbf{r}_i(\mathbf{x})- \text{mean}(\mathbf{R}_i)) \mathbf{V}_i$. Linear probes were trained and evaluated on these projections in few-shot setup. This setup ensures fixed-dimensional representations across layers, isolating the role of informativeness. We present the results in the table below.

Compared to the results before PCA, the mean accuracy differences are minor, particularly for Layer 6 (the highest-performing layer in both setups). This supports the conclusion that dimensionality alone does not explain the trends observed in layer-wise performance, as the fixed-dimensionality setup after PCA produces similar relative patterns across layers.

We're happy to engage in further discussion.

Thank you for insighftul comments. We address them below.

W1: Usage of "information content": relationship between accuracy and "information content".

Thank you for raising this point. We refer to “information content” as the representations’ informativeness about the task under distribution shifts. Accuracy is used as a proxy for this informativeness, as it directly reflects how well the features support the task. We focus on the lack of informativeness in the penultimate layer because its representations are optimized for the original task, achieving high accuracy in-distribution. As shown in Section 4.2.1 (Information Content for OOD Generalization at Last Versus Intermediate Layer), these representations often perform worse on OOD tasks compared to intermediate-layer probes, highlighting their limited robustness for OOD generalization and thus their lack of information content, even with abundant OOD data.

To further support this argument, we conducted additional experiments using non-linear probes (e.g., multi-layer perceptrons) on intermediate-layer representations, as described in Appendix C.2 (in the updated version of the PDF). The results showed negligible performance improvements compared to linear probes. This suggests that the lack of performance is not due to the probe’s limited capacity but rather the absence of learnable information in the representations.

W2: It may be difficult to find the "best" layer in practice

We agree that finding the best layer is not always straightforward. However, model selection typically involves a held-out testing set, which can include OOD datapoints [a, b, c]. Under this setting, selecting the best layer is analogous to selecting the best hyperparameters or training epochs when using a last-layer classifier. Since we assume access to already-pretrained models, this approach only requires training a set of linear probes, making it computationally similar to last-layer retraining methods [a].

[a] Kirichenko, Polina, Pavel Izmailov, and Andrew Gordon Wilson. "Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations." ICLR 2023 (2023).
[b] Sagawa, Shiori, et al. "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization." arXiv preprint arXiv:1911.08731 (2019).
[c] Gulrajani, Ishaan, and David Lopez-Paz. "In search of lost domain generalization." arXiv preprint arXiv:2007.01434 (2020).

W3: Lacking theoretical backing

We agree that a mathematical understanding of this phenomenon would be valuable. However, we believe this is challenging, as it requires developing insights specific to DNNs that extend beyond analyses of simpler, two-layer networks. Even for last-layer retraining methods, theoretical understanding is limited. Our empirical findings suggest that intermediate layers capture more general, task-invariant information and handle OOD data more similarly to ID data compared to the penultimate layer. Understanding how features evolve across layers, particularly why the penultimate layer sometimes also captures task-invariant features, remains an open question and is largely addressed in empirical studies at present.

Q1: Feasibility and appropriateness of feasibility score

The key idea is to measure how features change from ID to OOD data. This is done per group (or per class, in classification problems), accounting for how each sub-cluster within the group spreads under distribution shifts. The score reflects a layer’s ability to preserve features consistent across ID and OOD data. By normalising with within-group distances, the metric avoids bias from the variability of ID data within a layer's features. Additionally, the score aligns with observed layer behavior, where intermediate layers exhibit lower sensitivity and greater robustness, supporting its validity as a meaningful measure of sensitivity. For these reasons, we believe this score is appropriate for evaluating feature sensitivity.

Q2: Tasks other than image classification

While we don’t have results for tasks beyond image classification, we hypothesize that the outcomes depend on the nature of the supervision signal. Tasks with fine-grained supervision may benefit less from intermediate layers, whereas tasks with coarser supervision signals may rely on them more.

Q1: Which ResNets were used to produce the plots (Figures 3, 4)?

Figure 3: ResNet18. Figure 4: ResNet-50 (Waterbirds, CelebA) and ResNet18 (remaining plots). This information has been added to the updated manuscript, thank you for pointing this out.

Q2: Was any trend observed when comparing different ResNet sizes in Figure 4?

The influence of ResNet size appears minimal. In cases with limited training data for probes, the gap between ILCs and last-layer retraining methods depends more on the dataset than the architecture.

Q3: Can we really call "zero-shot" a setting in which the "OOD" data is composed of ID samples with added noise?

Thank you for this perspective. To clarify, in all "zero-shot" settings, we do not use any OOD data during training (e.g., no noise is added to ID samples during this phase). While we agree that noise addition conceptually represents a mild distribution shift, we note the following:

1. In CIFAR-10C/-100C experiments, we used the highest noise level (level 5).
2. Classifiers trained on noise-free CIFAR-10/100 datasets perform poorly on noisy datasets, with a performance drop of ~25% on CIFAR-10C (see Figure 1), illustrating the practical significance of this shift.

Given that the model was not originally trained on all noise variations and exhibits a drastic performance drop on noisy datasets, we believe this constitutes a valid OOD task.

Q4: Did the authors find that any particular architecture had intermediate representations consistently outperforming last-layer retraining?

Yes, intermediate representations consistently outperformed last-layer retraining across nearly all dataset-architecture pairs. ResNets, in particular, exhibited consistent trends in all experiments. The only exceptions were CelebA and Waterbirds, where intermediate representations performed similarly to last-layer retraining in the few-shot setting with the full OOD dataset used for training. However, in zero-shot settings and cases where only a fraction of the data was used, ILCs continued to outperform last-layer retraining. We hope this answers your question, and we are happy to provide further clarification if needed.

Q5: Did the best layer for a particular dataset seem to agree with the best layer for other datasets, for the same model?

Thank you for this interesting question. Figure 18 in the Appendix shows accuracies per layer in both zero- and few-shot settings. While the exact position of the best layer varies across datasets, it is typically located in the second half of the network, as discussed in Section 5.

Thank you for your insightful review.

W1: Sensitivity analysis lacks depth on why some intermediate layers are more stable than others in different types of shifts

Thank you for the suggestion. We agree that understanding why some intermediate layers are more stable than others is an important question, especially across architectures. Answering this is challenging, as even within a single dataset like CIFAR-10C, performance varies significantly across different types of noise (see Figure 15 in the Appendix). This suggests that the stability of intermediate layers depends on both the nature of the shift and the architecture.

We are actively analyzing these trends across architectures and will incorporate the findings into the paper.

W2: Analysis in Fig. 9 / Fig. 10 isn't clear

Thank you for the feedback. We clarify the intended points and outline the changes made to the paper for better clarity.

Quantitative Analysis (Figure 9) We argue that earlier layers exhibit decreasing separation or sensitivity, particularly for minority data, which we regard as OOD. This aligns with your observation that the trend primarily holds for minority groups. The same conclusion extends to other shifts, such as CIFAR-10C and CIFAR-100C, as shown in Figure 19 in the appendix. The key point is that for the minority group, the penultimate layer separates ID and OOD points most distinctly.

Qualitative Analysis (Figure 10) The plot shows train and test samples under PCA projections using the MultiCelebA dataset and a ResNet18 model. Thank you for noticing this missing detail. We have included this information in the updated version of the paper. We have also clarified the penultimate layer's separation for a minority group in the caption.

W3: Terminology: training points vs. testing points vs. probe points vs. OOD points

To clarify this, we have explicitly indicated in the few- and zero-shot sections whether $\mathcal{D}_{\text{probe}}$ is sampled from ID or OOD distributions. Additionally, we have referenced Section 3.1 in these sections and included a graphical representation of the data used in the probes (see Table 1).

W4: Relation to previous works

Thank you for these references. The key distinction is that the cited works primarily focus on generalization to novel class splits, whereas our work addresses scenarios where the task and visual variations remain closely aligned with the original distribution. We have updated the manuscript to relate our work to the previous studies.

W5: Authors only considered frozen, pre-trained weights in all scenarios studied, which is somewhat unrealistic or incomplete

We agree that fine-tuning pre-trained models is common in practice. However, all models in our experiments were tailored to their respective datasets. For example, CelebA and Waterbirds models were fine-tuned on their specific datasets starting from ImageNet-pretrained weights, while CIFAR-10 models were trained from scratch. This setup ensured that each model was well-aligned with its target dataset, enabling a consistent evaluation of intermediate-layer representations without introducing additional variability from further fine-tuning.

W1: Trends across architectures/datasets/shifts; Q5: Agreement of layers across shifts for the same architecture

We have extended our analysis to provide a fine-grained view of the alignment between ILCs performance across layers, datasets, and shifts for CIFAR-10C and CIFAR-100C. Detailed results are included in Appendix C.4 of the updated PDF.

Across all tested shifts on CIFAR-10C and CIFAR-100C for ResNet18, the intermediate layer ($\ell = 7$) always outperforms the penultimate layer ($\ell = 8$). Regarding agreement across datasets for the same shift, the best-performing layer on CIFAR-10C is often not the best-performing layer on CIFAR-100C.

Should additional questions regarding cross-transfer among shifts or datasets arise, we would be happy to engage in further discussion.

I thank the authors for their detailed response. I believe the paper has significantly improved after the authors addressed my concerns and those from other reviewers, so I am increasing my score to 6.

Thank you for an insightful review.

W1: It would be interesting to understand what happens in the last layers that reduces generalizability?

We agree that understanding this is important. Our findings suggest that the penultimate layer’s features are highly specialised for in-distribution tasks. This specialisation may reduce robustness to distribution shifts. The sensitivity analysis in section 5.2 supports this, showing that penultimate-layer features are more sensitive to shifts than those in intermediate layers. This remains an open question, and we would be interested in exploring it further.

Q1: Is the reason for the decrease in generalization due to skip-connections?

Thank you for raising this point. Skip connections propagate intermediate features through the network. In theory, this should preserve generalisable information. However, our results suggest that the penultimate layer aggregates features in a way that does not fully retain robustness. Figure 6 shows GoogLeNet performance, which does not use residual connections like ResNets but still exhibits a large gap in performance between intermediate and penultimate layers. This indicates that the observed behaviour is not solely due to residual connections. We do not have conclusive evidence linking skip connections to reduced generalisability, but this hypothesis is interesting and merits further investigation.

Thank you for insightful comments.

Q1: Why would the data be selected ID for the zero-shot scenario and OOD for the few-shot scenario?

In the zero-shot scenario, $D_\text{probe} \sim P_\text{ID}$, as no OOD data is used for training the probes—only the original training data that the backbone was trained on. In the few-shot scenario, $D_\text{probe} \sim P_\text{OOD}$, aligning with the last-layer retraining paradigm, where limited OOD samples are used for adaptation.

We’d be happy to discuss this further if we misunderstood your question.

Q2: Inclusion criteria of models and datasets isn't clear

Thank you for the question. We agree that the inclusion criteria should have been explained in more detail. The selection of models and datasets is closely intertwined, as we wanted to avoid training models on ID datasets ourselves. Each (model, dataset) pair typically requires specific recipes, and we aimed to prevent misrepresenting results by relying exclusively on publicly available pre-trained models. Essentially, we focused on distribution shifts where neural networks tend to underperform and selected datasets within each domain based on their popularity and usage in the community.

We deliberately excluded VTAB-1k datasets because they consist of subsets from multiple datasets, conflicting with our focus on evaluating models trained within a single domain. VTAB-1k evaluates performance across diverse datasets, making it more suitable for studying generalization across tasks rather than distribution shifts within a domain. This fundamental difference in scope is why we did not include VTAB-1k in our evaluation.

We have updated Section 4.1 to include the selection criteria for primarily using ViTs and ResNets due to their differing inductive biases under "DNN model usage and selection" and the dataset selection criteria in Table 2. Due to space constraints, adding a new subsection was challenging, so these updates were integrated into existing sections.

Q3: ILCs on generic pre-trained models on dataset shifts

Thank you for raising this interesting question. Our work closely follows the last-layer retraining literature, and we deliberately restricted the scope to distribution shifts within a dataset, rather than transfer across datasets.

Testing ILCs on generic pre-trained models like ImageNet-pretrained ViTs or DinoV2 would not align with our zero-shot framework, a key contribution of our work, where no OOD data is used for training. These models inherently involve pretraining on diverse datasets, which conflicts with the assumptions of our setup. While testing such models would be interesting, it falls outside the intended scope of our current study.

Dear reviewer TgHf,

Thank you for the comment. We amended the text to reflect your suggestion and additionally explicitly indicated the distributions from which the dataset is drawn in both scenarios. We would be happy to engage in further discussion if any questions arise.

I've checked the comments by the other reviewers, and I think I still recommend acceptance. Amongst the concerns, I think it is important that the authors put the answer to the question about feature extraction protocol from Reviewer P4oy into the paper.

Thank you for your insightful comments.

W1/Q1: Definition of “few-shot” may differ from conventional standards

We use the term "few-shot" to describe experimental setups where some OOD points are available at training time for probing. While Section 4.2.1 uses a significant portion of the test dataset, Section 4.2.2 explores settings with varying numbers of OOD points, including extremely low availability (as low as 1% of the test set). This corresponds to the following sample sizes per class:

• CIFAR-100C: 2 samples
• CIFAR-10C: 5 samples
• CMNIST: 25 samples
• Waterbirds, CelebA, MultiCelebA: 10 samples

We believe that sample sizes of this magnitude are consistent with the few-shot setting, as they represent minimal data availability while still enabling meaningful evaluation.

W2/Q2: Reason for ID-trained probes improving OOD generalization isn't clear

We agree that the result may seem counterintuitive. In Section 5.2 (Feature Sensitivity in Intermediate and Penultimate Layers), we define a global metric to measure how sensitive each layer is to distribution shifts. The analysis shows that intermediate layers are consistently less sensitive to such shifts compared to the penultimate layer.

With this setup in mind, our intuition is as follows: intermediate layers capture more general and robust features than the penultimate layer. In contrast, the penultimate layer is optimized for in-distribution performance and is therefore more sensitive to shifts. For example, if an image is perturbed by noise, the features in the penultimate layer may change significantly, while those in intermediate layers remain more stable. Classifiers built on these robust intermediate-layer features are better suited for generalization, even in the zero-shot setting. This is because the OOD representations lie closer to those of ID, enabling the classifier to correctly generalize to OOD points.

W3/Q3 The use of a single-layer linear classifier at an intermediate layer may restrict its ability to capture more complex feature relationships

Thank you for pointing this out. While our goal was to present a simple method and challenge previous works claiming the transferability of penultimate-layer features, we agree that incorporating a more complex classifier is an important consideration.

To address this, we conducted additional experiments on CIFAR-10C and CIFAR-100C using ResNet18 and ViT models with an MLP to assess both penultimate and intermediate layer representations. The results, detailed in Appendix C.2 of the updated PDF, showed negligible performance improvements over linear probes for the penultimate layer representations but some improvement for intermediate layer representations. This suggests that while intermediate layers benefit from more complex probes, the limited performance of penultimate layers is not due to the probe’s capacity but rather the absence of sufficient learnable information in the representations.

W4: Relation to previous work in Neural Collapse

Thank you for this excellent reference. We believe [1] complements our work, with key differences as follows:

1. [1] focuses on transfer to new datasets (e.g., from ImageNet-1k pre-trained models), whereas our work examines distribution shifts within the same dataset, including visual shifts (ImageNet, CIFAR) and subpopulation or conditional shifts.
2. [1] proposes fine-tuning the layer with the greatest neural collapse (NC$_1$) on the downstream dataset alongside a linear classifier on that layer’s features, whereas our approach keeps the model parameters fixed and does not involve any parameter updates.

To further explore the role of Neural Collapse in dataset-specific models, we followed the setup of [1] and measured NC$_1$ under CIFAR-10C and CIFAR-100C distribution shifts. These results, detailed in Appendix C.1 of the updated PDF, reveal that the penultimate layer consistently exhibits the highest NC$_1$, even under distribution shifts. This contrasts with the findings in [1], which report dataset-dependent variability in the most-collapsed layer. We hypothesize this difference arises because [1] studies general pre-trained models, while our experiments focus on dataset-specific models.

Since the discussion phase is closing soon, we wanted to check if our response has addressed your concerns. We’d be happy to hear your thoughts and engage in further discussion if needed.