Test-time Adaptation for Regression by Subspace Alignment

Dear reviewer gBNN,

Thank you for your careful reading.

Weakness 1

The description of Figure 1 is too simple to understand the procedure of the proposed method. Some key variables (i.e., gφ and hψ) should be introduced. For easy understand, I suggest the authors provide a step-by-step description of the workflow shown in the figure.

Thank you for your suggestion. We updated the figure by adding definitions of the key variables and revising the description to make our method procedure clearer in the revised paper.

Weakness 2

The used compared methods are all published before 2023, how the proposed method compares to or differs from some newer approaches, such as “Backpropagation-free Network for 3D Test-time Adaptation, CVPR 2024”, “Improved Self-Training for Test-Time Adaptation, CVPR, 2024”.

Thank you for your suggestion. Even in recent works, TTA for regression has not been explored yet. The suggested two papers are also designed for classification and include classification-specific components that cannot be rendered to regression such as entropy. We have compared TTA methods as much as possible in the experiment.

We added the suggested papers in the related work (Section 5.3) in the revised paper.

Weakness 3

Why no results for RSD are given in Table 3? Please provide the missing results for RSD in Table 3 or explain why these results were not included.

Thank you for your clarification. Although we tried RSD on the California Housing dataset, it was numerically unstable and did not work. As we discussed in Section 4.3.2, this is because RSD requires SVD on the target feature batch in every iteration. We observed that SVD resulted in convergence error on the California Housing dataset.

Weakness 4

What is the visualization effect of the proposed method? Provide T-SNE rendering.

Thank you for your suggestion. In Figure 4 of Appendix D.2, we visualized the features by PCA. We also computed the optimal transport distance (OTD) between the source and target features in the principal subspace and found that our SSA produces a smaller OTD, i.e., a smaller feature gap.

Here, we added UMAP visualization of top $K=100$ principal components in Figure 5 to see the relations not limited within top-2 principal components. We used UMAP instead of t-SNE because UMAP can first learn the mappng of the source featurs before TTA, which are shared among the methods, and then map additional features. We mapped the target features after TTA using the learned UMAP mapping. The target feature distribution becomes closer to the source one with our SSA.

Weakness 5

What is the limitation of the proposed method? Are there any situations where SSA might not perform well, or any assumptions it makes that might not hold in all regression tasks?

Thank you for your clarification. As we discussed in Appendix A, one limitation is that our SSA assumes a covariate shift, where $p(y|x)$ does not change. For distribution shifts where $p(y|x)$ changes, e.g., concept drift, we will need not only to align $p(z)$ but also to address the change of the label distribution. For concept drift, one may combine label shift adaptation techniques such as [1] and [2]. Addressing concept drift is one direction of future work.

[1] Park et al., Label Shift Adapter for Test-Time Adaptation under Covariate and Label Shifts, ICCV2023.
[2] Sun et al., Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlations, NeurIPS2023.

Dear reviewer xN4r,

Thank you for your insightful comments.

Weakness 1

In the related work, feature alignment methods for TTA are reviewed with the claim, “Although some of these methods are directly applicable to regression, we have observed that they are not effective or even degrade regression performance.” However, these methods are not included in the experiments, leaving this claim unsupported.

Thank you for pointing it out. We have already included feature alignment methods that can be directly rendered to regression as much as possible: BN-adapt and FR as shown in Tables 2-5. We have observed they degrade the performance when rendered to regression. The other feature alignment methods include classification-specific components that cannot be applied to regression, e.g., sample weighting by confidence in DELTA[1], class-wise feature statistics in CFA[2] and CAFA[3], and class-categorical prediction in CAFe[4].

[1] Zhao et al., DELTA: degradation-free fully test-time adaptation, ICLR2023.
[2] Kojima et al., Robustifying vision transformer without retraining from scratch by test-time class-conditional feature alignment, IJCAI2022.
[3] Jung et al., CAFA: Class-Aware Feature Alignment for Test-Time Adaptation, ICCV2023.
[4] Adachi et al., Covariance-aware feature alignment with pre-computed source statistics for test-time adaptation to multiple image corruptions, ICIP2023.

Weakness 2

In Table 1, the subspace dimension for Biwi Kinect is shown as ‘34.5’, which is unclear, as dimensions are typically integers.

Thank you for your clarification. For Biwi Kinect, we computed the numbers of the feature dimensions for the six models trained on the six combinations of the domain and targets, i.e., {male, female} $\times$ {pitch, roll, yaw}. We reported the mean of the six numbers of feature dimensions. We clarified this point in the revised paper. The actual number of the feature dimensions of each model is as follows:

Table A: The number of feature dimensions on Biwi Kinect.

Weakness 3

MLPs and CNNs seem to produce embeddings with different ranks, with MLPs showing less low-rank behavior. A discussion or remark on this difference could provide useful insights.

Thank you for pointing it out. For the California Housing dataset, we used an MLP with only five layers, which has a lower capacity than the ResNets used for the other datasets. One explanation for the MLP's less low-rank behavior is that the complexity of the task is high relative to the model capacity. For instance, when training the ResNet-26, which has high capacity, on the SVHN, which has low complexity, the subspace dimension is much smaller than the original dimensions.

We added the explanation to Section 4.3.1 in the revised paper.

Dear reviewer 66U8,

Thank you for your thoughtful feedback.

Weakness 1

SSA assumes covariate shift, where p(y|x) remains unchanged. The paper does not address distribution shifts where p(y|x) changes, such as concept drift, limiting its applicability in broader scenarios.

Thank you for pointing it out. As we discussed in Appendix A, one limitation is that our SSA assumes a covariate shift, where $p(y|x)$ does not change. But covariate shift is also one realistic distribution shift that has been actively studied in TTA [1]. For distribution shifts where $p(y|x)$ changes, e.g., concept drift, we will need not only to align $p(z)$ but also to address the change of the label distribution. For concept drift, one may combine label shift adaptation techniques such as [2] and [3]. Addressing concept drift is one direction of future work.

[1] Liang et al., A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts, IJCV, 2023.
[2] Park et al., Label Shift Adapter for Test-Time Adaptation under Covariate and Label Shifts, ICCV2023.
[3] Sun et al., Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlations, NeurIPS2023.

Weakness 2

While the paper mentions the selection of the parameter λ, it lacks a detailed discussion on the impact of other hyperparameters, which could affect the model's generalizability and adaptability.

Thank you for pointing it out. Although the only hyperparameter of our method is the number of subspace dimensions $K$, we performed additional investigation on the learning rate and batch size, which are important hyperparameters in practice. The following tables show the results:

Table A: Impact of the learning rate on SVHN-MNIST

Table B: Impact of the batch size on SVHN-MNIST

Table C: Impact of the learning rate on UTKFace (averaged over corruption types)

Table D: Impact of the batch size on UTKFace (averaged over corruption types)

Table E: Impact of the learning rate on Biwi Kinect (averaged over genders and targets)

Table F: Impact of the batch size on Biwi Kinect (averaged over genders and targets)

As we can observe from the tables, typical ranges of the learning rate and batch size produce competitive performance. For the batch size, a larger batch size results in better performance since the estimation of feature mean and variance becomes more accurate. But batch sizes $\geq 16$ produce competitive performance.

We also added more detailed results, i.e., corruption-wise on UTKFace and gender/target-wise on Biwi Kinect, in Tables 21-26 of Appendix D.8 in the revised paper.

Weakness 3

The paper does not discuss the computational complexity of SSA, particularly its performance with large-scale datasets, which is crucial for practical applications.

Thank you for pointing it out. Although benchmarking on large-scale datasets is crutial for practical applications, there are no large-scale regression datasets including distribution shifts like ImageNet family in classification. As our method can run in an batched online setting as shown in Tables 29-31 in Appendix D.10 of the revised paper, we believe that our method can scale up to large-scale datasets. Also, one may combine continual learning techniques like EWC [4] to stabilize adaptation. Constructing large-scale benchmarks for regression TTA and scaling up our method is future work.

[4] Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks.

Dear reviewer t9WM,

Thank you for your constructive comments.

Weakness 1 and Question 1

In Section 3.1 of the article, the use of "two diagonal Gaussian distributions" is not adequately justified, and the advantages and limitations of employing such distributions are not discussed.

Why are "two diagonal Gaussian distributions" used in Chapter 3? What is the rationale behind this choice?

Thank you for your clarification. Assuming Gaussian distributions is reasonable for the following reasons:

First, we can easily compute the KL divergence in a closed form by assuming the diagonal Gaussian distribution, as in Equation (3). Second, as shown in Figure 3, features follow a Gaussian-like distribution when projected onto the feature subspace detected by subspace detection. This is due to the central limit theorem described in Section 4.3.4, i.e., the features are more likely to follow a Gaussian distribution in the subspace as the number of the original feature dimensions increases. Moreover, since subspace detection uses the PCA, the features projected onto the subspace are decorrelated. Thus, assuming that each dimension is independent, i.e., diagonal, is also reasonable.

We added explanation to use diagonal Gaussian to Section 3.2 of the revised paper.

One limitation of employing Gaussian distributions is that features are not likely to follow Gaussian distributions when the number of dimensions of the original feature space is small. In such a case, one may employ another distribution, e.g., a histogram used in BUFR [1]. Addressing such a case is one direction of future work.

[1] Eastwood et al., Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration. ICLR2022.

Weakness 2

The proposed method in this paper exhibits limited originality, as its various components are commonly found in existing models.

Thank you for pointing it out. Our contributions include the finding that features of a trained regression model are distributed in only a small subspace. This finding is novel, and we believe that our method is also simple yet novel since it is based on the finding. Moreover, incorporating the idea of feature alignment in the subspace for TTA is also novel.

Weakness 3

The dataset utilized in this study has a relatively small sample size, which weakens the persuasiveness of the findings.

Thank you for pointing it out. We have evaluated methods on various regression datasets with distribution shifts as much as possible. Since TTA for regression is an unexplored field, constructing common regression benchmarks including large-scale datasets is an important future work.

there is a lack of comparative algorithms from the past three years, resulting in insufficient theoretical validation.

This is because there are no comparable baselines in our regression TTA setting except for BN-adapt and FR. We have compared methods that can be directly rendered to regression as much as possible. This is the first study that explicitly addresses TTA for regression, and our method can serve as a good baseline for future work in this setting.

Although we have discussed TTA methods from the past three years in Section 5.3, they include classification-specific components that cannot be rendered to regression, e.g., sample weighting by confidence in DELTA[2], class-wise feature statistics in CFA[3] and CAFA[4], and class-categorical prediction in CAFe[5].

[2] Zhao et al., DELTA: degradation-free fully test-time adaptation, ICLR2023.
[3] Kojima et al., Robustifying vision transformer without retraining from scratch by test-time class-conditional feature alignment, IJCAI2022.
[4] Jung et al., CAFA: Class-Aware Feature Alignment for Test-Time Adaptation, ICCV2023.
[5] Adachi et al., Covariance-aware feature alignment with pre-computed source statistics for test-time adaptation to multiple image corruptions, ICIP2023.

the measurement standard is solely based on R²; incorporating additional metrics such as RMSE and RMAE would enhance the evaluation.

Thank you for your suggestion. We added MAE to Tables 2, 3, 27 and 28 for SVHN-MNIST, California Housing, UTKFace, and Biwi Kinect, respectively, in the revised paper. Our method consistently outperformed the baselines also in the additional metric.