PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation

Thank you for your valuable and constructive comments. We appreciate your recognition of our novel and insightful perspective on CTTA based on the geometric structure of pre-trained weights, as well as your positive feedback on the clarity of our writing, the motivation conveyed through well-designed experiments and figures, and the comprehensive evaluation across multiple benchmarks with detailed ablation studies. Below, we address your concerns and questions individually:

Q1: Extension to other network architectures beyond Transformer

Thank you for your forward-looking suggestion. The key insight of PAID, preserving the pairwise angular structure between weight vectors to retain domain-invariant semantics, can be formally extended to convolutional networks, where each convolutional layer is also a parameterized linear transformation.

In a standard convolutional layer, each output channel corresponds to a filter of shape $(C_{\text{in}}, k_h, k_w)$. We flatten this into a vector of length $D = C_{\text{in}} \cdot k_h \cdot k_w$, treating each filter as a vector in $\mathbb{R}^D$. Stacking these column-wise yields a weight matrix $W \in \mathbb{R}^{D \times C_{\text{out}}}$, which we decompose as:

$W = M \odot \hat{W},$

where $\hat{W}$ contains unit-norm columns and $M \in \mathbb{R}^{1 \times C_{\text{out}}}$ stores the magnitudes. During adaptation, we freeze $\hat{W}$ and update $M$ along with a learnable orthogonal matrix $O \in \mathbb{R}^{C_{\text{out}} \times C_{\text{out}}}$, constructed via Householder reflections. The updated weights become:

$W' = M \odot (\hat{W} \cdot O).$

Here, $O$ operates along the output channel dimension. The spatial structure within each filter remains intact because we do not rotate across spatial positions. Notably, flattening convolution filters and treating them as vectors is a standard practice in low-rank adaptation and model compression literature. Reshaping back after rotation ensures that spatial locality is fully preserved during convolution.

To validate this extension, we follow standard CTTA setups used in prior works: WideResNet-28 on CIFAR-10-C, ResNeXt-29 on CIFAR-100-C, and ResNet-50 on ImageNet-C. We select representative and open-sourced baselines for comparison, using benchmark results that have been consistently reported and validated across multiple CTTA studies. The results show consistent performance gains across all convolutional backbones, indicating that PAID generalizes beyond ViTs and is also applicable to CNNs.

To further explain why PAID remains valid in convolutional networks, we revisit several prior studies on the geometric structure of convolutional filters. DCNet [18] shows that the convolution operation can be reformulated as an inner product and decomposed into magnitude and angular: the magnitude better models intra-class variation, while the angular captures semantic differences. SphereConv [15] further demonstrates that learning only the angular component of convolutional filters is sufficient for semantic classification. These findings indicate that the angular structure in CNNs also carries domain-invariant semantics. Therefore, applying PAID’s strategy to convolutional networks is not only formally valid, but also empirically supported by prior work.

Q2: Blurry text in Figure 5

Thank you very much for pointing out this typo. We will replace the figure with a clearer one and use a larger font in the revision.

(1) Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation, ECCV 2024.

(2) Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptation, WACV 2024.

Thank you for your valuable and constructive comments. We appreciate your recognition of our key empirical finding, as well as your positive feedback on our well-motivated approach, the comprehensive supporting experiments, and the clarity and organization of the paper. Below, we address your concerns and questions individually:

Q1: Theoretical analysis for why pairwise angular structure is robust to domain-specific shifts but sensitive to semantic changes

To address your question, we provide a formal analysis using a simplified linear classification model. Specifically, we prove that:

• As the domain-specific shift applies the same transformation to all input samples regardless of class, the optimal classifier in the target domain can be written as a shared linear transformation of the source classifier, under mild and mathematically defined conditions. This transformation preserves all pairwise angles between class weights.
• In contrast, semantic changes induce class-specific input transformations, which break this shared structure and necessarily alter the pairwise angular geometry.

This supports the core design of PAID: using a orthogonal rotation and magnitude adjustment is sufficient for adapting to domain-specific corruptions, but insufficient for adapting to semantic shift, where the angular structure must be re-learned. We now present the mathematical proof in detail.

---

Mathematical Formulation

Linear Classification Model

Let $x \in \mathbb{R}^a$ be the input vector, and for each class $c \in \{1, ..., C\}$, let $w_c \in \mathbb{R}^a$ be the classifier weight vector. The predicted class is:

$\hat{y}(x) = \arg\max_{c} w_c^\top x.$

We compare two settings:

---

Case 1: Domain-specific Degradation (Class-agnostic input transformation)

We assume the degraded input is obtained by applying the same transformation to all inputs:

$x' = d(x),$

and that $d$ is differentiable near each $x$. We approximate $d(x)$ by its first-order Taylor expansion around clean data:

$d(x) \approx Sx + q,$

where $S \in \mathbb{R}^{a \times a}$ is the Jacobian matrix at $x$, assumed to be invertible and independent of class $y$. We formalize this assumption as:

Assumption 1 (Class-agnostic linear perturbation):
There exists an invertible matrix $S \in \mathrm{GL}(a)$ such that for all $x$, $\quad d(x) = Sx + q, \quad \text{with } S \text{ independent of class } y$.

We define the optimal classifier in the target domain as the one that operates on $x' = Sx$ but maintains the same class boundaries in the original space. This gives:

$w'_c = S^{-\top} w_c, \quad \forall c.$

Now we examine the effect of this transformation on pairwise angles. We perform the polar decomposition:

$S^{-\top} = RH,$

where $R \in \mathbb{R}^{a \times a}$ is orthogonal $( R^\top R = I )$, and $H \in \mathbb{R}^{a \times a}$ is symmetric positive definite. We now examine the transformed weights:

$\tilde{w}_c = RH w_c.$

We compute the cosine similarity between any two transformed weights:

$\cos \angle(\tilde{w}_c, \tilde{w}_d) = \frac{(RH w_c)^\top (RH w_d)}{\|RH w_c\| \cdot \|RH w_d\|} = \frac{w_c^\top H^2 w_d}{\|H w_c\| \cdot \|H w_d\|}.$

To ensure angles are preserved, we introduce:

Assumption 2 (Near-isotropic perturbation):
The matrix $H$ satisfies $H^2 \approx \gamma^2 I$ for some scalar $\gamma > 0$.
This holds when the degradation has approximately equal effect in all directions (e.g., blur, light noise, JPEG), and ensures angle preservation:

$\cos \angle(\tilde{w}_c, \tilde{w}_d) \approx \cos \angle(w_c, w_d).$

Thus, under Assumptions 1 and 2, pairwise angular structure remains invariant under domain-specific shifts.

---

Case 2: Semantic Change (Class-dependent input transformation)

Suppose the semantic shift modifies each class's input differently:

$x' = d_y(x), \quad \text{with } d_y(x) \approx S_y x + q_y,$

where $S_y \in \mathrm{GL}(a)$ varies with class $y$. Then the optimal classifier becomes:

$w''_c = S _{y=c} ^{-\top} w_c.$

In this case, each weight undergoes a different transformation. Unless all $S_y$ are identical up to scaling or rotation, there exists no single matrix $T \in \mathbb{R}^{a \times a}$ such that $w''_c = T w_c$ for all $c$. Hence:

There is no shared transformation that preserves the entire angular structure. The pairwise angle matrix must differ from that in the original domain. This proves that semantic changes necessarily break the pairwise angular structure.

---

Final Conclusion

• When domain shifts are class-agnostic and locally differentiable, they can be represented by a shared invertible matrix $S$, and the optimal classifier corresponds to applying the inverse transpose $S^{-\top}$ to all weights.
• This transformation preserves angles up to a near-isotropic assumption, justifying why PAID only needs to learn a shared orthogonal rotation and magnitude adjustment.
• When the shift is class-dependent, this shared structure no longer exists, and angular geometry must change.

This provides a direct theoretical justification for your question and explains the observations of our paper.

Q2: Extension to convolutional or non-transformer architectures

Thank you for your forward-looking suggestion. The key insight of PAID, preserving the pairwise angular structure between weight vectors to retain domain-invariant semantics, can be formally extended to convolutional networks, where each convolutional layer is also a parameterized linear transformation.

In a standard convolutional layer, each output channel corresponds to a filter of shape $(C_{\text{in}}, k_h, k_w)$. We flatten this into a vector of length $D = C_{\text{in}} \cdot k_h \cdot k_w$, treating each filter as a vector in $\mathbb{R}^D$. Stacking these column-wise yields a weight matrix $W \in \mathbb{R}^{D \times C_{\text{out}}}$, which we decompose as:

$W = M \odot \hat{W},$

where $\hat{W}$ contains unit-norm columns and $M \in \mathbb{R}^{1 \times C_{\text{out}}}$ stores the magnitudes. During adaptation, we freeze $\hat{W}$ and update $M$ along with a learnable orthogonal matrix $O \in \mathbb{R}^{C_{\text{out}} \times C_{\text{out}}}$, constructed via Householder reflections. The updated weights become:

$W' = M \odot (\hat{W} \cdot O).$

Here, $O$ operates along the output channel dimension. The spatial structure within each filter remains intact because we do not rotate across spatial positions. Notably, flattening convolution filters and treating them as vectors is a standard practice in low-rank adaptation and model compression literature. Reshaping back after rotation ensures that spatial locality is fully preserved during convolution.

To validate this extension, we follow standard CTTA setups used in prior works: WideResNet-28 on CIFAR-10-C, ResNeXt-29 on CIFAR-100-C, and ResNet-50 on ImageNet-C. We select representative and open-sourced baselines for comparison, using benchmark results that have been consistently reported and validated across multiple CTTA studies. The results show consistent performance gains across all convolutional backbones, indicating that PAID generalizes beyond ViTs and is also applicable to CNNs.

To further explain why PAID remains valid in convolutional networks, we revisit several prior studies on the geometric structure of convolutional filters. DCNet [18] shows that the convolution operation can be reformulated as an inner product and decomposed into magnitude and angular: the magnitude better models intra-class variation, while the angular captures semantic differences. SphereConv [15] further demonstrates that learning only the angular component of convolutional filters is sufficient for semantic classification. These findings indicate that the angular structure in CNNs also carries domain-invariant semantics. Therefore, applying PAID’s strategy to convolutional networks is not only formally valid, but also empirically supported by prior work.

Q3: Typo in Figure 5

Thank you very much for pointing out this typo. We will correct this error in the revision.

(1) Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation, ECCV 2024.

(2) Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptation, WACV 2024.

Thank you for your valuable and constructive comments. We appreciate your recognition of the importance of the CTTA setting, the clarity of our writing, the thorough experimental validation across multiple benchmarks, and the novelty of applying weight decomposition in the CTTA context. Below, we address your concerns and questions individually:

Q1: What is the intuition behind the performance drop with small batch sizes

As shown in Figure 6 (c), most existing CTTA methods, including ours, exhibit a notable performance drop as the batch size decreases. This behavior is not specific to our approach, similar trends have been reported in prior works such as [24] (Figure 4 (d)) and [28] (Table 5), where small-batch scenarios consistently lead to degraded adaptation performance.

For our method in particular, the primary factor is the reliability of target feature statistics. In Equation (9), we perform alignment between source and target distributions using the mean and standard deviation computed from the current test batch. When the batch size is small, these estimates become highly variable and noisy, which results in unstable gradients and impairs the optimization process. This instability ultimately hinders adaptation effectiveness.

In contrast, larger batch sizes produce more reliable and smoother estimates of distributional statistics, effectively reducing variance in the adaptation signal. This leads to more stable parameter updates and improved learning dynamics. Empirically, we observe that performance becomes acceptable when the batch size exceeds 4, and saturates when the batch size is greater than 32.

To mitigate this limitation, we also explore a simple enhancement: applying an Exponential Moving Average (EMA) to the target statistics. This temporal smoothing helps stabilize the alignment objective under small-batch regimes and partially alleviates the performance drop when the batch size is 1 or 2. The corresponding results are provided below:

Q2: Clarification and Discussion on the Fairness of Baseline Comparisons and Hyperparameter Tuning

We sincerely thank the reviewer for raising this practically relevant concern, which prompted us to further reflect on the challenges faced by the CTTA community in real-world deployment.

Clarification on the fairness of baseline comparisons

The baseline results reported in our experiments are all based on benchmarks widely adopted in prior CTTA studies. These results have not only been repeatedly validated in multiple papers, but also follow the hyperparameter settings specified in the original papers or open-source code provided by the respective authors. For example, ViDA uses a learning rate of 1e-4 on CIFAR-C, but 5e-7 on ImageNet-C; C-MAE adopts 1e-5 on CIFAR-C, while using 1e-3 on ImageNet-C. Therefore, we ensure that all baseline comparisons are conducted under reasonable and recommended settings, which guarantees the fairness and reliability of our experimental results.

Discussion on hyperparameter sensitivity and manual tuning

We agree with the reviewer that manual tuning may present a limitation in real-world deployment. In fact, sensitivity to hyperparameters such as learning rate and regularization coefficients is a widely recognized challenge across CTTA community, as discussed in the prior work (1). We believe that improving the robustness of CTTA methods to hyperparameters, or exploring automated tuning strategies, is one of the most important directions for future research.

That said, we also note that manual tuning still holds practical value at the current stage and offers useful empirical guidance for deployment. Below we share our observation regarding hyperparameter selection in our method:

• Loss balancing coefficient λ: We found that image resolution significantly affects statistical variations. For low-resolution images (e.g., CIFAR with original size 32×32), upsampling to 384×384 amplifies local noise, making variance alignment more important. For high-resolution images (e.g., ImageNet-C with native resolution 224×224), global intensity shifts dominate, so aligning feature means becomes more crucial. This observation provides an empirical basis for choosing λ under different pre-processing conditions.

In addition, we observed that when using the same learning rate settings for both CIFAR-C and ImageNet-C, our method still achieves competitive performance. This demonstrates that our method does not rely heavily on excessive tuning.

Q3: Extension to convolutional networks

Thank you for your forward-looking suggestion. The key insight of PAID, preserving the pairwise angular structure between weight vectors to retain domain-invariant semantics, can be formally extended to convolutional networks, where each convolutional layer is also a parameterized linear transformation.

In a standard convolutional layer, each output channel corresponds to a filter of shape $(C_{\text{in}}, k_h, k_w)$. We flatten this into a vector of length $D = C_{\text{in}} \cdot k_h \cdot k_w$, treating each filter as a vector in $\mathbb{R}^D$. Stacking these column-wise yields a weight matrix $W \in \mathbb{R}^{D \times C_{\text{out}}}$, which we decompose as:

$W = M \odot \hat{W},$

where $\hat{W}$ contains unit-norm columns and $M \in \mathbb{R}^{1 \times C_{\text{out}}}$ stores the magnitudes. During adaptation, we freeze $\hat{W}$ and update $M$ along with a learnable orthogonal matrix $O \in \mathbb{R}^{C_{\text{out}} \times C_{\text{out}}}$, constructed via Householder reflections. The updated weights become:

$W' = M \odot (\hat{W} \cdot O).$

Here, $O$ operates along the output channel dimension. The spatial structure within each filter remains intact because we do not rotate across spatial positions. Notably, flattening convolution filters is a standard practice in low-rank adaptation and model compression literature. Reshaping back ensures that spatial locality is fully preserved during convolution.

To validate this extension, we follow standard CTTA setups used in prior works: WideResNet-28 on CIFAR-10-C, ResNeXt-29 on CIFAR-100-C, and ResNet-50 on ImageNet-C. We select representative and open-sourced baselines for comparison, using benchmark results that have been consistently reported and validated across multiple CTTA studies. The results show consistent performance gains across all convolutional backbones, indicating that PAID generalizes beyond ViTs and is also applicable to CNNs.

To further explain why PAID remains valid in convolutional networks, we revisit several prior studies on the geometric structure of convolutional filters. DCNet [18] shows that the convolution operation can be reformulated as an inner product and decomposed into magnitude and angular: the magnitude better models intra-class variation, while the angular captures semantic differences. SphereConv [15] further demonstrates that learning only the angular component of convolutional filters is sufficient for semantic classification. These findings indicate that the angular structure in CNNs also carries domain-invariant semantics. Therefore, applying PAID’s strategy to convolutional networks is not only formally valid, but also empirically supported by prior work.

Q4: Limited performance improvement in Table 1 and Table 4

We sincerely thank the reviewer for the rigorous examination of the results. In response to the concern, we would like to clarify its significance from two perspectives: empirical consistency and research value.

First, the performance improvements are consistent across datasets and tasks, rather than accidental fluctuations.

Our method achieves a +1.5% gain on the CIFAR100-C benchmark, and a +1.6% gain on CIFAR10-C. On the ImageNet-C benchmark, which is the largest and most diverse, our method maintains a 0.3% lower mean error than C-MAE. Moreover, on the ACDC segmentation benchmark, our method outperforms ViDA by +0.3% in mIoU. These consistent improvements across different datasets and tasks demonstrate the robustness and generalizability of our method, suggesting that its effectiveness does not rely on any data distribution from a particular benchmark.

Second, the core contribution of this paper lies in proposing a novel direction for CTTA research, which we believe carries substantial value.

While most existing methods focus on exploiting target stream data, PAID is the first to explore pre-trained weights as more than static initializations. By preserving the pairwise angular structure of weight directions during adaptation, PAID enables a structured update that respects the semantic organization encoded during pre-training. This perspective establishes a new design paradigm for CTTA, with the potential to inspire a range of principled adaptation strategies and lay a solid foundation for future performance improvements.

(1) On Pitfalls of Test-time Adaptation, ICLR 2023.

(2) Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation, ECCV 2024.

(3) Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptation, WACV 2024.

Dear Reviewer jqAQ,

The discussion phase ends in ~48 hours and you have not engaged with the rebuttal. It is a necessary and important part of the review process and the responsibility of a reviewer for NeurIPS 2025.

Please remember to acknowledge the rebuttal and update your review by the discussion deadline. I look forward to your updated review!

Recall that the rebuttal acknowledgement, final rating, and final justification are required by NeurIPS 2025.

Best,
Your AC

Thank you for your valuable and constructive comments. We appreciate your recognition of our work's novel perspective on CTTA from the view of weight-space geometry and the tight algorithmic design based on this insight. Below, we address your concerns and questions individually:

Q1: Clarity about the $Z_{t}^{T}$ in Eq. (9) and the procedure for computing source and target statistics

Thank you very much for pointing out this ambiguity. We agree that our notation in Eq. (9) was unclear and could lead to confusion.

To clarify: in $Z_{t}^{T}$, the subscript $t$ indicates that the features are from the target domain (in contrast to source domain $s$), and the superscript $T$ refers to the test-time step, meaning the index of the currently arriving target batch in the continual adaptation process.

For each incoming batch $B_{t}^{T}$, we extract features using the CLS token output after the final layer normalization and before the classification head in the ViT-Base. This CLS token serves as the feature representation for each image in the batch.

The source domain statistics $(\mu_s, \sigma_s)$ are computed in exactly the same way: we randomly sample 500 source images prior to test-time, extract their CLS token features from the same location, and compute their mean and standard deviation. These statistics are pre-computed once and reused throughout adaptation, with no further access to the source data.

We will revise Section 3.3 to explicitly clarify these. We will also release our code on GitHub so that the details can be better understood.

Q2: Extension to convolutional networks

Thank you for your forward-looking suggestion. The key insight of PAID, preserving the pairwise angular structure between weight vectors to retain domain-invariant semantics, can be formally extended to convolutional networks, where each convolutional layer is also a parameterized linear transformation.

In a standard convolutional layer, each output channel corresponds to a filter of shape $(C_{\text{in}}, k_h, k_w)$. We flatten this into a vector of length $D = C_{\text{in}} \cdot k_h \cdot k_w$, treating each filter as a vector in $\mathbb{R}^D$. Stacking these column-wise yields a weight matrix $W \in \mathbb{R}^{D \times C_{\text{out}}}$, which we decompose as:

$W = M \odot \hat{W},$

where $\hat{W}$ contains unit-norm columns and $M \in \mathbb{R}^{1 \times C_{\text{out}}}$ stores the magnitudes. During adaptation, we freeze $\hat{W}$ and update $M$ along with a learnable orthogonal matrix $O \in \mathbb{R}^{C_{\text{out}} \times C_{\text{out}}}$, constructed via Householder reflections. The updated weights become:

$W' = M \odot (\hat{W} \cdot O).$

Here, $O$ operates along the output channel dimension. The spatial structure within each filter remains intact because we do not rotate across spatial positions. Notably, flattening convolution filters and treating them as vectors is a standard practice in low-rank adaptation and model compression literature. Reshaping back after rotation ensures that spatial locality is fully preserved during convolution.

To validate this extension, we follow standard CTTA setups used in prior works: WideResNet-28 on CIFAR-10-C, ResNeXt-29 on CIFAR-100-C, and ResNet-50 on ImageNet-C. We select representative and open-sourced baselines for comparison, using benchmark results that have been consistently reported and validated across multiple CTTA studies. The results show consistent performance gains across all convolutional backbones, indicating that PAID generalizes beyond ViTs and is also applicable to CNNs.

To further explain why PAID remains valid in convolutional networks, we revisit several prior studies on the geometric structure of convolutional filters. DCNet [18] shows that the convolution operation can be reformulated as an inner product and decomposed into magnitude and angular: the magnitude better models intra-class variation, while the angular captures semantic differences. SphereConv [15] further demonstrates that learning only the angular component of convolutional filters is sufficient for semantic classification. These findings indicate that the angular structure in CNNs also carries domain-invariant semantics. Therefore, applying PAID’s strategy to convolutional networks is not only formally valid, but also empirically supported by prior work.

Q3: Motivation and sensitivity analysis of using householder reflection for orthogonal transformation

We thank the reviewer for this constructive question. We choose to use the Householder reflection primarily due to computational efficiency considerations. Below, we elaborate on our motivation through a comparison with alternative implementations and an analysis of sensitivity.

Alternative: cayley transformation

Another common mathmatics tool for constructing orthogonal matrices $R \in \mathbb{R}^{d \times d}$ is the cayley transformation, which maps a skew-symmetric matrix $A \in \mathbb{R}^{d \times d}$, where $A^\top = -A$, to an orthogonal matrix:

$R = (I + A)(I - A)^{-1}.$

This formulation underlies two recently proposed methods:

• OFT [20]: Constructs an orthogonal matrix by applying the Cayley transformation to multiple learnable skew-symmetric blocks.
• BOFT (3): Builds a butterfly-style composition of sparse Cayley-transformed blocks for efficient approximation.

Comparison of parameter and computational complexity

We refer to the analysis in Section 3.2 and Table 1 of [21], which compares these two types of methods:

• Parameter complexity: Both Cayley-based methods and the Householder-based method require approximately $\mathcal{O}(d^2)$ parameters.
• Computational complexity: Cayley-based methods involve matrix inversions and dense multiplications, resulting in $\mathcal{O}(d^3)$ complexity. In contrast, the Householder-based method only involves vector inner products and scale-vector multiplications, yielding a lower cost of $\mathcal{O}(d^2)$.

This one-order-of-magnitude improvement in computational complexity makes Householder reflection a more efficient choice for constructing orthogonal matrices.

Sensitivity analysis

We construct orthogonal matrices following [20] and (3), and conduct a comparative experiment on the ImageNet-C benchmark. All methods perform similarly, with only a slight performance drop, suggesting that the overall performance is not sensitive to the specific choice of orthogonalization strategy.

That said, we recommend the Householder reflection approach. Given its comparable expressiveness and parameter complexity to Cayley-based methods, its lower computational cost makes it more suitable for CTTA, where efficient online adaptation and reduced overhead are essential.

(1) Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation, ECCV 2024.

(2) Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptation, WACV 2024.

(3) Parameter-efficient Orthogonal Finetuning via Butterfly Factorization, ICLR 2024.