Un-Mixing Test-Time Normalization Statistics: Combatting Label Temporal Correlation

We appreciate your constructive comments and suggestions, and they are exceedingly helpful for us in improving our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Comparison w.r.t ([a] RoTTA, [b] NOTE).

Thank you for your comment. In the Introduction, we reference studies ([a] RoTTA, [b] NOTE) that address adaptation under non-i.i.d. data streams. NOTE introduces instance-aware BN (IABN), and RoTTA proposes a robust batch normalization (RBN) module, which uses global statistics to robustly normalize feature maps. However, as our results show, these normalization layers alone struggle in non-i.i.d test-time adaptation scenarios. Their effectiveness is partly due to their use of a memory bank for sampling category-balanced data, thereby simulating an i.i.d. test-time adaptation environment. In contrast, UnMix-TNS's core concept is to maintain both current and past statistics of online test features across its K different components. For each test sample, we refine the most closely aligned components, allowing for the estimation of unbiased BN statistics without relying on a memory bank. Our experiments demonstrate that UnMix-TNS outperforms RBN and IABN in non-i.i.d setups, even when integrated with RoTTA and NOTE, as shown in Tables 1 and 2. This highlights the unique contribution and robustness of our approach in these challenging adaptation scenarios.

Hyperparameter $\alpha$.

Thank you for your query regarding the choice of setting the hyperparameter $\alpha$. As mentioned in Appendix A.2 of our manuscript, we analyze the source variance within the context of UnMix-TNS. Our analysis indicates that the source variance is contributed by (1) the average variance of individual components, and (2) the variance of the means of individual components. By setting $\alpha=0.5$, we ensure that these two contributions to the source variance are balanced and have equal weight. This balance is crucial for the effective functioning of UnMix-TNS.

Hyperparameter $\lambda$.

Thank you for highlighting the need for a clearer explanation regarding the hyperparameter $\lambda$ in Section 2.2.3 of our paper. Equations (12) and (13) are derived based on the principles of momentum batch normalization (MBN), and the hyperparameter $\lambda$ is set according to the MBN guidelines proposed by Yong et al. (2020). The primary objective of MBN is to ensure a consistent noise level introduced by BN across varying batch sizes. This consistency is crucial to maintain equivalent noise levels in both small and large batch scenarios. The tuning of $\lambda$ is integral to achieving this standardization based on the specific batch size employed. For a more comprehensive understanding of $\lambda$ and its significance in our method, we have included an in-depth explanation in the newly added Appendix A.4 of the manuscript.

Method comparison: 11 in Tables 1 and 2, only 7 in video dataset (Table 3).

Thank you for your comment regarding the number of methods included in the comparisons for the video dataset (Table 3). We aimed for a fair and relevant evaluation by aligning our experimental setup with that of LAME (Boudiaf et al., 2022), selecting top-performing baselines for comparison. The methods listed in Tables 1 and 2, which are not included in Table 3, either necessitate extensive reimplementation for application to video datasets or are not directly applicable to such datasets. This decision was made to ensure the most meaningful and practical comparison within the constraints of our experimental setup.

Query on UNMIX-TNS: parameter updates, BN layer changes, and forward pass requirements.

In response to the reviewer's queries about the UnMix-TNS method, as a standalone method, UnMix-TNS is designed to update only its internal statistics and requires only a single forward pass. However, when integrated with other TTA methods, UnMix-TNS solely replaces the Batch Normalization (BN) layers and behaves like the replaced BN in the same manner implemented in those TTA methods, including parameter updates. For instance, in a TTA method like TENT, the affine parameters of the BN layers are updated, and UnMix-TNS will follow this same update protocol. Conversely, in methods like LAME, the focus is on updating only the internal statistics of normalization layers. In summary, UnMix-TNS can either operate independently with a single forward pass or align with the update mechanisms of other TTA methods it is paired with.

Discrepancy in author's claim and Equations (12) & (13) on updating BN statistics in UNMIX-TNS.

We acknowledge that our initial phrasing may have been misleading. In practice, we update all K BN statistics components, each with varying assignment probabilities. We have updated the manuscript to clarify this point more accurately.

Thank you for your constructive comments and suggestions; and they are exceedingly helpful for us in improving our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Applicability of UnMix-TNS beyond vision tasks, e.g., NLP.

Thank you for your comment regarding the integration of UnMix-TNS with convolutional layers in computer vision tasks. Our work is specifically tailored to the batch normalization layer in image classification models. While the extension of UnMix-TNS to NLP tasks is an interesting concept, it falls outside the scope of this paper. This decision is guided by the prevalent use of layer normalization in NLP, which differs from our focus on batch normalization. In summary, exploring UnMix-TNS in NLP context would require a different approach, considering the distinct normalization techniques used in NLP.

Discussion on how to choose $K$ and $\lambda$.

Thank you for pointing out the need for a clearer explanation of the selection of $K$ and $\lambda$ in our paper. Regarding $K$, its choice is crucial, as emphasized in our future work. A single UnMix-TNS component can efficiently represent several similar classes (e.g., car, truck, bus), allowing for an accurate approximation of the test distribution with a lower $K$ value. This approach is particularly beneficial for datasets with distinct class similarities. However, in datasets with a large number of classes like CIFAR100-C, DomainNet-126, or ImageNet-C, aligning $K$ with the number of classes may lead to inefficiencies and an increased memory footprint. For instance, setting $K$ to 100 in CIFAR100-C tests led to reduced performance. The optimal $K$ varies with the testing scenario; larger $K$ values are necessary for mixed domains with diverse styles, while smaller $K$ values are sufficient in single or continual domains due to reduced variability. In summary, choosing $K$ in UnMix-TNS is about balancing class/domain complexity with model efficiency, and we plan to refine these guidelines further in our future research.

Concerning $\lambda$: Equations (12) and (13) are derived based on the principles of momentum batch normalization (MBN), and the hyper-parameter $\lambda$ is set according to the MBN guidelines proposed by Yong et al. (2020). MBN aims to standardize the noise level introduced by Batch Normalization (BN) across different batch sizes, ensuring comparable noise levels between small and large batches. The tuning of $\lambda$ is integral to achieving this standardization based on the specific batch size employed. For a more in-depth explanation, we have added new information in Appendix A.4 of our paper.

Algorithm 1 in Appendix B.2: Concise but confusing with shape mismatch.

Thank you for your valuable feedback regarding Algorithm 1 in Appendix B.2. We understand your concerns about the clarity of the shapes used in the algorithm and apologize for any confusion caused. To address this, we have revised Algorithm 1 to ensure the correct alignment of tensor shapes.

In the updated algorithm, we have introduced the unsqueeze operation to adjust the dimensions of various tensors appropriately. The instance_mean is now unsqueezed along dimension 1, altering its shape to (B, 1, C). This adjustment allows for the correct broadcasting when calculating the hat_mean of size (B, K, C). Additionally, we have applied the unsqueeze operation to self.components_means along dimension 0, resulting in a shape of (1, K, C). Furthermore, we have unsqueezed $p$ along the last dimension, changing its shape to (B, K, 1). By incorporating all the above steps, we can now correctly combine these tensors, ensuring the resultant hat_mean has the intended size of (B, K, C). We hope that these clarifications and the revised algorithm will make the implementation more straightforward and eliminate any confusion.

Unclear dimension explanation in Appendix B.2 for torch.var_mean(x, dim=[2, 3]).

Thank you for your query. We would like to clarify that in this context, the variable $x$ indeed represents a batch of intermediate features within a Convolutional Neural Network (CNN). As is typical with CNNs, the tensor $x$ possesses four dimensions: batch size, number of channels, height, and width. Therefore, when we call torch.var_mean(x, dim=[2, 3]), the function calculates the variance and mean across the height and width dimensions of the tensor $x$. This operation effectively computes the instance-wise statistics for each feature in the batch, considering the spatial dimensions (height and width) of the CNN's intermediate feature maps.

Thanks to the authors for the additional clarification and experiments. Overall, I think addressing temporally correlated test data is a realistic and interesting research direction, and this paper did a good job tackling it. I have increased my rating to 8. I look forward to your future work, hopefully with real-world applications.

Before going into details, I will reiterate my recommendation that you should put the phrase "with negligible inference latency" somewhere in the abstract, or maybe even do an experiment comparing the inference latency with other methods like Tent. UnMix-TNS looks very complex and use cases like autonomous driving care about inference latency a lot, so it's good to know it admits an efficient implementation.

To address this, UnMix-TNS is designed to update the statistics of the active component slowly, using an exponential moving average. This method of updating helps in mitigating the risk of rapid shifts in the active $\mu_k$.

I was worried about a gradual, instead of rapid, shift in feature representation. As you pointed out, the cluster mean $\mu_k$ is pretty robust to rapid shifts thanks to the use of a moving average, but if the shift is gradual I'm not sure how a moving average helps. On the other hand, I should probably not worry too much about that, because we won't see a lot of samples sitting on the decision boundary in practice: those correspond to ambiguous samples that are information-theoretically hard to classify.

In line with your suggestion, we have conducted additional experiments on CIFAR10-C under mixed-domain scenarios.

Sorry for the confusion, but what I meant is "To what extent does each instance's cluster membership correlate with its class label?". However, your result is still helpful: I can infer that the answer is probably "not much", because each class can have multiple subclasses/domains, e.g. a dog in the snow may look more like a deer in the snow than a dog on the grass.

Thank you for your constructive comments and suggestions; and they are exceedingly helpful for us in improving our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Ambiguity regarding instance selection and potential bias.

We would like to clarify any confusion in the submitted paper regarding the potential for bias in instance selection. For a given batch of test feature-map instances, represented as $\mathbf{z}\in \mathbb{R}^{B\times C\times H \times W}$ (where $B$ is the number of instances, $C$ the number of channels, $H, W$ the width and height of an instance feature-map), we do not perform any selection of the test features for updating the components statistics. Instead, every individual feature map $z[b]$ $\in \mathbb{R}^{C\times H \times W}$ is soft-assigned to the $K$ components of UnMix-TNS. This assignment is based on the cosine similarity between the mean of each instance-wise feature map and the centers of the components (Equation 7). Based on this similarity given by $p_{b,k}$, we update the parameters $\mu_k^t$, $\sigma_k^t$ of all $K$ components (Equations 12 & 13 in the paper). We acknowledge the importance of this clarification.

More detailed theoretical insights to handling temporally correlated data, comparison with traditional methods and potential bias issue.

We appreciate your thoughtful review and valuable suggestion to provide a deeper theoretical understanding of how our model accounts for temporal correlation. In response, we have expanded our theoretical analysis. In our revised manuscript, we have added a dedicated section (A.3 in the Appendix) to address this topic comprehensively. We also compare UnMix-TNS with the standard test-time normalization method (TBN) in section A.3. Furthermore, we have highlighted why UnMix-TNS can help mitigate the bias introduced in the estimation of feature normalization statistics, which arises due to the time-correlated feature distribution.

Dealing with highly diverse or outlier instances.

We appreciate the reviewer's concern about handling highly diverse or outlier instances in test batches. We want to emphasize that we adopt the same setting as other test-time normalization methods for a fair comparison by assuming that temporally correlated test batches are free from outliers, such as instances from novel or unknown categories. This is an important consideration, as the presence of outliers may result in inaccurate normalization statistics estimations in UnMix-TNS, an issue that is also applicable to other methods.

Fortunately, we are already planning to extend our work to include detecting and excluding anomalies during the test-time adaptation process. The envisioned approach involves leveraging an anomaly detection score, calculated based on the distance metric of test instances to the nearest component in the UnMix-TNS and will utilize an optimal threshold to identify outliers effectively. By identifying and pruning outlier instances using this strategy, we aim to prevent the adverse impact these instances might have on adapting the model. This will significantly improve the robustness and accuracy of UnMix-TNS in dealing with diverse test batches. We have outlined this planned improvement as a key focus for our future work in the revised version of our manuscript (conclusion section).

Thank you for your constructive comments and suggestions. We have carefully addressed them. In the following, your comments are first stated and then followed by our point-by-point responses.

How does UnMix-TNS effectively handle mixed domain settings when its normalization suggests it treats multiple domains as a single one?

Thank you for your insightful comment. We agree that UnMix-TNS does not incorporate a specific module tailored for mixed domain settings. However, we want to shed light on the fact that UnMix-TNS shows robustness against temporal correlations (different values of Dirichlet coefficient $\delta$) of the test images with respect to their inaccessible labels compared to other test time normalization schemes.

The core functionality of UnMix-TNS lies in its ability to retain current and past statistics of online test features through its $K$ components, functioning as a time-decaying memory. In cases where test features are label-correlated, UnMix-TNS effectively simulates an i.i.d. scenario. It accomplishes this by estimating the current mean and standard deviations from its $K$ components, each representing different groups of test features encountered previously.

Additionally, the gradual update of the $K$ components over time enables UnMix-TNS to adeptly handle continual domain settings, where domain shifts occur sequentially rather than randomly. This capability was demonstrated in our experiments, showing UnMix-TNS's effectiveness not only in single domain settings but also in continual domain scenarios. Therefore, while UnMix-TNS may not specifically target mixed domain settings, its underlying design principles and operational mechanics provide flexibility and adaptability across varying domain configurations, including mixed domain. Furthermore, as supported by our experiments, our method's adaptability and performance in mixed-domain scenarios can be further enhanced by adjusting the momentum hyperparameter $\lambda$ and appropriately increasing the value of $K$ to encompass the style diversity in the mixed domain.

Finally, our claim regarding the mixed domain scenario has been revised in the updated manuscript.

Unclear experimental setting, with UnMix-TNS not surpassing "Source" in mixed domains and baseline performances falling behind those in Lim et al. (2023)

Thank you for your comment and for highlighting these concerns. Firstly, we apologize for any confusion caused by the experimental protocol reference. We followed the protocol set out in Marsden et al., 2023, not Lim et al., 2023. This error has been corrected in our revised manuscript.

Regarding the performance of baselines, it's important to note that Lim et al., 2023, primarily reported baseline performances in an i.i.d. setting. They only report the performance of their method for class imbalance for the single domain setting. Our approach with UnMix-TNS, while occasionally not surpassing the source model, does show consistent improvements over other test-time normalization schemes, particularly where these schemes may not have yielded significant enhancements. Additionally, it's important to recognize that our results are consistent with Lim et al., 2023's observation. They noted underperformance compared to the source model in class imbalance settings and indicated that relying solely on normalization techniques may not always lead to significant improvements in non-i.i.d settings.

However, UnMix-TNS stands out by offering notable improvements to test-time adaptation methods. In some cases, it even helps these methods outperform the source model in different scenarios, including mixed domain. For instance, when integrating UnMix-TNS with ROID, we observed a significant performance leap over the source model, a feat not achieved with TBN as the normalization technique. It is noteworthy that UnMix-TNS outperforms the source model in the mixed domains scenario on DomainNet-126 and CIFAR10-C datasets. While the results for mixed domains have been reported with $K=128$ in the manuscript, increasing $K$ to $K=1024$ shows even more impressive results on CIFAR100-C. Specifically, the error rate on CIFAR100-C decreases to 45.8%, outperforming the source model by 0.7%.

Nevertheless, we can enhance the performance of UnMix-TNS in mixed domain adaptation scenarios by tuning the $\lambda$ hyperparameter. As detailed in Appendix A.4, this adjustment, guided by momentum batch normalization (MBN) principles, ensures consistent noise levels across different batch sizes. $\lambda$ is calculated as $\lambda=1-(1-\lambda_0)^\frac{B}{B_0}$ where $\lambda_0$ and $B_0$ represent the ideal values, and $B$ is the actual batch size. In mixed domain scenarios, with more varied styles and higher noise, we suggest a modified formula $\lambda=1-(1-\lambda_0)^\frac{B}{B_0*15}$ considering $15$ as the number of corruptions. This adjustment lowered the error rate on CIFAR100-C to 45.9%, slightly outperforming the source model.

If the temporal correlation mainly...

It seems the point about temporal correlation was not completed. Could you please elaborate on this, so we can address it more effectively?

Choice of $K$

We appreciate your inquiry regarding the selection of $K$ in our UnMix-TNS framework. The determination of $K$ is a critical element, as we have previously mentioned in our discussion of future work. To begin, we highlight that a single UnMix-TNS component can sometimes effectively represent multiple similar classes (e.g., car, truck, bus). This means that even with a relatively low $K$ value, our model can approximate the mean and variance of the test distribution accurately. This approach is particularly beneficial when class similarities are pronounced. However, in scenarios with a large number of classes, such as CIFAR100-C, DomainNet-126, or ImageNet-C, setting $K$ equal to the number of classes can lead to inefficiencies. Specifically, it can result in an excessive memory footprint, which is impractical for many applications. For instance, in our experiments with CIFAR100-C, we observed that setting $K$ to 100 actually decreased performance.

Furthermore, the optimal value of K varies depending on the test scenarios. In mixed domain scenarios, where there is a diverse range of styles in the test distribution, a larger $K$ is necessary to capture this diversity effectively. This contrasts with single domain scenarios, where the domain remains consistent, or continual domain scenarios, where the domain shifts sequentially. In these cases, a smaller $K$ may be sufficient due to the reduced variety in the test distribution.

In summary, the choice of $K$ in UnMix-TNS is a balance between the complexity of the classes and domains and the practical considerations of model efficiency. While a higher $K$ may offer more nuanced representation in diverse settings, it also comes with trade-offs in terms of computational and memory requirements. Our ongoing research aims to refine these guidelines further and provide more concrete recommendations in various application contexts.

Minor Weakness

Thank you for your feedback regarding minor weakness. We acknowledge the pixelation issue of Figure 2 and have improved the dpi to enhance its clarity in the revised manuscript. Also, in the revised manuscript (Section 2.1, second paragraph, Test-time adaptation under label temporal correlation), we have clarified that it specifically refers to the temporal correlation of labels. We thank you for noting the font style inconsistency after Equation (1) on Page 5; we corrected this in the revised manuscript. Also, we appreciate your suggestion to replace $\sim$ with $\approx$ for clarity in denoting approximation rather than distribution. This change is implemented to enhance the precision of the expressions in the paper.

Equation (5)

In response to the reviewer's comment on Equation (5) regarding the distribution of $\mu_{k,c}^0$, we would like to clarify that $\mu_{k,c}^0$ is a scalar quantity representing the mean of the $k^{th}$ UnMix-TNS component for channel $c$, and $t$ is a scalar sampled from a Normal Distribution $\mathcal{N}(0,1)$. Given this, the vector $\mu_k$ does not lie on a line and has no low-rank structure.

Using cosine similarity instead of L2 distance?

We thank the reviewer's comment. The L2 distance is susceptible to the curse of dimensionality, and as the number of channels increases, the contrast between L2 distances ($b^{th}$ instance's statistics and the $K$ statistic components) tends to diminish. Consequently, it becomes necessary to decrease the temperature $\tau$ when computing the assignment probability $p_{b,k}$ in Equation (10) to avoid updating all $K$ components with the same weight. Such a distance function, therefore, requires careful setting of the temperature for every layer. On the other hand, cosine similarity does not suffer from this problem and only requires fixing one unique and shared temperature for all layers. Thus, we have opted to retain this score.

Dear Reviewer Yi31,

Thank you for your thoughtful comments. We appreciate your recognition of our revisions and value your suggestions for further enhancing our manuscript.

Mixed Domain: We are pleased our clarification on the mixed domain aspect was well-received.

Experimental Setting: We appreciate your acknowledgment of the clarification we provided on the experimental setting.

Temporal Correlation: Regarding UnMix-TNS, our aim was to ensure robustness against temporal correlations (different values of Dirichlet coefficient $\delta$) of the test images concerning their labels compared to other test time normalization schemes. If your reference to Test-Time Prior Adaptation methods pertains to label distribution shifts between source and target domain, this indeed offers a relevant yet distinct scenario from our current focus. We note your suggestion for further comparison in this area. However, the specific Test-Time Prior Adaptation method you referred to was not cited. It would be beneficial if you could provide more clarity on the approach. While we see the value in such a comparison, we must consider the scope and current focus of our manuscript to determine the feasibility of including this comparison. If not feasible at this stage, this will certainly be a consideration for our future work to enhance the robustness of our research.

Choice of K: We acknowledge your concerns about empirically selecting K in our clustering algorithm. While our ablation studies and further investigation and discussion have touched upon this, we appreciate your suggestion for further refinement and will continue to explore this in future work to enhance our algorithm's practicality and applicability.

Thank you once again for your valuable input and for your acknowledgment of the interesting idea and solid results presented in our paper.

Thank you for directing our attention to the survey paper. We acknowledge the importance of comparing our work to the methods outlined in this comprehensive survey. However, implementing a direct comparison with these methods would require installing dependencies and running code from numerous online repositories, a task that is not feasible within the current discussion period's timeframe. Despite this, we have taken steps to demonstrate the versatility and robustness of our UnMix-TNS in the Test-Time Prior Adaptation scenario.

One of our top-performing test-time adaptation baselines, ROID, incorporates an additional prior correction module during test-time (as detailed in Section 4.3 of their paper). This module is designed to adjust for prior shift based on the class distribution within a batch. We conducted an ablation study where we disabled their prior correction module (ROID-(Prior Correction)) and replaced its normalization layers with our proposed UnMix-TNS layer (ROID-(Prior Correction)+UnMix-TNS). The results, as shown in the table below (second row), indicate a significant performance gain over the ROID method without prior correction, even surpassing the performance of ROID with the prior correction module (ROID+(Prior Correction)) (third row). Furthermore, adding our normalization layer to the standard ROID (last row in the table) even further enhances the performance. These findings underscore the robustness of UnMix-TNS without the need for an additional prior correction module.

This compelling result motivates us to consider integrating UnMix-TNS into other related test-time adaptation methods in future work in scenarios slightly different but related to Test-Time Prior Adaptation.

We hope this additional insight and the efforts we have made in this direction positively influence your final rating and demonstrate our commitment to advancing the field of Test-Time Adaptation.

[1] Robert A Marsden, Mario Dobler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. arXiv preprint arXiv:2306.00650, 2023.