A Critical Look at Classic Test-Time Adaptation Methods in Semantic Segmentation

W1: The paper presents a study on TTA for segmentation. Yet the analysis is on a single scenario (cityscapes -> ACDC/rain/fog) and the insights minimal. It is hard to understand if this behaviour also generalizes in more cases. I would expect at least one or two more source/test datasets and domains. In its current version, and without any technical contribution, as a study, this paper is weak.

W2: The experiments in Fig 1c and 1d are not adequately presented. Specifically, 1c the experiments with LN/GN are highly unclear, while the description of 1d is very hard to understand. The text would need to be rewritten, explained clearly and perhaps with way more details in an appendix.

W3: Section 4 is also written in a confusing way, without giving proper background for the experiment in Table 2. Also, from Figure 2 it is not easy to compare single vs TS performance as one has to contrast nubmers from two tables. a plot would be far clearer

W5: the class imbalance analysis in sec5 is also not clear to me - it is a known fact that performance in the tail classes is lower and the results on how does adaptation affect that performance are unclear from fig 5.

• Though the three findings are useful for potential readers, these findings are not surprising.
• The insightful analysis related to the three issues about TTA segmentation is missing. -Though the authors point out the three issues, no potential of investigation direction is given. Therefore, the novelty of this paper is somehow limited to me.

1. The authors present a visual analysis of how mean and variance changes during online adaptation, but fail to explain why their results do not align with the true distribution of test data. Furthermore, the authors should also provide visual analysis for aligning batch norm statistics in classification tasks.
2. Similarly, the authors provide experiments that demonstrate the advantage of teacher-student models when pseudo-labels are inaccurate and find that they encounter bottlenecks when pseudo-labels quality are enhanced. However, they do not provide a more compelling, task-specific analysis for segmentation and do not provide a inspirational solution.
3. The details for data augmentation of ACDC-fog are missing.
4. As mentioned in the related work section, there has been some work focused on TTA segmentation such as [1].More discussions and analysis about these task-specific methods should be provided.

[1] Dynamically Instance-Guided Adaptation: A Backward-Free Approach for Test-Time Domain Adaptive Semantic Segmentation, CVPR2023

Insufficient Evaluation:

The paper investigates the effectiveness of the BN updating strategy in TTA classification tasks, predominantly within CNN-based architectures. However, it deploys Segformer-B5, a Transformer-based architecture, for segmentation TTA tasks, which employs fewer BN layers. This choice raises the question of whether the use of Transformer-based architectures, rather than the segmentation task itself, may contribute to the potential ineffectiveness of the BN updating strategy. It would be beneficial for the authors to provide more comprehensive insights into this matter.

Insufficient Observations:

The paper discusses patterns observed in classification tasks but does not adequately test these patterns against continuous (Wang et al., 2022) or persistent distribution shifts (Yuan et al., 2023). It is essential to expand the analysis beyond the context of fully test-time adaptation (Wang et al., 2021) to draw more meaningful conclusions regarding the broader applicability of the proposed strategies.

Minor Issues:

• Some figures and the text within them, such as Figure 2 and Figure 3, appear too small, which can hinder the readability of the paper. I recommend that the authors include more detailed images or visualizations in the supplementary material to enhance the clarity of their work.

• It appears that there might be a typo error in the statement "while the count of class 14 is $10^5$." It should be corrected to "while the count of class 19 is $10^5$."

(Improvement in writing)

The writing of this paper could be improved. For example, the following segment is copied three times, with very little changes, in the abstract and the introduction, and two times within the introduction itself, providing very little extra information.

Our comprehensive results have led to three key observations. First, the classic batch norm updating strategy, commonly used in classification TTA, only brings slight performance improvement, and in some cases it might even adversely affect the results. Even with the application of advanced distribution estimation techniques like batch renormalization, the problem remains unresolved. Second, the teacher-student scheme does enhance training stability for segmentation TTA in the presence of noisy pseudo-labels. However, it cannot directly result in performance improvement compared to the original model without TTA. Third, segmentation TTA suffers a severe long-tailed imbalance problem, which is substantially more complex than that in TTA for classification.

Also, adding background information about how the TTA methods work, in more detail, in the appendix would make the paper more self-contained. More information about the segmentation tasks/pretrained architecture in the appendix would also help.

(Misses literature on segmentation TTA)

The paper discusses test-time adaptation for semantic segmentation tasks, and shows that some popular TTA methods for image classification can be inadequate for the segmentation task. However, the paper fails to mention/analyze work done specifically for segmentation TTA. While it is possible that classification TTA methods are not sufficient for segmentation TTA, if there are specific methods for segmentation TTA, not discussing them makes the paper weaker.

I will mention a few examples here, but the list is not exhaustive.

1. SITA [1]. Looking at table 2 of this paper, it does seem TENT often does not perform well for segmentation. But SITA shows impressive numbers in this task. It also uses a batch size of 1 (single test instance).
2. [2] shows an interesting way of doing TTA for segmentation on night-time images using thermal information.
3. MM-TTA [3] shows interesting use of multi-modal information for 3D semantic segmentation.
4. Slot-TTA [4] uses a semi-supervised slot-centric scene decomposition model that at test time is adapted per scene through gradient descent on reconstruction or cross-view synthesis objectives.
5. Finally, [5] also works on how to adapt a segmentation model given a single unlabeled image without any other information.

(Possible improvements in experimental results)

For methods related to batch norm updating (Table 1), only TENT [7] and its variants are considered. There are other methods related to this strategy, for example BN [6], that have not been included in the table. I understand that BN is discussed in a later section for a separate context, but seeing how they compare to source-only (SO) and TENT on segmentation tasks can be important. Especially because comparison tables in SITA [1] show that BN performs better than TENT in segmentation tasks, so this feels like an incomplete/cherry-picked table.

(Other segmentation models)

The paper uses Segformer-B5 [9] as the pre-trained segmentation model. Experiments with more segmentation models will be important to establish the broad claims that this paper makes. We do not know if the results we are seeing are artifacts of this particular architecture/pre-training data, or are something more general. How about FAIR’s segment-anything [8] model/some other pretrained models?

(Suggestion for overall improvement)

While I appreciate the overall idea of this paper, i.e., analyzing failure of classification TTA methods on segmentation task, it feels incomplete without analyzing why certain segmentation TTA methods work well/what is missing from classification TTA methods. For example, if the paper shows the dynamics of SITA [1] or any of the other methods, show why it is performing well on segmentation tasks and what is different in them from image classification TTA methods, that would improve the paper vastly. SITA also seems to work well on classification tasks, so I feel it might not be the case that classification TTA methods just don’t work well on segmentation TTA tasks, rather something more interesting.