Open-World Test-Time Training: Self-Training with Contrastive Learning

• The writing could be improved. 1) In-text citations are missing parentheses. 2) The authors spend a lot of time explaining $\ell_2$-norm and other basic definitions, but don't provide a preliminary on test-time training, which is more interesting and requires some explanation. 3) There a bunch of typos all over the paper and missed dots.
• The results, although seem to improve, are still very close. Would be interesting to measure the standard deviation of model performance as opposed to just the mean.
• The evaluation leverages basic OOD datasets, such as CIFAR-10C and ImageNet-C. There exists other benchmarks that could be used and that are more interesting (other ImageNet variations, WILDS, etc.)

Weaknesses

• There are a lot of typos in this paper, for example,

  • Substitute $\mathcal{F}$ to $\mathcal{F}^\prime$ in (4).
  • Rewrite the inner product of positive pair as $v_i \cdot v_j^\prime$ or $v_i^Tv^\prime_j$ in (6).
  • $i$ should be subscript in (9).
  • $\hat{y}_i$ is more easily understood expression in (11).
  • The summation component does not rely on $i$ in the RHS of (12)?

• The paragraph under (15) is not clear enough. Referring to [1] may be a great way to improve readability. The overall readability of the paper is not very good, authors should carefully consider whether the use of notation is appropriate to improve the readability of the article, especially for usage of subscripts, like replacing $d_m$ into $d^m_i$, $M$ into $n_s^i$ in (1), giving the specific formula for $K$ of (12) are a little bit better, directly comparing the pipelines of OWDCL and OWTTT in a certain paragraph can help readers with relevant field foundations quickly understand the contribution of this paper.

• I can't understand why the authors of both OWTTT and this paper adopt the criteria of $Acc_N$. This quantity measures the goodness of recognizing strong OOD rather than the accuracy of strong OOD. In practice, it is possible to calculate the classification error for strong OOD data, isn't it?

1. While the paper is well-executed, its novelty seems quite incremental, with limited unique contributions. Much of the OWDCL framework builds directly on prior work, OWTTT, with NT-XENT serving as an existing method for contrastive learning. Additionally, the performance improvement is also limited, especially when additional augmentations are included.
2. The paper lacks in-depth explanation and empirical evidence to substantiate the claim that "misclassification of certain weak OOD classes as strong OOD results from insufficient contrastive information." Both theoretical support and experimental comparisons are necessary to highlight the effectiveness of contrastive learning in addressing this problem.
3. Contrastive learning for test-time training and adaptation is explored in various existing methods, such as Chen et al. (2022) and Wang et al. (2022). It would be helpful for the authors to provide a comparison to these methods and clarify the distinctions.
4. The current experimental setting may be too simple for the problem. Proof of the proposed solutions should be conducted on more challenging cross-domain datasets such as PACS and DomainNet.

[1] Li, Y., Xu, X., Su, Y., & Jia, K. (2023). On the robustness of open-world test-time training: Self-training with dynamic prototype expansion. Proceedings of the IEEE/CVF International Conference on Computer Vision, 11836–11846.

[2] Chen, D., Wang, D., Darrell, T., & Ebrahimi, S. (2022). Contrastive test-time adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 295–305.

[3] Wang, Q., Fink, O., Van Gool, L., & Dai, D. (2022). Continual test-time domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7201–7211.

1. The novelty is incremental. First, contrastive learning has been shown to be effective in many different tasks. In the TTA task, TTT++, AdaContrast also use contrastive loss in their work. Therefore, the authors need to clarify why contrastive learning used in this work can address OWTTT well and why others cannot achieve it. This analysis can be some theoretical derivations or some targeted contrastive experiments.
2. In Eq. 4 of the manuscript, I am wondering why the L2 norm is performed in the batch channel instead of the feature channel. What is the reason for this?
3. The formulae in the manuscript are unclear and misleading. For example, in Eq.12, the definitions of $k$, $c$ and $K$ are misunderstood. While $k$ is the pseudo label of the test sample and $K$ is the superscript of the sum function in Eq.12, $k$ is redefined as the number of sample pairs and $K$ is the class id in line 270.
4. This paper uses relatively simple data augmentation techniques to generate positive sample pairs. However, the effectiveness of these techniques may be limited when dealing with out-of-distribution (OOD) data that are highly complex or significantly different from the original data, such as different domains in PACS datasets.