IT$^3$: Idempotent Test-Time Training

We appreciate the reviewer’s insightful comments. We address the reviewer's concerns below and will revise the paper accordingly.

 

"The experiments primarily involve smaller datasets and omit standard OOD benchmarks like WILDS, leaving broader applicability and real-world relevance underexamined." / "limited set of baselines" / "The tasks used in experiments are relatively small-scale"How does IT³ scale in terms of memory and runtime when model sizes grow, and have you tested it on larger networks (e.g., ImageNet-scale) to confirm its practicality?"

Following the reviewer’s suggestion, we conducted additional experiments on ImageNet-C and compared also to TTA baselines: TENT, ETA and MEMO across various batch sizes. The results are PRESENTED IN THIS PLOT, which will be included in the final version of the paper. For each method and severity level, we present bars composed of 15 accuracy values—one for each corruption type. Our method consistently outperformed with a 3–5% improvement. While WILDS is clearly relevant, prior TTT works did not evaluate on it and we missed it. In the limited rebuttal time we had, we prioritized ImageNet-C and additional baselines experiments. We will test on standard OOD benchmarks when revising the paper.

Runtime analysis can be found in Appendix A. When revising, we will add analysis of memory consumption to all cases as well. One case for batch size 128 ImageNet-C in GB:

TENT: 4.8
ETA: 4.9
MEMO: 13.5 (3 augmentations)
ActMAD: 7.2
IT$^3$: 7.4
Vanilla Model: 4.5

 

"... does not include rigorous derivations confirming that test-time adaptation preserves idempotence on OOD samples."

Thank you for identifying this important issue. We acknowledge that the paper lacks a clear logical flow connecting idempotence to OOD adaptation. We provide here A SKETCHED LOGICAL CHAIN explaining exactly this. While not a formal proof, it is a rigorous description of this relation. This will be combined into the introduction in the paper. A detailed extended version will be added to the paper as appendix C. A very brief summary is below:

1. When using ZigZag approach to training, the discrepancy between recursive calls of the model, $|| y_1 - y_0 ||$, is highly correlated to how far out-of-distribution (OOD) an input is.
2. Thus, we can use $|| y_1 - y_0 ||$ as a loss, and reduce 'OODness'.
3. By doing this, we make the network more idempotent: $L_{\mathrm{IT^3}} = || y_1 - y_0 || = || f(x, f(x,0)) - f(x,0) ||.$
4. But properly minimizing $L_{\mathrm{IT^3}}$ is non trivial, so we employ the approach from IGN (Shocher et al.)
5. This can be understood as a projection onto a distribution. (Appendix B).

To empirically support this claim, we also analyzed the relationship between corruption severity and idempotence loss on ImageNet-C. The results are shown IN THIS LINK. Each point represents a batch of data, with the x-axis showing the computed idempotence loss and the y-axis indicating classification accuracy. As expected, higher severity levels correspond to greater idempotence loss. Notably, we observed a strong negative correlation between idempotence loss and performance (Pearson correlation: –0.94).

 

"Briefly discuss tuning hyperparameters (e.g., optimization steps, learning rate) for test-time adaptation to guide practical deployment."

Thank you for this important comment. We will expand appendix A to include implementation details. Code will be released upon acceptance.

 

"Could you clarify how you prevent confusion between the neutral zero and legitimate zero labels?"

We thank the reviewer for their important comment. The special 0 notation used does not represent an actual zero. Instead, it is a unique signal with the same dimensions as the labels, specifically chosen to differentiate it from actual label values. Our approach builds upon the ZigZag method from [1], where the choice of this signal is extensively discussed and justified. We understand that this notation may appear confusing, so we will revise the paper to include a clearer explanation to avoid any ambiguity.

 

"If the model’s initial prediction is incorrect, even with the frozen reference model, the iterative optimization can magnify the error signal." / "When the initial prediction for an OOD sample is substantially off, what mechanisms prevent... reinforcing incorrect outputs?"

This is a valid point. Even with the frozen network trick, it is not impossible for such event to occur. Empirically, such cases are rare in comparison to cases where the TTT session improved the prediction. The number of TTT iterations is very limited. The network is reset after each instance so no long term damage can really accumulate.

We sincerely thank the reviewer for their detailed and constructive feedback. We address the reviewer's concerns below and will revise the paper accordingly.

 

"Lack of justification for IT3’s generalization ability to out-of-distribution (OOD) data. While the paper presents IT3 as an effective method for handling OOD data scenarios, the justification for why enforcing idempotence leads to better OOD generalization. The intuition behind this connection needs to be more explicitly stated, and additional ablation studies could help clarify why this approach is effective beyond empirical results."

Thank you for identifying this important issue. We acknowledge that the paper lacks a clear logical flow connecting idempotence to OOD adaptation. We provide here A SKETCHED LOGICAL CHAIN explaining exactly this. This will be combined into the introduction in the paper. A detailed extended version will be added to the paper as appendix C. A very brief summary is below:

1. When using ZigZag approach to training, the discrepancy between recursive calls of the model, $|| y_1 - y_0 ||$, is highly correlated to how far out-of-distribution (OOD) an input is.
2. Thus, we can use $|| y_1 - y_0 ||$ as a loss, and reduce 'OODness'.
3. By doing this, we make the network more idempotent: $L_{\mathrm{IT^3}} = || y_1 - y_0 || = || f(x, f(x,0)) - f(x,0) ||.$
4. But properly minimizing $L_{\mathrm{IT^3}}$ is non trivial, so we employ the approach from IGN (Shocher et al.)
5. This can be understood as a projection onto a distribution. (Appendix B).

To empirically support this claim, we also analyzed the relationship between corruption severity and idempotence loss on ImageNet-C. The results are shown IN THIS LINK. Each point represents a batch of data, with the x-axis showing the computed idempotence loss and the y-axis indicating classification accuracy. As expected, higher severity levels correspond to greater idempotence loss. Notably, we observed a strong negative correlation between idempotence loss and performance (Pearson correlation: –0.94).

 

"Unclear contribution of exponential moving average (EMA) to the performance gains."

The online version, as described in sec. 3.3 differs from the base version because they aim at different scenarios. In the base version, the assumption is that each input is a separate test and has no correlation or information about other inputs. Thus the network weights are reset back to the state they were at the end of the pre-training phase. In contrast, the online version is intended for a continual learning data stream with correlation and somewhat smooth transitioning between inputs. In this case, instead of resetting after each input, we leave the weights updated from the previous inputs.

Since the data keeps shifting in this scenario, over time, $f_{\theta}$ may diverge far from $F$, making it irrelevant. Instead, we need an anchor that is influenced by a reasonable amount of data, yet evolves over time. Our solution is to replace $F$ with an Exponential Moving Average (EMA) of the model. This means $f_{\text{EMA}}$ is a smoothed version of $f_{\theta}$ over time.

Note that the online-IT$^3$ results, depicted in Table. 1 achieve higher scores than the base model. This is due to the inherent difference between the base and the online scenarios. A correlated data stream allows aggregating information during test-time and therefore expected to exploit it and perform better.

 

"Limited large-scale experiments... it would be beneficial to include results from larger datasets such as ImageNet." / "I think showing more high-dimensional (label) domain in the image domain will be great." / "Missing baselines... lacks comparison with augmentation-based baselines."

Following the reviewer’s suggestion, we conducted additional experiments on ImageNet-C and compared also to TTA baselines: TENT, ETA and MEMO across various batch sizes. The results are PRESENTED IN THIS PLOT, which will be included in the final version of the paper. For each method and severity level, we present bars composed of 15 accuracy values—one for each corruption type. Our method consistently outperformed with a 3–5% improvement.

 

"Computational overhead and fairness in comparisons... A fairer comparison should analyze compute overhead and consider normalized performance metrics that account for increased memory usage (or parameter count)."

Runtime analysis can be found in Appendix A. We will add analysis of memory consumption to all cases as well. One case for batch size 128 ImageNet-C, peak memory reserved with torch.cuda.max_memory_reserved() in GB:

TENT: 4.8
ETA: 4.9
MEMO: 13.5 (3 augmentations)
ActMAD: 7.2
IT$^3$: 7.4
Vanilla Model: 4.5

We sincerely thank the reviewer for their thoughtful and encouraging feedback. We address the reviewer's concerns below and will revise the paper accordingly.

 

"The authors claim that their method does not require designing additional auxiliary tasks or using extra data, making the adaptation on-the-fly. However, from my perspective, enforcing idempotency during training can also be considered an auxiliary task,...

Indeed, enforcing idempotence can be seen as an auxiliary task. However, it is not a domain-specific one. This is in contrast to approaches that use auxiliary tasks that are specific for data-type. Take for instance Sun et al., or Gandelsman et al. Their visual auxiliary tasks cannot be used on other types of data. Some others are architecture-dependent e.g., ActMAD (Mirza et al., 2023). Contrary to existing work, the IT$^3$ mechanism can be used out of the box for any task (we demonstrate image classification, aerodynamics prediction, aerial segmentation and tabular data) and any architecture (MLPs, CNNs, GNNs).

 

...and it also requires access to training data for supervised learning. Could the authors clarify this further?"

During the pre-training phase, IT$^3$ uses the training data like any other supervised-learning method. At test time, the challenge is defined so that there is absolutely no access to the training data. At any point, the only thing given is the current test input.

 

"For classification results, more comparisons with advanced fully test-time adaptation (TTA) methods, such as TENT, ETA, and DEYO, as well as evaluations on larger datasets like ImageNet-C, would strengthen the justification for the proposed method’s superiority."

Following the reviewer’s suggestion, we conducted additional large-scale experiments on ImageNet-C, comparing our method against popular TTA baselines including TENT, ETA, MEMO and ActMAD. The results are PRESENTED IN THIS PLOT, which will be included in the final version of the paper.

As shown in the plot, our method consistently outperforms the baselines across all severity levels and batch sizes, achieving a 3–5% improvement in accuracy. These results further highlight the effectiveness and scalability of our approach.

 

"For segmentation results, could the authors include additional evaluations on commonly used benchmarks such as KITTI-C and nuScenes? This would enable a fairer comparison, as the dataset used in this paper has not been widely adopted in previous works."

Thank you for pointing out these benchmarks. They are relevant. We missed them because prior TTT works don't use them. We intend to remedy this when revising but, unfortunately, training on them would take more time than we have for this rebuttal. Thus, to provide additional results during the rebuttal period, we prioritized experiments on ImageNet-C, as discussed in our response to the previous comment.

 

"The error bars in Figures 3, 4, and 8 are difficult to recognize and distinguish, particularly for color vision-deficient readers."

In the revised version, we will improve the clarity and accessibility of the error bars, following the approach used in PLOT. Specifically, we made the error bars bolder and more visible, used brighter, more distinguishable colors, and added explicit labels to differentiate the bars. These improvements will be applied to all relevant figures in the paper, along with better-written captions to clearly explain the contents of each plot and guide interpretation—ensuring improved readability, including for color vision-deficient readers.

 

"Additionally, could the authors provide more qualitative results instead of bar charts? The absolute differences in bar heights are not intuitive to interpret."

When revising, we will add an Appendix. D with some more qualitative segmentation results for the aerial photography road segmentation challenge. You can view these qualitative results via the FOLLOWING THIS ANONYMOUS LINK.

In the figure, we display the segmentation predictions of TTT, ActMAD, and IT$^3$ on OOD data. Quality scores are added on top of each prediction because they offer an objective measure of segmentation performance, helping to interpret the results more intuitively.

 

"Could the authors provide a computational complexity analysis comparing their method with existing baselines?"

Runtime analysis can be found in Appendix A. When revising, we will add analysis of memory consumption to all cases as well. One case for batch size 128 ImageNet-C, peak memory reserved with torch.cuda.max_memory_reserved() in GB:

TENT: 4.8
ETA: 4.9
MEMO: 13.5 (3 augmentations)
ActMAD: 7.2
IT$^3$: 7.4
Vanilla Model: 4.5

We sincerely thank the reviewer for their thoughtful and constructive feedback. Below we address all the comments, and will revise the camera-ready version accordingly.

 

"The paper's presentation lacks clarity. Specifically, one of its key claim is "if such a network is trained so that f(x, 0) = f(x, y) = y, then at test time the distance ||f(x, f(x, 0))−f(x, 0)|| correlates strongly with the prediction error". However, the authors only skim through it, which disrupts the logical flow."

Thank you for identifying this important issue. We acknowledge that the paper lacks a clear logical flow connecting idempotence to OOD adaptation. We provide here A SKETCHED LOGICAL CHAIN explaining exactly this. The introduction will be rewritten accordingly and a detailed extended version will be added to the paper as appendix C. In short:

1. When using ZigZag approach to training, the discrepancy between recursive calls of the model, $|| y_1 - y_0 ||$, is highly correlated to how far out-of-distribution (OOD) an input is.
2. Thus, we can use $|| y_1 - y_0 ||$ as a loss, and reduce 'OODness'.
3. By doing this, we make the network more idempotent: $L_{\mathrm{IT^3}} = || y_1 - y_0 || = || f(x, f(x,0)) - f(x,0) ||.$
4. But properly minimizing $L_{\mathrm{IT^3}}$ is non trivial, so we employ the approach from IGN (Shocher et al.)
5. This can be understood as a projection onto a distribution. (Appendix B).

 

"I'm interested to see ablation studies examining the impact of enforcing idempotence within the TTT framework."

We provide empirical evidence analyzing the impact of enforcing idempotence in Fig. 2 of the main paper. It shows histograms of the distance from idempotence across four scenarios. For the original training and validation data, the model demonstrates very high idempotence (with the validation set slightly lower than the training set). For Out-of-Distribution (OOD) data, the model initially shows near-random idempotence. However, after applying our TTT optimization, the distribution of OOD data is significantly "shifted" from this random state towards that of the training and validation sets. This supports our claim that optimizing for idempotence indeed makes the model behave on OOD data similarly to how it behaves on in-distribution data.

To further investigate this, we conducted additional experiments on ImageNet-C, measuring idempotence loss across various corruption types and severity levels. (SEE ANALYSIS FIGURE HERE). Each point in the plot represents a batch of data; the x-axis shows the computed idempotence loss for that batch, while the y-axis indicates its classification accuracy. The data covers 15 corruption types and 5 severity levels, which are visualized using the colorbar. As expected, higher severity levels of corruption correspond to larger idempotence loss, further validating the effectiveness of enforcing idempotence within the TTT framework and highlighting its strong negative correlation with model performance (Pearson correlation: –0.94).

 

"Section 2.1: The argue "TTT operates per-instance, with no assumption that future test data will be similar. So previous work have treated TTA and TTT as distinct paradigms" seems weak. As some works, e.g. [1] and [2] do consider TTT under online setting."

We conducted a new large-scale experiment on ImageNet-C that also includes comparisons to TTA methods TENT and ETA across various batch sizes. (SEE RESULTS HERE) These will be added to the main paper in the camera-ready version.

As the reviewer points out, there is previous work [1, 2] about the online setting that we present in sections 3.3 and 4.6. However, unlike in [1,2], ours is intended for continual learning. Furthermore, we want to make a distinction between TTA, along with its many variants, and online-TTT: While the latter is used in a data-streaming context the former is used on a given dataset or batch, or the training data at hand.

 

"In line 235 in the Experiments section: It's unclear which auxiliary task 'the original TTT method' refers to or whether other auxiliary tasks were evaluated."

Our apologies for this lack of clarity. By original TTT, we meant Sun et al. 2020. The auxiliary task used in their work is orientation prediction for images, with validation on image recognition benchmarks. We also compare to ActMAD (Mirza et al., 2023), which we adapted for other types of data and tasks.

 

"The authors should reorganize the presentation, as detailed in my "Other Strengths And Weaknesses" section."

Thank you for helping us improve the presentation of our paper. We will reorganize accordingly.