IT$^3$: Idempotent Test-Time Training

We sincerely thank the reviewer for their thoughtful feedback and recognition of the strengths of our work. We greatly appreciate their acknowledgment of the simplicity of our approach, the clarity of our writing, and the breadth of experiments demonstrating the versatility of our method. We have carefully addressed all comments and concerns, as outlined below.

• The method section, while simple, is a bit confusing. Why does the paper present two variants of IT$^3$? This questions why not leveraging the online version in all the experiments?

Thank you for highlighting this lack of clarity. The online version differs from the base version because they are designed for different scenarios. The main difference is that in the base version, the network weights are reset to their original state they were at the end of the pre-training phase. This assumes that each input is an independent test with no correlation or shared information with other inputs. In this scenario, using updated weights from previous examples would be considered cheating, as also noted in prior work (e.g., [1]).

In contrast, the online version assumes some correlation and smooth transitions between inputs arriving in a stream, representing a continual learning regime. Here, instead of resetting after each input, we allow the weights to remain updated from previous inputs. We will add a clarification to the paper to address this distinction.

• Is there an experimental setup where the offline one is much better than the online one? Why doesn't the paper present one method and treat the other as a special case/variant that is less powerful?

Please see the explanation for the above comment—the two versions are designed for two different scenarios. The online version is almost always better in terms of performance, as also noted in [1]. However, the online setup is not always feasible in real-world applications, particularly in cases where data arrives in isolated batches or where it is crucial to ensure that no information leaks from one input to another. In such cases, the offline version must be used to maintain these constraints.

For this reason, both versions are relevant and widely used in practice. In our experiments, we consider both setups as they correspond to two distinct real-world scenarios, allowing us to demonstrate the versatility and applicability of our method in varied contexts.

• TTT/TTT++[A] are directly comparable with IT$^3$, then I suggest to at least have them in all of the experiments with one-to-one comparison in terms of batch size and model architecture. Also, consider adding the performance of IT$^3$ under batch size=1 in Figure 3 results.

TTT and TTT++ are both domain-specific. Nonetheless, we include new experiments with these datasets in the next comment. Both TTT and TTT++ are designed for image data and technically cannot be applied to other types. Even within the image domain, they are unsuitable for some data types. For instance, for aerial photos, they may not work well as they rely on predicting orientation, which is poorly defined for aerial photography.

Thank you for suggesting adding batch-size=1 to the CIFAR figure. We have openly stated in the experiments section, the figure caption, and the limitations section that batch-size=1 for CIFAR is currently unsuccessful. Additionally, we note that in the original TTT, while only a single example is used, a batch of 32 augmentations is constructed to update the weights.

We sincerely thank the reviewer for their detailed and thoughtful feedback on our work. We deeply appreciate their recognition of the strengths of our approach, including the clarity of our presentation, the robustness of our experimental evaluation, and the novelty of applying idempotence to the TTT setting. We value their constructive suggestions and have carefully addressed all raised concerns, as outlined below.

• The main weakness I see in this work is the lack of proper baselines.

We appreciate the reviewer’s concern and acknowledge that this is an important point. To address this, we have included comparisons in Tables 2, 3, and 4, which evaluate performance on several datasets and provide a broader perspective on baselines. Our goal was to develop a TTT method that works universally across architectures and data types. Most prior TTT methods rely on task-specific self-supervised objectives, which limit their generality. For example, the original TTT [1] depends on the image rotation task, which is not only unsuitable for non-image domains (like graph or text data) but also inapplicable to certain image-based tasks, such as aerial segmentation, where orientation has no clear meaning. Similarly, TTT++ [2] uses a SimCLR-style loss, which faces similar limitations across various domains. These tables highlight that while some methods can be adapted, their reliance on task-specific objectives often limits their generalization, reinforcing the need for a universal approach like ours.

The only truly general TTT approach to date is ActMAD [3], which operates on feature-level statistics and is rightly identified as our main competitor. However, ActMAD is not directly applicable to tasks with variable input sizes, such as graphs with varying numbers of nodes or text tasks with varying token lengths. In such cases, modifications to ActMAD are necessary, such as using aggregated features (e.g., average pooling) instead of pixel-level feature statistics. While this is a straightforward adaptation, we observed in our experiments that it significantly deteriorates ActMAD's performance.

This limitation led us to conclude that there are currently no universally appropriate baselines for TTT tasks. Nonetheless, we understand the value of such comparisons for a more comprehensive evaluation. Below, we provide additional comparisons as requested by the reviewer, which we believe will contribute meaningfully to the discussion.

• The work makes claims about how idempotence can be seen as a generalisation of a projection and that it allows to map the OOD data to the in-distribution. Thus, while the authors do spend some time to explain why would intuitively their method works, they do not have any ablation studies to verify that these exact intuitions hold in practice.

In this plot, we show the distributions of idempotence loss for different subsets of the airfoils dataset: train, test, and OOD (both optimized and non-optimized versions). As seen, the train and test subsets exhibit lower idempotence loss values, while OOD samples have significantly larger losses. After optimization on OOD data, the idempotence loss values shift closer to those of the train and test subsets. This supports our claim that the optimization helps align OOD samples with in-distribution behavior, effectively demonstrating the role of idempotence as a projection-like mechanism in practice.

From a more theoretical perspective, idempotence can indeed be seen as a projection mapping, as originally observed in ZigZag [4], iterative models [5], and generative models [6]. In short, idempotence holds on in-distribution (ID) data but does not hold on OOD data. This behavior, similar to a projection mapping, is evident in the plot above and further supported by the table below.

For this table, we used the model trained on the data reported in our paper and computed the idempotence loss for each sample in the ID test set and OOD set, without performing any optimization. To demonstrate that idempotence losses for ID and OOD data are indeed distinct, we used the idempotence losses as an "out-of-distributionness" score and evaluated them on standard OOD detection metrics ROC-AUC and PR-AUC. Higher values of these metrics indicate better separation between in- and out-of-distribution samples, which implies larger differences in idempotence losses for OOD data. The results, shown in the table below, further validate our point.

Table 1. OOD Detection Performance Across Datasets. ROC-AUC and PR-AUC (in %) for CIFAR, Age Prediction, Airfoils, and Cars. Higher values (>80-90%) indicate better OOD detection and support the observation that idempotence holds for in-distribution samples but not for OOD samples, similar to projection mapping.

We sincerely thank the reviewer for their thoughtful feedback on our work. We appreciate their recognition of the simplicity and elegance of our approach, as well as its intuitive incorporation of idempotence into TTT. We value the reviewer’s insights and have carefully addressed all comments and concerns, as outlined below.

• In regression tasks, it is unclear how the model differentiates this zero from an actual label of 0.

We thank the reviewer for their important comment. The special 0 notation used does not represent an actual zero. Instead, it is a unique signal with the same dimensions as the labels, specifically chosen to differentiate it from actual label values. Our approach builds upon the ZigZag method from [1], where the choice of this signal is extensively discussed and justified. We understand that this notation may appear confusing, so we will revise the paper to include a clearer explanation to avoid any ambiguity.

• Online appears to rely significantly on the use of Exponential Moving Average (EMA). However, the authors did not provide a citation for this technique.

Thank you for pointing out. We will add citation for the mentioned paper [2]. It is indeed related and analyzes EMA. The origin of the technique is much older. We also want to point out that the reliance on EMA is not as strong as it may appear. The most important difference is- not resetting the weights after processing each input. Actually, in some cases we did set $\alpha=1$.

• In the first experiment (Section 4.1 on tabular data), the method of randomly substituting features with zeros in the test set may resemble a missing data scenario rather than a true distribution shift.

Thank you for making this distinction. Our goal in making this choice was to have a diversity of corruptions to showcase along the experiments. While acknowledging the distinction you make, we point that missing information makes a divergence from the training distribution, therefore can also be thought of as a distribution shift.

• There are unnecessary abbreviations throughout the manuscript. Figures 14 to 16 are not mentioned or referred to in the main text. Figures 12 and 15 have the exact same title. Some minor improvements and spelling corrections

Thank you for carefully reviewing and helping us improve our paper. In the revised paper these issues will be fixed.

• Do you use a special encoding or placeholder value instead of 0 for regression tasks?

Please see the explanation above: it is a special signal rather than an actual zero. For example, for tasks like aerodynamic predictions, we use placeholder values significantly different from the possible range of lift-to-drag and drag values to ensure clear differentiation.

• If the purpose of the neural "don't know" zero is purely for contrast and its specific value is not important, will it be more computationally efficient to use a representative value such as the median?

Thank you for the suggestion. However, the specific value is actually important. The model needs to clearly differentiate between the two inference modes: with and without additional information provided. For this reason, we use values that are significantly different from potential predictions. Additionally, this question pertains to the ZigZag [1] approach rather than the $IT^{3}$ method. We follow the guidelines outlined in the ZigZag paper, where this choice is extensively discussed.

• Could you provide guidance on selecting the value of or share experimental results demonstrating performance across different values?

Thank you for the suggestion. The ZigZag [1] paper provides detailed guidance on selecting placeholder values, emphasizing the importance of using values significantly different from valid predictions to distinguish inference modes effectively. Our experiments align with their findings, and we refer readers to their discussion for further details

• In EMA [2], when $\alpha=1$, the online IT$^3$ aligns with the offline version, and when $\alpha=0$, it encounters the collapse issue described in Section 3.2. Could you provide guidance on selecting the value of $\alpha$ or share experimental results demonstrating performance across different values?

The online version differs from the base version because they aim at different scenarios. The main difference is that in the base version the network weights are reset back to the state they were at the end of the pre-training phase. The assumption is that each input is a separate test and has no correlation or information about other inputs. The assumption for which the online version is made for, is correlation and somewhat smooth transitioning between inputs arriving in a stream. This is a continual learning regime. Therefore in this case, instead of resetting after each input, we leave the weights updated from the previous inputs. We have added a clarification to the paper.

We sincerely thank the reviewer for their detailed and thoughtful feedback on our work. We deeply appreciate their recognition of the strengths of our approach, including the clarity of our presentation and the robustness of our experimental evaluation. We value their constructive suggestions and have carefully addressed all raised concerns, as outlined below.

• Test-time training requires undesirable extra compute for test-time optimization of the whole model. How much additional compute is needed compared to running inference on the base model? How does this method scale with model size?

We thank the reviewer for their question about computational costs. Test-time training (TTT) inherently induces additional computational overhead because optimization steps are performed during inference. In our method, while we require an additional inference step (two forward passes), this is computationally efficient since forward passes are generally 3–4x faster than backward passes. Thus, the extra inference step adds relatively little overhead compared to the total cost of TTT methods. We added a discussion about efficiency to the revised paper.

Our method typically requires only 1–3 optimization steps, keeping the overall cost comparable to other well-known TTT methods. Below, we provide a comparison of inference times and performance improvements on out-of-distribution data for three approaches: the base model without optimization, the state-of-the-art TTT method ActMAD, and $IT^{3}$. As shown, while our method introduces no significant overhead compared to ActMAD, it delivers a substantial improvement in prediction accuracy (measured by MAE in Airfoils and Cars experiments and by Quality in Road experiment), whereas ActMAD achieves only marginal gains.

Table 1. Inference Time and Performance Comparison (OOD Airfoils)

Table 2. Inference Time and Performance Comparison (OOD Cars)

Table 3. Inference Time and Performance Comparison (OOD Roads)

We will add this table to the revised version. We acknowledge that both our method and TTT methods in general introduce computational overhead during inference. This is a shared limitation across existing TTT approaches. Addressing this challenge and developing TTT methods that avoid such overhead entirely is an important research direction and could lead to significant advancements in the field.

Regarding scalability with model size, our method works effectively with larger models, as demonstrated on aerial segmentation models with millions of parameters. By requiring only two forward passes and a small number of optimization steps, our approach remains computationally efficient even for such large-scale architectures, making it competitive with traditional TTT methods that often involve several backward passes.

• The paper lacks analysis that explains how or why idempotent training is expected to improve out-of-distribution analysis. Further investigation and ablations that provide intuition for how this method works would be valuable.

In the revised version, we will add a more concrete explanation about the reasons we expect idempotence to improve OOD performance. In the introduction lines 77-86 we build the intution towards the motivation to enforce idempotence. There are two prespectives to view this relation. The first one is that the idempotence term rises when minimizing the measure for OOD-ness from [1]. Then having idempotence as the objective makes sense according to [2] - The joint internal representation of (x,y) is projected onto the manifold of representations of (x,y) pairs that are in the data distribution. (More detailed explanation is in the repsponse to your last comment below).