Asynchronous Perception Machine for Efficient Test Time Training

List of References in Rebuttal

[R1] Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback

[R2] Continual Test-Time Domain Adaptation

[R3] Robust Test-Time Adaptation in Dynamic Scenarios

[R4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models”.

[R5] NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation

[R6] Robust Test-Time Adaptation in Dynamic Scenarios

[R7] SoTTA: Robust Test-Time Adaptation on Noisy Data Streams

[R8] Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors

[R9] Tent: Fully Test-time Adaptation by Entropy Minimization, ICLR 2021

We thank the reviewer for providing us the chance to improve our work. Please find our responses:

On improving related-work As reviewer KZWi points out, we will discuss more recent tta/prompting/source-free domain-adaptation approaches. We have noted to add differences w.r.t feature refinement/distillation: a.k.a $L_2$ mimicking feature grids vs boltzmann-temperature-matching logits.

On Clarifications between ttt/tta, comparisons with more state-of-the arts We apologize for the confusion between test-time-training (TTT)/test-time-adaptation (TTA), and would like to clarify the experimental setting of test-time-training and its connections to test-time-adapation works, relative novelty of apm, after checking the shared works. Thank you for sharing these works and we will add these works to related work too.

• Existing works in TTA train on a source dataset and then adapt them for testing. APM, which follows TTT, works without such source-training by just initializing with randomized weights for only 1 test sample. TTA adapts to the target distribution (using all test samples), whereas TTT adapts to each sample independently.
• TTA methods use clean datasets (eg c10) during train, and adapt on corrupted versions(c10c), which might limit its potential to deploy on some other dataset [R1,R2,R3,R5,R6,R7], because they might require training dataset-specific linear-probe.
• Some tta methods update statistics on "batch of test-set". APM can work with weights randomly-initialized after each sample.
• APM is computationally efficient, since it uses only 1 conv layer and 5 mlp layers(tab 2, fig4), while some models might use higher-parameterized network like a ViT, or a diffusion-model [R1].

We observe that [R1] also shows performance for TTT and below we compare APM with [R1] on datasets where APM performed well. We note our baseline numbers match that of [R4]. APM presents competitive performance, on bigger datasets like Imagenet. It is important to note that APM is much more efficient than [R1] (1conv/3MLP vs. ViT+Diffusion model). We will add these and other comparisons to Tab 3. Thank you so much for pointing this out.

• In some cases, methods like [R9] do show source-free domain-adaptation, but on smaller datasets. In tab 1-2, we have shown results on larger datasets like imagenet-c without imagenet pretraining on APM, as also duly noted in line 169.
• We will be grateful to mention that APM is a fresh-perspective on machine perception, with the ability to do location-based processing.

Details on vision-encoder We apologize for this confusion. It refers to the ViT/Dinov2 vision encoder. We will add the explanation in the paper.

On misleading trained column in tables, confusion on ttt-mae We apologize for the confusion caused by this. TTT relies on two phases 1) train model with ssl task + labels on train set. 2) test time -> use ssl task only. For eg., ttt-mae is initialized with pretrained imagenet-1k weights (sec 4.1 of their paper) and needs ssl objective (masking) during ttt. However, APM is initialized with random weights, and no explicit ssl-task is required. We have renamed trained column to $P$ in all tables and clarified in caption: A ✓ in P means that method leveraged pre-trained weights on a clean variant of train set aka, Image-net and downstream-ttt on corrupted version. We provide more results in global rebuttal pdf with updated captions. We will update tables in the main paper also.

On Adding ttt-specific ablations We will be grateful to present this ablation with a variable number of ttt iterations on dtd below. Next, ttt with varying apm parameters is presented in tab 7 of paper. Further, apm's property of one-sample-overfitting during ttt (line 250), was evaluated in tab 5a).

At 25th iteration, we can obtain 49.5, and performance degrades over lesser ttt iterations as pointed out by the respected reviewer KZWi. We tried to discuss the computational efficiency of apm, and some baselines in table 4.

On Questions: on components in fig 1 and their computations Components in fig 1 refer to a new architecture called APM inspired from GLOM [10]. It contains a trigger column T to which a given image is routed. The computation it does is a folding-unfolding process to yield location-aware columns. Each column queries an MLP and decodes a location-specific feature. These features are gathered on the output via statistical-running-average to yield a representation. The final representation is used for zero-shot classification via textual-encoder of a teacher in the contrastive-space. More technicalities are discussed in our reply to the respected reviewer KZWi.

Thank you so much for your time and we will be happy to respond to further clarifications.

Sincerely,

Authors

While I have no doubt about the performance and efficiency of the APM, I have a serious concern that misunderstanding the terminologies and previous work's experiment settings may affect the reliability of the work. The misunderstanding implies that you never read any of TTT or TTA papers in detail and completely misunderstand their settings. My arguments are as follows:

Response to "TTA adapts to the target distribution (using all test samples), whereas TTT adapts to each sample independently.

This is completely wrong. Most TTA do not adapt to the whole test set. Many of them adapt to the batch samples or even to instance. Moreover, "TTT adapts to each sample independently" is completely wrong. It is unnecessary that TTT has to either adapt to one sample or the entire batch or test set. TTT/TTA lies on a special case of Domain Adaptation whereas the model has no access to the source data (source-free) but has to adapt to the target data during the test time.

I suggest you read through all the papers about TTT and TTA. You will see some people in the community use TTA and TTT interchangeably [2]. I can see no significant difference between to the general setting between TTT and TTA. However, the pioneer like TENT [3] discusses a small difference between TTT and TTA. TTT requires the adjustment of training loss (adding an auxiliary loss term) while TTA only introduces the new loss during the test time.

I can refer to "Network details" on section 3 of the pioneer test-time training (TTT) [1] (which some people also classify it as test-time adaptation [2]). They use different ResNets trained on ImageNet and CIFAR10. ImageNet-trained ResNet is evaluated on ImageNet-C and the same for CIFAR10. Regarding your method, as you use pretrained Vision Encoder as teacher, the vision encoder is already trained on ImageNet. This means your method also uses the knowledge from the training distribution extracted by the teacher model.

[1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts

[2] A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts

[3] Tent: Fully Test-time Adaptation by Entropy Minimization

We thank the respected reviewer for their valuable time, and efforts in helping us to improve our work. We are grateful for the chance to engage deeply in the inner workings of APM.

In our reply to the respected reviewer Kzwl On memory/states and auto-regression in APM, we mentioned substituting the word autoregression with sequential, because the trigger column $T$ unfolds to yield location aware columns, i.e. $T_{ij} = (T|p_{ij})$, where $p_{ij}$ is the generated hard-coded positional encoding as being used in transformers[3], and neural fields[2]. These $T_{ij}$ columns are responsible for sequential querying of the MLP.

Mathematically, the MLP predicts a location specific feature $f_{ij}=MLP(T_{ij})$ as a consequence of a forward-pass as each column gets pumped through the net. The MLP is queried sequentially with different columns $T_{ij}$ until the locations $1 \leq i \leq H$, $1 \leq j \leq W$ get exhausted, where $H,W$ are the dimensions of the input image $I$.

In the shared code in supplemental, MLP does not contain explicit recurrence, because the trigger column $T_{ij}$ carries the entire sequence $T$. The $p_{ij}$ in $T_{ij}$ guides the MLP to decode location-specific feature $f_{ij}$, which is a form of feature-expression as also previously hypothesized in GLOM [1] (our changes over Section 2.1 and Figure 3 in the GLOM paper).

We can also gain more depth by looking at Fig 3 of the GLOM [1]. The subtle differences are, 1) [a b] in that figure is the trigger column $T$ in APM, which carries the whole image $I$. 2)$x4$ is substituted by a positional encoding $p_{ij}$. 3) In addition to decoding location-specific RGB, we also decode higher dimensional features $f_{ij}$, for $1 \leq i \leq H$, $1 \leq j \leq W$ which allows APM to do field-based-decoding, which we leveraged for downstream-classification. We have also added a detailed pseudo-code for operation of APM during test-time-training in Pseudocode.

The behaviour of APM is then akin to neural fields[2] (as suggested by the respected reviewer doB6), for eg, neural fields fire iid rays into the MLP shared across all input rays and decode location-specific rgb. Similarly, APM shares a MLP across all columns and decodes location-specific features. While neural fields work for 3D view synthesis, APM works for 2D image perception, with potential for extensions to other higher-dimensional input-percept, as also duly noted in Sec 11 of the GLOM paper[1].

As promised, we will also refine the paper-writing for better engagement with a prospective reader.

Yours sincerely,

Authors.

[1] Hinton, Geoffrey. "How to represent part-whole hierarchies in a neural network." Neural Computation 35.3 (2023): 413-452.

[2] Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.

[3] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. Advances in neural information processing systems. 2017;30.

We thank the reviewer for their timely response to our rebuttal. We are grateful for the reviewer's insight and help in improving our work.

We agree with the reviewer that the differences between TTA/TTT are indeed subtle/minor. As this difference do not play an important role in our contributions, we will refrain from differentiating between TTT/TTA works in our paper, and add more TTA works in related work and comparisons. We have shared comparisons with the recent TTA work aka Diffusion-TTA, and we will also add other such works.

In our work, we have followed the experimental setup of TTT-MAE [NeurIPS 2022] and TPT [NeurIPS 2022]. Furthermore, we will be grateful to mention the reference to TENT [Neurips 2020] in TTT-MAE paper: "Other papers following [42] have worked on related but different problem settings, assuming access to an entire dataset (e.g. TTT++ [28 ]) or batch (e.g. TENT [48 ]) of test inputs from the same distribution. Our paper does not make such assumptions, and evaluates on each single test sample independently".

Finally, we also point towards mention of TENT in the TPT paper: "However, TENT needs more than one test sample to get a non-trivial solution."

We would like to clarify a few other points:

• APM does not rely on any pretraining since weights are drawn from a normal distribution. While APM does rely on CLIP-pretrained weights, we do not perform any training/pretraining on ImageNet. We would be grateful to mention that CLIP was trained with 400M image-text pairs from OpenAI, and Openclip variant leveraged dfn5b. Furthermore, APM performs competitively on CLIP baselines and other methods which use CLIP weights.

"Moreover, "TTT adapts to each sample independently" is completely wrong. It is unnecessary that TTT has to either adapt to one sample or the entire batch or test set"

We agree with the reviewer here that TTT does not have to adapt to a single sample. Existing works like TTT-MAE[1] and others have focused on adapting to a single sample, as noted in their paper "By test-time training on the test inputs independently, we do not assume that they come from the same distribution".

We will be happy to discuss any further concerns,

Yours sincerely,

Authors.

[1] Gandelsman, Yossi, et al. "Test-time training with masked autoencoders." NeurIPS2022.

[2] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." NeurIPS2022

[3] Sun, Yu, et al. "Test-time training with self-supervision for generalization under distribution shifts." International conference on machine learning

p.s. other references follow paper order.

Yours Sincerely,

Authors

R1: Einstein, Albert. "On the electrodynamics of moving bodies."

R2: Geoff Hinton at Stanford, YouTube video, timestamp 49:07, https://www.youtube.com/watch?v=CYaju6aCMoQ

R3: LA Homogeneous Universe of Constant Mass .

R4: Neumann, John. "Theory of self-reproducing automata."

R5: Shannon, Claude Elwood. "A mathematical theory of communication."

R6: Hinton, Geoffrey E., et al. "The 'wake-sleep' algorithm for unsupervised neural networks."

R7: Bengio, Yoshua. "The consciousness prior."

We thank the reviewer for giving us their valuable time. Please find our responses:

On using the local patch variable $I_{xy} in the fig1 , but not in the methods. Fig 1 meant to be a general case of APM's operation. In methods, APM doesn't use $I_{xy}$. We were showcasing that positional codes break symmetry even without additional patch-prior and disentangle information from global $I$. Ablations in Tab 5b), validate local-patch-injection improving from 96.5-> 96.8 on C-10 .

On Whether unfolded state is being feed-forwarded through mlp or the folded state? Trigger column T exists in the folded state after a fwd pass(fig 1b), when knowledge has 'collapsed' into the weights of conv layer. When fwd-pass begins, T opens up (unfolds), and 'all' columns are forwarded through mlp. We apologize, since the middle column in fig 1a) is shown as folded and lent itself to confusion. We will update it.

On memory/states and auto-regression in APM APM encodes memory in 1) weights of T, and 2) weights of MLP. Both T and MLP are shared across all the locations. Weight-sharing induces the distributed-memory to learn relevant synaptic-patterns [11].

Auto-regression unrolls a shared-decoder over time. In contrast, APM holds the whole sequence in $T$, and directly hops onto a space/time-step [R1] via a location-column. Recurrence/feedback-loops are compensated for by a form of feature-expression ([12] & our changes over GLOM [10] sec 2.1, para 2. Positional code gives the notion of distance from starting point T, and iid columns help ensure parallel-processing of columns as in the third line of fig1 caption).

We convey this by choosing sequential. We will replace auto-regression in fig1a) and line 65.

On Details of folding-process/the gather step Folding operates at input of APM at the end of fwd-pass. Since a column $T_{ij}= (T|p_ij)$, and $p_{ij}$ is governed by hard-coded sinusoids, each $p_{ij}$ can be thought of as being annihilated during folding[14-16], aka deleting $p_{ij}$. Gradients from all $T_{ij}$ then flow back in towards the (center-of-mass[R3]/reservoir[R4]) column T[10].

'Gather' happens during the unfolded phase at mlp output: for a particular generated column $T_{ij}$, the net is predicting the feature $f_{ij}$. Gather estimates statistically running average of $f_{ij}$, as different $T_{ij}$ get pumped. Gather's output is used for final classification.

On Sources of patch-level features during supervision: We will clarify that patch-level features are the last-layer of a teacher and supervised via $L_{2}$ constraint even further in ablation section(line 231).

On APM's claims:percept is a field: We will rephrase as "APM is a step towards validating if input percept is a field". We will revise line 176 as: apm can work with randomly initialized weights, but requires a single representation distilled from a model trained at scale.

On apm's reliance on a teacher: We performed an additional experiment by removing the teacher from the apm, and doing rgb reconstruction on coco-train. L_2 rgb loss on coco-val fell to 0.0027. Training took far longer than if feature-vectors were also distilled from a teacher. Higher-dimensional vector-spaces carry more bits[R5], but incur significant randomness. Supervision from a stronger teacher compensates and guides apm to correct points in subspace. Progressing from vit b->h in tab1, makes apm more competitive.

We will add limitation: a.k.a reduced performance as cls token's dimension reduces.

On Using prompt-ensemble on the textual side: apm's imagenet-experiments in tab 1 used 80 hand-crafted prompts, same as CLIP. Openclip vit-h baseline also used the prompt ensemble, where apm also obtains competitive performance.

On Intuition behind final apm features surpassing teacher, stopping constraints: APM relies on distillation, where students have been observed to outperform their teachers. In fig 3, over 250 ttt iterations, the loss dropped from 1e-3 to 1e-12. We had to reduce the learning rate by a factor of 0.1 every 50 iterations. Whilst small, indeed it is not perfectly zero. One reason might be finite 64-bit precision of neural synapses and low learning rates. We waited-long to creep gradients into the net, as in [11]'s footnote 20.

One might build additional constraint in a student (aka APM) to estimate when its own fantasies[18]/predicted semantic-features are better than the teacher's and stop dynamically [R6]/recurse. Notions on soft decision-making for higher-level cognition are embedded in [R4,R7].

On Multi-iteration pseudo-code for ttt: Please find an anonymous code-linkPseudocode. We will update the paper.

On Quantitative effects of lower number of ttt iterations We provide the ablation in global rebuttal pdf in Table 1.

On Improving related work We will discuss more ttt/prompting/tta approaches. We shall add differences w.r.t feature refinement/distillation: a.k.a $L_2$ mimicking feature grids vs boltzmann-temperature-matching logits[3].

On Extending limitations sectionWe apologize if our limitations came across as optimistic future work. We have noted APM's limitation: APM requires multiple iterations for test time training, which might increase inference time in time-sensitive scenarios. It would be very interesting to evaluate its performance leveraging other zero-th order optimization techniques [11].We also report mean/std of ttt on dtd with variable its/seeds in global pdf. We also provide more results on imagenet-c with additional noise severities (1-4).

On Adding future work We will add apm's potential on dense tasks.

Yours Sincerely,

Authors

p.s please find references in comments

Thank you for your time and efforts in evaluating our work and providing positive and constructive feedback. We will be happy to address your comments:

Illustrative example, “self-driving car trying to make sense of the visual world while it is raining, but there were no such training scenarios in its training data”

We thank the reviewer for this insight, and we will replace this with a better example. We agree that TTT decisions are not made instantaneously in one iteration, in the cases where the net was not-pretrained on any source data, which also happens to be the setting where APM was evaluated. Some of the experiments did showcase APM being able to semantically cluster on COCO-val, after training on large datasets like COCO-train (Fig 5). It would be an interesting future-work to see optimizations such source-training brings towards reducing required ttt-iterations. APM is also computationally efficient (tab 4) compared to other baselines. It would be very interesting to evaluate its performance leveraging other zero-th order optimization techniques [R1,R2]. As [R3] also points out, such efficient-optimization would indeed be a good application for helping make instantaneous-decisions. We thank the reviewer for this observation, and will discuss this in future work also.

Analogies in sec 3.1 Thank you for this comment. As also pointed out by multiple-reviewers, we will shift these analogies to supplementary. We thank the reviewer for the suggestion to link our insights to 3D novel view synthesis. Indeed, given a single 3D-spatial coordinate of a pin-hole camera, neural fields operate by shooting i.i.d rays into the scene and leveraging the MLP to decode rgb. In a similar way, APM leverages location independent columns, a shared MLP, to decode location-specific features. While neural fields work for a 3D scene, apm works for a 2D percept, on a collection of images. We will add such a technical-analogy to our paper. We will correct the spelling mistake on line 17, and add the word 'nets'.

We thank the reviewer for giving us their valuable time, and will be grateful for the chance to engage in further discussions.

Sincerely,

Authors

[R1] Hinton, Geoffrey. "The forward-forward algorithm: Some preliminary investigations." arXiv preprint arXiv:2212.13345 (2022).

[R2] Malladi, Sadhika, et al. "Fine-tuning language models with just forward passes." Advances in Neural Information Processing Systems 36 (2023): 53038-53075.

[R3] Sun, Yu, et al. "Learning to (learn at test time): Rnns with expressive hidden states." arXiv preprint arXiv:2407.04620 (2024).

We thank the reviewer for helping us improve our work. Please find our responses below:

[1] On processing patches of an image one at a time/lack of novelty

We apologize that the novelties of our work were not clear. A CNN filter in any layer could process any patch by directly sliding on that region and performing convolution. Therefore, it can be said to still do asymmetric processing. However, for features to flow in the next layer, the whole CNN filter operation in the current layer will need to be finished thereby raising a waiting issue.

APM improves upon this by 1) breaking symmetry by location-aware columns, and 2) layer skipping: which allows apm to learn a mapping from input pixels to the last layer feature of a model via co-distillation. We will highlight these aspects more in the abstract.

Furthermore, as pointed out by reviewer KZWi, we will also focus on the ability of apm to recover scene-semantics from a global cls token and its potential for learning part-whole hierarchies via ssl-tasks.

[2] On improving related work by adding more works We agree that our related work could be improved with works from other relevant areas such as test-time-adaptation as mentioned by reviewer PHih. Furthermore, as pointed out by reviewer KZWi, we will add more prompting approaches. We have also made a note to add an additional section clarifying the feature refinement/distillation.

[3] On the presented biological analogy in section 3.1 We apologize if our biological analogy motivated by GLOM came across as non-technical. We will shift biological insights (sec 3.1) to supplementary. We have provided more clarifications on trigger columns, folding-unfolding operation in our responses to reviewer KZWi. We have also added pseudo-code illustrating apm's operation during TTT (https://anonymous.4open.science/r/apm_rebuttal-F3D1/ttt_pseudocode.png).

[4] On providing precise-hyperparameters/train/test-details for reproduciblity Taking inspiration from [38], we provide Reproducibility Statement: In order to ensure the reproducibility of our experiments, we have shared the code in supplementary during the review process. The code, model weights shall be released publicly post-review. We will also share a docker image.

Hyperparameter tables: Code has been shared in supplemental.

Architecture details: With input dimensions $h, w, c$ and feature dimension $d_p$: dimensionality of positional encoding. $s$: stride of convolutional filter in encoder, $d_c$: dimension of the CLS token of the teacher on which APM learns.

[5] On extensions of apm to few-shot testing cases We appreciate the reviewer's forward-outlook of apm's application beyond ttt (line 216 of paper). We will add an additional discussion on few-shot-testing in future work.

We will be happy to respond to further clarifications. Thank you so much for your valuable time,

Yours Sincerely,

Authors

p.s. references follow paper order