Token Bottleneck: One Token to Remember Dynamics

We sincerely thank Reviewer 3Edo for the thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.

W1. Clarification on the description of the decoder architecture

We apologize for the confusion. During the decoding process, the visible and masked target patches are concatenated with the bottleneck token and then fed into the self-attention layer. We promise to address the confusions in the descriptions.

W2. Comparison with i-BOT, DINOv2 and T-CORE

We provide the comparison results of DINOv2 [11], iBoT [15], and T-CORE [16] in five tasks in Franka Kitchen. Since we have provided the performance of DINOv2 in Franka Kitchen in the supplementary material (Table A), we directly report the results in Table R13. Moreover, we further evaluate the performance of iBoT and T-LORE. As shown in Table R13, our ToBo surpasses DINOv2, iBoT, and T-LORE on all the five tasks. We promise to provide the performance of DINOv2, iBoT, T-LORE in other benchmarks in the revised version.

[Table R13. Comparison of ToBo with DINOv2 and T-CORE on Franka Kitchen]

References

[11] Oquab et. al., "DINOv2: Learning Robust Visual Features without Supervision", arXiv:2304.07193.

[15] Zhou et., al., iBOT: Image BERT Pre-Training with Online Tokenizer, ICLR 2022.

[16] Liu et. al., "When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning", CVPR 2025.

We sincerely thank Reviewer dztw for the thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.

W1. Comparison with DINO v2 (beyond franka-kitchen), I-JEPA, and V-JEPA

Following the comments, we provide the comparison results of DINOv2 [11], I-JEPA [12], V-JEPA [13]. We provide the performance of I-JEPA and V-JEPA for five tasks of Franka Kitchen in Table R10. Moreover, since we have provided the performance of DINOv2 in Franka Kitchen in the supplementary material (Table A), we additionally compare ToBo with DINOv2 on 'relocate' and 'pen' tasks in Adroit in Table R11. As shown in Table R10, our ToBo surpass DINOv2, I-JEPA, and V-JEPA on all the tasks in Franka Kitchen. Moreover, ToBo also outperforms DINOv2 on all the tasks in Adroit. These results further support the effectiveness of ToBo. We promise to provide the performance of DINOv2, I-JEPA, V-JEPA in other benchmarks in the revised version.

[Table R10. Comparison of ToBo with DINOv2, I-JEPA, and V-JEPA on Franka Kitchen]

[Table R11. Comparison of ToBo and DINOv2 on Adroit]

W2. Scalability of our method when trained on more samples

Following the advice, we verify the scalability of ToBo when trained on more samples. To this end, we compare ToBo models pre-trained on Kinetics-400 and Kinetics-600, a larger dataset that includes Kinetics-400, for 100 epochs. Table R12 showcases that the ToBo model pre-trained on Kinetics600 surpasses the ToBo model pre-trained on Kinetics-400in all tasks except for the "Sdoor Open" task, where both models perform comparably. This results validate the scalability of ToBo for more samples.

[Table R12. Scalability of our method when trained on more samples]

W3. Clarification for much smaller improvements on label propagation

The improvements in video label propagation are smaller because it requires dense and spatially distributed predictions, which do not directly align with our SSL objective. In contrast, the strong gains in robotics stems from the nature of our SSL objective, which enforces conservative summarization of observed scenes into a bottleneck token, a representation better suited for action prediction under partial observability.

The following are responses to the sub-questions:

Response to "Is the agent's policy architecture being more suited to the proposed SSL method?" The observed improvements in robotics tasks are not due to any special advantage in policy architecture. Our method directly adopts the policy network design and the evaluation protocol from prior works [3], without any modification specific to our SSL objective. Importantly, all compared SSL methods share the same policy architecture, ensuring a fair comparison. Therefore, the gains are not attributable to an architectural bias toward our method.

Response to "Is the agent policy architecture the same for all the robotics tasks?" Yes, the same policy network architecture is used across all robotics tasks. This uniform setup eliminates architectural variation as a source of performance gaps. Although the training mechanisms of models vary across SSL methods, all models share the same structure, which enables the use of a single policy design across the pre-trained models.

References

[1] He et. al., "Masked Autoencoders are Scalable Vision Learners," CVPR 2022.

[2] Gupta et. al., "Siamese masked autoencoders," NeurIPS 2023.

[3] Jang et. al., "Visual representation learning with stochastic frame prediction", ICML 2024.

[6] Kay et. al., "The kinetics human action video dataset," arXiv:1705.06950, 2017.

[11] Oquab et. al., "DINOv2: Learning Robust Visual Features without Supervision", arXiv:2304.07193.

[12] Bardes et. al., "Revisiting Feature Prediction for Learning Visual Representations from Video", arXiv:2404.08471.

[13] Assran et. al., "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture," ICCV 2023.

Thank you for responding to authors. Would you mind to do the "Mandatory Acknowledgement" for the review? Did the author's response change your score?

-- Your AC

Thank you for the thoughtful feedback! We're glad that our response addressed your concerns. Please feel free to let us know if there are any remaining points we can further clarify!

We sincerely thank Reviewer W34j for the thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.

W1. Ablation studies on key design choices

We agree that the commented factors are important for understanding the behaviors of ToBo. Thus, we conduct ablation studies on the suggested aspects. All the ablation studies are conducted using models pre-trained for 100 epochs.

• Ablation study on the number of bottleneck tokens

  • We conducted an ablation study on the number of bottleneck tokens, varying it among {1, 2, 4, 8}. As shown in Table R1, we observe that using a single token generally yields the best performance across tasks. These results demonstrate that conservative summarization without separate storage is beneficial for understanding the current observation, thereby improving action prediction in robotics

• Ablation study on the impact of temporal difference

  • We conducted additional experiments where the maximum frame interval is varied among (48, 96, 144). As shown in Table R2, the models are encouraged better to learn to capture dynamic scene evolution when trained with moderate temporal differences, not too short to include meaningful changes and not too long to break temporal coherence.

• What if there is no temporal difference between the frames?

  • Following the comment, we applied our method using the same frame for both the source and target scenes. As shown in Table R3, we found that our method still works even without temporal difference, surpassing other baselines (e.g., MAE [1], SiamMAE [2], and RSP [3]) with significant gaps across the tasks. However, its overall performance degrades compared to original ToBo since it loses the opportunity to learn how to capture dynamic evolutions from consecutive scenes. This highlights the importance of temporal contrast for effective pre-training of ToBo.

• What if multiple source frames are given?

  • We further compare ToBo with the model pre-trained using multiple source frames. Specifically, we randomly sample 4 frames as the source frames and pre-train the model for 100 epochs under the same training recipe to ToBo. As shown in Table R4, the multi-frame based model surpasses other baselines (e.g., MAE [1], SiamMAE [2], and RSP [3]) in most of the tasks. However, despite requiring higher pre-training cost, it underperforms compared to ToBo across all robotics tasks. These results suggest that while it is possible to extend ToBo to multi-frame settings, such naive extension may encounter potential new challenges, leading to suboptimal performance.

[Table R1. Ablation study on the number of bottleneck tokens]

[Table R2. Ablation study on the impact of temporal difference]

[Table R3. Analysis on the importance of temporal difference]

[Table R4. Analysis on the multiple source frame]

W2. Comparison of training and inference flops

We conducted FLOPs evaluation for both training and inference to quantitatively compare the computational cost of each model, as summarized in Table R5.

• During inference, all models use the same backbone architecture and input resolution without any input masking, resulting in identical inference FLOPs at the same model scale (e.g., 4.6 GFLOPs for ViT-Small).
• During training, ToBo, MAE [1], and SiamMAE [2] show similar computational costs while RSP [3] requires substantially more computation of 32.5 GFLOPs due to its complex decoding mechanisms. When considering computational costs with downstream performance (e.g., performance in Franka Kitchen [4]), these results further support the effectiveness of ToBo, which achieves a strong balance between efficiency and performance.

[Table R5. Comparison of training and inference FLOPs and downstream performance in Franka Kitchen ]

W3. Comparisons of ToBo and MAE on Kinetics-400

Following the comment, we compare the action classification accuracy of ToBo and the baselines (i.e., MAE [1], SiamMAE [2], and RSP [3]) on the Kinetics-400 dataset [6]. To this end, we fine-tune all models under a reduced setting of 40 epochs, due to the limited rebuttal period. As shown in Table R6, while the performance gaps are relatively smaller than those appears in our main downstream tasks in the main paper, ToBo achieves the highest accuracy among all models, demonstrating its effectiveness on this standard video classification benchmark.

[Table R6. Comparison of ToBo and other baselines in action recognition in Kinetics-400]

W4. Comparison with straightforward baselines

We apologize for the confusion. We would like to clarify that MAE [1] in the tables of the main paper is indeed a straightforward baseline that performs masked image modeling independently on both input views. We will improve the description for the baselines.

References

[1] He et. al., "Masked Autoencoders are Scalable Vision Learners," CVPR 2022.

[2] Gupta et. al., "Siamese masked autoencoders," NeurIPS 2023.

[3] Jang et. al., "Visual representation learning with stochastic frame prediction", ICML 2024.

[4] Gupta et. al., "Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRL 2019.

[5] Russakovsky et. al., "Imagenet large scale visual recognition challenge," IJCV 2015.

[6] Kay et. al., "The kinetics human action video dataset," arXiv:1705.06950, 2017.

Thank you again for your constructive comments! We're glad that your questions have been addressed. Are there any remaining points we can further clarify? Please feel free to let us know if there are any remaining concerns!

We sincerely thank Reviewer GhiU for the thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.

W1. Clarification on concerns regarding variance

• While addressing your comment, we found that the variance of ToBo in the DMC benchmark [7] was incorrectly reported as 8.6, when it should have been 0.86. We will correct this error (8.6 --> 0.9) in Table 2 of the main paper, as reflected in Table R7.
• ToBo may tend to exhibit relatively high variance in RLBench [10]. However, we believe this reflects a broader trend where model variances vary significantly depending on the evaluation benchmark. In fact, on other benchmarks such as CortexBench [8] (Table R7) and Franka Kitchen [4] (Table R8), ToBo shows lower or comparable variance than other methods except “Sdoor Open”. Moreover, even in the "Sdoor Open" task, ToBo records a variance of 7.1, which is lower than that of SiamMAE (7.9) and CropMAE (8.1). These results suggest that variance is task-dependent, and ToBo maintains stable performance in several benchmarks despite relatively higher sensitivity in certain RLBench scenarios

[Table R7. Experimental results on vision-based robot policy learning on CortexBench.]

[Table R8. Experimental results on vision-based robot policy learning on Franka Kitchen.]

W2. Supplement to the limitation part

Thank you for the advice. We improved the discussion in the limitation part: Due to the resource constraints, we both did not check the scalability of our method beyond huge scale and did not explore beyond the commonly used input resolution of 224x224. Additionally, our study focused on a simplest setting involving two dynamic scenes to learn temporal dynamics. Extending our pipeline to multiple frames setting would be an interesting direction for future work, solving new potential challenges emerging from multi-frame setting.

W3. Extension of ToBo to multi-frame-based approach

Following the comment, we compare ToBo with the model pre-trained using multiple source frames. Specifically, we randomly sample 4 frames as the source frames and pre-train the model for 100 epochs under the same training recipe to ToBo. As shown in Table R9, the multi-frame based model surpasses other baselines (e.g., MAE [1], SiamMAE [2], and RSP [3]) in most of the tasks. However, despite requiring higher pre-training cost, it underperforms compared to ToBo across all robotics tasks. These results suggest that while it is possible to extend ToBo to multi-frame settings, such naive extension may encounter potential new challenges, leading to suboptimal performance.

[Table R9. Comparison with the naive extension of ToBo to multi-frame-based setting]

References

[1] He et. al., "Masked Autoencoders are Scalable Vision Learners," CVPR 2022.

[2] Gupta et. al., "Siamese masked autoencoders," NeurIPS 2023.

[3] Jang et. al., "Visual representation learning with stochastic frame prediction", ICML 2024.

[4] Gupta et. al., "Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRL 2019.

[7] Tassa et. al., "dm_control: Software and tasks for continuous control," arXiv:2006.12983, 2020.

[8] Majumdar et. al., "Where are we in the search for an artificial visual cortex for embodied intelligence?," NeurIPS 2023.

[9] Eymaël et. al., "Efficient image pre-training with siamese cropped masked autoencoders," ECCV 2025.

[10] James et. al., "Rlbench: The robot learning benchmark & learning environment," IEEE Robotics and Automation Letters, 2020.