UniViT: Unifying Image and Video Understanding in One Vision Encoder

W1: U-RoPE and VideoRoPE share a common goal but differ in approach, and should be discussed as complementary methods.

A1: Thank you for your suggestion. Both U-RoPE and VideoRoPE target temporal positional encoding stability, though our design is tailored for unified image-video representation learning.

VideoRoPE is a representative and impactful work that redesigns the RoPE mechanism to enhance modeling capacity for very long sequences, especially in large multimodal models. In contrast, our approach focuses on unified modeling for vision foundation models, and introduces a simple yet effective solution by re-allocating frequency bands to decouple spatial and temporal encodings. This design improves both temporal stability and fine-grained semantic sensitivity without altering the core RoPE structure.

We believe these two approaches are complementary, each suited to different use cases and architectures. We will include a more comprehensive discussion of their similarities, differences, and applicability in the revised version of the paper.

W2: Slot-VLM also targets structured semantics via slot separation, and its conceptual overlap should be acknowledged.

A2: Thank you for pointing out the omission of Slot-VLM, a highly insightful and relevant work. Slot-VLM leverages Slot Attention to explicitly separate object-centric (Slow Slots) and event-centric (Fast Slots) representations. It represents a significant advancement in structured semantic modeling, offering explicit semantic disentanglement, dynamic reasoning, and strong interpretability, an increasingly important direction in video understanding.

In contrast, our method employs offline global multi-granularity clustering to model object-level and event-level semantic structures. Our approach is more focused on learning a shared semantic structure across modalities through dense pseudo-labeling. Nonetheless, the instance-level semantic separation in Slot-VLM offers a fresh and inspiring perspective.

We are particularly interested in the possibility of integrating dynamic slot assignment mechanisms with our offline clustering framework, which may further improve the model’s generalization and structural awareness. This is a direction we have not explored yet, and we plan to investigate it in future work.

We will include a discussion of Slot-VLM and cite it appropriately in the revised version to acknowledge its contributions.

W3: Key baselines like InternVideo and OmniMAE are missing, limiting the strength of empirical comparisons for unified vision modeling claims.

A3: Thank you for your valuable suggestions. We acknowledge that several important baseline models were missing in the original submission. To address this, we have added additional experiments under the full attentive probing protocol, including comparisons with OmniMAE as a strong unified image-video pretraining baseline. The results are shown below:

We will continue to improve the completeness of our experimental comparisons in future revisions.

Regarding InternVideo, we note that it involves a more sophisticated post-training stage and uses distillation from VideoMAE, making direct comparison with our method, which is trained fully from scratch, somewhat unfair at this point. Nonetheless, we are committed to transparent benchmarking and have trained a new version of UniViT under similar pretraining conditions (including SSV2), but without any post-training. The attentive probing results are as follows:

These results demonstrate UniViT’s strong performance across diverse benchmarks, even without relying on extra post-training stages. We will include these expanded comparisons and clarify training protocols in the revised paper to enhance fairness and reproducibility.

W4: While the paper is generally clear, some key implementation details are implicit.

A4: Thank you for pointing out the ambiguity in our explanation. Your understanding is absolutely correct, image inputs are handled in the same way as 1-frame segments within the VS² module. We apologize for the lack of clarity in our original description and will revise the terminology and implementation details in the updated version to eliminate any confusion.

Q1: Could you explain why you chose the 3/4 split for frequency allocation in U-RoPE, and how the use of U-RoPE compares to VideoRoPE in terms of performance or suitability across different tasks

A1: Thanks for the interesting question. We chose the 3/4 frequency allocation split in U-RoPE based on the following considerations:

1. In our training and downstream evaluation settings, video sequences are relatively short. Therefore, fewer dimensions are needed for modeling temporal position information.
2. In contrast, spatial semantics (e.g., textures, object boundaries) tend to be more complex. Allocating more frequency components to the spatial dimension helps preserve high-frequency spatial details, enhancing the model’s spatial resolution and local perceptual sensitivity.

Unlike VideoRoPE, which redesigns the RoPE structure itself, U-RoPE preserves the original RoPE mechanism and achieves spatiotemporal decoupling via frequency reallocation. This makes our approach simpler, more efficient, and easier to generalize across model architectures. As shown in Fig. 3 (c) and Table 3 (b), U-RoPE demonstrates modest yet consistent improvements over M-RoPE. We believe this design provides better task adaptability with minimal implementation overhead.

Q2: How are image inputs integrated into the VS² module? Are they treated as single-frame segments consistent with the video processing pipeline? This should be made explicit.

A2: Yes, thank you for raising this point. We apologize for not making this clear in the paper. Your understanding is absolutely correct, image inputs are treated exactly the same as single-frame segments within the VS² module. We will explicitly clarify this implementation detail in the revised version to avoid any confusion.

Q3: Why is InternVideo not included in Tables 1 and 2, despite being a strong video-only model with image transfer performance?

A3: Thank you for your thoughtful review. We acknowledge that InternVideo is a strong video-centric baseline with impressive image transferability. In our main tables (Tables 1 and 2), we opted not to include models that benefit from additional supervised post-training (e.g., on K700 or SSV2), in order to maintain a consistent and fair evaluation protocol focused on zero-shot and pretraining-only settings. This choice was intended to better highlight the generalization capabilities of our model under comparable conditions.

We recognize the value of including strong video baselines for comprehensive evaluation. Accordingly, we provide a variant of UniViT trained under comparable pretraining conditions to InternVideo (with SSV2 included), yet without additional post-training. The results are reported in our response to W3 to facilitate a fairer comparison under aligned training protocols.

Q4: Will you consider adding OmniMAE, OmniVL, and Omnivore as baselines or at least explicitly positioning UniViT relative to them?

A4: We have addressed this question in our earlier response above. Thank you again for the suggestion.

Q5: Have you explored how sensitive your clustering strategy is to hyperparameters (e.g., number of clusters or k-means initialization)?

A5: We appreciate your insightful comment. You are right that the earlier version of our paper lacked a detailed analysis of clustering sensitivity. To address this, we conducted an ablation study focused on the number of cluster classes. The results are shown below:

We observed that performance peaks around 1 million clusters, and then gradually declines as the number increases. We believe the degradation is due to several factors:

1. As the number of clusters increases, each cluster contains fewer samples, weakening semantic cohesion and reducing the effectiveness of pseudo-labels.
2. Excessively fine-grained clustering can lead the model to overfit local patterns, especially for short or semantically redundant video segments.
3. A larger number of clusters increases the chance of noisy or unstable pseudo-labels, which may hurt contrastive training and generalization.

Based on this analysis, we chose 1M clusters as the default configuration, striking a good balance between performance and computational cost. In future work, we plan to explore dynamic cluster allocation strategies to further improve clustering quality and model robustness.

Q6: Training efficiency and overhead of VS² and clustering vs. contrastive/masked modeling

A6: Thank you for bringing up this comprehensive question. Compared to contrastive models such as PE-CLIP, our method is more computationally efficient, as we do not require a text encoder, which significantly reduces the training overhead while still achieving better performance.

When compared to masked modeling approaches, although our framework introduces slightly higher computational overhead due to the clustering and VS² module, we observe a substantial performance improvement. For example, VideoMAE-g reports an average performance of 35.3%, whereas our method achieves 73.1%, demonstrating a significant gain in representation quality.

We believe this trade-off is favorable and will provide additional implementation details in the version to clarify the cost-performance balance.

W1: Discrepancy in Baseline Results: There are notable inconsistencies between the reported results and the original papers for several baselines.

A1: Thank you for your careful review and for pointing out the ambiguities in our experimental setup. We have addressed your concerns from the following three perspectives: (1) evaluation protocol differences across baselines, (2) probing vs. full-data settings, and (3) unified comparison with and without multi-view testing.

1. Unlike works such as V-JEPA, which evaluate models using multiple spatial and temporal views, a strategy known to improve results through test-time ensembling, we intentionally adopted a simpler evaluation protocol that uses only a single center spatial crop and a single temporal clip. This minimal setup is designed to transparently assess the intrinsic quality of the learned visual representations, without relying on auxiliary inference strategies. Our objective is to isolate the encoder’s contribution and enable fair, consistent comparisons across all baselines under identical conditions.

2. We apologize for the confusion caused by the lack of detailed explanation in the main paper. Due to space constraints, we did not explicitly specify that all baselines were evaluated under a unified probing protocol to ensure consistency. To further clarify, we standardized our evaluation setup and also reproduced results under the commonly used multi-view configurations for reference. Specifically, we tested two setups: (1) 16 frames, 1 view, 1 clip, and (2) 16 frames, 2 views, 3 clips. All models were trained for 20 epochs using the same backbone, input resolution, and optimization configuration to ensure a fair and controlled comparison. This design helps disentangle the true representational quality of the model from improvements attributed to test-time ensembling or view aggregation.

3. Thank you also for pointing out that we did not clarify the probing strategy on video. In our original submission, all models were evaluated using a consistent 50-shot attentive probing setup, which emphasizes representation quality under limited supervision. In contrast, results such as the 35.6% reported by DINOv2 on SSV2 were obtained using the full training set, which naturally leads to higher performance. To illustrate this, we reproduced DINOv2’s results under both settings. As shown in Table 2, DINOv2 achieves 19.6% with 50-shot probing and 35.2% under full-data training, confirming that the performance gap is due to differences in evaluation protocol rather than implementation discrepancies. We adopted a uniform probing setup across all baselines to ensure fair comparison, and we hope this clarification helps contextualize the reported results.

Table 1: Attentive probing results under 50-shot and full-data settings, showing UniViT's significant improvement and competitive performance across SSV2 and K400.

Table 2: Linear probing results on SSV2 under 50-shot and full-data settings, illustrating the performance gap caused by limited data.

These results show that the performance gap mainly stems from probing vs. full-data settings. Under full supervision, UniViT achieves comparable or superior results, with discrepancies from original reports within 1%, confirming the reliability of our evaluation. We will release our full evaluation code for reproducibility.

W2: Weak Temporal Modeling: Regarding W1, on motion-centric benchmarks such as Something-Something V2.

A2: Thank you for your patient and detailed review. We next provide a more detailed analysis to address your concern regarding temporal modeling performance.

1. V-JEPA was pre-trained directly on the SSV2 dataset, which inevitably introduces a degree of data leakage. In contrast, we intentionally excluded SSV2 from our training data to more accurately assess the generalization ability of our model.
2. To further avoid overfitting on evaluation benchmarks, we trained our video models exclusively on a further refined subset of the InternVid dataset, without mixing in SSV2. Despite this, our model demonstrates competitive performance. For example, UniViT-L achieves 32.1% on SSV2, outperforming PE-Core-G (23.7%), which similarly does not use SSV2 in training, and even surpassing VideoMAE-G (27.9%), which does include SSV2 during pre-training.
3. To directly address your concern, we also conducted experiments with SSV2 included in the training set. As shown in the table below, our model achieved a +15% performance gain on SSV2 in the 50-shot attentive probe setting, exceeding the performance of V-JEPA-L. Notably, performance on other benchmarks remains stable or even improved, indicating our model's strong robustness and generalization.

Table 3: Results with SSV2 included in training, where UniViT-L outperforms V-JEPA-L across all benchmarks, demonstrating strong generalization.

The results show that including SSV2 in training leads to improved performance on SSV2 without compromising other benchmarks. UniViT-L not only surpasses V-JEPA-L on all datasets but also maintains strong generalization across diverse video tasks.

4. To further validate our temporal modeling capabilities, we evaluated our models on the Charades dataset under the OpenTAD[2] framework. Importantly, neither model was fine-tuned on the downstream task, yet our model shows significantly stronger performance compared to V-JEPA-L on Charades dataset:

Table 4: Temporal localization on Charades, where UniViT-L consistently outperforms V-JEPA-L, indicating stronger temporal modeling.

This substantial improvement in mAP and consistent gains across all tIoU thresholds demonstrate that UniViT captures richer temporal structures even without task-specific fine-tuning. These results further validate its effectiveness in zero-shot temporal understanding.

W3: Missing Full Finetuning Comparison:he paper lacks a comparison under full fine-tuning settings, which is critical for comprehensive evaluation of the learned representations in downstream tasks.

A3: Thank you very much for raising this important point. Our primary goal is to validate the robustness of UniViT by evaluating it under more challenging and diverse scenarios.

While our initial focus was on linear and attentive probing to highlight the transferability of the frozen backbone, we recognize that different methods may exhibit varying behavior under full fine-tuning, especially in terms of task-specific adaptation and upper-bound performance. Therefore, in response to your suggestion, we conducted additional full fine-tuning experiments across a wide range of downstream tasks, covering video datasets (e.g., SSV2). These experiments were designed to assess not only accuracy improvements but also the adaptability of the unified architecture under task-specific supervision.

The table below is a partial summary of the updated results, demonstrating that UniViT performs competitively or even outperforms strong baselines like V-JEPA under full fine-tuning settings.

Table 5: Full fine-tuning results showing UniViT-L is competitive with V-JEPA-L on SSV2 even without pre-training on SSV2.

These results confirm that UniViT not only offers strong frozen representation quality but also adapts effectively under full supervision. We believe this addresses your concern and demonstrates UniViT’s competitiveness in both transfer and fine-tuned settings. Additional results on more benchmarks will be included in the final version.

We also acknowledge that VideoRoPE [1] is a very strong and well-designed method. We will include it in the discussion in the revised version of the paper. In the updated manuscript, we will provide the full table with complete results and clearly annotate all experimental configurations (e.g., number of frames, views, training protocols, etc.) to support a comprehensive and transparent comparison, including a detailed discussion of VideoRoPE.

We sincerely hope that our detailed responses and newly added experimental results effectively address your concerns and help to clarify the strengths and robustness of our approach. We would greatly appreciate it if you would consider updating your rating and confidence accordingly.

References:

[1] VideoRoPE: What Makes for Good Video Rotary Position Embedding?

[2] OpenTAD: A Unified Framework and Comprehensive Study of Temporal Action Detection

W1: Limited novelty.

A1: We understand the concern that the contribution of our work may appear limited. We would like to clarify several key points that demonstrate the conceptual and novelty of our work:

1. Unlike MLCD, which is primarily designed for image-language tasks, our proposed UniViT introduces a unified Transformer-based architecture capable of jointly handling images and videos event representations. Rather than simply stacking image processors on top of video sequences, UniViT unifies the input encoding scheme, spatiotemporal interaction modules, and training objectives across different modalities.
2. A direct extension of MLCD to video often leads to degraded image performance. Increasing the proportion of image data to compensate hampers video understanding. To resolve this conflict, we introduce a novel multi-granularity event modeling approach that balances image and video training. This not only preserves image classification performance but also achieves SOTA results on video benchmarks. Furthermore, it enables the model to generalize across varying temporal scales, a capability beyond MLCD.
3. UniViT consistently achieves superior performance across a wide range of downstream tasks, including image classification, video understanding, and action recognition. These results validate the practical effectiveness of our unified architecture and demonstrate its potential as a general-purpose vision foundation model.

We will further revise the paper to clearly articulate the motivation and conceptual innovations over MLCD and related work. We sincerely thank the reviewer for raising this point and helping us improve the clarity of our contributions.

W2.1: Lack video-only baselines, and image performance is not clearly superior.

A2.1: Thank you for your suggestion.

1. Our method focuses on extending the video capabilities of the Unified framework, while preserving image performance. As shown in our experiments, our performance on image tasks is competitive with MLCD, while achieving significant improvements in temporal understanding, such as a 10% gain on SSV2. To further address this concern, we have added comparisons with MLCD and UniViT on several image benchmarks, as shown below: | Method|Aircraft|DTD |Cars|Birdsnap|Pets| |-|-|-|-|-|-| | MLCD-L|86.7|80.9|82.8|86.6|89.2| | UniViT-L|87.1|81.4|82.3|86.7|88.6|

2. To address the lack of comparisons with video-only models on image tasks, we further evaluate such models under the same attentive probing setting as V-JEPA on image benchmarks. Despite not being tailored for image, these video models provide a reference for image generalization. The results below demonstrate that UniViT achieves clearly superior performance, further supporting its effectiveness in unified representation learning. | Method|Places|iNat|Imagenet| |-|-|-|-| |V-JPEA-L|66.4|60.3|72.6| |VideoMAE-H|59.1|65.5|72.3| |UniViT-L|70.8|86.4|84.8|

W2.2: 50-shot attentive probing is uncommon

A2.2: Thank you for pointing out the ambiguity in our experimental setup.

1. Our evaluation protocol follows the 50-shot attentive probe setting introduced in V-JEPA, aiming to measure the model's transfer ability and data efficiency.
2. Regarding the Food dataset, both zero-shot and full-attentive probe results for PE-Core reach the same reported accuracy of 96.9%. We believe this is because PE-Core/G already achieves near-saturation performance on Food, leaving little room for further improvement through probing. This also indicates the strong generalization capability of PE-Core in the zero-shot setting.
3. Additionally, we re-evaluated PE-Core and reproduced an accuracy of 95.8%, which is close to the originally reported value.
4. For video datasets, we also adopted a single-view attentive probing setting, training for 20 epochs using the full dataset. The results are summarized below: | Method|SSV2|K400|K600|K700| |-|-|-|-|-| | V-JPEA-L|66.7|75.9|77.7|66.5| | UniViT-L|51.3|85.5|86.1|75.9|

W2.3: SSV2 result in VideoMAE

A2.3: We sincerely appreciate your detailed review.

1. Regarding the SSV2 performance, we would like to clarify that the results reported by VideoMAE are based on pretraining only on SSV2, which may lead to overfitting.
2. In contrast, our UniViT is pretrained on large-scale multimodal data without including SSV2, aiming to evaluate its generalization capability.
3. To further assess the effectiveness of our method, we conducted an additional experiment by including SSV2 in the training set and reported linear probing results as follows: |Dataset|UniViT-L|VideoMAE|AdaMAE|MoCo v3| |-|-|-|-|-| |SSV2|47.1|38.9|40.1|33.7|

W3: Wavelength mitigate periodic oscillation

A3: Thank you for raising this insightful question. We want to explain that our design is driven by two considerations:

1. We aim to support much longer video inputs in future settings enhancing the model’s ability to capture long-range dependencies without segment-level fragmentation, which often results in information discontinuity. To accommodate this extended length, we cannot rely solely on wavelength-based approaches.
2. Our dense labeling strategy is designed to reduce redundancy between adjacent frames and enhance local discriminability. While increasing the base wavelength slows the phase progression and might reduce aliasing, it also risks making the model less sensitive to subtle temporal changes, counter to our goal of capturing fine-grained temporal dynamics.

We will elaborate on these design choices more clearly in the revised paper.

W4.1: Definitions of "object" and "event"

A4.1: Thank you for pointing this out. We sincerely apologize for the confusion. Our choices stem from the nature of the clustering process. In our framework, image-level clustering tends to produce object-level semantics, such as dresses or cakes, which are static and visually coherent. In contrast, clustering of video clips captures temporal dynamics, often corresponding to event-level semantics like a foot-washing ceremony or cement mixing. These inherently involve temporal evolution and are characterized as "events." Our goal is to enable the model to learn global representations for longer clips, while shorter sequences help learn fine-grained sub-event representations (such as steps). To support this, we perform offline clustering with up to 1 million clusters, enabling fine-grained labeling of video segments. This provides the model with richer pseudo-supervision and helps capture subtle temporal patterns.

W4.2: What is the VS2

A4.2: Apologize for it. The core idea of VS² is to better model temporal structure by dividing a video into multiple event segments of varying lengths before the encoder. To improve temporal discrimination across events, we manipulate the positional encoding such that frames from different events are spaced farther apart in the positional domain, while maintaining the relative structure within each event. For example, consider where a 16-frame video is divided into four 4-frame events. Instead of applying standard positional encoding from 0 to 15, we encode each 4-frame event as [0+i16,1+i16,...] for the i-th event. This effectively enlarges the relative distance between different events, encouraging the model to capture fine-grained intra-event details. As for the aggregation function, it is implemented as an attentive pooling module, which maps variable-length feature sequences into a fixed-dimensional representation.

W4.3: What is Figure 3d intended to convey?

A4.3: Apologies for the earlier confusion regarding Fig. 3d. Its purpose is to measure how similar a video representation is to a single frame from the same video, high similarity implies weak temporal modeling. UniViT shows lower similarity than other models, indicating stronger temporal awareness and better distinction between static and dynamic content.

W4.4: What is the initial feature extraction model?

A4.4 Thank you for your question. The initial features used during the clustering stage are extracted using the CLIP model, following the same setup as described in the MLCD paper to ensure a fair comparison. We chose CLIP for its strong semantic generalization capabilities, which helps improve the quality of the clustering results.

Q1: Common protocol

A1: Thank you for the helpful suggestion. We agree that a more comprehensive comparison between UniViT and representative video encoders is important to assess the effectiveness of UniViT. To address this, we have additional evaluations of UniViT under common protocols. The results are summarized below:

Q2: Why spatial do not cause the issue

A2: Thank you for your thoughtful question. We would like to clarify that high-frequency periodic oscillation is not inherently problematic, representing a certain bias. Under a shared RoPE strategy, different dimensions behave differently. Prior works have shown:

1. Lower dimensions are sensitive to local semantics and relative distances.
2. Higher dimensions are suited for capturing long-range dependencies.

In space, high frequencies aid fine detail modeling; in time, they can disrupt temporal coherence. To ensure stable temporal representation, we assign lower frequencies to the temporal axis. We'll clarify this design choice in the revision.

Q3: The paper need to be polished.

A3: Thanks for your suggestion. We appreciate your feedback and will carefully revise the writing for clarity and polish in the final version.

Dear Reviewer,

Thank you once again for your thoughtful and constructive feedback. We sincerely appreciate the opportunity to further clarify key aspects of our work.

We hope the additional explanations help convey the motivation and significance of our unified framework, particularly in addressing the semantic and architectural challenges posed by integrating image and video understanding. If any part remains unclear or would benefit from further elaboration, we would be more than happy to provide additional analysis or examples. We would greatly appreciate it if you please consider increasing the score in light of these clarifications and additional insights.

Best regards,

The Authors

Dear Reviewer mwRn

We sincerely thank you once again for the time and effort you have devoted to reviewing our work and for the constructive and detailed feedback you provided throughout the discussion phase. Your engagement has been invaluable in helping us refine the presentation and clarify the contributions of our framework.

We would like to kindly check whether our latest responses have fully addressed your concerns. If there are aspects requiring further elaboration, such as the key contributions we highlighted, the role of multi-temporal scale beyond data augmentation, or our temporal modeling results, we would be glad to provide additional explanations or supporting evidence. In particular, we have conducted targeted experiments to address your concern regarding performance on SSV2. While V-JEPA was pre-trained directly on SSV2, we intentionally excluded SSV2 from our training data to more rigorously evaluate generalization. Even under this stricter setting, our model achieved competitive results, outperforming models trained without SSV2 and in some cases even surpassing those trained with it. Furthermore, when including SSV2 in training, our approach delivered a +15% improvement on SSV2 in the 50-shot attentive probe setting, while maintaining or improving performance on other benchmarks.

Beyond SSV2, we have also validated the temporal modeling capability of our framework through extensive experiments on diverse datasets such as Charades under the OpenTAD framework, where our model significantly outperforms strong video baselines without task-specific fine-tuning. These consistent gains across multiple datasets further confirm the robustness and generalization ability of our unified design. Our aim is to ensure that the novelty and technical merits of our framework are conveyed with complete clarity.

In your latest comment you also mentioned raising your score in light of our clarifications. We truly appreciate this recognition. We have noticed that the updated rating does not yet appear in the system and wondered if this might have been an oversight.

Thank you once again for your thoughtful consideration and valuable time. We greatly appreciate your contribution to improving the quality of our work.

Best regards,

The Authors

W1: The method's reliance on offline clustering with embeddings from a separate pretrained model is a significant weakness, as it may introduce biases from the initial model and limits the framework's ability to learn end-to-end.'

A1: Thank you for your thoughtful examination of this key design aspect. We acknowledge the concern that relying on offline clustering with embeddings from a separately pretrained model could introduce biases from the initial model and may, to some extent, limit the end-to-end trainability of our framework. We would like to clarify the motivation and benefits behind this design choice:

1. Compared to alternatives such as distillation or masked modeling, our clustering-based approach is actually less dependent on the initial model. We do not directly mimic its features; instead, we use the semantic embeddings only to generate compact and robust pseudo-labels through clustering, which then guide downstream training.
2. Clustering allows us to explicitly model hierarchical semantic structures, e.g., object-level and event-level concepts, which are difficult to capture directly through most self-supervised objectives.
3. The offline clustering significantly reduces computational cost during training, enabling us to scale effectively to large mixed image-video datasets.

We also recognize the limitations of this design in certain scenarios, such as domain generalization or task transferability. To address this, future work will explore integrating the clustering objective into an end-to-end trainable pipeline, for example, through online clustering or proto-token matching, to make the pseudo-labeling process learnable, reduce dependency on the initial model, and improve generalization. We will include a discussion of this trade-off and future direction in the final version of the paper.

W2: The research suffers from a major reproducibility issue, as the authors state that the code and a key dataset are proprietary and cannot be released, making it difficult for the community to verify the results or build upon the work.

A2: Thank you for your feedback. After careful consideration and discussion, we commit to releasing all pretrained models and code to support the development of vision foundation models and multimodal large language models (MLLMs). We believe this will provide valuable resources to the community for both verification and further research. Additionally, we will open-source the complete evaluation pipeline to ensure full reproducibility of our results.

W3:The ablation study on the number of cluster classes suggests that performance peaks at 1 million classes and then declines, but the reason for this decline is not fully explored.

A3: Thank you for raising this important point. You are right that the earlier version of our paper lacked a detailed analysis of clustering sensitivity. To address this, we conducted an ablation study focused on the number of cluster classes. The results are shown below:

We observed that performance peaks around 1 million clusters, and then gradually declines as the number increases. We believe the degradation is due to several factors: As the number of clusters increases, each cluster contains fewer samples, weakening semantic cohesion and reducing the effectiveness of pseudo-labels. Excessively fine-grained clustering can lead the model to overfit local patterns, especially for short or semantically redundant video segments. A larger number of clusters increases the chance of noisy or unstable pseudo-labels, which may hurt contrastive training and generalization. Based on this analysis, we chose 1M clusters as the default configuration, striking a good balance between performance and computational cost. In future work, we plan to explore dynamic cluster allocation strategies to further improve clustering quality and model robustness.

Q1: How does the quality of the initial feature extractor impact final performance, and was online or iterative clustering considered to allow joint adaptation of encoder and cluster assignments?

A1: Thank you for the important question. Our use of offline clustering based on pre-computed features is a deliberate design choice aimed at reducing reliance on the initial model while maintaining training efficiency. Unlike distillation or MIM, we don’t directly mimic features but instead use semantic embeddings to generate pseudo-labels via clustering, mitigating initial bias. In addition, this choice brings several practical advantages:

1. On large-scale image-video mixed datasets, offline clustering significantly reduces computational overhead during training. Unlike online or iterative clustering approaches, we avoid repeatedly alternating between clustering and training, which can be costly and unstable, especially under limited computing resources.
2. Maintaining global cluster centers allows us to bypass the need for extremely large batch sizes often required by contrastive learning or distillation methods. This leads to a more lightweight and flexible training paradigm.

As part of future work, we are planning to explore two lightweight alternatives to enable better co-evolution:

1. Periodically update cluster centers using exponential moving average of features during training, without full re-clustering, to balance adaptability and efficiency.
2. Design learnable or adaptive strategies to update cluster centers throughout training, enabling the model to better adjust to evolving feature distributions.

Q2: What is the intuition behind this specific allocation, and how sensitive is the model's performance to different frequency distribution ratios between the spatial and temporal components?

A2: Thank you for the great question. Our decision to allocate a larger portion of the frequency spectrum to the spatial dimensions (75%) and a smaller portion to the temporal dimension (25%) is based on two core intuitions:

1. Compared to spatial resolution, videos typically have shorter sequence lengths in the temporal axis. Hence, fewer dimensions are needed to encode temporal positions effectively.
2. Allocating more dimensions to spatial encoding preserves high-frequency spatial details, which improves spatial semantic representation and enhances local perceptual sensitivity. To assess the sensitivity of the model to this frequency allocation, we conducted an ablation study on the UniViT-S model using different frequency split ratios. The results on SSV2, K400, and ImageNet are as follows: |Frequency (Temporal/Spatial) |SSV2| K400| ImageNet | |-|-|-|-| |5:6|16.6|67.9|80.8| |3:4|16.8|67.6|80.4| |1:2|16.7|67.6|80.1|

These results indicate that while the model is relatively robust to small variations in the split ratio, the 75/25 allocation offers a good trade-off between spatial and temporal modeling performance. We will add this discussion to the revised version for clarity.

Q3: The paper successfully unifies image and video modalities. Could you discuss the primary challenges you foresee in extending the UniViT framework to other data types, such as audio or text, to create an even more broadly unified "n-modality" encoder?

A3: Thank you for the inspiring question. While UniViT is primarily designed to unify visual modalities, we are indeed interested in extending it toward a more general "n-modality" unified encoder. However, this direction introduces several fundamental challenges:

1. Visual data are continuous signals in space and time, while text is a discrete sequence of symbols, and audio is a continuous waveform or spectrogram. These modalities differ significantly in representational structure, resolution, and temporal granularity, making unified modeling inherently difficult.
2. Images and videos can share a common tokenizer, but it is much more challenging to use the same tokenizer across modalities like speech and text.

Despite these challenges, we believe UniViT provides a strong foundation. In future work, we are interested in exploring joint multimodal pretraining, shared cluster anchors, or cross-modal alignment heads as promising directions toward a truly unified multimodal framework. We are very excited by this possibility and would be happy to further discuss any ideas you might have for building such a generalized system.

Q4: How were the different temporal scales for multi-event segments (e.g., 1, 2, 4, 8, 16 frames) chosen, and is there an optimal combination or a point of diminishing returns when adding more scales?

A4: Thank you for your question. We will explan for it. In particular, using 2-frame segments contributes little to performance improvement. We believe this is because such short clips are insufficient to capture meaningful event-level semantics, and thus provide limited benefit in modeling video dynamics. However, we still include 1- and 2-frame segments for an important reason: they help better align video segments with single-frame image representations, especially under the 1-frame setting. This alignment is critical for achieving unified modeling between images and videos. The following table presents our ablation results.

These results confirm that while longer event segments (e.g., 4 or 8 frames) contribute most to video understanding, including shorter ones (1–2 frames) still offers gains by improving alignment across modalities. If we further increase the temporal length of input sequences, it remains an open question whether additional temporal scales would continue to yield substantial benefit. We consider this a promising direction for future work.

Thanks for your rebuttal. The authors address most of my concerns and I will keep my rating. I suggest the authors should include the new discussion proposed in the rebuttal. And I hope the authors would open-source both pretrained models and training code.

W1: The method is claimed to learn object-level representations for images and event-level representations for videos. Although ablations show that combining both improves performance, further analysis is needed to verify that the learned features (or clusters) truly capture these properties. In particular, it should be analyzed what kind of feature representations can be obtained for long-term and short-term events.

A1: Thank you for pointing out the need for deeper analysis of the learned object-level and event-level representations. We acknowledge that interpreting feature behavior across different temporal scales is important, and we have added detailed visualization and clustering analyses to better understand how our model captures both short-term and long-term event structures.

Our current evaluation primarily focuses on quantitative results (e.g., performance on K400 and SSV2) and qualitative clustering visualizations (Fig. 6 and Fig. 7), which demonstrate the model’s effectiveness in unifying image and video representations. Our goal is to learn representations at multiple semantic granularities: for example, from single-frame inputs to 16-frame clips, we expect the model to capture both global semantics and fine-grained object-level details.

To analyze this, we examined cluster visualizations. We observed that object-level clusters often capture single-object semantics (e.g., "cake", "wedding dress") or simple multi-object scenes (e.g., "a child with a backpack"), while event-level clusters correspond to action or event semantics (e.g., "mixing cement"). This suggests the model is indeed separating visual concepts across semantic levels. We will include representative cluster visualizations in the final version to further support this analysis.

While our current analysis offers initial insights, it does not yet provide fine-grained verification of the temporal specificity of event representations, particularly in distinguishing long-term versus short-term dynamics. Our framework is designed with the intent that long-range segments encode high-level, holistic event semantics, whereas short-term segments capture finer-grained procedural steps or sub-actions. To facilitate this, we adopt an offline clustering strategy with a high-capacity vocabulary (1 million clusters), enabling the generation of detailed pseudo-labels that promote segment-level semantic learning. We consider this an important direction for future research and will provide a more in-depth discussion on temporal decomposition analysis in the final version of the paper.

W2: As the authors also point out, since clusters are determined offline, biases from initial feature extraction remain, posing challenges related to domain shift.

A2: Thank you for your constructive comment, this indeed touches on a key limitation of our method. As you pointed out, the clustering process is performed offline based on initial features, meaning the model cannot dynamically update representations during training. This can potentially introduce bias from the initial features and affect generalization, especially under domain shifts. However, we would like to clarify that generating pseudo-labels via clustering introduces only a weak and indirect dependency on the initial model, in contrast to direct distillation from a teacher model. This mitigates the influence of the initial features to some extent. Our choice of an offline clustering strategy was a trade-off between training stability and computational efficiency. Compared to iterative clustering methods like DeepCluster [1], our approach maintains scalability and avoids the instability that may arise from end-to-end joint optimization on large-scale datasets. We recognize that the risk of bias still exists. To address this, we carefully selected the initial model based on findings from the MetaCLIP [2] paper, opting for CLIP, which has been shown to be among the least biased available models. Incorporating iterative or dynamically updated clustering could further enhance semantic modeling and cross-domain robustness. This is indeed a key direction for our future work, and we will make this limitation and its potential solutions clearer in the revised paper.

Q1: I would like the authors to address the weaknesses I pointed out.

A1: Thank you again for your thoughtful feedback. We have carefully addressed each of the concerns you raised, including verifying the object-level and event-level representations, analyzing short-term and long-term temporal structures, and mitigating biases introduced by offline clustering. We hope our detailed responses and new experimental results help resolve the issues and demonstrate the strengths and robustness of our approach. We would be grateful if you could consider updating your rating and confidence accordingly.

Q2: The paper states that a 1:1 ratio between images and videos was maintained during training. Does this ratio refer to the number of images versus the number of video frames (e.g., 2 K videos × 16 frames = 32 K frames, which would not match the number of images), or is it defined by the number of tokens after aggregation with VS²? Please clarify this design choice and the reasoning behind it.

A2: Thank you for the detailed question. We follow the first interpretation you mentioned, the 1:1 ratio refers to the number of images versus the number of video frames. For example, 2K videos × 1 clip (e.g., 16 frames) are treated as 2K video samples, matched with 2K image samples. We conducted preliminary ablation studies to compare different image-to-video data ratios. Our design prioritizes improving performance on video tasks while maintaining competitive image performance, aiming to enhance video modeling without sacrificing image understanding, as compared to MLCD. The table below shows the results on UniViT-S with varying data ratios.:

Table 1: Ablation results on different image-to-video sample ratios, showing that a balanced 1:1 ratio provides a good trade-off between video and image performance.

These results suggest that using a balanced 1:1 image-to-video sample ratio yields the most favorable trade-off between video and image performance. While increasing the video ratio (1:3) slightly improves SSV2 performance, it leads to a drop in image accuracy. Conversely, increasing the image ratio (3:1) compromises video modeling. This supports our design choice of maintaining a balanced data composition to ensure strong video understanding without significantly degrading image-level representation learning.

References:

[1] Deep Clustering for Unsupervised Learning of Visual Features

[2] Demystifying CLIP Data