OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation

We deeply thank reviewer 5fwS for their valuable opinion and constructive feedback on our work. We are encouraged that they found our approach “novel”, our paper quality “excellent”, the topic “timely and highly relevant”, our architecture and ideas “clear”, and our experimental results “complete and convincing”. In the following, we address their questions:

The encoded embeddings are augmented with a learnable modality-specific auxiliary signal. The need for this auxiliary information is presented very briefly: please improve it with clearer examples or references to similar approaches in the literature.

We thank the reviewer for the opportunity to clarify the auxiliary embeddings.

As shown in our ablations (L283–286), adding learnable bounding box embeddings improves performance (e.g., ~+1% in skeleton action recognition). These embeddings serve a similar role to positional encodings in Transformers—here, encoding the spatial location of a person in the frame (e.g., (x, y, w, h)).

This spatial signal helps the predictor establish pixel-to-keypoint correspondence and disambiguate multiple people in a scene. Without it, the model may associate skeletons to the wrong subject, especially in crowded scenes (e.g., predicting joints for the wrong person when several are present).

This idea parallels techniques in multi-human trajectory forecasting [1] and pose estimation [2], where auxiliary spatial cues improve identity tracking. While we use bounding box embeddings in this work, the framework generalizes—e.g., using voice ID tags in future audio extensions.

We will revise the manuscript to make this rationale and flexibility clearer.

Sect. 3 (Methodology) is very dense and mixes architectural design and training approach: for clarity and reproducibility, it would be useful to substitute or integrate a textual description of the more algorithmic steps with pseudo code (e.g., the details in Cross-Modal Masking Strategy paragraph)

We appreciate the reviewer’s careful read and constructive feedback.

Indeed, we recognize the opportunity to improve the representation of some of the dense technical details of the methodology section. To address this, we have included a high-level pseudo code outlining the core steps of OSKAR’s training pipeline. In the camera-ready version, we will further improve the organization of the architectural and training descriptions and provide a more detailed and structured pseudocode in the Appendix.

# Inputs:
# modalities = {
#   'modality_1': input_1,
#   'modality_2': input_2,
#   ...
# }
# total_visible_token_budget: e.g., 128
# total_target_token_budget:  e.g., 128

# Step 1: Tokenize and embed inputs
tokens = {}
for modality, input_data in modalities.items():
    raw_tokens = Tokenizer[modality](input_data)
    tokens[modality] = Embed(raw_tokens, modality_tag=modality)

# Step 2: Sample visible and target tokens under fixed budgets

modalities_list = list(tokens.keys())
num_modalities = len(modalities_list)

visible_props = Dirichlet(alpha=[0.5] * num_modalities).sample()
target_props  = Dirichlet(alpha=[0.5] * num_modalities).sample()

visible_counts = AllocateTokens(
    visible_props,
    total_visible_token_budget,
    min_tokens=MIN_VISIBLE,
    max_tokens=MAX_VISIBLE
)

target_counts = AllocateTokens(
    target_props,
    total_target_token_budget,
    min_tokens=MIN_TARGET,
    max_tokens=MAX_TARGET
)

# Sample tokens per modality based on allocated counts
for modality in modalities_list:
    visible_tokens[modality], target_tokens[modality] = SampleTokens(
        tokens[modality],
        num_visible=visible_counts[modality],
        num_target=target_counts[modality]
    )

# Step 3: Encode visible tokens with shared fusion encoder
fused_latents = FusionEncoder(visible_tokens)

# Step 4: Encode full tokens with modality-specific teacher encoders (for targets)
with no_grad():
    teacher_outputs = {}
    for modality in target_tokens:
        teacher_outputs[modality] = TeacherEncoder[modality](tokens[modality])

# Step 5: Predict masked target tokens using modality-specific predictors
predictions = {}
losses = {}
for modality in target_tokens:
    predicted = ModalityPredictor[modality](fused_latents)
    target_indices = target_tokens[modality].indices
    target = teacher_outputs[modality][target_indices]
    predictions[modality] = predicted
    losses[modality] = LossFunction(predicted, target)

# Step 6: Backpropagate total loss
total_loss = sum(losses.values())
total_loss.backward()

A point worth mentioning in the imitation is the problem of unpaired multimodal data. It would be interesting to understand if the added flexibility of OSKAR can be exploited to pretrain on a dataset with unpaired and missing modalities.

Indeed, the design of OSKAR is very friendly to datasets with missing modalities. In fact, even in the current version trained on paired multimodal samples, the Dirichlet sampling strategy often omits a subset of modalities completely in some batches stochastically, effectively simulating unpaired settings. This demonstrates that OSKAR already operates under a partially unpaired regime and can be extended to train on unpaired or heterogeneous datasets without modification. This opens the door for training on unpaired samples from any dataset with any number of modalities.

We sincerely thank reviewer 5fwS for their time reading this rebuttal. We are committed to incorporating their comments on the auxiliary embeddings, methodology section, and training on unpaired data in the camera-ready version of our paper.

[1] Saadatnejad, S., et al. (2024). Social-transmotion: Promptable human trajectory prediction. ICLR.

[2] Xu, Y., et al. (2025). DynPose: Largely Improving the Efficiency of Human Pose Estimation by a Simple Dynamic Framework. In CVPR.

We thank reviewer XjPx for their insightful comments and constructive feedback. We are glad that they recognize the “nice writing and presentation”, and our method being able to “effectively integrate the features of action videos and texts”, and achieve “competitive performance on downstream tasks”. We are also encouraged that they are able to “clearly understand the framework”. In the following, we address their concerns:

1. Validation on Large-Scale Data

Lack of validation on large-scale data. I think effectiveness on large-scale data is the key to current representation learning, enabling it to be better applied to VLM or VLA tasks. The paper shows competitive performance on small-scale data, but there is still a gap compared to other methods that use more data.

We acknowledge the reviewer’s concern regarding evaluation at scale. To address this, we pretrained OSKAR-G (1.3B) ahead of the rebuttal, and we evaluated it across widely accepted benchmarks.

• #PT. samples denotes the number of pretraining samples

OSKAR-G further bridges the gap with methods using more data:

• On action recognition: OSKAR-G achieves 77.9% on SSv2 (new SOTA) and scores within 0.8% of InternVideo-v2 on K400, despite pretraining on 40x less data.
• On vision-language tasks: OSKAR-G surpasses VAST on MSRVTT-QA, and comes within 5 points of GRAM on VATEX T2V retrieval, despite pretraining on only 36% of GRAM’s data size.

While we could not train on even larger datasets due to the limited compute budget and the small rebuttal window, we believe that OSKAR is already trained at a decent scale and is competitive with methods using more data on popular evaluation datasets.

Further, we highlight the following aspects of OSKAR:

• Consistency: OSKAR not only matches the performance of prior methods with similar model and data scales (Tables 1–5), but also competes closely with much larger expert models across 5+ downstream tasks—with consistency. This is achieved despite OSKAR being pretrained for general-purpose use, without any task-specific customisations or distillation from any external models.

• Scalability and efficiency: OSKAR achieves these results while being a data-, parameter-, and label-efficient learner. For example, with only 5% of the labels, it surpasses VideoMAEv2 by 26.5% and V-JEPA by 2.6% (see main paper Figure 3).

• Multimodality and generalization: The major contribution of OSKAR lies in its unified pretraining, generic use, and strong performance. Unlike the methods in comparison, it has the extra benefit of working seamlessly across other tasks (ex, SOTA performance in skeleton action recognition).

In summary, in addition to performance improvements, OSKAR excels in consistency, multimodality, generalisation, efficiency, and scalability. We believe that these features make OSKAR a practical and valuable foundation for the community.

2. SOTA Comparisons

The baseline methods used for comparison are somewhat outdated and do not fully reflect the current state-of-the-art. For example, in an action recognition task, there is no Comparison of current SOTA models(InternVideo[1], Unmasked Teacher[2]，Hiera[3]). These situations also exist for other tasks:Text-to-video retrieval(GRAM, VATEX[4]), VideoQA(VAST[5]).

We appreciate the reviewers concern on SOTA comparison. To directly address this concern, we have now included all the suggested baselines—InternVideo, Unmasked Teacher, Hiera, GRAM, and VAST—in our comparison tables above. We also added other recent models for completeness. Across tasks, OSKAR-G is on par with these methods despite using up to 40x less data and no external distillation.

We would like to clarify that, in our main paper, we limited comparisons to methods of similar model scale, input resolution, and data budget, to ensure a fair and meaningful evaluation. We excluded methods that:

• Are initialized or distilled from external models, e.g., InternVideo, TeachCLIP;
• Employ higher input resolutions.
• Or involve larger model or dataset scales, e.g., GRAM, VAST, InternVideo-v2.

Please note that this way of ensuring fairness by limiting comparisons is a common practice (ex, in [8]) . Additionally, in our appendix (Tables 12-18), we already compare to some of these recent methods mentioned by the reviewer (UMT, InternVideo) with clear notes on scale, resolution, and supervision differences to maintain transparency.

That said, the reviewer’s concern is very well acknowledged; we are committed to updating our camera-ready version to resolve it.

3. Use of Modality-Specific Tokens in the Fusion Encoder

Q1: For the fusion encoder, are there special tokens used as identifiers for tokens of different modalities when they are input?

Modality identity is encoded using learnable modality-specific embeddings. OSKAR learns per-modality embeddings that act like soft identifiers, allowing the model to distinguish and integrate different modalities during attention. Please refer to L106-115 (main paper, methodology sec. 3) for more details.

4. Impact of Mask Rate on Training and Convergence

Q2: The impact of mask rate on model training and performance. The author did not discuss the mask rate in the paper. Will using too high a mask rate cause the model to fail to converge or reduce performance?

The mask rate is implicitly determined by the fixed token budget, which controls how many tokens are visible during training (ablated in Table 9). For instance, 128 visible tokens out of 2,640 (1568 video + 816 skeleton + 256 text) yields ~95% masking, which performs well across tasks.

As the reviewer correctly expected, pushing to 99% (32 visible tokens) leads to degraded performance. This aligns with prior work (e.g., VideoMAEv2 [11]), which caps masking at ~95% to combat temporal redundancy and improve efficiency without hurting performance.

Unlike these methods, OSKAR’s input and target tokens are multimodal, enabling cross-modal interaction and improving generalisation across diverse tasks. We study these benefits in Table 6 (ablation section 5).

5. Symmetric vs. Asymmetric Predictor Design

Q3: In the self-supervised training paradigm of mask reconstruction, asymmetric encoders and decoders[1] are usually used, such as MAR. In my understanding, the modality predictor plays a role similar to the decoder in the framework of this article. The author kept the structure of the predictor consistent with the encoder in the experiment. Why is this? Please experiment or discuss this.

The reviewer correctly understands the similarities between the decoder in MAE and predictor in OSKAR. We include further ablations below (with encoder depth=8, width=512) to clarify the effect of an asymmetric predictor. We observe that a deeper predictor noticeably improves performance (+3.0 → +3.8), while increasing width has marginal effect.

• Metric: average gain over baseline across three tasks as described in L239-241, Sec. 5.

Given that the predictor output dimension must match the target encoder's latent dimension, we adopt a symmetric predictor to avoid redundant projection layers and maintain architectural simplicity. Importantly, unlike MAE, our predictor performs semantic latent prediction across modalities, requiring sufficient capacity for generalization.

We sincerely thank reviewer XjPx for their time reading this rebuttal. We hope we sufficiently addressed their major concerns. We are committed to incorporate their feedback on evaluation at scale, SOTA comparisons, and predictor design in the camera-ready version of our paper.

[6] Wang, R. et al. (2023). Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In CVPR.

[7] Wang, Y. et al. (2022). Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191.

[8] Wang, M. et al. (2025). Action Detail Matters: Refining Video Recognition with Local Action Queries. In CVPR.

[9] Chen, S. et al. (2024). Cosa: Concatenated sample pretrained vision-language foundation model. ICLR.

[10] Tian, K. et al. (2024). Holistic features are almost sufficient for text-to-video retrieval. In CVPR.

[11] Wang, L. et al. (2023). Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR.

[12] Kunchang Li et al. (2023). UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer. In ICCV.

[13] Hur, C., et al (2025). Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions. In CVPR.

We thank reviewer 1kEh for their valuable feedback. We are glad that they found our paper to be “written clearly and concisely”, our experimental results “rigorous and detailed”, our technical details to be ”specific” and “detailed”, and our method “flexible and extensible”, “shows promising generalization performance”, and “demonstrates effective multimodal grounding, high transferability, and scalability.” In the following, we discuss their raised question on LLMs:

In the Video QA section, only a BART-based language module was used. I am curious whether OSKAR can be effectively adapted to existing LLMs, which would further enhance the significance and effectiveness of this method.

We thank the reviewer for highlighting the potential of adapting OSKAR to large language models (LLMs). Although our current experiments use a lightweight BART decoder for VideoQA, this choice was made to isolate and evaluate OSKAR’s fused representations. In fact, OSKAR is inherently compatible with LLMs and designed to serve as a modality-agnostic fusion encoder for multimodal reasoning.

Recent works such as BLIP-2 [1], Flamingo [2], and MiniGPT-4 [3] have shown the effectiveness of freeze-then-adapt approaches for LLM integration. OSKAR naturally aligns with this paradigm: Its fuse-then-predict architecture already learns to integrate video, skeleton, and text into a shared semantic space. The encoder is decoupled from task-specific heads, enabling its outputs to be interfaced with LLMs via adapters or direct token-level alignment.

Importantly, OSKAR provides not just compatibility but added value over prior visual backbones typically used in LLM-based systems. Most existing approaches rely on frozen image-text models (e.g., BLIP-2[1] and CLIP [4]) that struggle with rich temporal dynamics or motion reasoning. In contrast, OSKAR’s pretraining explicitly fuses temporally-aware, multimodal inputs—including motion and language—into a coherent representation space. This makes it particularly well-suited for tasks that require temporal abstraction, action understanding, and long-range multimodal context, which are difficult to capture with static image features alone. This modularity means OSKAR can act as a drop-in fusion backbone for future LLM-based systems—offering general, scalable multimodal representations with minimal changes.

While LLM integration is not explored in this version due to resource constraints, we are actively pursuing this direction. We appreciate the reviewer’s insight and will add a discussion to emphasize OSKAR’s role as both a standalone multimodal learner and a valuable complement to LLM-based architectures.

We thank reviewer 1kEh for their time reading our response.

[1] Li, J., et. al (2023, July). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML. PMLR.

[2] Alayrac, J. et al. (2022). Flamingo: a visual language model for few-shot learning. NeurIPS.

[3] Zhu, D., et al. (2024). Minigpt-4: Enhancing vision-language understanding with advanced large language models. ICLR

[4] Radford, A. et al. (2021, July). Learning transferable visual models from natural language supervision. In ICML. PmLR.

We thank reviewer 376k for their insightful comments. We are glad that they recognize the “strong performance”, “extensive ablation studies”, “practical and scalable design”, “manageable” compute, and the flexibility to train on unlabelled data in our method. In the following, we address all concerns and questions:

1. Statistical Significance of Performance Gains

Standard errors are not reported, but performance over other, closely performing models do not seem to be statistically or significantly significant. Most of them range from 0.3 to 1.0 percentage points. I tend to look at differences under 1 or 2 points as likely not statistically significant in the absence of variance measures. + The authors gave a reason why there are no error plots. But the results look suspiciously non-significant. It might be worth the computational cost to verify OSKAR's dominance. Or, make a statement saying even if OSKAR's gain are minor, there are other, architectural reasons for using it.

To address the reviewer’s concerns regarding statistical significance, we conducted additional ablation experiments (with the setting described in L239–241, Sec. 5). Specifically, we pretrained with 5 different random seeds and evaluated each model 5 times with varied evaluation seeds across 3 downstream tasks (total: 5 pretrains × 15 evals = 75 runs). We report the mean ± standard error of absolute percentage point improvements over the baseline.

OSKAR’s improvements consistently exceed baseline variability. For example, the VidCls gain (+3.18) surpasses the baseline’s standard error (0.04) by almost 80×, far beyond the 95% confidence interval—indicating statistically robust gains across all 75 evaluations.

While we could not re-run similar analysis on the large scale (10M+ samples) due to the extremely high compute cost and short rebuttal window, we would like to highlight the following points:

1. Consistent and significant gains: OSKAR either exceeds or matches prior methods across 5+ downstream tasks (Tables 1–5) with consistency, often exceeding the 2-point threshold noted by the reviewer. For example,

   • Improves over VideoMAE-B by +3.5 on AVA Action Detection (Table 2)
   • Surpasses OmniVL (+2.6, MSRVTT) and Lavender (+4.3, MSVD) in T2V retrieval (Table 3)

2. Efficiency and scalability: OSKAR is data-, parameter-, and label-efficient. For example, under limited supervision (5% of labels), it outperforms VideoMAEv2 by +26.5% and V-JEPA by +2.6% (see Figure 3).

3. Multimodality and generalization: OSKAR provides unified, task-agnostic pretraining and generalizes across diverse tasks; for example, unlike the methods in comparison, it achieves SOTA in skeleton action recognition.

In summary, OSKAR's contribution extends beyond performance improvements, its real power lies in its consistency, multimodality, generalisation, efficiency, and scalability.

2. Interaction Effects in Ablation Studies

Ablation studies seem thorough, but they do not take into account interactions. They are also highly dependent on the order in which features were added or removed.

We agree that analyzing component interactions is critical. To address this, we conducted a factorial-style ablation, jointly varying key components: EMA update parameter (λ), attention routing (fusion, target, predictor), and input/target token counts.

• +++ and ++ denote λ = 0.998 and λ = 0.9998, respectively.
• C and S denote cross-modal all-to-all token interactions and separate intra-modality token interactions in the attention routing, respectively.
• Metric: average gain over baseline across three tasks as described in L239–241.

The best settings (Rows 3–4) combine fast EMA, separate target attention, and asymmetric tokens—showing our design choices are complementary, not just additive. We adopt Row 3 by default for the best performance/efficiency tradeoff. This confirms that gains hold across interaction axes, addressing concerns about order dependence.

3. Momentum Tuning and Generalization

Fine-tuning of momentum may complicate generalizing to other new contexts.

We understand the reviewer’s concern that EMA parameter λ values can impact generalization. Please note that careful tuning of customized EMA update parameters per modality is NOT required in OSKAR. In fact, we adopt a shared λ by default, which already achieves competitive performance across all downstream tasks (Tables 1–5).

When adding a new modality or context, momentum fine-tuning—like any other hyper-parameter—remains an optional enhancement. Our default setting already performs well across diverse contexts. As noted in the conclusion (Lines 300–303), we view momentum tuning as a promising direction for future work and plan to investigate adaptive or learned momentum strategies to further boost generalization and robustness without manual tuning.

4. Token Budget and Modality Scaling

The authors acknowledge that a limitation of the paper is that only 3 modalities were tested. How would performance change because of the fixed budget. Or is there some dynamic re-allocation of the budget across modalities as some modalities appear to be more important?

We break down your question into two parts and run targeted ablations to answer each separately.

First, does the fixed token budget hurt performance as you add more modalities?

Our additional ablation studies in the table above show that:

• Adding modalities under small fixed budgets (e.g., 64 tokens) does not always improve performance (Rows 1–3), likely due to token competition. For example, adding text in Row 3 (vs. Row 2) does not increase the average score, yet it expands applicability—enabling VideoQA support (+2.7, row 3).

• As the budget increases (Rows 4–6), adding all three modalities yields noticeable gains (+2.6 → +3.8, rows 4 → 6), peaking at 128 tokens. Beyond that (Rows 7–8), improvements plateau, suggesting diminishing returns. This suggests 128 tokens is a sweet spot—large enough to represent all modalities well, yet small enough to remain efficient and computationally practical.

These findings suggest that modality inclusion is helpful when sufficient capacity is available.

Second, dynamic re-allocation of the budget:

We conducted further ablation experiments to compare three token allocation strategies: Balanced, Weighted, and Dirichlet sampling. Balanced allocation gives equal tokens per modality, offering moderate gains (+2.3). Weighted allocation favors larger modalities like video, improving VidCls (+3.4) but hurting text-rich tasks like VidQA, lowering the average gain (+1.9). Dirichlet sampling (adopted by default in OSKAR) stochastically varies token ratios per batch (while ensuring token balance over the entire training), enhancing diversity and generalization, with the best overall performance (+3.8). Overall, dynamic and modality-aware strategies like Dirichlet sampling are most effective, especially in imbalanced or noisy multimodal settings.

5. Limitations and Failure Cases

Can you provide cases where OSKAR did not perform as expected, or any limitations you encountered while experimenting?

OSKAR excels at understanding tasks, but its design inherently limits generation. We explored video generation from OSKAR’s latent space, but results lacked the visual fidelity and temporal coherence of state-of-the-art diffusion models. This is expected, as OSKAR’s objective prioritizes semantic abstraction over pixel detail, enabling strong transfer to understanding tasks. Nonetheless, its powerful latent representations are promising for future work with generative decoders.

We sincerely thank Reviewer 376k for their thoughtful and constructive feedback. We hope that we sufficiently addressed their major concerns. We are committed to incorporating their feedback—particularly on statistical reporting, component interactions, and generalization—in the camera-ready version.