Vintix: Action Model via In-Context Reinforcement Learning

Thank you for your review. Based on your feedback, we have identified these key points:

1. How does Vintix compare to the latest action models?
2. Does Vintix learn to adapt or does it mimic expert through task identification?
3. How does the quality of dataset impact performance?
4. Providing more detailed supplementary materials.

We include responses and experiments conducted within the rebuttal period. If anything is missing, feel free to reach out — we will respond as promptly as possible

---

How does Vintix compare to the latest action models?

To broaden the comparison beyond JAT, we include expert-normalized scores for REGENT on MuJoCo and MetaWorld — the two domains shared across all works (Table 1, Table 2)

Vintix outperforms both JAT and REGENT in normalized scores but lags behind REGENT on entirely unseen tasks. Notably, unlike REGENT, Vintix received no demonstrations for these tasks and operated in a fully cold-start setting.

---

Does Vintix learn to adapt or does it mimic expert through task identification?

Exp 1

We evaluated whether AD-style training on noise-distilled trajectories outperforms behavioral cloning with context, using a model identical to Vintix but trained solely on expert data. Both models were evaluated using the procedure described in lines 238–239 of the paper.

Figure 1 shows that ED converges to ~0.8, while AD reaches 0.97 (MuJoCo) and 0.95 (MetaWorld). This highlights the importance of policy-improvement data for self-correcting behavior. ED shows stable performance in MuJoCo and surprisingly improves with more shots in MetaWorld, likely due to task inference still occuring under a shared encoder (requiring 4 episodes in context) but still underperforms AD despite being trained on the same amount of data.

Exp 2

Secondly, we examine whether Vintix relies on the reward signal for self-improvement during inference. To test this, we re-trained Vintix without access to rewards and compared it to the original model using the cold-start inference on tasks from the MuJoCo and MetaWorld domains.

Figure 1 shows that reward feedback is crucial: the masked-reward variant performs worse asymptotically in both domains and converges more slowly on MetaWorld. These results suggest that training on data reflecting policy improvement is essential for enabling in-context reinforcement learning

---

How does the quality of dataset impact performance?

To evaluate the impact of dataset quality, we collected a MuJoCo dataset using an untuned noise decay schedule, which led to non-smooth policy improvement on some tasks. We re-trained the AD model on both datasets and compared them using the cold-start evaluation

Figure 2 shows that certain tasks were strongly affected by poor decay scheduling, while others remained stable. Figure 3 indicates that models trained on lower-quality data exhibit weaker asymptotic performance

Despite reaching an expert-normalized score of 0.81, these findings highlight the importance of using high-quality data with smooth, progressive improvement to maximize performance

---

Providing more detailed supplementary materials

We provide a link to an anonymous repo with Vintix code and training dataset. Extra supplementary material is available in paper’s appendix

---

Discussion of sequence based RL approaches With the rise of Transformers for modeling sequential data, several works (1, 2) formulated MDP as a causal sequence modeling problem. (1) focused on reward conditioning treating each MDP element as a separate token, while (2) applied beam search over discretized SAR tuples.

Subsequent research has expanded this area by making models that maximize returns (3), adapting DT to online learning (4), and replacing the Transformer with SSM backbones like Mamba (5).

On suboptimal demonstrators We did not filter out failed expert trajectories to avoid biasing the dataset, particularly in cases where failures may be correlated. Instead, we addressed noisy behavior by further training the demonstrators.

On generalization to unseen tasks

1. Scaling the dataset AD benefits from a large number of tasks to generalize effectively. We plan to expand the dataset with new domains.
2. Domain-invariant architecture Using VLA-like models to map modalities into a shared embedding space may improve cross-domain transfer and reduce reliance on task identification.

Thank you for the review. Based on your feedback, we believe the central point of discussion is the practicality — in a broad, real-world sense — of the proposed approach. In particular, we address the following raised topics:

• Do the selected domains represent a broad spectrum of challenging, real-world-relevant tasks?
• Does the proposed data collection method offer advantages over existing approaches (e.g., learning histories from PPO or optimal action labels as in DPT)?

While our work focuses on simulated environments and does not yet make claims about real-world transfer, we argue below that both our choice of domains and our data collection methods are well-justified. Moreover, we believe they provide a strong foundation for the continued development of action models within the framework of in-context reinforcement learning.

---

Do the selected domains encompass a broad spectrum of difficult and real-world tasks?

The chosen tasks for experimental validation are overly standardized and fail to encompass a broad spectrum of extreme or unconventional scenarios. This narrow focus could severely limit the applicability of the method in complex real-world environments.

When benchmarking models of this kind, it is important to balance standardized environments—which ensure reproducibility and fair comparison—with task suites that reflect real-world complexity. A central goal of our work is to contribute datasets and results to the broader community, laying a foundation for further scaling and development.

This objective imposes constraints on domain selection: environments must be open-source and widely adopted by the research community. While MuJoCo is included as a field standard, the remaining domains were chosen for their practical relevance and the complexity of the challenges they present:

• Meta-World ML45: A highly challenging benchmark where state-of-the-art online Meta-RL algorithms achieve a success rate of just 0.4 (Shala et al., 2025). It is widely used and practically motivated, with over 490 citations since 2024.

• Industrial Benchmark: A synthetic suite that models industrial optimization problems with complex dynamics, heteroscedastic noise, delayed multi-objective rewards, and partial observability. Notably, the benchmark was explicitly designed to test RL algorithms under conditions resembling real-world industrial control problems.

• Bi-DexHands: Grasping is a core challenge in robotics, critical for tasks in human-centric environments (Billard et al., 2019). Despite extensive research, it remains difficult due to the high-dimensional action space. Bi-DexHands offers a diverse suite of grasping tasks, and is recognized as it was accepted to last year’s NeurIPS Benchmarks and Datasets track last year.

---

Does the proposed collection method offer advantages over the existing ones (e.g, learning histories from PPO or optimal action labels as in DPT)?

The paper fails to clearly elucidate the practical advantages of Continuous Noise Distillation in real-world scenarios. Besides, the paper lacks detailed discussion regarding the potential impact and value of this method in tangible applications.

Continuous Noise Distillation significantly simplifies dataset collection for Algorithm Distillation. In vanilla AD, a new RL agent must be trained for each learning history, often requiring millions or billions of steps. This process is time-consuming, unpredictable, and limits scalability.

In contrast, Continuous Noise Distillation enables efficient, controllable data collection using only a demonstrator policy. Users can set the length of each learning history, reducing computational overhead and enhancing practicality in time- and cost-sensitive scenarios.

The dataset construction and utilization process is complex and potentially cumbersome, posing substantial challenges for practical implementation. This intricate dependency on specific steps and environments restricts the generalizability and reproducibility of the method.

While dataset construction does require some effort, it remains relatively simple compared to other In-Context RL methods. It only needs demonstrator policies and environment access. By contrast, AD requires full RL learning histories, and DPT needs expert-provided target actions. Generalist agents like JAT, GATO, and Baku also rely on expert demonstrations. Although simplifying data collection for In-Context RL is an important direction, it is beyond the scope of this work.

Thank you for the time and effort you devoted to reviewing our paper. We have identified the following key issues for further discussion:

Is it possible to include experiments comparing the proposed approach with vanilla AD to provide stronger evidence?

Vintix diverges from standard AD through noise-distilled data methodology, replacing RL learning histories because:

• Zisman et al. (2024a) link noise distillation to in-context learning emergence
• It improves trajectory control by avoiding lengthy, sample-inefficient PPO histories, which demand large computational resources and exhibit unstable convergence
• It permits low-cost dataset generation via open-source expert policies

Vintix functions as a standard AD framework applied to cross-domain, noise-distilled datasets. Recollecting dataset for vanilla AD requires retraining 1157 (87 tasks * 13.3 trajectories) RL agents—computationally unfeasible for rebuttal timelines

What evidence might be presented to demonstrate that standardizing reward functions with task-specific factors positively influences model performance?

We conducted a hyperparameter-tuning experiment on the Humanoid task. Over 50 hyperparameter configurations were evaluated. Figure 1 presents the results, showing that using a non-standard reward scale yielded better scores. Before training Vintix,we performed analogous sweeps on each task. We don’t claim that modifying reward scales always leads to better models,but observed performance gains under our chosen settings

What is the key novelty of the paper that makes its technical contribution sufficiently substantial?

This work advances beyond noise integration in distillation by offering empirical principles for selecting ε-decay functions. We show noise-augmented distillation enables dynamic self-correction during inference, particularly in tasks with complex dynamic, high-dimensional actions, and partial observability. Also, we have open-sourced collected datasets for In-Context RL. To our knowledge, it was previously undertaken by the JAT, and their dataset comprises expert trajectories only.

Our model training prioritized challenging domains over simplistic ones. The MetaWorld ML45 benchmark, a prominent robotic manipulation suite with 490+ citations since 2024, presents significant difficulty: state-of-the-art online MetaRL methods achieve only 0.4 success rates (Shala et al., 2025). Industrial Benchmark introduces synthetic tasks simulating industrial optimization challenges, characterized by complex dynamics, heteroscedastic noise, delayed rewards and partial observability. Bi-DexHands addresses robotic grasping (a critical capability for human-environment interactions) through tasks testing RL and MetaRL algorithms against high-dimensional action space limitations. To our knowledge, no prior work has applied methods beyond behavioral cloning on such challenging cross-domain datasets

What is the performance of ADε in comparison with other methods?

Comparison with other offline MetaRL approaches is indeed a valuable experiment. Vanilla AD is discussed in the first paragraph. AMAGO is an online off-policy MetaRL approach limited to a single domain, while Vintix is offline and trains on fixed data. AMAGO is both difficult to replicate and challenging to adapt to the offline setting. DPT is unable to learn in-context in partially observable MDPs (Appendix H of Nikulin et al.(2024)). Bi-DexHands and Industrial Benchmark are partially observable

We implemented DPT and trained it on MuJoCo. We also trained Vintix only on MuJoCo, using the same transformerparameters. Figure 2 shows that DPT’s training loss exceeded Vintix’s and displayed spikes. Validation followed the same procedure with an empty initial context. As shown in Figure 3,DPT performs poorly, necessitating further tuning.No prior work, to our knowledge,has applied DPT to MuJoCo tasks, leaving it for future study

What is the difference between "group identifier" and "task identifier"?

The task identifier is a unique ID assigned to each task in the dataset. The group identifier is an ID indicating whether a group of tasks shares the same observation and action spaces (in terms of dimensionality and the semantic meaning of each channel)

Why were some JAT experts retrained to collect the dataset?

During data collection, we assessed JAT demonstrations and found several low-performing experts (zero success rates). Prioritizing dataset quality and model evaluation against expert benchmarks, we retrained selected experts. However, fair comparison remained possible through score normalization against updated benchmarks, intentionally lowering Vintix’s scores relative to JAT. This imposed stricter evaluation conditions on our model

Thank you for the review. Based on your feedback, we believe the main points of discussion can be distilled into the following:

• Does Adaptability (or inference-time learning) happen, or is it just Task Identification (context conditioning)?
• How to scale to non-vector-based/proprioceptive modalities, e.g., images?

Below, we present our responses and the experiments we conducted during the limited rebuttal phase. If there’s anything we’ve overlooked or if further clarification is needed, please let us know—we’ll respond promptly within the available timeframe.

---

Does Adaptability (or Inference-Time Learning) Happen, or Is It Just Task Identification (Context Conditioning)?

You’re right to highlight the challenge of disentangling adaptability (i.e., inference-time learning) from task identification via conditioning. This is a recurring issue in meta-learning research, especially when attempting to distinguish in-weights learning from in-context learning at scale.

To ensure that our results are not simply a consequence of task identification — and to explore the trade-off between in-context learning and in-weights learning — we perform several key ablations of our proposed approach:

• Algorithm Distillation (AD) with Masked Rewards — The model is trained on noise-induced improvement trajectories, but with rewards masked out during training.
• Expert Distillation (ED) — The model is trained exclusively on expert-level trajectories, similar to JAT/GATO-style behavior cloning.

These ablations help shed light on the contribution of improvement trajectories and reward signals, thereby going beyond task identification and supporting the case for learning dynamics.

---

Vintix vs. Expert Distillation

We trained a model with the same architecture as Vintix (transformer backbone, encoders, loss function), but only on expert demonstrations (ED). We evaluated both ED and Vintix using the cold-start procedure (lines 238–239 in the paper).

Results (link) show that Expert Distillation underperforms relative to AD on both domains. ED reaches an average expert-normalized score of 0.8 on MuJoCo and Meta-World, while AD achieves 0.97 and 0.95, respectively.

This suggests that the structure of policy improvement in the dataset is valuable for enabling self-correcting behavior and high performance. Notably, ED’s performance in MuJoCo stays flat across different shot counts, but improves in Meta-World as more episodes are provided.

This implies ED partially learns task identification—especially challenging in Meta-World, where a shared encoder spans tasks. Our findings suggest ED requires ~4 episodes to infer the current task. However, even after identifying the task, ED fails to reach AD’s asymptotic performance, despite having the same data volume.

---

Vintix vs. Algorithm Distillation with No Rewards

In the second experiment, we aim to assess whether Vintix is utilizing both structure of policy improvement and the reward function—in other words, whether the reward signal contributes to the model's ability to self-improve during inference.

To investigate this, we re-trained the Vintix model without access to rewards and compared its performance to the original version of Vintix (trained with rewards), using the previously described cold-start inference procedure on training tasks from the MuJoCo and Meta-World domains.

The evaluation results (link) suggest that the reward signal plays a role in achieving performance comparable to the demonstrator. AD trained with masked rewards performs worse asymptotically across both domains and shows slower convergence on the Meta-World domain.

These findings indicate that reward feedback is essential for effective self-improvement during inference, supporting the view that supervised training on a dataset containing policy improvement and rewards enhances the model’s in-context reinforcement learning capabilities.

---

How to Scale to Non-Vector-Based/Proprioceptive Modalities (e.g., Images)?

Vintix currently operates on proprioceptive inputs, extending it to image-based observations is great direction for future work. To extend it to image-based observations with parametric variations, a natural approach would be to use a vision foundation model (e.g., BLIP) to encode visual input on the environment side.

Variations such as object color or distractors can be applied prior to encoding and will be captured in the resulting image embeddings. These can then be passed to Vintix’s MLP encoder as n-dimensional inputs.

This preserves the existing pipeline structure: only the image encoding is delegated to the environment. As before, training requires noise-distilled policy improvement data; inference remains unchanged.