AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning

We sincerely appreciate your thorough and insightful review! We would like to address your concerns with the following responses:

It would be valuable to see (a) what the differences in asynchronous tolerance and (b) the AReaL tradeoffs between speedup vs stability & performance gain look like across algorithms with different variants of advantage bootstrapping.

We conducted additional experiments using the RLOO advantage bootstrapping method. Due to the tight rebuttal timeline, we ran 1.5B models in a local 8 GPU setting with 8k context length and various staleness values. The following evaluation results demonstrate that RLOO exhibits slightly better asynchronous tolerance to vanilla PPO.

The reviewer raises an important direction for future research. AReaL modifies the PPO/GRPO algorithm because the importance sampling term naturally supports asynchronous off-policy training. Beyond PPO-based workflows, it would be valuable to investigate asynchronous tolerance for REINFORCE-like and other off-policy algorithms. We consider this an important avenue for future work.

Many works have reported that reasoning-style RL on Qwen models behaves very differently to models from other families $1, 2$ . Given this, I would be interested to see at least the experiments in Figure 5 (a) and (b) extended to another model family.

Thank you for this valuable suggestion! We should clarify that there exists a paradigm difference: references $1,2$ mainly investigate reasoning patterns starting from base or instruction-tuned models, while AReaL focuses on RL fine-tuning of long CoT models (e.g., distilled from OpenAI-o1/DeepSeek-R1). After distillation, the reasoning pattern should be less affected by the base model. To validate this point, we conducted additional experiments with DeepSeek-R1-Distill-Llama-8B in the math setting. The following results demonstrate that AReaL's effectiveness transfers across different model families:

We primarily used the Qwen family for experiments because: (1) it offers a range of model sizes (from 1.5B to 32B), which is convenient for both ablation studies (Figure 5) and scalability validation of AReaL, and (2) the Qwen family provides readily available distilled long CoT models for RL training (i.e., DeepSeek-R1-Distill-Qwen). While running ablation studies on other model families (e.g., Llama 8B) is more computationally expensive, we are prepared to conduct these experiments when additional computational resources become available and will include these results in the camera-ready version.

How exactly does AReaL completely avoid batching?

AReaL avoids generation batching within the RL pipeline, but the inference engine will automatically handle request batching on GPU. To be more specific, instead of calling batched generation with engine.generate, AReaL asynchronously sends individual generation requests to a remote HTTP server. The server (e.g., vLLM or SGLang) handles request batching according to its internal scheduler logic and executes batched computation on the GPU, then returns HTTP responses individually to the RL pipeline. Therefore, from the perspective of the RL pipeline, AReaL completely avoids batching during rollout while still benefiting from the server-side batching computations.

We appreciate the reviewer's detailed and constructive feedback! We would like to address your concerns one by one in the following responses:

Table 1 reports results only on LiveCodeBench.

During the rebuttal period, we conduct additional evaluations in the coding domain and present the results below. We will include these results in the appendix of the camera-ready version.

Experiments rely on clusters of many GPUs, with no guidance for small-scale academic setups (< 16 GPUs) where the staleness/throughput trade-off may differ.

Thank you for this important observation! We conducted a series of experiments using the Qwen 1.5B distilled model with 8k context length and batch size 64×16 on 8 GPUs, testing various staleness values. The following table shows the experimental results. We observe that our preliminary conclusions from the large-scale setting (Table 2) generally align with findings using fewer GPUs.

We note that fine-tuning strong reasoning models with RL is typically resource-intensive $1$ , so training with 8 GPUs has to use smaller context lengths, and the final performance may not match that obtained from large-scale distributed training. We consider improving the efficiency of asynchronous RL training for smaller setups an important future direction.

$1$ https://github.com/giterinhub/DeepScaleR-1.5B-Preview

Could you discuss how practical it would be to conduct research on these types of methods for smaller-scale experiments (4-8 GPUs)?

AReaL provides benefits for long CoT and agentic settings regardless of experimental scale. When trajectory length variance is high, synchronous systems can easily encounter resource underutilization issues while waiting for the longest rollout to complete. AReaL eliminates this idle time through continuous batching and rollout interruption, thereby improving overall throughput (Figure 3).

Researchers may have concerns about the usability and customizability of asynchronous systems. However, the algorithmic implementation of AReaL does not differ significantly from synchronous counterparts. Users simply need to switch between synchronous and asynchronous rollout methods. The algorithm orchestration code can remain identical for both synchronous and asynchronous RL:

dataloader = StatefulDataloader(dataset)  
for step in range(max_steps):  
    if async_training:  
        batch = rollout.prepare_batch(dataloader, workflow=workflow)  
    else:  
        data = next(dataloader)  
        batch = rollout.rollout_batch(data, workflow=workflow)  
    # Prepare training inputs  
    adv_batch = actor.compute_advantages_and_returns(batch)  
    batch['advantages'] = adv_batch['advantages']  
    # Execute PPO update  
    stats = actor.ppo_update(batch)  

This design ensures that researchers can easily adopt AReaL in their existing workflows with minimal code modifications.

We sincerely thank the reviewer for their supportive comments! We hope the following responses address your concerns:

I noticed that many data points for VeRL are missing or marked with "-" in Table 1

We acknowledge that despite our best efforts to tune VeRL before paper submission, our VeRL reproduction did not achieve comparable results to AReaL in some cases. Consequently, in the draft, we included results from third-party forks of VeRL, such as DeepScaleR $1$ and DeepCoder $2$ , which typically incorporate customized modifications to improve training stability.

Currently, we are re-running experiments with the latest version of VeRL to address these missing data points. However, given the computational expense of these experiments (e.g., the 7B math model requires over 10,000 GPU hours to complete), we have not been able to produce valid results during the rebuttal period. We will make every effort to include these amended results in the camera-ready version.

$1$ https://huggingface.co/agentica-org/DeepScaleR-1.5B-Preview

$2$ https://www.together.ai/blog/deepcoder

It would be helpful if the authors could include results across multiple random seeds. This would provide a clearer picture of how VeRL's final performance compares to AReaL.

We completely agree that including multiple random seeds would strengthen the robustness of our results. In the current draft, AReaL uses a fixed random seed of 1 for all training experiments. The evaluation protocol samples 32 responses with different random seeds and reports the average accuracy, following common practices adopted in the community $3$ .

We plan to re-run experiments with multiple random seeds to test AReaL's robustness once we have available computational resources. We commit to updating these results in the camera-ready version.

$3$ https://github.com/QwenLM/Qwen2.5-Math

VeRL occasionally outperforms AReaL. Does this imply that training on the most recent data may yield better model performance than using stale samples, at least within the current implementation?

We agree with the reviewer about this observation. In general, training with more recent data does yield better performance. However, the algorithmic modifications in AReaL increase tolerance to staleness, enabling us to achieve comparable performance even with staler data.

We note that the end-to-end comparison between VeRL and AReaL may not clearly showcase the effect of data staleness due to numerous implementation differences, such as the use of log-probabilities produced by the inference engine and different training/inference backends. Each of these factors can introduce numerical issues that influence final performance. Based on Table 1, we conclude that VeRL and AReaL demonstrate comparable algorithmic performance.

Regarding the effect of data staleness, Table 2 presents a more controlled ablation study. The results show that staleness levels of 1, 2, and 4 all achieve similar final performance to the synchronous oracle (staleness=0) across multiple evaluation benchmarks. These results demonstrate AReaL's increased tolerance to data staleness.

Thank you for the detailed and insightful review! We would like to address your concerns with the following responses:

The paper does not release any code.

We have provided an anonymous link in Appendix A. We apologize that we cannot repost the link in this rebuttal response, as it is strictly prohibited by the rebuttal guidelines.

Concerns about the generalizability of AReaL across different model architectures.

During the rebuttal period, we conducted additional experiments using the DeepSeek-Distilled-Llama-8B model, which is a long-CoT model based on Llama 3.1 8B. We matched the experimental configuration with the Qwen 7B math model from Table 1, and the results are presented in the following table:

The results demonstrate that AReaL generalizes effectively across different model families.

The system's current reliance on SGLang may act as a barrier to integration and adoption.

We primarily incorporated and modified SGLang to implement the interruptible rollout workers, but AReaL is not restricted to specific inference and training backends. We commit to supporting additional backends in the future, including PyTorch FSDP for trainer and vLLM for rollout inference worker. This expansion is currently part of our ongoing internal development efforts and will be uploaded to an open-source repository in the future.

Some system-level design choices, such as the fixed ratio between inference and training, are based on heuristics without dynamic tuning or adaptation. A more adaptive approach could further improve throughput and robustness in varied deployment scenarios.

The reviewer's observation is absolutely correct! We currently use a heuristically tuned inference-to-training ratio designed to saturate training GPUs. This ratio could be further optimized (decreased) based on specific training configurations and would benefit significantly from dynamic adjustment during training, particularly when context lengths change during RL training.
We will discuss these limitations in the appendix and consider these important ideas for future work. Thank you again for this insightful suggestion!