BLEUBERI: BLEU is a surprisingly effective reward for instruction following

We appreciate the reviewer’s recognition of our work, including the interestingness of using BLEU as a reward signal for alignment, and our extensive experimental scope. Below, we respond to each of the weaknesses and questions in detail.

Weaknesses (same as questions)

1. BLEU’s effectiveness is dependent on reference quality
We acknowledge this limitation, but we also want to highlight that obtaining high-quality references is relatively straightforward. Such data can come from two sources:

• Synthetic outputs from frontier LLMs: Generating synthetic data has become a standard component of modern LLM training pipelines (Lambert et al., 2024; Qwen Team, 2024). These synthetic outputs have proven to be highly effective in practice.
• Ground-truth responses from existing datasets: Numerous publicly available datasets already provide high-quality reference answers. Prominent examples include Tulu 3 (Lambert et al., 2024), OpenOrca (Mukherjee et al., 2023), UltraChat (Ding et al., 2023), and Alpaca (Wang et al., 2022).

In contrast, training reward models demands extensive human-labeled preference data, which is costly and time-consuming. Reference-based approaches like BLEUBERI offer a more accessible and efficient path to alignment, demonstrating that even simple metrics like BLEU can serve as practical alternatives to reward models.

References:

• Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024)
• Qwen2.5 Technical Report (Qwen Team, 2024)
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (Ding et al., 2023)
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (Mukherjee et al., 2023)
• Self-Instruct: Aligning LM with Self Generated Instructions (Wang et al., 2022)

2. A baseline that feels missing is one that’s based on an LLM as a judge -- tasked with the same/similar task as BLEU
We agree that using an LLM-as-a-judge to compare responses against a reference could, in principle, provide a more semantically nuanced reward than BLEU. However, incorporating it during training introduces potential drawbacks that may undermine the primary appeal of reference-based metrics (simplicity and efficiency):

• LLMs are much more expensive to run than BLEU and similar metrics like ROUGE. Additionally, if the LLM judge generates long-form justifications, it will be much slower than reward models.
• Reference-based metrics like BLEU provide a fully transparent and deterministic reward signal, whereas LLM-based judgments are inherently opaque.

To clarify, our focus on BLEU throughout the paper stems from its unique position among reference-based metrics: it has a particularly simple form, yet it can already be surprisingly effective. Our intent is not to argue that BLEU is the optimal reference-based reward function. Rather, we demonstrate that even such a basic metric can rival the performance of learned reward models (which are essentially finetuned LLM judges). We hope this insight can encourage further exploration of other reference-based metrics.

We appreciate the reviewer’s recognition of our work, including the originality of using BLEU as a reward signal for alignment, the clarity of our writing, and the thoroughness of our analyses, including the agreement study, synthetic data experiments, alternative metrics, and qualitative behavior analysis. Below, we respond to each of the weaknesses and questions in detail.

Weaknesses

1. One may wonder if the advantages of using BLEUBERI are an artifact of the Qwen model.
We acknowledge that Qwen is a much stronger base model than Llama, and thus it is easier to get RL to work on the Qwen models compared to Llama. Prior work has observed similar findings in RL experiments in other domains (Shao et al., 2025). In our experiments, we apply a brief stage of SFT to Llama before running RL to provide the model with a basic understanding of instruction-style data. A more detailed discussion can be found in Section 3.2, Line 218-223. Importantly, our results in Table 1 show that on Llama-3.1-8B (with SFT initialization), BLEUBERI can achieve competitive performance with GRPO-RM and SFT, suggesting that the effectiveness of BLEUBERI is not merely an artifact of the Qwen base model.

References:

• Spurious Rewards: Rethinking Training Signals in RLVR (Shao et al., 2025)

2. More insights into how different post-training methods lead to different behaviors.
We agree that using reference-based metrics as a reward is like a middle ground between SFT and RL with reward models. To summarize, our current findings suggest that:

• SFT produces the most verbose and repetitive outputs, followed by BLEUBERI, then GRPO-RM (Table 13).
• BLEUBERI-trained models produce more factually grounded outputs than SFT-trained and GRPO-RM-trained models (Section 4.2).

To the best of our knowledge, our work is the first to explore applying simple reference-based metrics to improve general-domain instruction following. We hope these findings lay the groundwork for more systematic explorations into how different forms of supervision shape model behavior in instruction following tasks.

Questions

1. BLEUBERI with synthetic data feels like some sort of distillation on a high level. Have you compared it with distillation methods empirically?
We agree that BLEUBERI shares a high-level resemblance to knowledge distillation. In our current experiments, SFT acts as the closest analog to distillation, though it does not incorporate teacher logits in the way classical distillation methods do. As shown in Table 1, BLEUBERI consistently outperforms SFT across benchmarks. This is likely because SFT tends to induce broad behavioral shifts, whereas GRPO facilitates more targeted, fine-grained adjustments (Rajani et al., 2025).

In the next revision of the manuscript, we plan to include a standard distillation baseline for comparison.

References:

• Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them (Rajani et al., 2025)

2. What will happen if you do not filter out easy data when training with BLEUBERI? Are there some common reward hacking patterns?

We report results from training with BLEU on easy, medium, and randomly sampled data in Table 4 (Appendix B.2). While random sampling improves over the base model, training on the hardest examples yields significantly better performance. This is in line with findings from prior work (Qwen Team, 2025).

Regarding common reward hacking patterns, we do observe that models trained on easy data tend to produce more repetitive outputs compared to those trained on harder examples. We will try to quantify this better in the next version.

References:

• Qwen3 Technical Report (Qwen Team, 2025)

3. In the agreement study in Sec. 2, the best method is to use BLEU + RM. Have you tried similar approaches during training? Will it also lead to the best performance?
Yes, we report training results using the combined BLEU+RM reward in Table 11 (Appendix B.10). Interestingly, while this reward showed the highest agreement with human preferences in the evaluation study (Section 2), the BLEUBERI model slightly outperforms the BLEU+RM-trained model in terms of benchmark performance.

For the BLEUBERI and BLEU+RM models in Table 11 (both trained on 1,000 examples), we performed bootstrapping to estimate confidence intervals (please refer to our response to Reviewer R3fD for details). The performance difference between the two models appears modest, as their confidence intervals frequently overlap. Furthermore, we did not tune the reward weights in the BLEU+RM setup, which could leave room for further optimization.

4. How much markdown formatting do you have in your training data? Does that explain the trends in Sec. 4.1?
Among the 5,000 training examples used for each model in Table 1, here is the percentage of data where the reference response contains markdown formatting:

• Qwen2.5-7B: 21.8%
• Qwen2.5-3B: 24.0%
• Llama-3.1-8B (SFT init.): 13.8%

While markdown is not overwhelmingly dominant in the training data, its presence likely contributes to the trends in Section 4.1 (Table 13). GRPO-based methods may amplify features that correlate with higher rewards (e.g., structured formatting), so even modest amounts of markdown in the training data could be reinforced. In contrast, SFT matches references token-by-token, which may underemphasize sparse formatting cues, leading to lower markdown usage in outputs.

5. I don’t get the “amplifies subtle biases” part in line 291. Can you elaborate?
In line 291, “amplifies subtle biases” refers to the phenomenon observed in Zhao et al. (2025), where RL post-training tends to magnify preferences or stylistic patterns already weakly present in the pretrained model, even if they are not strongly expressed in the base model outputs. We will revise this sentence in the next version and add the appropriate citation for clarity.

References:

• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining (Zhao et al., 2025)

We appreciate the reviewer’s recognition of our work, including BLEU’s computational efficiency compared to reward models, the strong performance of BLEUBERI-trained models across four benchmarks and multiple base models, and the improved factual grounding achieved through BLEUBERI training. Below, we respond to each of the weaknesses and questions in detail.

Weaknesses

1. This approach is applicable only when high-quality references are available to compute BLEU against.
2. The strengths still outweigh the above weakness, given that collecting reference data can be often more cost-effective than calling the reward models, and one can also leverage already existing datasets with references.
We acknowledge this limitation, but as the reviewer also noted, obtaining high-quality references is relatively straightforward. Such data can come from two sources:

• Synthetic outputs from frontier LLMs: Generating synthetic data has become a standard component of modern LLM training pipelines (Lambert et al., 2024; Qwen Team, 2024). These synthetic outputs have proven to be highly effective in practice.
• Ground-truth responses from existing datasets: Numerous publicly available datasets already provide high-quality reference answers. Prominent examples include Tulu 3 (Lambert et al., 2024), OpenOrca (Mukherjee et al., 2023), UltraChat (Ding et al., 2023), and Alpaca (Wang et al., 2022).

In contrast, training reward models demands extensive human-labeled preference data, which is costly and time-consuming. Reference-based approaches like BLEUBERI offer a more accessible and efficient path to alignment, demonstrating that even simple metrics like BLEU can serve as practical alternatives to reward models.

References:

• Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024)
• Qwen2.5 Technical Report (Qwen Team, 2024)
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (Ding et al., 2023)
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (Mukherjee et al., 2023)
• Self-Instruct: Aligning LM with Self Generated Instructions (Wang et al., 2022)

Questions

1. What would happen if experimented with ROUGE (more recall focused) instead of BLEU (more precision focused), or use both?
We currently discuss alternative metrics like ROUGE-L in Section 2.2 and Appendix B.10. Below, we provide some new experiment results with ROUGE-L and a BLEU + ROUGE-L ensemble metric on the Chatbot Arena analysis:

From these new results, we find that with stemming enabled and more references available, ROUGE-L can achieve human agreement comparable with BLEU. Combining BLEU and ROUGE-L also yields strong agreement results. We will add and discuss these results in the next version.

2. How sensitive is it to use BLEU with different hyper-parameters, e.g., different n-gram length?
We present new experiment results below, which shows that BLEU performance is relatively stable across different maximum n-gram orders, as long as the order is greater than 1.

For the standard setup with max ngram order set to 4, we also experimented with both smoothed and unsmoothed BLEU variants and found that using smoothing is crucial for achieving high human agreement. With smoothing, the agreement is 72.4%. Without smoothing, the agreement drops to 54.7%. Smoothing prevents BLEU scores from collapsing to zero when higher-order n-gram matches are sparse, even in outputs that are meaningfully aligned. In all experiments in the paper, we apply smoothing (as noted in Footnote 2 in the paper). We will add and discuss these results in the next version.

We appreciate the reviewer’s recognition of our work, including BLEUBERI as a cost-effective alternative to GRPO-RM for alignment, the novelty and interestingness of our findings, the value of our collected synthetic data, and the informative comparison of different metrics. Below, we respond to each of the weaknesses and questions in detail.

Weaknesses

These numbers suggest that BLEUBERI is not consistently more verbose than GRPO-RM, and overall differences between the two are relatively modest.

On length bias:

• We note that our benchmarks try to mitigate length bias by design. Both ArenaHard v1 and v2 apply style control, which control for factors like length and markdown usage. WildBench similarly has a length penalty mechanism, which converts small wins/losses into ties if one response is much longer than the other.

On repetition bias:

• As shown in Table 13, BLEUBERI has lower repetition rates than SFT and rates that are generally close to—or only slightly higher than—those of GRPO-RM. Despite this, our human evaluation (Section 4.3) finds BLEUBERI model outputs to be just as preferable as GRPO-RM’s. This indicates that the minor verbosity and repetition seen in Table 13 do not substantially affect the perceived quality of responses.

2. Insufficient Discussion of Alternative Rule-Based/Verifiable Metrics
We currently discuss alternative metrics in Section 2.2 and Appendix B.10. We agree that further exploration of such metrics could provide valuable insights. Our primary goal, however, was to demonstrate that even simple metrics like BLEU can be effective. Given our resource constraints, we focused on BLEU in the training experiments due to its simplicity and surprisingly strong performance. To the best of our knowledge, our work is the first to apply simple reference-based metrics to enhance general-domain instruction following. We hope this serves as a foundation for future research on rule-based or verifiable reward functions.

2.1. “It is hard for me to recognize why ROUGE-L, BERTScore, BLEUrt are not competitive to BLEU. Also, the ensemble of rewards should also be considered.”
ROUGE-L and BLEU + ROUGE-L ensemble (new experiments)

• We ran additional experiments on the Chatbot Arena dataset with ROUGE-L. With stemming enabled and multiple reference responses, ROUGE-L can achieve human agreement levels comparable to BLEU. Moreover, combining BLEU and ROUGE-L (with stemming) as an ensemble reward function by averaging also yields strong agreement results. These findings will be included and discussed in the next version of the manuscript.

BERTScore

• To clarify, results in the current version of the manuscript indicate that BERTScore is competitive with BLEU in the Chatbot Arena analysis (Figure 1) and in training experiments (Table 11, Appendix B.10). However, due to computational constraints, we did not pursue extensive training with BERTScore.

BLEURT

• This current manuscript does not mention BLEURT. We leave the exploration of more reference-based metrics like this to future work.

3. Concerns with Human Evaluation Reporting (line 320)
Thank you for raising these concerns. Our reported soft preference rate was not intended to imply BLEUBERI’s superiority, but rather to reflect how often its outputs were at least as good as GRPO-RM’s (i.e., including ties and wins). This metric indicates general parity, not dominance. We acknowledge it could be misleading without context and will revise the manuscript to clearly distinguish it from strict preference counts.

Regarding the Cohen’s Kappa of 0.34, we note that our annotation task required subjective judgments on highly similar, high-quality responses to complex prompts from four benchmarks (MT-Bench, ArenaHard v1/v2, WildBench). These often involved technical or domain-specific content, making clear preferences difficult. The frequent use of the “tie” option by one annotator further suggests the closeness in output quality. Prior studies have also observed that subjective evaluation tasks often yield lower inter-annotator agreement (Wang et al., 2024; Plank, 2022).

Our human evaluation supports the key finding that BLEUBERI-trained models can rival reward-model-trained counterparts in output quality. We will revise the evaluation section to better highlight its limitations, clarify metric interpretation, and contextualize their relevance to our conclusions.

References:

• The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation (Plank et al., 2022)
• HelpSteer2: Open-source dataset for training (Wang et al., 2024)

Questions

1. Interest in seeing the results on larger base model sizes (32B)
We understand that results on a much larger base model would offer valuable insights into the effectiveness of simple reference-based reward functions like BLEU. However, due to limited compute resources, we leave such explorations to future work. We would like to highlight again that all training experiments (exploratory and final) cost about 1357 GPU hours (around $4750 USD), which represents a substantial computational cost for an academic lab.

2. More direct quantification of GRPO algorithm speedup between RM-8B and BLEU
For two runs trained with BLEU and RM-8B as the reward, the average reward computation times over 250 steps (profiled by TRL) were:

• RM-8B: ~1.15s
• BLEU: ~0.03s

Despite BLEU being much faster, both runs took roughly the same wall-clock time (~2h45m). In our setup, the main bottleneck was output generation per group, which was slow due to using standard Hugging Face Transformers without fast inference backends like vLLM. As a result, BLEU’s speed advantage didn’t significantly impact end-to-end training time.

However, in more optimized setups with faster generation or larger group sizes, BLEU’s efficiency could offer greater benefits. Not having to load a reward model should free up memory for larger batch sizes. In our experiments, we use identical hyperparameters for BLEUBERI and GRPO-RM for fairness. But in practice, using lightweight metrics like BLEU could enable better scaling.

We appreciate the reviewer’s recognition of our work, including BLEUBERI’s competitive performance with GRPO-RM, BLEUBERI’s cost efficiency, and our strong results in both factuality and human evaluations. Below, we respond to each of the weaknesses and questions in detail.

Weaknesses

1. Reliance on reference quality is the biggest concern
We acknowledge this limitation, but obtaining high-quality references is straightforward. Such data can come from two sources:

• Synthetic outputs from frontier LLMs: Generating synthetic data has become common in LLM training pipelines (Lambert et al., 2024; Qwen Team, 2024). These data can be highly effective in practice.
• Ground-truth responses from existing datasets: Many publicly available datasets already provide high-quality reference answers. Prominent examples include Tulu 3 (Lambert et al., 2024), OpenOrca (Mukherjee et al., 2023), UltraChat (Ding et al., 2023).

In contrast, training reward models demands extensive human-labeled preference data, which is costly and time-consuming. Lightweight reference-based metrics like BLEUBERI offer a more accessible and efficient path to alignment.

References:

• Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024)
• Qwen2.5 Technical Report (Qwen Team, 2024)
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (Ding et al., 2023)
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (Mukherjee et al., 2023)

2. Absence of statistical significance results
We thank the reviewer for raising this point. In response, we reran benchmark evaluations with bootstrapping and report results below. Overall, we find that:

• Both BLEUBERI and GRPO-RM show consistent improvements over the base model across all three model types, as suggested by minimal or no overlap between their confidence intervals and those of the base model.
• Between BLEUBERI and GRPO-RM, the confidence intervals almost always overlap, suggesting that BLEUBERI performs on par with GRPO-RM overall.

We will add these results into the next version of the manuscript.

Benchmark results with bootstrapping:

Arena-Hard v1/v2 already include bootstrapping in their official code: they draw 100 resamples and re-fit a Bradley-Terry model for each, which is computationally intensive, hence the cap at 100. We reran the scripts and report the median point estimates and 95% confidence intervals (CIs). For MT-Bench and WildBench, we implemented a similar procedure with 1000 full-size prompt resamples per model, reporting the median score and two-sided 95% CI.

3. Limited experiments on exploring effects of scaling model size, data volume or extensive training time
In our preliminary experiments, we trained for multiple epochs but observed no significant performance gains for either BLEUBERI or GRPO-RM. One possible explanation is that larger models might benefit more from longer training. Regarding data volume, we note that our setup aligns with that of the Qwen3 pipeline (Qwen Team, 2025), which trains exclusively on 4,000 challenging samples, as noted on Line 204.

Overall, we agree that more scaling experiments can provide helpful insights. However, due to limited compute resources, we leave such explorations to future work. We would like to highlight again that all training experiments (exploratory and final) cost about 1357 GPU hours (around $4750 USD), which represents a substantial computational cost for an academic lab.

References:

• Qwen3 Technical Report (Qwen Team, 2025)

4. Known issues in mathematical reasoning tasks
We assume the reviewer is referring to the results shown in Figure 7, which shows that on the Math/Reasoning domain, the gap between BLEU’s and the two reward models’ human agreement rates is large. We would like to reiterate that the primary focus of this work is on general-domain instruction following. For structured domains like math, it is straightforward to develop reward functions that checks the correctness of the final answer. If model outputs consisted solely of final answers, BLEU would essentially function as a correctness-based reward. However, many Math/Reasoning examples in the Chatbot Arena dataset include intermediate reasoning steps, which reduce BLEU’s effectiveness as a proxy for correctness. In our experiments, we chose not to optimize for any specific domain.

Questions

1. How would this work be adapted to dealing with mathematical reasoning tasks? Any initial experiments or thoughts?
Building on the discussion of Weakness 4, one potential way to improve both general-domain and math performance is to combine BLEU and a correctness reward into a single objective function, where BLEU could be used to assess the quality of reasoning traces. We plan to explore this approach in a future version of the manuscript.

2. How does variability in reference quality impact the effectiveness of the training process
We provide training results with different references in Table 9 in the Appendix. These results generally align with our findings from Section 2.2 (Line 124), suggesting that stronger reference models will lead to better results, which is intuitive. As discussed under Weakness 1, obtaining high-quality references is straightforward.

3. Can BLEU be prone to more subtle reward hacking?
We interpret the reviewer’s comment as referring to subtle, hard-to-detect artifacts. As noted in Section 4.1 (Lines 280–294), GRPO-RM models often begin with “Sure!”, while BLEUBERI models favor “Certainly!”—though this trait is not unique to BLEUBERI. Importantly, human evaluations (Section 4.3) rate BLEUBERI and GRPO-RM responses as equally preferable, and benchmark results (Table 1) and factuality assessments (Section 4.2) show that any such artifacts do not meaningfully impact perceived quality.

4. Can this metric work well in multi-step reasoning tasks? Why was there a performance decline with enforced CoT reasoning? Could you elaborate please?
Prior work shows that explicit reasoning does not always help, especially when instruction adherence is the main goal. In our experiments, enforcing CoT reasoning reduced performance, consistent with Li et al. (2025), who found that CoT can distract from constraint-relevant tokens and introduce errors. Sprague et al. (2024) similarly observed that CoT primarily benefits math and symbolic tasks, with little or no gain and sometimes harm on general instruction-following. Our BLEUBERI results align with these findings, supporting the idea that reasoning should be applied selectively.

It is possible that larger models or longer generation lengths may allow more effective use of reasoning in general instruction-following tasks. We plan to explore this in a future version of the manuscript.

References:

• When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs (Li et al., 2025)
• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (Sprague et al., 2024)

Dear Reviewer R3fD,

Thank you once again for your thoughtful review. If you have any follow-up questions or concerns regarding our rebuttal, we would be happy to provide additional clarification before the discussion period concludes.

To summarize, we have included new benchmark evaluation results with bootstrapping and clarified several of the points you raised. In the next version of the manuscript, we also plan to incorporate training results on larger models and more data, as per your suggestion.

If our rebuttal has addressed your concerns, we would be grateful if you would consider adjusting your score accordingly. We sincerely appreciate your time and helpful feedback!