EvolvedGRPO: Unlocking Reasoning in LVLMs via Progressive Instruction Evolution

We sincerely thank the reviewers for their exceptionally thorough and detailed comments! We will explain your concerns point by point.

Q1: In the core reward function, Equation (9), the symbol format' appears without any definition or explanation provided anywhere in the paper. Additionally, the explanation for the test variable is also same problem.

A1: Thank you for pointing this out.

format refers to a reward score of format, which is designed to encourage the model to produce structured outputs, particularly those involving chain-of-thought reasoning.

test denotes the number of evaluation trials used to assess accuracy, aligning with the number of responses generated in a reinforcement learning batch. The expression using max in the goal ensures at least one correct answer post-question augmentation, preventing variance collapse.

We will include a more detailed description in the revised version to improve accessibility and clarity.

Q2: Clarification of Multi-modal Instruction Editing.

A2: We clarify that the core contribution of EvolvedGRPO lies in its formulation of a model-agnostic, multimodal, self-evolving reinforcement learning framework applicable across diverse domains (e.g., physics, chemistry) using various tools while continuously improving through self-enhancement. Therefore, we emphasize the overall self-evolving pipeline rather than any domain-specific instantiation. For mathematical tasks, EvolvedGRPO adaptively selects appropriate modules for generating text editing instructions and employs external tools (e.g., Prisma Art, OpenCV, and Neural Style) to increase the visual complexity of mathematical images. This part was included in the appendix for brevity; however, we will incorporate a detailed description of the specific tools into the main text in the revised version.

Q3: A clear algorithmic flowchart or pseudocode is currently missing from the main body of the paper.

A3: The algorithm flowchart is currently included in Appendix A.4: PseudoCode (located in Supplementary Material.zip) due to space constraints. It will be moved to the main body of the paper in the revised version.

Q4: The presentation of results in Table 1 could be misleading.

A4: In earlier evaluations, we cited results from original papers due to the lack of standardized evaluation protocols. To ensure fairness and provide a comprehensive comparison, we have carefully re-evaluated the results using consistent prompts across all models. The updated results are displayed below:

Table 1: Expanded version of the main table

Q5: Mismatch Between Motivation and Method's Effect

A5: Thank you for raising this important question. We will address this from three points:

1. First, we clarify that EvolvedGRPO is designed to autonomously explore and escalate problem complexity while maintaining logical correctness. It uses the visual perception output as an intermediate state to construct longer CoT. The ultimate goal is to effectively disentangle the complex relationship between vision and text, seamlessly integrating perception into subsequent logical steps.

2. At the training level, the subsequent augmentation task (e.g., a calculation) imposes stringent precision requirements on the initial perceptual output (z). Any slight inaccuracy in perception is significantly amplified in the final result (D). By managing this process with a carefully designed reward function and filtering mechanism, we leverage this error amplification effect to generate a stronger, more explicit gradient signal. This, in turn, compels the model to disentangle the visual and textual information with much higher precision.

3. To demonstrate the framework's generality and effectiveness, we leverage Qwen2.5-VL-7B-Instruct to perform complex editing instructions across various real-world vision-language tasks. The results provide strong evidence that our method effectively decouples visual and textual information in diverse scenarios, leading to a significant enhancement of the model's comprehensive reasoning abilities.

   Table 2: Avg performance of all domains on EvolvedGRPO disciplinary reasoning

   Models	Physics	Chemistry	Biology	Avg.
   OpenVLThinker	53.8	60.6	65.0	59.8
   MM-Eureka	56.2	65.2	65.2	62.2
   Qwen2.5-VL-7B-Instruct	45.4	56.4	54.0	51.9
   +GRPO	51.6	57.1	58.6	55.8
   +EvolvedGRPO	61.5	65.5	63.9	63.6
   +EvolvedGRPO w/iter 1	51.4	56.4	57.2	55.0
   +EvolvedGRPO w/iter 2	54.2	60.8	57.6	57.5

   Table 3: Avg performance of all domains on EvolvedGRPO general medical reasoning

   Models	MedXpertQA-MM	OmniMedVQA	Avg.
   Qwen2.5-VL-7B-Instruct	20.3	58.4	39.4
   +GRPO	23.8	61.0	42.4
   +EvolvedGRPO	24.3	62.8	43.6

Q6: The paper lacks the necessary analysis to prove that this method actually solves the problem it set out to address: reward convergence and training stagnation.

A6: First, we have solved reward convergence by creating a dynamic problem space. As shown in the reward changes provided in the appendix, our adaptive difficulty mechanism ensures that once the model's performance stabilizes on a given problem set, the framework automatically increases the complexity and reasoning chain length. Quantitatively, we further present the gradually changing trends of multiple samples during the enhancement process in the following table:

Table 4: Variables during training

As shown in the table, EvolvedGRPO prevents the model from converging to a single optimal policy for a static task distribution, forcing it to constantly adapt and learn.

Second, the practical result of overcoming reward convergence is the alleviation of training stagnation. We provide direct evidence of this by comparing our method against the baseline GRPO framework.

Third, as direct evidence against reward convergence, we analyze the reward variance for different valid responses to the same problem. We first randomly sampled a set of N base problems. For each problem, we generated multiple distinct yet valid solutions (e.g., different Chains of Thought) using two methods: (a) for the original problem (baseline), and (b) for the problem after being enhanced by our framework. We then calculated the reward for every solution and computed the variance of these rewards for both sets in the table. We can provide experimental evidence to demonstrate that a complete improvement has been implemented.

Q7: The paper's image manipulation method (e.g., adding noise or filters) is conceptually very similar to that of NoisyRollout

A7: Although both our EvolvedGRPO and NoisyRollout edit images, the idea and methodology are different.

In contrast to NoisyRollout, which injects random noise for specific geometric tasks, our method performs semantically meaningful, complex image edits coupled with multi-round text-image interactions, aiming to enhance model capabilities on general-purpose, real-world datasets. (b) Furthermore, in a direct comparison using only image editing, our method outperforms NoisyRollout by an average of 1.0%, highlighting the benefit of our approach. In addition, we have added comparisons with NoisyRollout and our own ablation studies on the lower-level components to better assess the contribution and importance of each part of the system as follows:

Table 5: Extended ablation experiments of EvolvedGRPO

I thank the authors for their detailed rebuttal. The newly provided re-evaluation of baseline models and the comprehensive ablation study have substantially improved the experimental rigor of the paper and successfully addressed some of my initial concerns.

However, I still think that the paper lacks clarity in its methodological description.

Additionally, the central claim that the method solves reward convergence and training stagnation remains unsubstantiated. The evidence provided in the rebuttal is unconvincing, and a promise of future analysis is insufficient at this stage.

Due to these fundamental issues, I will maintain my original rating.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Thank you for your valuable suggestions. We will address your concerns in two areas: methodological clarification and unsubstantiated claim on solving reward convergence and training stagnation.

Methodological Clarification:

In GRPO training for the reward of sampled responses $\\{r\_i\\}^{G}\_{i=1}$, the advantage of each response is:

While the advantages are used to calculate the updated gradients:

During the multi-round GRPO training with fixed data, the increase in accuracy and the training on fixed problems cause the standard deviation to decrease to zero, leading to reward convergence and, ultimately, training stagnation and collapse. To address this, we introduce EvolvedGRPO, which incorporates adversarial data augmentation to introduce additional information during multi-round training. We use the following reward to train a question generation model, which generates editing instructions based on external knowledge (eg, math concepts):

We aim to maximize the correctness of step-by-step edits after deconstruction while minimizing the accuracy of answers to synthesized enhancement problems (but ensuring at least one correct answer).

Unsubstantiated Claim on Solving Reward Convergence and Training Stagnation

Next, we will explain, from both theoretical and experimental perspectives, how we have addressed the issue of reward convergence and training stagnation.

1. Theoretical Analysis

For training data that the model has repeatedly encountered, the accuracy can approach $100\\%$ over multiple learning iterations for part of the data. In such cases, since each evaluation samples only $5$ responses and the reward is variance-normalized, repeated training causes the responses to converge. When all $5$ responses are correct, the variance becomes $0$ and is thus ignored, leading to training stagnation. Furthermore, in order to increase accuracy, the model may omit the reasoning process entirely or produce an incorrect reasoning process that nevertheless yields the correct answer, resulting in negative rewards that can cause training collapse.

In contrast, EvolvedGRPO introduce additional informational complexity during multi-round training, thereby increasing the data's informational entropy and increasing reasoning complexity. This process reduces the model's accuracy, strategically promoting an increase in cognitive load that drives the model to engage in more nuanced multi-step inferential reasoning, preventing training stagnation and fostering further reasoning learning. Furthermore, the integration of additional information and the escalation of difficulty within the synthetic data forces the model to extend its reasoning processes, impeding direct answer memorization and reliance on spurious memorization of erroneous inferential paths and thereby mitigating the risk of training collapse.

2. Experimental Analysis

First, we want to demonstrate that we have addressed Reward Convergence. The practical result of overcoming reward convergence is the alleviation of training stagnation. First, we provide the average variance calculated from the sampling at each corresponding training step, with each training round consisting of 30 steps:

We can observe that EvolvedGRPO slows down the decrease in variance, thereby increasing the proportion of effective data and helping to mitigate reward convergence. Additionally, we calculate the average reward for the corresponding intervals:

We observe that EvolvedGRPO exhibits greater fluctuations during each training round, as it can consistently improve after data augmentation. Specifically, we found that the reward in EvolvedGRPO fluctuates continuously, with the average reward sharply decreasing after each data augmentation before gradually increasing. In contrast, the reward growth in GRPO slows down and eventually stagnates.

The analysis confirms that by generating more complex problems, EvolvedGRPO allows for a richer variety of high-quality solutions, indicating a more diverse and nuanced reward landscape that encourages exploration over exploitation.

Second, we want to demonstrate that we have addressed training stagnation. Previously, we compared the accuracy on the validation set and the average inference token length during multi-round training for GRPO and EvolvedGRPO, as shown in Figure 2 of the main text. We observe that after 30 steps, the inference token length gradually stabilizes during the second and third rounds of training, and the score growth on the validation set slows down. This demonstrates that EvolvedGRPO significantly enhances the model's reasoning ability during multi-round training, preventing the training stagnation observed with GRPO.

Additionally, we have provided an external comprehensive visualization of the entire training process. During training, GRPO shows little to no improvement after $100$ steps and collapses entirely after approximately $600$ training steps. The response length drops to fewer than $20$ tokens over the next $600$ steps, and the average accuracy decreases by close to $60\\%$, leading to an almost complete loss of reasoning capability.

In contrast to GRPO, our EvolvedGRPO maintains a stable improvement in the model’s reasoning capability during training, with accuracy exhibiting continual fluctuations across iterations and the number of reasoning tokens increasing steadily.

We sincerely appreciate the reviewer's insightful and detailed feedback. We address each of your questions individually below.

Q1: The instruction evolution mechanism still relies heavily on implicit assumptions about controllability and reward fidelity.

A1: We will focus on analyzing from three aspects: controllability, fidelity, and generalization.

(1.1) Controllability: Our framework enables fine-grained control over reward dynamics by modulating the depth and structure of multimodal reasoning. Increasing the number of inference steps—particularly in long reasoning chains—yields more informative reward variations than shallow perturbations. We further guide the learning process using MLLM-extracted abstract concepts and adversarial editing instructions for text and image editing. The resulting reward trends, which initially decline and then steadily rise across training rounds, reflect the model’s progressive improvement. Appendix D visualizes this evolution, highlighting the controllability of our approach.

(1.2) Fidelity: To ensure data fidelity, we combine a multi-step filtering pipeline with a stepwise Hallucination Estimator to suppress errors and hallucinations during training. In addition, Manual evaluation further confirms the generation of high-quality, semantically valid data throughout the process.

(1.3) Generalization: We conducted additional experiments on several other real-world vision-language tasks, demonstrating that our approach can be effectively adapted to different tasks with minimum modifications.

Q2: Have you examined whether small perturbations in instructions (e.g., minor edits in visual modality) produce meaningful reward signal changes?

A2: We address this concern from two perspectives.

(2.1) Since reward signals in multi-modal reasoning are closely tied to the depth and structure of the reasoning process, changes in entropy at both the visual and textual semantic levels can effectively modulate the reward landscape. Specifically, altering the number of inference steps—particularly in extended reasoning chains involving over 4096 tokens—tends to induce more substantial and informative reward variations than shallow input perturbations. Appendix D visualizes the evolution of reward signals across multiple training rounds, further supporting this observation. For completeness, we plan to integrate these results into the main text in future revisions.

(2.2) We leverage a MLLM to extract abstract concepts and generate adversarial editing instructions, which are then used to guide both text and image editing for multi-modal enhancement. During each round of training, the reward associated with data augmentation initially drops significantly, then gradually increases as training progresses, accompanied by the continuous improvement of the model's capabilities.

Q3: Could the reward collapse still persist in late training?

A3: Thank you for the concerns about reward collapse. Compared to the original GRPO training, our method not only effectively addresses the issue of reward stagnation caused by repeated training, and also mitigates the reward collapse typically observed after a fixed number of training steps. Specifically, the original GRPO model encounters reward stagnation after approximately 100 steps and collapses entirely after around 600 training steps. In contrast, our EvolvedGRPO framework can maintain stable performance throughout the training process without exhibiting any signs of reward collapse.

Q4: While hallucination reduction is mentioned, how robust is the filtering pipeline?

A4: Thank you for the concerns regarding the reduction of hallucinations.

(4.1) Firstly, the filtering pipeline enhances the validation of each step through multi-step disassembly, which can indeed robustly reduce errors. Moreover, the problem generation model incorporates a Hallucination Estimator at each step of reward calculation, ensuring that hallucinations are minimized as much as possible during each training round.

(4.2) To evaluate robustness, we randomly sampled 100 instances and analyzed the distribution of three types of cases across multiple training rounds: (1) valid and effective questions that contribute positively to training; (2) neutral questions that are ineffective but filtered out without impacting training; and (3) erroneous questions that introduce negative effects. The results are as follows:

Table 1: Human evaluation results for hallucination detection

Q5: Can the framework actively regulate difficulty based on performance plateaus? If so, how is this determined?

A5: The framework can proactively adjust the difficulty. When increasing the difficulty for the question generation model, the reward is computed as the maximum between the actual answer accuracy and 1/test (where test denotes the number of test samples). This guides the difficulty to converge toward 1/test. As shown in the appendix D, when 1/test = 0.2, the answer accuracy effectively converges slightly above 0.2. Additionally, the number of editing steps is increased once the reward stabilizes and the length of the reasoning chain remains unchanged, allowing for further difficulty scaling.

Q6: Could you elaborate on the application potential in real-world settings such as visual question answering for medical or industrial imagery?

A6:

We trained on multiple real-world vision-language tasks by replacing mathematical concepts with their corresponding domain-specific counterparts. With the further development of image editing technology, we embed the Flux.Kontext image editing model into the training framework for image editing, to enable richer and more customized image enhancements. The results are as follows:

Table 2: Avg performance of all domains on EvolvedGRPO disciplinary reasoning

Table 3: Avg performance of all domains on EvolvedGRPO general medical reasoning

Thank you for your response and for carefully considering both our rebuttal and the broader reviewer discussion. We fully respect your decision to uphold your original evaluation.

At the same time, we would like to kindly confirm whether all of your concerns have been fully addressed. If there are any remaining issues or clarifications needed, we would be more than happy to provide further responses.

Thank you again for your time and thoughtful review. Wishing you all the best in your future research endeavors.

We sincerely appreciate your constructive and insightful comments! We will explain your concerns point by point.

Q1: More clarification of Image Editing Pipeline.

Thank you for the valuable feedback. In Appendix F, we list external image processing tools such as Prisma Art, OpenCV, and Neural Style. It is evident that the current image editing tools lack the capability to modify complex elements, such as auxiliary lines, hence our focus on modifications related to flipping, cropping, and style transfer. Nevertheless, EvolvedGRPO, being model-agnostic and a self-enhancing framework, can integrate more advanced visual editing techniques such as Flux.Kontext, in a resource-efficient manner.

Thanks to editing techniques like replacement and rotation from Flux.Kontext, we are able to generate varied physical or medical images, enhancing the multimodal difficulty and thereby assisting EvolvedGRPO in achieving better performance improvements. EvolvedGRPO demonstrates strong performance in both complex physical domains (+22.5% in Table 4) and medical domains (+10.7% in Table 5).

Q2: Robustness of EvolvedGRPO.

A2: As you suggested, we have reproduced the results in three runs for robustness of our experiments. We evaluate the base model Qwen2.5-VL-7B-Instruct, strongest baseline NoisyRollout, and our method EvolvedGRPO with three different seeds. The results are as follows:

Table 1: Multiple evaluations of NoisyRollout and EvolvedGRPO

Q3: Analysis of judgement and reward of EvolvedGRPO.

A3: We apologize for any confusion regarding the judgement and reward. We would like to address your concerns from two perspectives.

(3.1): Firstly, we would like to clarify that GPT4o is only used to determine whether the answer result is the correct answer. For evaluation, we extended the results of using human evaluation to perform multiple random seed evaluations and calculate the mean. The full results are shown below:

Table 2: Evaluation of EvolvedGRPO via multiple human annotators

(3.2): When using reward for mathematical problems, only functional rule-based mathruler are used to calculate higher required rewards, without using LLM to determine whether the answer is correct. In addition, for more complex tasks, rewards can be computed using the base model, thereby avoiding bias preferences in GPT-4o without relying on external models.

Q4: Ablations are high-level only.

A4: Thank you for the suggestion. We have added ablation studies on the lower-level components to better assess the contribution and importance of each part of the system:

Table 3: Extended Ablation Experiments of EvolvedGRPO

As observed, EvolvedGRPO significantly outperforms most ablation settings across all tasks. With the integration of image and text editing, the capabilities of Qwen2.5-VL in logic-based tasks continue to improve, especially after incorporating text editing. This enhancement predominantly stems from the increased complexity of training data due to cascaded text edits, which in turn strengthen the model's reasoning abilities.

Q5: Have you attempted to use it with real-world vision language tasks?

A5: Thank you for your thoughtful suggestions. Thanks to the scalability of EvolvedGRPO, we have successfully applied Qwen2.5-VL-7B-Instruct to real-world vision-language tasks in the next table. In our testing, EvolvedGRPO has excelled in various fields such as physics, chemistry, and biology, consistently outperforming baseline methods in disciplinary reasoning and general medical QA tasks. This confirms the model’s effectiveness and adaptability in real-world applications.

Table 4: Avg performance of all domains on EvolvedGRPO disciplinary reasoning

Table 5: Avg performance of all domains on EvolvedGRPO general medical reasoning

We sincerely appreciate your decision to improve the rating after reviewing our rebuttal. In the revised version, we will incorporate the additional experiments and refine the text accordingly. Thank you once again for your thoughtful review and support.

We sincerely thank you for the valuable comments! We are encouraged to see that our work can enhance subsequent research endeavors. We will address your concerns point by point.

Q1: Limited Scope of Demonstrated Instruction Evolution.

A1: Thank you for raising the important concern regarding the scope of demonstrated instruction evolution in our manuscript. We first leverage Qwen2.5-VL-7B-Instruct to perform more abstract (e.g., conceptual reasoning) and commonsense (e.g., real-world knowledge)-based editing instructions, and then showcase the universal adaptability and scalability of our evolved instruction generation process.

(1.1) Evaluation on More Real-World Vision-Language Tasks Using Qwen2.5-VL

We conduct instructional evolution on various disciplinary reasoning and general medical QA tasks with Qwen2.5-VL-7B-Instruct.

The results are presented in Tables 1 and 2. Using Qwen2.5-VL-7B-Instruct as the backbone, EvolvedGRPO continues to demonstrate its effectiveness across multiple real-world domains such as physics, chemistry, and biology, outperforming all baseline methods on both disciplinary reasoning and general medical QA scopes. This confirms that EvolvedGRPO is model-agnostic and effective across various types of real-world vision-language tasks.

Table 1: Avg performance of all domains on EvolvedGRPO disciplinary reasoning

Table 2: Avg performance of all domains on EvolvedGRPO general medical reasoning

(1.2) Analysis on the scalability of EvolvedGRPO

EvolvedGRPO has demonstrated robust performance across multiple domains by employing a single iteration in its operations. As suggested, we have conducted additional experiments with more iterations to examine its scalability. The outcomes showcase the universally adaptable and scalable nature of our evolved instruction generation process, as detailed in the last two rows of Table 1.

Q2: Concern over Quality Control and Cumulative Error.

A2: Thank you for your questions. We will address them from three points:

(2.1) In our process reward function, we explicitly penalize large discrepancies between step t and step 0 through model-inferred output discrepancies instead of naive rule-based methods. In this way, we adaptively reduce the cumulative error introduced at each step and mitigate the overall error accumulation in multi-step enhancement, even in more complex physical (+22.5% in Table 1) and medical domains (+10.7% in Table 2).

(2.2) To better illustrate the effectiveness of quality control, we present detailed visualizations of reward dynamics during the multi-round training process in Appendix D. During each round of training, the reward associated with data augmentation initially drops significantly, then gradually increases as training progresses.

(2.3) To better illustrate the reduction of Cumulative Error, we quantitatively analyze the accumulation of errors over time t during the progressive enhancement of batch samples. We measure this by the inverse of the similarity between step t and step 0 using human evaluation. The results are presented in Table 3.

Table 3: Cumulative error during training