Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executors

We thank the reviewer for the insightful feedback and valuable suggestions. Below, we address each raised point individually:

Misleading framing of "Weak-for-Strong":

• Clarification and justification of the terminology and reported performance differences: Answer: Thanks for the question. First we have to clarify that, for MATH, we only use questions with the highest difficulty level (5), while in Qwen's report, they report the number on the whole dataset. If evaluating on full dataset, GPT-4o-mini performs much better than Qwen2.5-7B-Instruct. In most of the tasks, GPT-4o-mini has a much better performance than Qwen-7B except for GSM8K, where they have comparable performance since Qwen has been post-trained on similar sources. We already have experiments using GPT-4o and Claude-3-5-sonnet as executors in Appendix E.2, which are definitely strong models compared with 7B Qwen. We here also provide more experiments using GPT-4o, GPT-4.1-mini and o3-mini-medium as executors on AIME dataset.

Clarity and Design Choices:

We thank the reviewer for suggestions to enhance our paper clarity.

• Motivation and explanation for “example workflow” (line 115):

  Answer: The “example workflow” serves two roles.

First, it anchors the format of the workflow, showing the correct function signature and how to utilize LLM call helper function agent.call_json_format_llm. This helps avoid syntactic errors or LLM call errors. Here we show an example:

def workflow(agent, task: str):
    instructions = "Requirements:\\n1. Please explain step by step."
    messages = [{"role": "user", "content": f"# Your Task:\\n{task}"}]
    response = agent.call_json_format_llm(
        messages=messages, 
        temperature=0.5, 
        num_of_response=1,
        agent_role="expert", 
        return_dict_keys=["reasoning", "answer"], 
        instructions=instructions.strip(),
    )
    
    return_dict = response[0]
    return_dict["answer"] = str(return_dict.get("answer", ""))
    return return_dict

Second, the example workflow will be executed on validation set before the first iteration to provide feedback. Our rewards are based on current as well the history workflows. The example workflow enables the system to compute reward signals even before the meta‐agent has generated any workflows.

• Definition and granularity of task T: Answer: For the Task description $T$ included in meta-agent's state, it's text that briefly describe the task, the input and output format. We will list all $T$s to Appendix.

  As for the specific task, it depends on different datasets. For our current QA benchmarks, the task will simply be the questions. For example, a math question is a task. But if we extend it to more complex agentic benchmark, a task can be a complex one that might need multiple turns, tool usages, etc.

• Clarification on public/private validation sets (line 139): Answer: Thanks for pointing out. We will add more details for public/private sets in our paper. Public and private validation sets are created by randomly splitting the original validation data. During training, validation accuracy is computed only on the private split, while case studies (including prompts, model answers, ground-truths, etc) are drawn from the public split (10 validation samples). By exposing the meta‐agent to case studies only from public examples, we prevent it from overfitting to validation samples. In practice, we have observed instances where the meta‐agent attempts to generate workflows that simply reproduce known question‐answer pairs from public examples. Because such memorization does not improve accuracy on the private split, these behaviors receive low rewards and are thus discouraged.

• How iteration is skipped and fallback options (line 143): Answer: During data collection we sample m candidate workflows. Any candidate that fails to execute is simply discarded. If the agent only samples one action at each iteration (without best-of-m) or all m candidates fail (we never see this happens though), the iteration will be skipped, which means we keep the same state to resample actions but the count of iteration will plus one. In fact, with our self-correction mechanism, these cases rarely happens in . And during testing time, with our trained models, we never witness such thing happens. We do not have other fallback options and do not inject incorrect workflows to the history because they will propogate errors.

• Clarification of test-time procedures: Answer: We provide details of test-time procedure in Implementation Details in Section 3.1. Similar to ADAS and AFlow, W4S samples a trajectory at test time with only one action in each iteration. The difference is that ADAS and Aflow requires 30/20 iterations each while W4S only requires 10 iterations to converge. The input is just what you said, the task description, instructions, example workflow, feedback of the specific benchmark. We will add this information to Section 3.1.

Need for Qualitative Analysis:

Thanks for the suggestion to enhance our paper. Here we provide more qualitative analysis to further prove the effectiveness of our framework.

• Cost–benefit trade-off: The extra training/validation overhead noted by the reviewer is modest compared with manual prompt engineering. Although our method, as well as other automated workflow design methods require additional training and validation, automated workflows are still cost-efficient since human labor is the most expensive. It takes time for human to try different prompting methods, come up with new approaches, refine prompts for specific tasks.

• How the learned workflows differ from canonical prompting: Canonical techniques such as chain-of-thought or program-of-thought supply a single, static prompt; in contrast a W4S workflow is a program that decides, step by step, which tool (strong LLM call, Python, majority vote, fallback policy, translator, etc.) to invoke next, conditioned on previous outputs. For instance, on GSM-Hard the agent first generates a concise textual reasoning, then dispatches the resulting numerical expression to the Python interpreter for exact arithmetic, and finally wraps the answer with majority-voting. If code execution fails, it falls back to CoT with role-playing; on MGSM it inserts an initial translation stage that never appears in English-only datasets. These multi-tool pipelines emerge automatically and differ markedly from hand-written program-of-thought prompts, which typically mix reasoning and calculation inside a single LLM call.

• Discussion on failure cases:

  • Validation over-fitting. Early agents occasionally hard-coded answers for samples in case studies. Splitting validation into public/private subsets and rewarding improvement on the private set mitigates the issue.
  • Exploration collapse into resampling loops: We discovered that occasionally Weak coders sometimes inflate “n-best” parameters instead of changing logic. Using best-of-m approaches along with utilizing models better at code can mitigate this issue. Besides, we set a hard limit on the inference time to avoid from too many API calls in one workflows. Future steps can consider improve rewards to reward more creative workflows.
  • Syntax/runtime errors: Prior to RLAO, weak models are prone to make simple coding mistakes. Implementing a decent self-correction mechanism has almost solved this issue.

• Impact of helper functions: The most useful helper fuctions are LLM call function and Python interpreter. For tasks like MATH, the agent learns to use answer extraction function to extract answers in the response and then performs majority voting or debating. The helper function can also be seen as tools for meta-agent. Users can develop more helper functions if more complex agentic tasks needs.

  Here we provide statistics of the number of helper function used for different tasks. We observe that workflows generated for different tasks include at least two helper functions.

Incorrect description of RWR:

• Clarification of the offline RL approach and single-iteration EM implementation: Answer: Thank you for pointing this out--indeed the original Reward Weighted Regression (RWR) was presented as an online EM-style: each iteration collects fresh trajectories and then performs a return-weighted maximum-likelihood (M-step) update.[1,2]. However, the RWR idea is not limited to online interaction and can be applied in an offline RL setting as well, either by collecting a single batch of trajectories to train a policy via one round of reward-weighted regression or by reusing the same offline datasets for multiple rounds of training. This “one-shot” variant has been widely adopted and explicitly described as offline or batch RL in prior work. For example, [3] re-uses a single batch of demonstrations for multiple RWR iterations, [4] describe a single-round reward-weighted fine-tuning of a diffusion model, which is essentially an offline RWR. [5-6] also collects one batch of trajectories for training LLMs. Our “Weak-for-Strong” setting matches these precedents: we collect one batch of trajectories (to avoid prohibitive API cost), weight them by the strong model’s reward signal, and perform a single supervised update. Empirically this already captures most of the potential gain, so further online iterations were unnecessary. We will revise the manuscript to read like: "We employ the offline variant of Reward-Weighted Regression—a single M-step on a fixed dataset ([4-6])."

[1] Reinforcement Learning by Reward-weighted Regression for Operational Space Control [2] Reward-Weighted Regression Converges to a Global Optimum [3]Reward-weighted regression with sample reuse for direct policy search in reinforcement learning [4] Aligning text-to-image models using human feedback. [5] Recursive Introspection: Teaching Language Model Agents How to Self-Improve [6] Aligning Language Models with Offline Learning from Human Feedback

Choice and sophistication of evaluated tasks:

• Justification for chosen benchmarks and potential exploration of more complex tasks like SWEBench: Answer: Thanks for the suggestion. The tasks we use are following popular previous works (AFlow, ADAS), and other concurrent work (MaAS, ScoreFlow, MAS-GPT, etc). In order to test our framework on more difficult benchmark, we here provide experiments on AIME which is challenging even for the latest models.

We thank the reviewer for pointing out SWE-bench. However, due to time limit, we cannot complete the experiments on SWE-bench during rebuttal. As mentioned in limitation part, we believe that extension to more complex agentic tasks like SWE-bench is a straightforward and interesting future step, which might involve some new improvements on the current framework like integrating more tools, memory, etc.

Additional Questions from Reviewer:

• Typo in Figure 2 caption: Answer: Thank you for pointing out the typo. We have corrected it in our paper.

• Correction for equation below line 158: Answer: Thanks for pointing out the issue. The equation should be $a_i^* \,=\, a_i^{\arg\max_{k \in [1,m]} v_i^k}$. We have corrected it in our paper.

• Explanation on how best-of-m selection enriches the training dataset: Answer: At every iteration the meta-agent draws m candidate workflows from its current policy and evaluates each on a held-out validation shard. We then log all m (state s, action a, reward r) triples but advance the search with only the highest-scoring candidate. This simple procedure enriches the offline buffer in two complementary ways:

  • Built-in contrastive signal. Because the m actions share the same state, the buffer now contains one positive Δ-reward example alongside up to m – 1 neutral or negative siblings. The critic can therefore learn “this specific edit raised the strong model’s score, these nearly-identical edits did not,” which sharpens the advantage estimate far more than seeing isolated successes or a sea of uniformly low-return steps.
  • Curriculum through winner continuation. Retaining only the best action to form the next state steers the roll-out toward promising regions of the workflow space. Subsequent m-tuples are sampled around progressively stronger baselines, so the agent observes a sequence of harder-to-improve contexts—an implicit curriculum that accelerates convergence and reduces variance.

  Without best-of-m, a single draw per state would often yield an uninformative failure (no clue what would have worked) or a rare success with no contrasting negatives (no gradient direction).

  Here we provide new experiments of W4S with best-of-m and without best-of-m (sample one action at each iteration). For the latter, we collect more trajectories to keep number of training data the same.

Table: Performance of W4S on GPQA and GSM Hard. The Executor is GPT-4o-mini.

We appreciate the reviewer's detailed feedback and believe these clarifications significantly strengthen the paper. We will include the new results and analysis in our paper.

We thank the reviewer for the insightful comments and are pleased that most of your concerns are addressed. Below, we provide (1) a detailed comparison of workflows produced by W4S against competitive methods (ADAS and AFlow), and (2) extend our analysis to a realistic agentic task scenario using the GAIA benchmark.

comparison with competitive methods

We compare the workflows generated by W4S, ADAS, and AFlow on MBPP (coding task) and DROP (reasoning task):

On DROP, ADAS generates 30 intricate workflows for 30 iterations that still underperform compared to simple CoT SC. For our W4S, the generated workflow is also similar to CoT SC, but with a more detailed and instructive prompts and suitable temperatures. Since W4S utilizes best-of-m selection and conducts RL finetuning, it learns that for this task, combining resampling with ensemble or even simply majority voting is better than other complex structures, it also learns to optimize prompts and temperatures.

Besides, we want to highlight that automated workflow generation via these different methods eliminates expensive manual validation. Although manually created workflows might appear similar, the process to achieve such optimized hyperparameters, prompts, and combination of techniques manually is highly time-consuming and costly. Furthermore, W4S often uncovers novel and task-specific combinations of existing methods that are non-trivial to discover manually.

• MBPP coding task

Workflow generated by ADAS

def forward(self, taskInfo):
   # Step 1: Initial Competitive Stage
   expert_instruction = "Please independently think step by step and provide a solution based on your expertise."
   expert_roles = ["Software Engineer", "Data Scientist", "Python Developer", "Algorithm Specialist"]
   expert_agents = [LLMAgentBase(["thinking", "answer"], "Expert Agent", role=role) for role in expert_roles]

   # Gather initial solutions
   initial_solutions = [agent([taskInfo], expert_instruction) for agent in expert_agents]

   # Step 2: Role Adjustment and Feedback Structuring
   feedback_instruction = "Evaluate the effectiveness of the provided solution and rank its usefulness."
   feedback_agent = LLMAgentBase(["feedback"], "Feedback Agent")

   feedbacks = [feedback_agent([taskInfo, solution[0], solution[1]], feedback_instruction)[0] for solution in initial_solutions]
   ranked_solutions = sorted(zip(feedbacks, initial_solutions), key=lambda x: float(x[0].content) if x[0].content.isdigit() else 0, reverse=True)

   # Adjust roles: prioritize collaboration with top solutions
   top_solutions = [solution for _, solution in ranked_solutions[:2]]
   role_adjustment_instruction = "Based on ranking, guide collaboration to refine top solutions."
   refined_solutions = []
   for solution in top_solutions:
       refined_solution = LLMAgentBase(["thinking", "answer"], "Refinement Agent")([taskInfo] + solution, role_adjustment_instruction)
       refined_solutions.append(refined_solution)

   # Step 3: Collaborative Refinement Stage
   collaboration_instruction = "Collaborate and integrate the refined insights into a cohesive solution."
   collaboration_agent = LLMAgentBase(["thinking", "answer"], "Collaboration Agent")
   collaboration_thinking, collaboration_answer = collaboration_agent([taskInfo] + [info for solution in refined_solutions for info in solution], collaboration_instruction)

   # Step 4: Final Decision
   final_decision_instruction = "Converge the collaborative insights into a final coherent answer."
   final_decision_agent = LLMAgentBase(["thinking", "answer"], "Final Decision Agent", temperature=0.1)
   thinking, answer = final_decision_agent([taskInfo, collaboration_thinking, collaboration_answer], final_decision_instruction)
   return answer

Q3: Only one model is evaluated as trained weak meta-agent:

We thank the reviewer for pointing out this question. We here provide more experiments using different weak models as meta agents.

Table: Testing performance with GPT-3.5-Turbo as Executor.

‘Weak for strong’ narrative is unconvincing:

Thanks the reviewer for the insightful question. We here provide detailed explanation to clarify your concerns.

• Contribution of novel approach and SOTA results: Our primary contribution is the novel method using RL to train meta-agent to harness strong executors. Our SOTA results are meaningful contributions to the community. We will reduce emphasis on the "weak for strong" framing to avoid confusion.

• Would a trainable strong meta-agent be better? In principle, fine-tuning strong models as a meta-agent could yield marginally better workflows, since it might generate more fluent code or choose subtler prompt strategies. In practice, however, two factors limit this advantage:

  1. The meta-agent’s task is primarily workflow composition rather than deep problem solving. Once the agent can (i) write syntactically correct Python and (ii) reason about what techniques to use and how to improve, further raw language ability or reasoning abilities yields diminishing returns.
  2. The most capable LLMs today—o3, Gemini 2.5-Pro, Claude—are not finetunable at all; even open-weight 70B checkpoints are unaffordable to train for most practitioners.

• Cost analysis in a realistic usage scenario. It's true that the cost of generating workflows can be amortized, but this amortization is not as straightforward as it might seem. For real application, the execution of strong models typically uses API calls (even for a large open-sourced model), the cost will be the API credits. But to train a strong open-sourced model, one has to use unaffordable GPU compute along with training-time cost.

• Clarification or justification of 'Weak for Strong' terminology: (Burns et al. 2024) used “weak-to-strong generalisation” for human labels supervising super-human tasks. Our setting is different yet analogous: a cheap, trainable LLM (weak) learns a policy-level skill that boosts a untrainable, frozen or even ununderstandable LLM (strong). The scientific question in both cases is the same: can limited supervision bootstrap a more powerful system than the supervisor itself? The question arises in Burns et al. (2024) because they focus on the analogy of humans supervising superhuman models; in our context, it emerges from the practical constraint of strong models being untrainable and increasingly opaque to human understanding as their capabilities grow. If this wording remains confusing, we are open to alternatives (e.g., “Open-for-Closed”) that better reflect the practical constraints of unmodifiable, high-capacity LLMs.

Additional Questions from Reviewer:

• How is w_0 chosen? Sensitivity to initial workflow (w_0):

  Answer: $W_0$ is chosen by domain:

  • MBPP, HumanEval: CoT

  • DROP, MMLU-Pro, GPQA: CoT, CoT SC

  • Math benchmarks: CoT, CoT-SC, and a CoT-with-code variant that runs the derived expression in Python.

  Compared with ADAS which utilizes nine initial workflows, W4S necessitates fewer instantiations (1-3). We here provide more results on sensitivity to $W_0$:

  Table: Performance on DROP using different $W_0$

  $W_0$	DROP
  CoT	87.9
  CoT SC * 5	87.3
  CoT + CoT SC	87.5

  Table: Performance on MBPP using different $W_0$

  $W_0$	MBPP
  CoT	86.8
  CoT SC * 5	84.8
  CoT w/ Test	87.6
  CoT + CoT SC * 5 + CoT w/ Test	87.9

  Table: Performance on MGSM using different $W_0$

  $W_0$	MGSM
  CoT	65.5
  CoT SC * 5	66.8
  CoT w/ code	62.5
  CoT + COT SC + CoT w/ code	68.2

  The reason why we seed math tasks with both a plain CoT template and a CoT-with-code template is to let the meta-agent sees syntactically correct examples of how to utilize LLM call helper function to get text or code. When we provided only plain CoT, the agent still attempts to add code generation, but its early scripts frequently crashed with syntax errors.

• (Table 1) Dataset used for SFT on 4o-mini vs. W4S w/SFT:

  Answer: Thanks for the question, but there might be a misunderstanding. In table 1, Finetuned GPT-4o-mini is trained on validation samples rather than serving as the meta-agent, so it's reasonable that the SFT tuned GPT-4o-mini falls short. However, it's true that we can finetune GPT-4o-mini as meta-agent, but the trajectories should not be the same as used for W4S with 7B Qwen since those trajectories are collected using 7B Qwen itself. We could use GPT-4o-mini as meta-agent to collect trajectories and filter out those with low rewards (since now we do SFT). Here we provide the corresponding experiments.

  Table: Accuracy performance on different benchmarks. Executor is GPT-3.5-Turbo.

  	MGSM	GSM Plus	GSM Hard	GSM 8k	SVAMP
  ADAS	57.4	53.4	34.5	61.1	82.8
  W4S (7B Qwen)	68.2	66.2	61.8	86.5	84.2
  W4S (Finetuned GPT-4o-mini)	68.6	64.5	63.4	85.7	85.5

  Since finetuning API requires more API cost, training a weak model serve as more efficient and effective approach.

• Reasoning for different model use in Tables 1 and 2 (4o-mini vs. 3.5 Turbo):

  Answer: We use different strong models as executors to show that our method performs well regardless of model use. We choose these two models because ADAS utilized 3.5 Turbo and AFlow utilizes GPT-4o-mini. We also provide results using GPT-4o and Claude as strong executors in the Appendix.

• (Table 3) Explanation for W4S's faster wall-clock time compared to AFlow:

  Answer: Yes, AFlow executes sequentially while ours benefit from parallelization.

• Workflow for GSMHard: Explanation of prompt choice:

  Answer: In this case, the meta-agent utilizes the provided helper function agent.execute_code(code). This helper function has the following desciption shown in the prompt (can also check Appendix A.2): "agent.execute_code(code): Execute the code and return the output. The code MUST contain a solution function. The output of execute_code(code) will be the return value of the solution function if the code is executed successfully or raise an exception."

We greatly appreciate the reviewer’s feedback and are confident these clarifications strengthen the paper significantly.

We thank the reviewer for the thoughtful and insightful comments that make our paper better. Below, we address each raised point individually.

Q1: Unclear why the system is better:

Clarification on code-based vs. graph-based workflows: Answer: Graph-based systems like AFlow use a fixed library of nodes (3 to 4 nodes for one task), each encapsulating hard-wired code and a fixed prompt. The meta-agent cannot modify a node’s internal logic or introduce entirely new modules. As [1] already observed, its action space is narrow and prone to premature convergence on suboptimal templates.

By contrast, W4S treats workflow design as a programming task: the meta-agent writes arbitrary Python code that can call strong models, combine steps, and implement error handling on the fly:

• Unrestricted module creation. The agent can implement unrestricted techniques by calling strong models. On MMLU-Pro it spontaneously spawned role-specialised sub-agents (“scientist”, “reason expert”, etc) by specialised system prompt and fused their answers. On multilingual task, the meta-agent happens to create a translator module that translates original prompt to English version. Besides, for mathematical tasks like GSM Hard, the meta-agent introduces error handling of code execution.
• Editable implementations of the same primitive. Even for the same techniques like “CoT SC”, W4S can re-implement it. For example, on MATH, the agent discovered that extracting the final numbers from each chain and doing simple majority-voting outperforms using ensemble which feeds the entire reasoning trace into another LLM call. The latter is what AFlow’s built-in Ensemble node does. But on benchmarks like DROP, the meta-agent learn to use ensemble instead of majority voting of final answers.
• Granular prompt and hyper-parameter control. Temperatures, sample counts, agent roles (system prompts) and etc are micro-adjustments in Python that contributes to improvement.

Thanks for pointing out this question, we will include the comparison and analysis to appendix of the paper.

[1] ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization

What does the model learn during training (MATH example): As mentioned in the last question, for MATH dataset, the meta-agent in W4S learns that COT with final answer majority voting performs better than COT with ensemble (which is conventional COT SC does), it also learns that directly solving the problem performs better than python code generation since the answers in MATH are often LaTex expressions. Below is the best workflow generated by AFlow within 20 iterations for MATH, which is a conventional COT prompting followed by python code generation.

XXX_PROMPT = """
# Please provide a detailed solution to the following problem, ensuring each mathematical step is explained.
#
# Solve it.
#
# """
class Workflow:
    def __init__(
        self,
        name: str,
        llm_config,
        dataset: DatasetType,
    ) -> None:
        self.name = name
        self.dataset = dataset
        self.llm = create_llm_instance(llm_config)
        self.llm.cost_manager = CostManager()
        self.custom = operator.Custom(self.llm)
        self.programmer = operator.Programmer(self.llm)

    async def __call__(self, problem: str):
        """
        Implementation of the workflow
        """
        # Step 1: Obtain initial solution using Custom operator
        custom_response = await self.custom(input=problem, instruction=prompt_custom.XXX_PROMPT)
        initial_solution = custom_response['response']

        # Step 2: Cross-verify with Programmer operator
        programmer_response = await self.programmer(problem=problem, analysis=initial_solution)
        final_answer = programmer_response['output']

        return final_answer, self.llm.cost_manager.total_cost

GSM Hard vs. MGSM performance differences: (1) MGSM is a multilingual math task that also requires multilingual abilities. As shown in Figure 4, the case-study workflow for MGSM and followed by code generation (2) GSM-Hard replaces the numbers in GSM8k with much larger ones, turning questions into pure long-arithmetic challenges where resampling and prompting methods all fail but do benefit from code execution. The workflow generated for GSM-Hard utilizes code generation with error handling mechanisms, which improves the performance largely.

Contributing factors to performance improvements: The performance improvements stem from detailed prompts, hyperparameters (number of samples, temperatures, etc), suitable combinations of different techniques, different agent role playing (implemented with system prompt), different implementations of the same techniques.

Q2: Limited to well-known reasoning and QA tasks:

• Potential risk of solution leakage into workflow training:

  Answer: Here we clarify why our results are unlikely to come from the meta-agent merely “reciting” templates that appeared in its pre-training data:

  1. Direct leakage of workflow code is impossible since the meta-agent never sees any explicit mapping of the form '<task, strong mode> --> workflow'.
  2. Since the reward depends on relative improvement, simply reproducing a canned program offers no advantage once it has already plateaued; the policy is pushed to modify the previous script, not to replicate something it might have read in pre-training.
  3. Previous work like ADAS and AFlow utilize GPT-4o or Claude as meta-agent, whose pre-training corpora are strictly larger than any open-weight 7B model, but they still underperform W4S when used inside ADAS or AFlow.

• Detailed comparison of dataset differences (GSM Plus, GSM8k, GSM Hard, SVAMP): Answer: We here provide comparison of 5 datasets.

We here provide results using different workflows that are gerated for GSM Plus or GSM Hard.

Table: Performance on GSM Plus and GSM Hard using different workflows. The Executor is GPT-3.5-Turbo.

As shown in the table, the performance differs when the GSM Plus workflow is applied to GSM Hard, and vice versa. We will update this comparison as well as the detailed workflow code in our paper.

We are really grateful to the reviewer for the thoughtful and insightful comments to make our paper better. Below, we provide new experiment results and address each raised point individually.

Q1: Comparison with training strong models w/ RL

The reasons why we did not compare with directly training strong models on a training dataset with more advanced training algorithms is that (1) Since we only utilize a small validation set, it's unfair to compare our method with strong models trained on full training dataset. (2) We acknowledge that with proper training (e.g., RL), trained strong models are expected to outperform W4S. However, our objective is not to outperform a fully-trained strong model, but to approach that upper bound when full training strong models is impractical or even impossible. This is motivated by common real-world cases when commercial models are closed-sourced or comparable open-source models demand prohibitive compute.

Q2: meta-agent w/o RLAO already improves performance

1. Why the “raw” W4S meta-agent is already strong on some datasets

Even without RLAO, W4S has intrinsic advantages over existing baselines.

• Unconstrained code-represented workflows. Unlike AFlow (graph-level, only 3 or 4 nodes for each task) that suffers from early convergence to sub-optimal, hard-wired templates, W4S lets the meta-agent write arbitrary Python code that calls the strong model to complete the task. This vastly enlarges the search space and encourage creativity.
• Short, information-dense context. W4S maintains a short, information‑dense history window rather than accumulating a long transcript of past iterations (as ADAS does). By focusing the agent’s attention on the most recent, salient workflows—augmented with case studies of past failure modes—W4S avoids overwhelming its context with irrelevant history (we show in Appendix E.1 that ADAS performs no better than random sampling).
• Self-correction mechanism. This ensures that simple coding errors do not derail the search: if a generated workflow fails on a single validation example, the meta‑agent is prompted to fix it up to three times. In contrast, AFlow lacks any error‑handling, so mistakes tend to propagate through subsequent iterations.
• Besides, we utilize code-tuned weak model Qwen2.5-Coder, which exhibits relatively good coding ability.

These factors let W4S beat baselines “out of the box.” on some benchmarks.

2. What RLAO still adds and experiments with different weak models

• RLAO amplifies weaker agents: To further prove the effectiveness of RLAO, we here provide more experiment results using different weak models as meta-agent. It's evident RLAO improves performance of all weak models and is more effective when the weak model is weaker (Llama, 3B Qwen). This confirms that RLAO behaves like a proper policy‑improvement algorithm: the weaker the initial policy, the more room there is to improve.

Table: Accuracy Performance of W4S using different models as meta-agents. The executor is GPT-3.5-Turbo

• Faster convergence and higher stability: As shown in Figure 3, W4S w/ RLAO is more stable with faster convergence, leading to sequential improvements compared with baselines and untrained versions.

3. Take-aways The baseline-beating performance without RL is a design achievement, not evidence that RLAO is unnecessary. RLAO still provides (i) consistent extra accuracy on high-ceiling tasks, (ii) large relative gains for truly weak models, and (iii) faster, more stable optimisation – all with <1 GPU-hour of training. We will add the expanded ablation table and statistical tests to the camera-ready version.

Q3: "unseen" math tasks seem similar to the seen tasks

• Generalization beyond mathematics: Table 1 already reports transfer from the same meta-agent to different domains, where W4S still outperforms automated baselines.
• "unseen" math tasks are genuinely different: Although in table 2, all the tasks are basically math ones, they have subtle differences. e.g.(1) MGSM is a multilingual math task that also requires multilingual abilities. As shown in Figure 4, the case-study workflow for MGSM begins with a Translator module, which never appears for the other datasets (2) GSM-Hard replaces the numbers in GSM8k with much larger ones, turning questions into pure long-arithmetic challenges where resampling and prompting methods all fail but do benefit from code execution. GSM-Plus perturbs and re-phrases GSM8K items, introducing distracting facts and adversarial choices; success relies on logical robustness rather than large-number arithmetic, where solely relying on python calculator is not enough. The resulting workflows demonstrate differences and confirm adaptation.

We will update a more detailed qualitative analysis and comparison with different generated workflows in camera-ready version.

Q4: Lacks comparison between the RLAO algorithm and other algorithms used for improving LLM coding and tool use.

Thanks for your suggestion! We here provide detailed comparison and analysis on why RLAO is novel. The discussion here will be updated in later version of our paper.

Comparison with other algorithms: Unlike RLHF, which depends on human‑labeled preferences or a learned reward model, RLAO computes rewards directly from a strong model’s validation scores—no separate reward network is required. Unlike “RL with verifiable rewards,” which optimizes absolute task accuracy, RLAO reinforces any workflow change that yields relative improvement, regardless of final performance. Methods such as Toolformer, ReAct, and LATS teach tool‑calling through self‑supervision or prompt engineering but do not learn a multi‑turn policy to refine workflows.

Key novelty of RLAO:

1. novel application of RL: Rather than optimizing a model’s raw task‑solving ability, RLAO optimizes the weak agent’s skill at understanding the task and generating and improving workflows that call the strong model.
2. Rewards are computed by comparing strong‑model validation performance before and after each proposed workflow, eliminating the need for human preferences or an explicit reward network.
3. RLAO is multi-turn algorithm. And the training is entirely offline: since executing the strong model is expensive, RLAO avoids inefficient online updates (e.g., PPO) by using a batch of recorded workflows and their validation outcomes.
4. Best-of-m selection: accelerates convergence by focusing learning on the most promising trajectories in each iteration; enriches the trajectories by including both good and bad attempts

In summary, RLAO is novel in what it optimises (ability to generate and improve workflow), how it learns (offline reward-weighted RL), and where it is applied (weak agents steering much stronger executors)—a setting not addressed by existing coding or tool-use RL methods.

Q5: Unclear if the method generalizes to other weak models

Our framework is model-agnostic; it treats the weak meta-agent as a black-box policy that produces workflow code, then refines that policy with RLAO. We initially reported Qwen2.5-Coder-7B-Instruct because it's one of open-weight 7B models that has been post-trained on code, which makes our training more efficient and effective. Weaker models may spend more time on correcting format or syntax errors. To further prove the efficacy of our method, we here provide experiments with another three weak models.

Table: Accuracy Performance of W4S using different models as meta-agents. The executor is GPT-3.5-Turbo

All four weak models outperform both strong-model optimisers (ADAS/AFlow) on at least one benchmark, demonstrating model-agnostic generalisation. Llama performs worse than other models. However, we currently stop after a single offline-RL pass; additional collect-and-retrain iterations will close the gap further.

Q6: inference-time compute

We enforce a hard cap during execution. If a run exceeds the budget, execution is aborted and the trial is skipped. However, we did not explicitly penalise cost in the reward, so it's possible that the meta-agent learns to scale up inference-time compute. Therefore, we conduct cost analysis which demonstrates that our method does not require more inference-computing than baselines. Incorporating cost into framework could be an interesting future step to enhance the application.

Q7: Can datasets solved by the weak coding model directly?

We follow previous automated workflow generation work like ADAS and AFlow to choose the datasets used in our paper (e.g., DROP, MATH, MBPP, etc). As mentioned in limitation section, implementing more complex tasks where weak models cannot solve at all will be a interesting future step. Besides, here we provide experiments on a quite difficult dataset AIME that the weak coding model cannot solve at all. We use AIME23 as validation set and test on AIME24.