We are grateful for the reviewer’s constructive remarks and suggestions. We respond to each point in the following:

Weakness: Clarity and Completeness of Method Presentation

Thank you for the insightful suggestion.

Regarding the missing details and ablation study, we have included the training overhead in Appendix Section C.7 and Table 16. In addition, we present ablation studies in the appendix: Section C.4 introduces various iterative training methods, and Section C.5 compares different controller model backbones. We agree that presenting these results in the main paper would improve readability. Accordingly, we will revise the main text to briefly summarize the training cost across different tasks and highlight the effectiveness of different controller architectures and training methods.

For clarification of sections and figures, we will explicitly reference the relevant components in Figures 2 and 3 within Sections 2 and 3.3 to improve clarity and navigation for readers.

Regarding other formatting issues, we will increase the spacing between table captions in Table 4 and body text to enhance visual clarity. As for the set notation in line 83, we acknowledge that it is not strictly accurate: while sets imply unordered, distinct elements, the guidance steps $g_i$ in our case represent an ordered sequence that may contain repetitions. To address this, we will rephrase it as “a sequence of T intermediate guidance steps $g_1$, $g_2$,…, $g_T$” to better reflect the intended meaning.

Question 1: Exam Task Difficulty without Optimization

We manually reviewed all the tasks in the Exam benchmark and identified the primary reasons why the unoptimized controller fails while the optimized controller succeeds.

To recap, the controller model is responsible for decomposing the task into intermediate steps, which are then used to guide the black-box LLM in invoking functions from the Agent class to solve each step.

In many failure cases, the unoptimized controller produces incorrect step formulations. For example, when the task requires examining a pencil using a desklamp, the unoptimized controller might output the step: “examine the pencil with the desklamp.” This leads to an error since the Agent class has no function named examine. In contrast, the optimized controller reformulates the step as: “take the pencil with the desklamp.” which correctly guides the black-box LLM to place the pencil under the desklamp, thus effectively performing the examination without triggering an error. This type of semantic mismatch accounts for most of the unoptimized controller's failures.

Additionally, the unoptimized controller sometimes fails to produce sufficiently detailed decompositions. For instance, if the task is to look at an alarm clock under a desklamp, the correct sequence of steps should be: (1) take the alarm clock, (2) turn on the desklamp, and (3) look at the alarm clock under the light. However, the unoptimized controller may directly attempt to “look at the alarm clock under the desklamp” without taking the alarm clock or turning on the light, resulting in an execution error.

Question 2: Consistency and Automation of Initial Prompting

Thank you for the insightful question. We want to clarify this question from two perspectives:

1. There are no "hand-crafted" prompts— the initial prompt is simply used to present the task across all baselines.
2. The inputs and outputs to the black-box LLM follow the same prompt format in both our method and the baselines, ensuring that all approaches are solving the same task under consistent conditions.

Below we elaborate on each point.

First, we emphasize that the initial prompt is not hand-crafted for specific tasks; rather, it serves as a general instruction to indicate the problem. Specifically, it can be simple while still providing sufficient context. In AlfWorld, for instance, we use a simple instruction:

Decompose the task into steps. First give a general plan of how you would solve the task, then for each step you plan to take, mark with '[Step xx]'.

We accompany this with a single one-shot example. Despite its simplicity and lack of task-specific hand-crafting, this initial prompt, when paired with our optimized controller, enables the white-box model to effectively guide the black-box LLM and achieve strong performance.

Additionally, regarding the difference between our pipeline and direct prompting baselines, we would like to clarify that the intermediate prompt (i.e., the input to the black-box LLM) closely resembles the format used in direct prompting. For example, in AlfWorld, the black-box LLM is required to annotate each action step before generating code, effectively guiding the code generation process. Similarly, the intermediate prompt produced by our controller model serves the same purpose and follows the same format. As a result, both approaches are solving the same task under equivalent prompt structures.

Question 3: Generalization and Human Dependency in New Tasks

We appreciate your insightful suggestion regarding extending the framework to new tasks requiring human feedback. To this end, we additionally implement the framework on code generation, using the APPS [1] benchmark, a large-scale and challenging code generation benchmark comprising 10,000 difficult programming problems. While this allows for scalable and fully automated preference annotation, it is worth noting that the test cases themselves are human-authored, and thus implicitly reflect human judgment regarding correctness.

​ In this setup, the controller model is tasked with generating high-level observations of the problem, which are then provided as guidance to a black-box LLM to assist in generating the final code solution. Due to time constraints, we were unable to optimize the intermediate prompts; However, our approach still outperforms strong baselines. Specifically, we compare against the following: (1) direct prompting, (2) a black-box + black-box pipeline using the same prompt as our method, and (3) an unfinetuned controller + black-box setting. For both our method and direct prompting, we also incorporate a reflection strategy to further boost performance. All experiments use Qwen2.5-7b-instruct as the controller model and GPT-4o-mini as the black-box LLM, with the controller trained via DPO on collected preference data.

Here, Strict Accuracy refers to solving a problem by passing all test cases, while Test Coverage denotes the average pass rate across test cases. Despite the lack of optimal intermediate prompt design, our method surpasses the strongest baseline by 0.7% in the single-pass setting, and this margin increases to 1.3% with four reflection attempts.

Compared to simpler domains such as planning, code generation poses significantly greater complexity and requires more careful design of intermediate prompts. We believe that with more refined prompt engineering, our framework could yield even stronger results in this setting.

[1] Measuring Coding Challenge Competence with APPS. (Hendrycks et al., 2021)

We hope our responses address the reviewer’s concerns and further highlight the value of our work. Please feel free to let us know if any questions remain.

We are grateful for the reviewer’s constructive remarks and suggestions. We respond to each point in the following:

Weakness 1: Preference Annotation Cost

We first emphasize that our method is compatible with both single-turn DPO and iterative DPO, and other RL algorithms for the white-box LLM training, with both human ranking and automatic verifier. In fact, in our experiments, we already exploit the verifier in learning.

For example, in planning tasks like AlfWorld, preferences are determined based on whether the generated plan leads to successful task execution automatically. In reasoning tasks such as GSM8K, preferences can be inferred by comparing the final answer against the ground truth. In personalization tasks like LaMP-1/2M/2N, preferences are derived based on comparing the final answer against the ground truth; For LaMP-3/4, preferences are derived based on comparing MAE/Rouge-L scores.

This design enables scalable and reproducible preference annotation without requiring manual human supervision, thereby making our framework more practical and efficient for real-world applications.

Question 1 & Question 2: Comparison with Online RL and Direct Output Training

Thank you for the question. We have included additional evaluations of the LLaMA3-8B-Instruct model on the GSM8K and GSM-Hard datasets. Following the same experimental settings as outlined in the paper, we employed a three-shot prompt design and incorporated closed-loop refinement via PAL during the inference stage.

We evaluated the performance of both the vanilla LLaMA3-8B-Instruct model and the DPO fine-tuned LLaMA3-8B-Instruct model as well as SFT fine-tuned LLaMA3-8B-Instruct model. As shown in Table 1, even when utilizing a relatively less powerful black-box model such as GPT-3.5, our white-box + black-box framework outperforms the powerful SFT fine-tuned white-box model by 6.4% on the GSM8K dataset and 6.3% on the GSM-Hard dataset. These results strongly demonstrate that our black-box + white-box approach is significantly more effective than directly fine-tuning the white-box model.

Table 1: Comparison with fine-tuned white-box models on GSM task

The reason is that solving the task often requires advanced reasoning capabilities, which are typically beyond the capacity of a white-box LLM at the 8B scale. In contrast, black-box LLMs have already acquired extensive knowledge during pre-training. Through reinforcement learning-based tuning, the white-box LLM can more effectively learn how to elicit and leverage this embedded knowledge from the black-box LLM, leading to improved accuracy.

Thank you once again for your insightful review. We appreciate your feedback on our work. Feel free to let us know if you have any further questions, and we are happy to discuss further.

Thank you for your insightful comments and for taking the time to review our paper. We respond to your suggestions as follows:

Weakness 1 & Question 1: Generalization to Unseen Tasks

We appreciate your suggestion to extend the framework to other task domains. We emphasize that our method is widely applicable to different problems and we additionally test the methods on code generation, using the APPS [1] benchmark, a large-scale dataset with 10,000 challenging programming problems.

In this setup, the controller generates high-level observations of the problem, which are then used to guide a black-box LLM in producing the final code. Due to time constraints, we were unable to optimize the intermediate prompts; However, our approach still outperforms strong baselines. Specifically, we compare against: (1) direct prompting, (2) a black-box + black-box pipeline using the same prompt as our method, and (3) an unfinetuned controller + black-box setting. For both our method and direct prompting, we also incorporate a reflection strategy to further boost performance. All experiments use Qwen2.5-7b-instruct as the controller model and GPT-4o-mini as the black-box LLM, with the controller trained via DPO on collected preference data.

Here, Strict Accuracy refers to solving a problem by passing all test cases, while Test Coverage denotes the average pass rate across test cases. Despite the lack of optimal intermediate prompt design, our method surpasses the strongest baseline by 0.7% in the single-pass setting, and this margin increases to 1.3% with four reflection attempts.

Compared to simpler domains such as planning, code generation poses significantly greater complexity and requires more careful design of intermediate prompts. We believe that with more refined prompt engineering, our framework could yield even stronger results in this setting.

[1] Measuring Coding Challenge Competence with APPS. (Hendrycks et al., 2021)

Weakness 2: Computational Overhead of Iterative Optimization

We acknowledge that iterative DPO introduces additional computational overhead compared to single-turn DPO.

We first emphasize that our method is compatible with both single-turn DPO and iterative DPO, and other RL algorithms for the white-box LLM training.

Our experiments suggest that even a single iteration of DPO is sufficient to yield strong performance. For example, as shown in Table 4, the second iteration only brings a marginal improvement of 1.49% over the first, while a single-turn DPO already outperforms the strongest baseline by 6.72%. This strong performance of single-turn DPO is also evident in both reasoning and personalization tasks, as demonstrated in Table 3 and Table 1, respectively.

We exploited iterative DPO in our setting to make the algorithm is compatible with both offline setting (by a single-turn upon the provided samples) and on-policy setting (with data collected through the current policy).

Weakness 3 & Question 2: Missing Comparison with Adaptive Prompting Methods

We appreciate the suggestion to include additional comparisons with adaptive prompting methods. In response, we now compare our approach with APE[2] and PHP[3] on GSM8K reasoning tasks, as shown in Table 1. It is worth noting that DSP[4] involves additional tuning via SFT and/or RL; thus, we include PHP as a more comparable baseline.

Table 1: Comparison with baselines on GSM8K

All experiments are conducted using GPT‑3.5‑Turbo as the black-box LLM. Our method consistently outperforms baseline approaches on GSM8K.

APE optimizes prompts by searching over a candidate pool generated by an LLM, while PHP incrementally uses previously generated answers as hints to guide the model toward the correct solution. Although these methods share our goal of prompt optimization via LLMs, they both rely on training-free, in-context learning strategies.

In contrast, DSP utilizes a small policy model to generate static clues, without modeling multi-turn interactions between the controller and the black-box LLM. Our method introduces a trainable white-box controller that not only performs prompt optimization but also engages in multi-turn interaction with the black-box model. This design offers a more powerful and flexible mechanism for guiding the black-box LLM and further enhancing its capabilities. We will incorporate these comparisons into our revised work.

[2] Large language models are human-level prompt engineers. (Zhou et al., 2022)

[3] Progressive-hint prompting improves reasoning in large language models. (Zheng et al., 2023)

[4] Guiding Large Language Models via Directional Stimulus Prompting (Li et al., 2023)

Weakness 4 & Question 3: Limited Analysis of Intermediate Guidance

We appreciate the helpful suggestion. We have included a case study on intermediate guidance for three tasks in Appendix Section I. Here, we provide a more focused and detailed analysis of the quality of intermediate guidance across the three task domains.

In Alfworld, the controller plays a key role in generating a high-level plan for the given task, which is then used to guide a black-box LLM in executing the required actions. For example, given the task "put a hot potato in the garbage can", the controller produces the following decomposition:

General plan: Retrieve a list of receptacles to locate a potato, bring it to a microwave for heating, and then place it into a garbage can.
Step 1: Get a list of likely receptacles containing a potato.
Step 2: Search through the receptacles until a potato is found.
Step 3: Pick up the potato.
Step 4: Go to a microwave and heat the potato.
Step 5: Go to a garbage can and dispose of the potato.

With this structured guidance, the black-box LLM is able to successfully complete the task.

In contrast, when prompted to solve the task directly without decomposition, the LLM tends to follow an incorrect default action. For example, "go to a stoveburner to heat the potato", which consistently fails with the error: "I cannot heat the potato 2 using the stoveburner."

In GSM8K, the controller is responsible for generating intermediate reasoning steps that guide the black-box LLM to arrive at the correct final answer. Take the following math problem as an example:

"Every day, Wendi feeds each of her chickens three cups of mixed chicken feed... In the morning, she gives her flock 6,887,483 cups of feed, and another 25 cups in the afternoon. If the flock contains 20 chickens, how many cups of feed does she need to give in the final meal of the day?"

Without decomposition, the black-box LLM incorrectly attempts to reverse-engineer the feed-per-chicken values from the morning and afternoon feed, resulting in a negative and invalid final answer (e.g., -8.0).

In contrast, with controller-generated intermediate guidance:

1. Calculate the total amount of feed given in the morning and afternoon.
2. Determine the total daily feed requirement per chicken.
3. Subtract the given feed from the total to find the final meal's requirement.

The black-box model is able to reason in the correct direction, yielding the correct solution structure.

In LaMP-1, the task for the controller is “Write a summary, in English, of the research interests and topics of a researcher who has published the following papers.” The black box model will answer the question about the author's reference choice based on the summary. So the performance heavily depends on the summary.

An example of the summary: “The researcher's work spans several domains within wireless sensor networks, network communications, multimedia, and social network analysis. Specifically, the research interests include:...” High-quality summaries generated by the controller are able to provide good and direct guidance for the answer.

In summary, M-Pilot is able to generate high-quality intermediate responses, and thus improve the overall performance on various tasks.

Question 4: Failure Cases of Controller Guidance

We manually reviewed all intermediate guidance across the AlfWorld and GSM-Hard evaluation sets. In AlfWorld, we observed that whenever M-Pilot fails, direct prompting also consistently fails, highlighting the strong quality of the intermediate guidance.

In GSM-Hard, we identified 11 out of 1,319 cases where M-Pilot fails while direct prompting succeeds. For example, consider the question:

Last Friday, 13 of the 82 teachers at Rydell Elementary School were sick. There were 2,655,185 substitute teachers called in to help. How many teachers were at school that day?

M-Pilot decomposes this into:

1. How many teachers were present at school that day?
2. How many teachers were not sick?
3. How many teachers were not called in as substitutes?

We found that the third question is ill-posed and semantically meaningless, which misleads the black-box LLM and leads to an incorrect answer.

Overall, we find that M-Pilot only degrades performance when it generates clearly misleading intermediate guidance, which occurs infrequently in our experiments, and is limited to extremely challenging tasks like reasoning. In the aggregate, M-Pilot still brings a significant improvement across the entire dataset.

Thank you again for your review. We hope our response could address your concerns. If you have any further questions, we would be happy to discuss them further.

Thank you so much for your positive and encouraging feedback. Below, we address your comments and suggestions in details:

Weakness 1: Few-shot Learning Experiments

Thank you for the suggestion. Below, we present few-shot experiments conducted on GSM8K, AlfWorld, and LaMP. In these experiments, the controller model is fine-tuned with varying numbers of preference pairs (1, 2, ..., 128). For comparison, we also include the performance of the unfine-tuned controller as well as the controller fine-tuned on the full dataset. The results are summarized as follows:

AlfWorld:

GSM:

LaMP: Accuracy for LaMP-1/2M/2N, MAE for LaMP-3 (lower is better), Rouge-L for LaMP-4

We use Qwen2.5-7B-Instruct as the controller model across all tasks. For the GSM8K and LaMP series, GPT-4o-mini serves as the black-box LLM, while GPT-3.5-turbo-0125 is used for AlfWorld.

From the results, we observe that on unfamiliar datasets such as AlfWorld, increasing the number of preference pairs significantly boosts performance, with a +9.0% gain when scaling from 1 to 128 pairs. In contrast, on tasks the controller is already familiar with, such as GSM8K, the performance improvement under the few-shot setting is marginal (less than 1.0%).

For LaMP-1, training on a single sample is able to obviously improve the performance, while for others the performance gradually improves with increasing data points.

This can be explained by the content generated by the controller. Compared to other LaMP tasks, in LaMP-1, the controller generates a summary, which has a greater impact on the black-box model’s ability to solve the problem. Even with very few samples, the controller can still significantly improve performance through training.

Weakness 2: Controller Capacity on Complex Tasks

Thank you for the insightful suggestion. We agree that the white-box capability might be a potential bottleneck for extremely complicated situations. However, we conjecture that for most of the cases, a medium size LLM is enough for task decomposition.

To this end, we conducted additional experiments on APPS [1], a large-scale and challenging code generation benchmark consisting of 10,000 difficult programming problems.

In this setup, the controller model is tasked with generating high-level observations of the problem, which are then provided as guidance to a black-box LLM to assist in generating the final code solution. Due to time constraints, we were unable to optimize all the intermediate prompts; However, our approach still outperforms strong baselines. Specifically, we compare against the following: (1) direct prompting, (2) a black-box + black-box pipeline using the same prompt as our method, and (3) an unfinetuned controller + black-box setting. For both our method and direct prompting, we also incorporate a reflection strategy to further boost performance. All experiments use Qwen2.5-7b-instruct as the controller model and GPT-4o-mini as the black-box LLM, with the controller trained via DPO on collected preference data.

Here, Strict Accuracy refers to solving a problem by passing all test cases, while Test Coverage denotes the average pass rate across test cases. Despite the lack of optimal intermediate prompt design, our method surpasses the strongest baseline by 0.7% in the single-pass setting, and this margin increases to 1.3% with four reflection attempts.

Compared to simpler domains such as planning, code generation poses significantly greater complexity and requires more careful design of intermediate prompts. We believe that with more refined prompt engineering, our framework could yield even stronger results in this setting.

[1] Hendrycks, Dan, et al. "Measuring coding challenge competence with apps." arXiv preprint arXiv:2105.09938 (2021).

Weakness 3: Missing Intermediate Guidance Examples in Main Content

We appreciate the suggestion and agree that including key examples of intermediate guidance can enhance the clarity of our approach. Accordingly, we will move Figure 5 into the main paper to improve readability and better illustrate the guidance process across different task types.

Question 1: Justification for White-box Controller vs. Engineered Prompting using Black-box LLM

Thank you for the insightful suggestion! Below we first provide a general explanation, followed by a detailed example from AlfWorld to support our response.

Overall Benefit – Effective Problem Decomposition with Feedback:

As described in our abstract, we treat the black-box LLM as an environment and the white-box LLM as a controller. The white-box model decomposes tasks and queries the black-box model. The black-box model then either interacts with the environment or compares its outputs to ground truth, returning feedback. This feedback serves as a supervisory signal to train the controller, allowing it to learn which decompositions are most effective. As a result, the trained white-box controller produces decompositions that better guide the black-box model to solve tasks. This is what black-box model alone, without feedback, cannot achieve.

Example – AlfWorld Task: "Put a clean mug in coffee machine" Given this task and the environment state:

White-box Controller's plan:

General plan: I need to get a list of receptacles to find a mug, take the mug to a sinkbasin, clean it, and put it in a coffee machine.
Step 1: get a list of receptacles where a mug is likely to appear.
Step 2: go to each receptacle in the list until seeing a mug.
Step 3: identify the mug I just found and take it.
Step 4: go to a sinkbasin to clean the mug.
Step 5: go to a coffee machine and put the mug in it.

Black-box Controller's plan:

General plan: I need to find a clean mug and put it in the coffee machine.
Step 1: Get a list of receptacles where a clean mug is likely to be found.
Step 2: Go to each receptacle in the list until finding a clean mug.
Step 3: Identify the clean mug and take it.
Step 4: Go to the coffee machine.
Step 5: Put the clean mug in the coffee machine.

The black-box Controller mistakenly assumes a clean mug is readily available, skipping the cleaning step. In contrast, the white-box Controller correctly decomposes the task to include cleaning, leading to a successful outcome. This highlights the value of decomposition guided by environment feedback.

Question 2: Latency Introduced by Controller Pipeline

Thank you for the insightful question regarding the latency of our framework. We acknowledge that our method introduces additional latency to some degree. Specifically, training the white-box controller on ~2k preference pairs takes approximately 1 hour using two GPUs. For inference, the framework is roughly 2× slower than direct prompting. For example, on GSM8K without any parallelization, our evaluation pipeline takes 159 minutes, compared to 75 minutes for direct GPT-4o-mini prompting.

However, this added cost brings substantial benefits: the training adapts the output of LLMs to specific direction, with much less computation consuming compared to post-training, while the multi-step process provides more refined controlling ability, compared to a single direct prompt. In fact, our framework significantly enhances the performance of the black-box LLM, yielding improvements of 3.19% in reasoning, 7.46% in planning, and 5.82% in personalization. In this sense, the increased time can be viewed as a trade-off for better performance.

Importantly, the white-box controller only needs to be trained once. We also show that it generalizes well and can be transferred to new tasks in a plug-and-play manner, further improving overall efficiency.

Thank you again for your review. We hope our response could address your concerns. If you have any further questions, we would be happy to discuss them further.

Dear Reviewer qqQB,

Thank you for taking the time to review our work and for providing such valuable feedback! We will make sure to incorporate the additional claims and experimental designs in future revisions.

During the rebuttal phase, in the Few-shot Learning Experiments section, we conducted additional experiments across all three tasks—GSM8K, AlfWorld, and LaMP. The results show that even a small number of preference examples can lead to significant improvements in unfamiliar domains like AlfWorld, whereas the impact is less on more familiar tasks such as GSM8K.

In the Controller Capacity on Complex Tasks section, we extended our approach to the code generation domain. Despite the limited rebuttal time, we demonstrated consistent performance gains over strong baselines, further validating the generalizability and effectiveness of our method.

To clarify the advantage of our approach over traditional black-box prompting, we emphasized that our method can leverage supervisory signals from the environment, which direct prompting cannot. We also provided a detailed example on the AlfWorld task to illustrate how this advantage translates into more effective problem decomposition.

In the Latency section, we conducted a quantitative analysis comparing runtime and effectiveness between our approach and black-box baselines. We explicitly highlighted the trade-off between latency and performance, demonstrating that the increase in computation is justified by the substantial improvement in results.

Thank you again for your constructive suggestions!

Please let us know if you have any further questions. If you feel that our response addresses your concerns, we would be grateful if you would consider raising your rating for our paper. We truly appreciate your consideration.

Best Regards,

Authors of Matryoshka Pilot