Thank you for your detailed suggestions and comments. Please find the corresponding responses as follows:

[Weakness 1]

Thank you for the suggestion. First, it is crucial to underscore the fundamental difference between our approach and conventional black-box prompt optimization. Rather than optimizing prompts specifically for individual tasks, our focus is on developing a generalized model that can generate prompts applicable to a wide range of tasks. In traditional black-box prompt optimization, fine-tuning an instance often results in limited transferability to other scenarios. In contrast, our objective is to deliver a more versatile and adaptable personal assistant capable of supporting users across various contexts and applications.

In [1], BPO also employs a RL method to train a prompt optimization model, aligning the outputs of black-box LLMs more closely with human preferences across multiple instruction datasets by optimizing the input prompts.

The key differences are as follows:

1. BPO's training data is derived by filtering open-sourced feedback data, whereas our approach uses outputs directly generated by a white-box LLM, offering broader applicability.
2. BPO trains using single-turn interactions, while our method incorporates multi-turn interactions, allowing for a more comprehensive optimization process.

Thanks again for your insightful suggestion, and we will add this meaningful and highly relevant discussion and citation into our paper.

Reference:

[1] Jiale Cheng et al., Black-Box Prompt Optimization: Aligning Large Language Models without Model Training, ACL 2024

We would like to thank the Reviewer for the insightful feedback. Your suggestions for clarification and improvement are constructive. Please find the responses below:

[Weakness]

Thank you for the suggestion. From a motivational perspective, our primary goal is to enable model personalization. While existing black-box LLMs excel across various tasks, they lack the ability to be trained for personalized usage. Through our proposed controller-generator framework, we can train a personalized white-box LLM to guide the black-box LLM, enabling personalized interactions and usage.

Additionally, we selected the LLaMA3-8B-Instruct model as the controller because it is significantly more lightweight compared to black-box LLMs like GPT. Our research shows that even a lightweight model can achieve state-of-the-art performance across diverse tasks, including reasoning, planning, and personalization, effectively balancing high performance with practical convenience.

[Questions]

It is also interesting to compare the results of directly using two black-box models with those achieved by our controller-generator framework. In the following sections, we will first present the superior performance of our approach compared to the two black-box models across various tasks, followed by an overall analysis of the results.

GSM

Alfworld:

LaMP:

Note: For tasks in ALFWorld, we adhered to the setup used in Adaplanner [1]. Since the prompt instructions are highly dependent on the type of black-box LLM, certain tasks cannot be solved when using GPT-4o-mini as the black-box LLM. Therefore, we standardized GPT-3.5-turbo as the black-box LLM for these experiments.

As shown, compared to different two-black-box configurations (e.g., GPT-3.5-turbo-0125 and GPT-4o-mini), our framework consistently achieves either comparable or significantly better results in most cases. We attribute this to the following key reason:

Effective Problem Decomposition with Feedback: As highlighted in our abstract, we treat the black-box LLM within our framework as an "environment" and the white-box LLM as a "controller." The white-box LLM decomposes the problem and inputs it to the black-box LLM. The black-box LLM interacts with the environment or compares its output against ground truth, returning the feedback as a supervisory signal. This feedback helps filter high-quality problem decompositions to train the white-box LLM. Consequently, the trained white-box LLM generates problem decompositions that more effectively guide the black-box LLM to solve tasks. In contrast, a black-box LLM alone lacks environment feedback and cannot perform equally effective problem decomposition.

Reference:

[1] Sun, Haotian, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. "Adaplanner: Adaptive planning from feedback with language models." Advances in Neural Information Processing Systems 36 (2024).

Thank you for your detailed suggestions and comments. Initially, we would like to clarify the focus of this paper on driving black-box LLMs to obtain desired capabilities following a series of existing works [1][2][3]. Currently, SOTA black-box LLMs still show better performances than open-source LLMs on different tasks, including all three tasks we use in the paper [4][5][6], which motivates us to find a valid method to drive the generation of black-box LLMs to achieve better performance on specific tasks. By leveraging a lightweight white-box LLM to control the black-box LLM, we can demonstrate the black-box LLM's broad applicability and exceptional performance across various tasks. Additionally, by training the white-box LLM to drive black-box LLM, we enable personalized utilization of the black-box LLM to better suit specific needs. Experiments also show that with the same budgets in model parameters and training data, our white-box LLM + black-box LLM framework outperforms simply fine-tuning the white-box LLM. Please find the detailed corresponding responses below:

[1] Huang, Yangsibo, Daogao Liu, Zexuan Zhong, Weijia Shi, and Yin Tat Lee. "$k$ NN-Adapter: Efficient Domain Adaptation for Black-Box Language Models." arXiv preprint arXiv:2302.10879 (2023).

[2] Sun, Haotian, Yuchen Zhuang, Wei Wei, Chao Zhang, and Bo Dai. "BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models." In Forty-first International Conference on Machine Learning.

[3] Zhuang, Yuchen, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. "Hydra: Model factorization framework for black-box llm personalization." arXiv preprint arXiv:2406.02888 (2024).

[4] https://paperswithcode.com/sota/arithmetic-reasoning-on-gsm8k

[5] https://paperswithcode.com/dataset/alfworld

[6] Salemi, Alireza, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. "Lamp: When large language models meet personalization." arXiv preprint arXiv:2304.11406 (2023).

[Weakness 1]

Thank you for your valuable feedback. We would like to emphasize that the primary contribution of our work lies in treating black-box LLMs as environments and training a lightweight white-box LLM controller to guide the outputs of black-box LLMs in alignment with specific preferences. This enables controllable multi-turn generation and enhances the capability of black-box LLMs in handling complex, long-horizon tasks. Currently, state-of-the-art black-box LLMs outperform open-source LLMs across various tasks, such as Code Generation on HumanEval [1]. This motivates us to develop an effective approach to better control the generation of black-box LLMs for specific domains or tasks.

From a generalization perspective, existing works, such as Adaplanner[2], Hydra[3], focus on limited use cases. In contrast, our method has proven effective across a broader range of tasks, including reasoning, planning, and personalization—covering a scope that no prior work has comprehensively addressed. Moreover, compared to previous approaches, our proposed method requires a lighter-weight white-box controller and a smaller amount of data.

Additionally, our approach is highly flexible, allowing it to be applied to different black-box or open-source LLMs in a plug-and-play manner. Finally, within our framework, the intermediate responses generated by the white-box LLMs can take various forms beyond CoT generation and planning generation. We appreciate the reviewer's insightful questions and will explore additional potential representations in future research.

Reference:

[1] https://paperswithcode.com/sota/code-generation-on-humaneval

[2] Sun, Haotian, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. "Adaplanner: Adaptive planning from feedback with language models." Advances in Neural Information Processing Systems 36 (2024).

[3] Zhuang, Yuchen, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. "Hydra: Model factorization framework for black-box llm personalization." arXiv preprint arXiv:2406.02888 (2024).

Hi Reviewer 22gq,

We have carefully considered your valuable feedback and have submitted a detailed response addressing the points raised in your reviews. We believe our response clarifies several aspects of the paper, highlights its contributions, and our additional work (attached above and in the updated paper) addresses your concerns. It would be great if you could take some time to review our responses and let us know your feedback.

Thanks in advance,

Authors of paper Martryoshka: Learning to Drive Black-Box LLMs with LLMs

We sincerely appreciate Reviewer tH4H for the insightful feedback. Your suggestions and insights are very helpful for further enhancing the submission quality. Please find the responses below:

[Major concerns 1]

Thank you for the suggestion. In practice, we only utilized the GPT-4o model in the DPO training set for the LaMP dataset. This was because the LaMP dataset lacks ground truth annotations, and in cases where the original black-box model could not generate sufficient positive data, we leveraged the GPT-4o model to produce additional positive samples needed for DPO training. Therefore, the use of the GPT-4o model was simply a design choice for data annotation.

If alternative options were available, such as high-quality human-annotated data, ground truth chain-of-thoughts from GSM8K, or interaction-based feedback from Alfworld to determine the quality of data, we could entirely bypass the use of the GPT-4o model.

In fact, even without the GPT-4o model, we can still achieve sufficiently or better positive results on the LaMP dataset. As shown in Table A, although slightly lower than the metrics reported using GPT-4o for LaMP-1 and LaMP-2, the results get even better on LaMP-3 and LaMP-4 and remain significantly better than other baselines on all LaMP tasks. As for the fraction, we observe that less than 5% data is from gpt-4o, further claiming that this setting is not needed. This demonstrates that our approach works well even without the assistance of a powerful model. In this case, where the powerful model is removed, it further highlights that the improvement in experimental performance stems from self-improvement rather than being directly distilled from a powerful model like GPT-4o.

Table A: Experimental results on the LaMP benchmark without GPT-4o-based labeling

[Major concerns 2]

We apologize for the confusion. On page 26 of the updated manuscript, we have included a new case study for the GSM_Hard task. In this case, Matryoshka showcases its ability to reflect on issues in the previous decomposition and provide a more precise re-decomposition, ultimately guiding the black-box LLM to successfully solve the problem.

This new case highlights two key points:

1. Using a white-box LLM to drive a black-box LLM is a highly promising direction, significantly outperforming CoT and PaL methods that rely solely on the intrinsic capabilities of the black-box LLM.
2. The white-box LLM's ability to learn within this framework is equally promising. Through learning, Matryoshka acquires reflective capabilities and the ability to revise previous decompositions—skills that the vanilla LLaMA3-8B-Instruct lacks—further enhancing its ability to guide the black-box LLM in problem-solving.

Besides, we have also updated the white-box LLM multi-turn interaction prompts for both the AlfWorld and GSM tasks in Appendix G.

[Major concerns 3]

The data generation cost was calculated by aggregating the total token consumption statistics provided by Azure API and subsequently applying the cost per token (gpt-3.5-turbo-0125, gpt-4o-mini) as specified in the OpenAI official documentation. The cost for processing 1M tokens, as detailed in Table B, served as the basis for this calculation.

Table B: Detailed API cost per 1 million tokens [1]

For the AlfWorld dataset, the entire training set consists of 8,808 samples. On average, using GPT-3.5-turbo to sample 100 examples costs approximately $3.20**, making the estimated cost for complete data collection **$282.

For the GSM8K dataset, the full training set comprises 7,473 samples. The average cost for sampling 100 examples using GPT-3.5-turbo is $1.215**, resulting in an estimated total cost for data collection of **$90.80. In comparison, fine-tuning the gpt-3.5-turbo with the Azure OpenAI service costs $216.50, and requires hourly payment in deployment for inference.

For LaMP-1, LaMP-2M, LaMP-2N, LaMP-3, LaMP-4, we use gpt-4o-mini for data generation. The total costs are separately $6.144, 1.882, 2.348, 8.111, 10.022, with 5252, 2719, 2369, 8506, 12518 generated data samples.

During the training phase, we used four H100 GPUs for two rounds of DPO training. The process took approximately 1.5 hours for AlfWorld and GSM8K, resulting in a total training cost of 6 GPU hours. It took separately 4 gpu h, 4 gpu h, 4 gpu h, 8 gpu h, 12 gpu h for the training process of LaMP-1, -2M, -2N, -3, -4.

Reference:

[1] https://openai.com/api/pricing/

[Major concerns 4]

We are grateful for the reviewer's insightful suggestion. We envision our model as a universal white-box assistant capable of being seamlessly integrated into new tasks in a plug-and-play manner while effectively guiding large models. We are currently exploring strategies to achieve this goal and will continue to pursue lightweight, versatile control capabilities for smaller models in future research.

[Minor concerns]

We first thank the reviewer for the great insights. The citation of relevant papers on collecting on-policy data is indeed important, and we will add the relevant citations to our revised version.

Specifically, [1] mentions that online DPO performs better than offline DPO. The insight/observation is compatible with our work, which utilizes online DPO for multiple iterations for continual enhancement of black-box LLM capabilities. Also worth mentioning, our method can accommodate different design choices for DPO, SimPO and so on, while our key focus is the proposed “LLM drives LLM” controller.

[2] and [3] both stress the importance of on-policy/online data, from practical and theoretical perspectives. We will add discussions on these relevant work in our revised version.

[5] and [6] offer interesting insights that there are various ways of training and controlling with regard to what to train and potential applications.

Thanks again for your insightful suggestions, and we will add these meaningful and highly relevant discussions and citations into our paper.

Reference:

[1] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, https://arxiv.org/abs/2404.10719

[2] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, https://arxiv.org/abs/2404.14367

[3] The Importance of Online Data: Understanding Preference Fine-tuning via Coverage, https://arxiv.org/abs/2406.01462

[4] Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents, https://arxiv.org/abs/2408.07199

[5] Red Teaming Game: A Game-Theoretic Framework for Red Teaming Language Models, https://arxiv.org/html/2310.00322v3

[6] Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation, https://arxiv.org/abs/2403.19103

Dear Reviewer tH4H,

Thank you very much for taking the time to review our rebuttal and offering insightful feedback. Also, thank you for raising the score!

Please don't hesitate if you have any further questions.

Best regards,

Authors