Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL

We deeply appreciate your time and effort devoted to reviewing our paper. And we are happy to have your acknowledgment of the significance and soundness of our work. Your insightful comments on the relationship between Prompt-OIRL and the online methods remind us to better present and contrast those approaches in our revision, and we have added a corresponding discussion section to our revised manuscript in Appendix D (New).**

In the following, we would start by noting that the online methods are the foundations for our query-dependent offline approach, and additional prompts discovered by online algorithms can improve the performance of Prompt-OIRL.

1. Additional prompt generated by the online method improves Prompt-OIRL’s performance upper bound.

As the number of available prompts increases, the best-of-n performance among those prompts will be monotonically better. In addition to the evidence we have shown in the paper (Figure 6), the same conclusion can be drawn by comparing the performance upper bound of Prompt-OIRL on different tasks.

In the table below, we use UB-Train-6 to denote the best performance (performance Upper-Bound) that can be achieved using the training prompts on the test queries, and use UB-Test-10 to denote the best performance that can be achieved using the 10 test prompts. Finally, we use UB-16 to denote the best performance that can be achieved using all those 16 prompts.

From the results, we can observe that the best performance can be achieved by the test prompts are usually worse than the best of the training prompts — this is not surprising given the fact that those training prompts are generated and optimized by experts or online algorithms. That said, we can also observe that selecting from the combination of all those 16 prompts can always lead to a significant improvement. Such an observation verifies the idea that adding additional prompts that are generated by the online prompts, either to the training set or to the test set, can be beneficial to Prompt-OIRL.

2. Monotonic Performance Improvement with the Number of Prompts.

To add additional empirical evidence to our discussion, we can observe from Figure 6 in the main text that all methods, including all baseline methods and our proposed method, can have improved performance when more available prompts are available.

The table below contextualizes the averaged performance improvement over different LLMs and tasks. We use “Improv. w/ +train” to highlight the improvement achieved using 6 training prompts as compared to only using 1 prompt. We use “Improv. w/ +test” to highlight the improvement achieved using additional test prompts as compared to only selecting from the training prompts (i.e., the improvement of Prompt-OIRL over BoTr Eqn(2)).

The monotonic improvement in all those comparisons demonstrates the benefit of integrating additional prompts, which can either be discovered through an existing online algorithm or crafted by human experts.

Once again, we thank the reviewer for their effort in improving our work. If there should be any remaining concerns or questions, we are keen to do our utmost to address them.

We deeply appreciate the insights you've shared during the review process. Following our revisions and previous responses, we are genuinely curious if we have adequately addressed the concerns you raised.

We would appreciate it if you could kindly let us know if there were any further questions. In the extremely limited time remaining, we are still eager to do our utmost to address them!

We extend our gratitude to the reviewer for their insightful feedback. As we approach the conclusion of the discussion period and have yet to receive further comments, we would like to summarize the initial feedback from the reviewer and our subsequent responses.

In their initial review, the reviewer acknowledged the soundness of our contribution by conducting a thorough empirical study and detailed ablations in an offline setting. They recognized the effectiveness of our approach in utilizing query-level prompts to reduce costs and improve the arithmetic reasoning capabilities of LLMs. They raised a concern regarding the connection between Prompt-OIRL and the online prompt-generation methods.

To address the reviewer’s concern about the link with the online methods,

1. Analytically, We emphasized that Prompt-OIRL is not in direct competition with online methods but rather builds upon their generated knowledge. This approach is akin to standing on the shoulders of giants; it improves upon the existing prompts by considering query dependency in test time, whether they are generated by online algorithms or domain experts. This is evident in Figure 6, where an increase in the number of prompts correlates with consistent improvements in performance. This demonstrates Prompt-OIRL's capacity to draw valuable insights from online methods and exhibit monotonically better results.

2. Synthetically, in our previous response, we posited that assuming an oracle reward model, the incorporation of more prompts at test time consistently enhances the performance upper bound. This holds true even if the additional prompts sourced from online methods are suboptimal. Thus, our study highlights the importance of diverse prompting strategies. This concept echoes our initial motivating example where different queries benefit from varying prompts, a finding supported in the concurrent literature [OPRO].

3. Empirically, we underpinned our reasoning with experimental evidence, demonstrating that:

• (1). Increasing the number of prompts sourced from online algorithms during training consistently enhances the performance of Prompt-OIRL and other baseline methods. This improvement underscores the value of enriched demonstration data derived from online algorithms.
• (2). Increasing the number of prompts sourced from online algorithms during testing leads to consistent improvements in query-dependent prompt optimization. This is clearly observed when comparing the outcomes of Prompt-OIRL — which selects the most effective prompt from both training and testing sets using its learned reward model — against BoTr Eqn(2), which limits its selection to training prompts. The performance gains observed in Prompt-OIRL affirm the benefits of integrating additional prompts from online sources.

We trust that the aforementioned clarifications sufficiently address the reviewer's concern. We are grateful for the opportunity to elaborate on these aspects and to underscore the importance of our findings in contributing to the field.

We would greatly appreciate any additional feedback from the reviewer on ways to further enhance the quality of our manuscript. Despite the extremely limited time remaining, we remain committed to doing our utmost to address any additional concerns or suggestions the reviewer may have.

Reference

[OPRO] Large Language Models as Optimizers (OPRO) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen arXiv 2023.

We thank the reviewer for their insightful comments, and their acknowledgement of the soundness, presentation, and contribution of our paper. We aim to address all the individual points in your review here, but please also see the revised manuscript for changes (highlighted in red).

Q1. While the method looks promising, I still expect to see potential discussion about the limitations, e.g., stability of inverse RL？

A1.

Thank you for raising the concern about stability and robustness. In general, our offline-IRL approach enjoys high stability since the reward modeling step is based on gradient boosting methods and optimized through supervised learning. Similar to other offline-RL literature, learning with a fixed dataset also avoids the dilemma of exploration-exploitation trade-off which is usually challenging in the online approach. (e.g., in OPRO, APE, and APO [1-3]). In our work, we applied xgboost with a universal hyper-parameter setting to demonstrate the robustness of our approach.

A temporary challenge for Prompt-OIRL’s wide application lies in the availability of open-accessible data. We will elaborate on this point with more details in A3 below.

Q2. The scope is limited by arithmetic reasoning but the title seems a more generic framework that can be used to solve more broader tasks across different NLP tasks.

A2.

Thank you for the suggestion. To avoid potential misunderstanding, we have updated our title with a subtitle in our revision to better highlight the use case studied in our work. The updated title is

Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL — A Case Study on Arithmetic Reasoning

We deeply appreciate the insights you've shared during the review process. Following our revisions and previous responses, we are genuinely curious if we have adequately addressed the concerns you raised.

We would appreciate it if you could kindly let us know if there are any further questions. In the extremely limited time remaining, we are still eager to do our utmost to address them!

Thank you for the insightful comments and your acknowledgment of the novelty and contribution of our work.

In our initial manuscript, we focused more on illustrating the key insight of leveraging offline IRL in prompt evaluation and optimization, and deferred the implementation details into the appendix due to the space limit of the main text. As pointed out by the reviewer, we agree expanding those details can further enhance the understanding of our work. As a consequence, we have updated those sections with expanded details. We have also updated the paragraph on training-test prompt generation in our updated version to enhance clarity.

In the following, we aim to address all the individual points in your review. Please also see the revised manuscript for changes (highlighted in red). As because of your insightful feedback, the clarity of our work has been significantly improved.

I want to thank the authors for going to great lengths to provide a detailed rebuttal, addressing all my questions and concerns. I believe especially the results on generalization across datasets and strong performance even on small datasets to be promising signs for the practical applicability of this approach.

I have thus raised my score.

I would still encourage the authors to address the following points:

• Highlight the fact that they use an LLM generated embedding as input to their XGBT in Section 3 Step 2.
• Include a direct comparison of oracle performance with their approach and BoTr Eqn. 1 to better contextualize the performance improvement realized by their method.
• Update Figure 7 with relative scales.

Q.3 A Nearest-Neighbor Baseline

A.3

We thank the reviewer for pointing out the nearest-neighbor method as an additional practical baseline. We would begin with a discussion on the differences between such a potential baseline and Prompt-OIRL, followed by experiment results.

1. The Nearest-Neighbor method can not be generalized to new prompts. The idea of using a nearest-neighbor approach can be an alternative to using a parameterized model in selecting from the training prompts. However, it can not be generalized to unseen prompts due to a lack of support.

2. As a non-parametric method, the Nearest-Neighbor approach requires memorization of the training embeddings, which can be expensive when the number of demonstrations increases.

Empirically, we implemented this idea with GPT-3.5-turbo on the MAWPS dataset. To be specific, for every test query, we look up the training queries’ embedding to find the closest neighbor of this test query’s embedding, and select from the training prompt(s) that has successfully prompted correct answers on such a training query.

The results are shown in the table below:

The nearest-neighbor approach, as a query-denpendent baseline, outperforms the BoTr Eqn(1) baseline, and achieves on-par performance as BoTr Eqn(2). However, it underperforms Prompt-OIRL as the generalization ability of the learned reward model in Prompt-OIRL further enhanced its prompting performance.

Actions taken: We analyzed the potential and challenge of applying the nearest neighbor method as an alternative approach to parametric models used in Prompt-OIRL, and provided additional experiment results to verify the idea.

Q.4 What are the values of K, M and P in appendix section C.1?

A.4

In Appendix C.1, N is the number of samples (query-answer pairs) in the training dataset, M is the number of samples in the held-out test dataset, K is the number of training prompts, and P is the number of test prompts.

For all datasets, we experiment with different choices of K = [1, 2, 3, 4, 5, 6]. (i.e., the x-axis of Figure 6: number of training prompts, ranging from 1 - 6). And use a total number of P=110 held-out prompts.

For GSM8K, there are N=7473 samples used for training, and M=1319 samples for testing. For SVAMP, there are N=15000 samples used for training, and M=4690 samples for testing. For MAWPS, there are N=6000 samples used for training, and M=1685 samples for testing.

Actions taken: We have updated our description in Appendix C.1 to enhance the clarity. Our revision is highlighted in red.

Once again, we thank the reviewer for their effort in improving our work. If there should be any remaining concerns or questions, we are keen to do our utmost to address them.

Q.2 Details of the offline dataset and the reward model.

A.2

We thank the reviewer for pointing out the importance of reward modeling details. Due to space constraints in the main text, we deferred the detailed information about the offline dataset and the discussion on the reward model to the appendix. To make our main text better self-contained, we have revised our manuscript and provided the essential empirical implementation of the reward model in the main text.

In addition, we agree with the reviewer that providing a more detailed empirical comparison and analysis will be invaluable not only for enhancing the understanding of our method but may also contribute to the general community in understanding the learning with embeddings. Therefore, we have also updated our appendix and provided more details on the reward modeling to including the following empirical evidence:

1. MLPs easily converge to the trivial classifiers

For the MLP model, we have tried different choices of its hyper-parameters, including unit numbers, layers, various drop-out choices, and dual-channel architecture with each channel process query and prompt embeddings individually. However, we find all of those choices tend to converge to the trivial solution that predicts either all 0 or all 1 for the binary classification task. Such a reward model does not have the prediction ability in inference time. In practice, with such a reward mode, we can do nothing better than select the best-performing prompt on the training dataset. Therefore, the performance of BoTr Eqn.(1) represents the best-achievable performance of using a dummy MLP reward model.

2. XGBoost is robust on hyper-parameter for reward modeling

For the xgboost method, we set a universal hyper-parameter for all tasks and LLMs. When deploying Prompt-OIRL in practice, a case-by-case engineering on the hyper-parameters on reward modeling should be able to further boost the performance of the algorithm, yet this is out of our research scope. In our paper, we experiment with a single universal hyper-parameter setting for all LLMs and tasks, demonstrating the robustness of the proposed method.

3. Instance-wise prediction of outcome is better than pair-wise prediction (preference-based learning)

Another alternative for reward modeling is based on preference-based learning, which is effective in some conventional inverse RL tasks [1. T-REX]. To be specific, for every training query $x^{(i)}$, there may exist multiple prompts (denoted as $p_+^{(i)}$) that lead to a correct answer, and some other prompts ($p_-^{(i)}$) that lead to a wrong answer.

We can also organize the offline prompt demonstration in the form of preference-based dataset $(x^{(i)}$ , $p_-^{(i)}$, $p_+^{(i)} )$, and learn from those pair-wise preference data. In this pair-wise preference approach, the learned reward model takes both prompts and the query as input, and outputs the preferred prompt. Given a new query in the inference time, such a reward model can be applied to all K candidate prompts with K-1 comparisons to find the best prompt. (so it is more computationally expensive than the direct reward modeling method used in our work). We empirically studied whether such an approach can lead to better performance on the MAWPS dataset with the GPT-3.5-turbo model. The results are shown in the table below:

It can be concluded that the pair-wise reward model can not achieve better performance than the direct reward modeling used in Prompt-OIRL.

All our offline datasets and code for processing those datasets, as long as the MLP implementation and pair-wise reward modeling implementation will be made publicly available and contributed as an asset for future research.

Actions taken: We have updated our manuscript accordingly to include extended details for the reward model training. Revisions have been highlighted in red in our updated manuscript.

Reference

[1. T-REX] Brown, Daniel, et al. "Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations." International conference on machine learning. PMLR, 2019.

We thank the reviewer for the insightful comments and for their acknowledgment of the contribution, novelty, and significance of our work. We aim to address all the individual points in your review here, but please also see the revised manuscript for changes (highlighted in red).

Q.1 How does the proposed method perform when the offline prompt-alignment dataset is small?

A.1

Thank you for highlighting this concern. We agree that the applicability of our idea in settings when demonstration data is scarce (i.e., when the knowledge of effective prompting strategy is limited) is important.

In our experiments, we consider the empirical setting when having access to a different number of expert prompting demonstrations. Specifically, the scarce demonstration setting presented in Figure 5 (left panel) experiments with only 1 (out of the 6) training prompt.

In such a setting, as there is only a single expert prompt available during training time, the best-of-training time prompt (BoTr) strategy must choose to use this training prompt, and so does the LLM-Confidence. Differently, with a reward model training using the demonstration data collected from a single expert-crafted prompt, Prompt-OIRL can generalize its evaluation ability on held-out prompts and achieve significantly improved performance.

In Figure 6, we change the availability of training prompts by the x-axis. We can observe that the performance improvement over baseline methods is most significant when only a single training prompt is used. Not surprisingly, when the number of human-crafted prompts increases, the best-of-training performance is greatly improved.

Therefore, in practice, Prompt-OIRL shines even with a limited number of demonstration prompts, proving its great potential to be applied to settings when expert prompting knowledge is relatively scarce.

Actions taken: We’ve updated our manuscript and improved the clarity in conveying such a point. We appreciate the reviewer for commenting on this point, as because of it, the clarity of the significance and general applicability of our work has been enhanced. Revisions have been highlighted in red in our updated manuscript.

We deeply appreciate the insights you've shared during the review process. Following our revisions and previous responses, we are genuinely curious if we have adequately addressed the concerns you raised.

We would appreciate it if you could kindly let us know if there are any further questions. In the extremely limited time remaining, we are still eager to do our utmost to address them!

We are deeply appreciative of all the reviewers for their insightful feedback and constructive comments, which have been pivotal in enhancing our work. By diligently addressing the concerns raised by the reviewers, we are confident that the quality of our paper has been substantially improved.

We are particularly thankful for the additional feedback provided by our reviewers during the author-reviewer discussion period. We are encouraged to see all of our reviewers acknowledge our response in addressing their initial concerns and have reached a consensus on the acceptance of our work.

During the discussion period, some of our reviewers provided additional suggestions, which have been invaluable in further refining our manuscript. In addition to the revisions listed above, we have also implemented the following updates

1. We have elaborated on the usage of embeddings in Section 3 to enhance clarity and understanding, following the suggestion of reviewer mPPy. (page 5)
2. We have integrated Figure 6 with the oracle baseline, to more effectively illustrate the performance gains and the prospective impact of Prompt-OIRL, as suggested by reviewer mPPy. (page 7)
3. The original Figure 7 has been replaced with a table, offering a clearer representation of Prompt-OIRL's cost-efficiency across various settings, as per the suggestion of reviewer mPPy. (page 9)
4. We have incorporated a discussion on the Nearest-Neighbor approach as an additional alternative for reward modeling. This inclusion aims to ensure the completeness of our study and to facilitate ease in future analyses, as recommended by reviewer wFdE. (page 23-24)
5. Our discussion on the interplay between Prompt-OIRL and online approaches has been expanded in Appendix D, inspired by the insights from reviewer xV9W. (page 29-30)

All of those revisions are highlighted in blue in our latest manuscript.

Finally, we would like to highlight the contributions of Prompt-OIRL.

Our work presents four key technical contributions:

1. Formally, we identify the overlooked query-dependent prompt optimization objective and its challenges, and introduce Offline Inverse Reinforcement Learning to integrate rich human expertise as a systematic approach.
2. Methodologically, we introduce Prompt-OIRL to first perform a query-dependent offline prompt evaluation with a learned reward model and then perform offline prompt optimization to improve prompting performance.
3. Practically, we highlight the existence of offline datasets generated --- as by-products when existing prompting strategies are benchmarked on open-access tasks --- can be directly adopted for Prompt-OIRL learning.
4. Empirically, we validate the efficacy and efficiency of Prompt-OIRL in offline prompt evaluation and optimization through experiments with 3 distinct LLMs, namely GPT-3.5-turbo, LLaMA-2-7B-Chat, and TigerBot-13B-Chat, across 3 arithmetic datasets: GSM8K, MAWPS, and SVAMP.

Furthermore, our work also contributes to the community by highlighting the (overlooked) importance of offline LLM-interaction datasets in the era of LLM research.

• The significant performance improvement of Prompt-OIRL over the baselines demonstrates the efficacy and efficiency of learning with those offline demonstration datasets. Therefore, we advocate for the broader LLM research community not only to release their code but also their interactive logs with LLMs. This practice will enhance reproducibility in research and foster further applications, as demonstrated in our work.

In closing, we wish to express our deepest gratitude to the reviewers for their thorough and thoughtful engagement with our work. We remain committed to advancing the quality of our paper, and welcome any additional suggestions or feedback.

Thank you once again for your invaluable contributions to our work.