PLHF: Prompt Learning from Few-shot Human Feedback

Thank you for the thorough review and the constructive feedback toward our paper. For the mentioned concerns, please see the following responses.

1. For the addressed issue of whether an LLM can produce discriminative scores based on next-token prediction, we acknowledge that our paper might not be a universal solution to the core problem of LLMs. However, the main purpose of our work is to provide a framework that performs prompt optimization efficiently in terms of the number of human feedback calls, when there is no explicit well-defined metric for the target task. Overall, though we do not have theoretical proof toward the model effectiveness (at least for now), we still want to share our findings as a new possibility to solve the problem in this paper.

2. The reviewer’s concern in the diversity, quality, and downstream task coverage of the human scoring is absolutely right. The effectiveness and accuracy of the finalized prompt depends on the quality of human feedback. However, as long as the scores (or say, labels) are provided by real humans, almost all prompt optimization methods suffer from the same human labeling quality issue, because the prompting itself is fundamentally based on labeled observed/training samples. Since the proposed method is a framework for prompt optimizations, we would like to address that the mentioned concern is considered out-of-scope in our work.

3. We greatly appreciate your feedback towards the readability of our manuscript. We will attempt to improve the descriptions and organizations for the mentioned parts.

4 & 5. We agree that the Experiments section has a room for improvement. We indeed plan to add more baselines, comparisons and analyses to justify the effectiveness of our prompt optimization framework. Due to the problem setting, it might not so suitable to include the mentioned Vicuna Eval and Self-instruct Eval datasets into the comparisons. For this part, we will make it clearer in the later versions.

Again, we greatly appreciate your review!

Thank you for the review. We would like to address the mentioned issues as follows.

1 & 2. We appreciate your feedback toward the paper readability. We will improve the notations and descriptions.

3. We would like to clarify --- the tackled task in this paper is in fact providing output evaluations (i.e., the scores) acting as from the real humans (since in the mentioned cases, there is no explicit well-defined metric for prompt optimizations). Therefore, whether the humans have biases on the scores is not considered as a concern in our work. After all, the main purpose of our framework is to perform prompt optimization to fit the human preference pattern, based on human feedback, so we assume that the human feedback is correct.

4. Since we adopt GPT-4o as the pseudo-human judge (described in Sec 4.3) to provide evaluations, we tend to adopt a weaker LLM (GPT-3.5 in our case) as the base LLM --- to imitate the relationship of LLMs and real human, where human experts should be more authoritative than LLMs. Nevertheless, we agree that we should consider other LLMs as the base LLM in our experiments.

We greatly appreciate your constructive review. Each mentioned point from the reviewer definitely makes us realize that our manuscript could be refined with more enhancements. For our answers to the questions, we summarize as follows.

• We will further expand the experiments to include more analyses toward model effectiveness and robustness.
• In a new version of our paper, the details of dataset pre-processing will be provided in the appendix section.
• More insights and examples are also planned to be presented in the appendix section.
• For the term “domain 1” mentioned on Line 320, it is originally from the dataset description of the AES-ASAP dataset (Ben et al. 2012). Since some of the essays have scores in two different domains, we specify that we consider the human-labeled score in domain 1. We will make the related descriptions more self-contained to avoid unnecessary confusion in the future version of our manuscript.

We are sincerely grateful for the reviews toward our work. For the concerned issues:

1. As we mentioned throughout the entire paper, the proposed framework is especially designed for few-shot prompt optimization. While we acknowledge the idea of comparing our model with a wider range of prompt optimization methods, the original setting of the task is that only few shots of samples are available for prompt optimization. We will improve our writing to mitigate the ambiguity on the scope.
2. Prompt Optimization with Human Feedback is indeed a related work providing another strategy to leverage human feedback to perform prompt optimization for LLMs. However, since the source paper is not officially published yet when we made this submission (the manuscript of “Prompt Optimization with Human Feedback” is also submitted to ICLR 2025 this time), based on the convention, we believe that it is reasonable to not include such work in our submission.
3. To our best knowledge, with the constraint of few-shot samples and no explicitly available metric to evaluate the LLMs’ outputs, there is no existing iterative method to perform prompt optimizations. For the theoretical analysis, we will consider it as future work. Similar to other work developing LLM usages, empirical results are often prior than the theoretical proof for a new concept. The main purpose of our paper is to demonstrate a novel possibility to tackle the mentioned issues.
4. Indeed, agreed with the reviewer, we are considering to include other LLMs are the base LLM in our experiments to enhance the coverage of our experiments.

Thank you for your recognition of our work and for your insightful reviews. For the mentioned weaknesses, our responses are as follows.

1. We agree with the point that we should include various LLMs to act as the base LLM in our experiments. However, since we consider GPT-4o as the pseudo-human judge to make overall evaluations, we tend to select a weaker LLM (GPT-3.5 in our case) as the base LLM (to mimic the relationship of LLMs and real human, in our target scenarios, human experts should be more accurate than LLMs). But yes, we will add more alternatives to replace the GPT-3.5 and GPT-4o, respectively, in our future experiments.
2. We appreciate the suggestions of making our novelty clearer. For the mentioned question, we would like to explain that we only require human annotations for the score of each training sample. The idea is that scoring an input-output pair (for the evaluator E) is relatively easier than providing suggested output for a given input (for the responder R). But we agree with the reviewer, providing the human annotations as additional context for both E and R is also a possible way to perform prompt optimizations with human feedback.
3. Yes, we sincerely accept the idea of including detailed the initial prompts and the corresponding optimized prompts for each task in our experiments. We have started to prepare it and we expect such results will be available as the appendix of our paper in the upcoming camera-ready version.
4. We also greatly appreciate the suggestion of adding the plots showing the performance changes with various number of training samples in the analysis part for the baseline models. Extra plots will enhance the clarity of our existing comparisons between the proposed framework and the baseline combinations.