Thank you for your review. Here we address both your concerns in the weakness (W1-W3), and the ones in questions (Q1, Q2), which we found is highly correlated to the weakness.

W1. Q2. We combine these two points together as they both refer to how the Act framework, i.e., the setting of LLM Agents, relates to our paper. There are two main differences caused by such settings. First of all, LLM agents are normally used in complex tasks where the final checker might not be available or might be very pricey. This is why much previous work in APE cannot be used in these tasks—because of the cost. The second difference is that even if there is a final checker in the Act framework, it could be much less informative than intermediate feedback, and thus the need for intermediate feedback is very important, which is why our method focuses on summarizing the intermediate feedback as guidance. There is also a minor point that is not very general to the whole domain but affected our design choice: some tasks in this domain require a strict format to interact in the Act framework. If we allow the prompt optimizer to change randomly, it will waste a lot of time changing to incorrect formats. To resolve this, our method focuses on changing one step in the CoT steps.

W2. Q1. These two points concern the clarity of the methods section. We have revised the method description in the updated version of the paper. Due to the page limit, we were unable to include all the prompts, which are highly relevant to the details in the methodology section, in the main paper. However, we have updated the paper to make this section easier to read and more self-contained. We invite you to review the updated version to verify these changes. Here, we provide direct answers to the specific points you raised:

• We found the use of the term “summarization team” to be confusing, as the summarizer is still an LLM. We have changed it to “summarizer.”
• The “focus point” is indeed a crucial detail in the optimizer prompt. We have included the prompt in the appendix and have highlighted the relevant part in the updated version of the paper.
• Regarding the feedback provider, we deliberately kept the description high-level, as our approach does not impose any specific constraints on this component. However, we acknowledge that this led to an undesirably vague description, rather than helping readers understand the generalizability of our method. We have revised the wording in the section you mentioned and clarified this point in a separate paragraph in the methodology section later in the paper.
• Regarding the algorithm box, we have included pseudocode in our appendix. While our method uses LLMs and the prompts themselves cannot be included in the code due to their length, we do not see a significant difference between the pseudocode and the workflow figure already included in the paper. As this is a paper on large language models, we believe it is unnecessary to introduce notations and formulations that do not improve readability. Therefore, in this updated version, we still have not included any formulations.

W3. Thank you for pointing this out. We found this sentence very misleading as our original point to make is to highlight the summarization step rather than the batch. We have updated our paper accordingly to avoid this misunderstanding.

Thank you for your review. Here we respond to the weaknesses you listed one by one:

W1. We agree that our method, REPROMPT, incorporates knowledge from the self-reflection phase into the prompt engineering phase. This approach helps to avoid early-stage errors and, more importantly, reduces the randomness observed in methods like ReAct and Reflexion, where specific scenarios may not receive adequate feedback. By summarizing common issues, such as budget balancing, REPROMPT ensures these are addressed in the prompt, allowing ReAct and Reflexion to focus on case-specific issues, like unexpected price spikes for hotels in certain cities on particular days. Since this was a common question among reviewers, we have included this discussion in our revised paper, which we invite you to review for further clarification. Specifically, our approach operates at a level above Reflexion by summarizing feedback generated by Reflexion and optimizing the prompt accordingly. This explains why REPROMPT outperforms in cases like TravelPlanner, where the feedback generator is Reflexion—because Reflexion summarizes ReAct, and REPROMPT, in turn, summarizes Reflexion. Moreover, we highlight that our paper is the first to address prompt engineering in settings where the final checker is unavailable during optimization. This, as noted in your Q1, is a novel contribution. We consider our main contribution to be the study of this unique setting, with the adaptation of Reflexion to APE being a natural development within this context.

W2. Thank you for your suggestion. We have changed correspondingly to help the readers understand the analogy, and the detailed analogy in Sec. 3.2. Again, we invite you to take a look at the corresponding paragraph, especially the second paragraph of section 3.2, to verify our change. And we have also provided a copy of that paragraph to the Q2 below for your reference.

W3. The feedback in the PDDL benchmark is provided by human experts, which is very time-consuming. The average judging speed is approximately 20 responses per hour per expert, making it too slow to be used as an intermediate process for APE. For TravelPlanner, while the checker is available, the dataset authors have explicitly stated that such a symbolic method should not be used in the process (https://github.com/OSU-NLP-Group/TravelPlanner?tab=readme-ov-file#%EF%B8%8Fwarnings). This is also why the other work [3] achieves much superior results, as they use external code executors that encode the constraints to facilitate the generation process, which completely undermines the original purpose of the dataset. To adhere to the requirements and purpose of the datasets and avoid exploiting their limitations, we do not compare that paper as a baseline.

Q1. There are numerous cases where an accurate final evaluator is highly costly, making it impractical for most people to use them frequently during training or, in our case, while optimizing the prompt. This is especially common in high-specialization domains that demand extensive domain knowledge. For example, PDDL generation, included as a dataset in our paper, requires such expertise. Similar challenges arise with LLM applications in fields like physics, chemistry, and other scientific disciplines. This need for high-quality data generation has led to the emergence of startups like scale.ai, which support training LLMs by providing reliable datasets. There are also scenarios where a final evaluation might be entirely absent, as seen in applications like ChatGPT, and particularly in GPT's tools. In these situations, users interact with the application, sometimes providing intermediate feedback, but often leaving without clarifying whether they received a satisfactory response or abandoning the interaction due to dissatisfaction with the model’s ability to assist further. Our method allows such applications to optimize their prompts even in the absence of explicit final evaluations, which was previously unachievable.

Thank you again for your prompt response. We are pleased to see that you agree with our previous discussion. Here again we address your remaining concerns in W1 and W3.

W1. We apologize if our previous response may have confused you by conflating the baselines and the dataset settings (and perhaps also due to the fact that we needed to split the initial rebuttals into two parts because of character limits). However, we would like to kindly refer you to our previous response to Q3, where we clarified why there are no baselines in the PDDL generation benchmark. In TravelPlanner, we have already included PromptAgent as our baseline, which seems to contradict your claim that “there don’t seem to be any baselines.” If you are referring to other baselines outside the APE domain, please let us know, and we would be happy to include them if they could significantly improve our paper.

W3. While the baseline discussion is included in our response to W1 above, here we further highlight the difficulty caused by the lack of access to verifiers and other informations. To illustrate this, we would like to first quote the example question again, along with the initial prompt of TravelPlanner provided by the dataset creator:

Please create a travel plan for me where I'll be departing from Washington and heading to Myrtle Beach for a 3-day trip from March 13th to March 15th, 2022. Can you help me keep this journey within a budget of $1,400?

You are a proficient planner . Based on the provided information and query , please give me a detailed plan , including specifics such as flight numbers (e.g., F0123456 ) , restaurant names , and hotel names . Note that all the information in your plan should be derived from the provided data . You must adhere to the format given in the example . Additionally , all details should align with common sense . Attraction visits and meals are expected to be diverse . The symbol '-' indicates that information is unnecessary . For example , in the provided sample , you do not need to plan after returning to the departure city . When you travel to two cities in one day , you should note it in the ' Current City ' section as in the example ( i.e., from A to B). Solve this task by alternating between Thought , Action , and Observation steps . The ' Thought ' phase involves reasoning about the current situation . The ' Action ' phase can be of two types : (1) CostEnquiry [ Sub Plan ]: This function calculates the cost of a detailed sub plan , which you need to input the people number and plan in JSON format . The sub plan should encompass a complete one - day plan . An example will be provided for reference . (2) Finish [ Final Plan ]: Use this function to indicate the completion of the task . You must submit a final , complete plan as an argument .

As one can see in these two parts, while some common-sense constraints like “Within Sandbox,” “Complete Information,” and “Diverse Restaurants/Attractions” are explicitly included in the prompt, other important common-sense constraints such as “Non-conflicting Transportation,” “Within Current City,” and “Minimum Nights Stay” are never addressed in the prompt. Providing the information and evaluator ahead of time exposes these hidden requirements to the LLMs, and giving LLMs hints about these hidden common-sense constraints should also be considered within the prompt space. These constraints are so hidden that they are still not included in the optimized prompt, even after applying our RePrompt method or the baseline PromptAgent. We believe this is part of the reason why, in [3], the common-sense constraints macro pass rate is as high as 40.6%, while our method only achieved 6.11%. In comparison, in [3], the authors only reached a 39.4% pass rate for hard constraints, whereas Reflexion alone achieved a 33.89% pass rate. This small gap is likely due to receiving more accurate feedback during the process, which helps eliminate hallucinations (one of the key errors identified in both our Section 5 and the original TravelPlanner paper) and other errors. However, knowing what is being tested and what needs to be considered during reasoning significantly contributes to the greatly improved result on the common-sense pass rate and, consequently, the final pass rate.

Again, we hope this helps address your concerns. Please let us know if you have any further questions.

W1. Because our paper is set in a unique context where the solution checker is either unavailable (as in TravelPlanner) or very expensive (as in PDDL generation) during the generation/training process of an LLM agent task, the two datasets were selected accordingly. At the time of our submission, we did not find any other publicly available datasets that fit this scenario. However, we believe the current two models already cover two categories of tasks that align with this setting: 1) The PDDL generation task provides precise but costly feedback, testing the exact translation abilities of LLMs—a crucial skill for generating correct code; 2) In contrast, the TravelPlanner environment, using Reflexion, provides low-cost but less accurate feedback without access to ground-truth information. Furthermore, due to the high cost of running LLM-agent tasks, many well-regarded papers, such as ReAct, also test only on two datasets, as we do in our paper.

W2. We do not think our improvement scale on the datasets is unsatisfactory. For Guan et al. (the PDDL generation task), we have reduced approximately half of the total number of errors, even in the household domain you mentioned. The remaining errors, as discussed in Section 5 of our paper, are highly case-specific and theoretically challenging to resolve at the prompt level. For the TravelPlanner benchmark, we need to emphasize that the recent work you referred to is one of the studies that explicitly hacks the checker during the generation phase. They use external code executors to facilitate the generation process, which completely undermines the original purpose of the dataset, as clearly stated in the TravelPlanner repository: https://github.com/OSU-NLP-Group/TravelPlanner?tab=readme-ov-file#%EF%B8%8Fwarnings. In this case, their improvement scale is entirely incomparable to works like ours that adhere to the requirements.

The results on the Travel planning domain highlights my concern as well, on the belief that intermediate feedback can replace a final critic.

Our method, REPROMPT, uses the knowledge fetched in self-reflection phased into the prompt engineering phase. On the one hand, this will avoid early-stage errors, and more importantly, this will reduce the randomness caused by ReAct and Reflexion where some errors might not receive good enough feedback in a specific scenario. By summarizing the feedback, REPROMPT ensures that all common problems, such as balancing a budget, are highlighted in the prompt, leaving the tasks of ReAct and Reflexion to address more case-specific issues, such as a budget constraint caused by the significantly high price of hotels in a certain city on a specific day. We invite you to review the converged prompt provided in the appendix to see how constraints become more explicitly addressed after prompt optimization.

Therefore, depending upon an LLM may not be robust in identifying these "impactful parts" and some explainable AI strategy maybe better suited.

We agree with you that, given the current capabilities of LLMs, they may not reliably identify these “impactful parts.” However, similar to many other AI/ML research areas, our work focuses on finding ways to automatically solve problems and reduce the human labor involved. On a broader level, the entire domain of Automatic Prompt Engineering (APE) aims to replace manual prompt engineering, even though APE is not necessarily superior yet. We believe that, as one of the works in this pioneering research direction, our study can serve as a step toward achieving a point where automatic processes outperform human efforts.

Can the authors point me to details on their "summarizing team"

We apologize for the confusion. We use the word “team” in this context—its only occurrence in the paper—to differentiate it from the other components of the optimizer: the actor and the feedback generator. Therefore, this part is still performed by an LLM, which is fully automated and does not require IRB approval. For your reference, the prompt for the summarizer is provided in the appendix, and we have also discussed how summarizers are designed in our method section. We have updated our paper to remove the potential confusion here.

Thank you for your review. Here, we address your concerns one-by-one.

Weakness

1. Thank you for pointing this out. We apologize for any confusion caused by our previous version, and we have updated the paper accordingly. Unfortunately, due to page limitations and the nature of LLM agents requiring multiple iterations—further multiplied in batch settings like REPROMPT, which can lead to hundreds of feedback rounds—it is not feasible to include even a single example that illustrates each part of our algorithm. Additionally, due to the page limit, we find it impossible to include the prompts in the main paper, which we understand is a vital part of LLM-based papers. We invite you to review our updated paper, which we believe will help clarify your questions.

2. We agree that all the related work you mentioned is relevant, as our work lies at the intersection of self-reflection and prompt engineering. However, we believe the unique domain of our work also differentiates it from previous work. Here is our detailed comparison, highlighted in bold, for each point you raised.

2A. Indeed, there have been some works, if not many, that use a similar idea of ``gradient-based” method for prompt optimization for LLMs, however, none of these works are in the domain of LLM ** agents**, and consider the ReAct/Reflexion framework that is essential in getting a good performance in the domain.

2B. Thanks for pointing out this related work. We have added this work to our paper. Our paper differs from the existing method both in the fact that we are considering ReAct in the loop, and the detailed way of how the summarizer is constructed and used by the optimizer. More specifically, compared to PromSt, their method focuses on human-provided feedback, which is much more accurate than ours. In contrast, our optimizer can support less informative feedback as well.

2C. Our method REPROMPT uses the knowledge fetched in the self-reflection phase within the prompt engineering phase via summarization. This avoids early-stage errors, and more importantly, reduces the randomness caused by ReAct and Reflexion, where some errors might not receive adequate feedback in a specific scenario. By summarizing the feedback, RePrompt ensures that all common problems, like balancing budgets, are highlighted in the prompt while leaving the job of ReAct and Reflexion to address more case-specific issues, such as budget constraints caused by the significantly high price of hotels in a certain city on a specific day.

Questions

1. Convergence means the prompt does not change after another round of updates. Similar to convergence in standard ML model training, the converged prompt does not necessarily lead to perfect performance in all cases and could still have feedback. However, with this remaining feedback, our optimizer will determine whether the summarized feedback is already included in the current prompt or is case-specific, and it will ignore this feedback accordingly.

2. We have provided the optimizer prompt in our appendix, and the three common questions are explicitly included there. We chose to include them in the prompt because we found these are problems that frequently appear. Optimizers do not always solve these common problems in one trial and may waste many iterations before arriving at similar solutions. To reduce the cost and improve the process's efficiency, we chose to include them in the optimizer prompt. However, theoretically, not adding them does not change the final performance or conclusion.

3. Sorry for the confusion. We have updated the related paragraph in the new version. To directly answer the question: we provide REPROMPT and the low-level solver with examples of the task, but not in the exact scenario they are facing. We do allow in-context learning (i.e., examples in the prompt) in the initial prompt, and as discussed in the paragraph of original line 243, the corresponding paragraph remains unchanged. In the PDDL generation task, we generate the response only once without previous trials in each specific scenario.