Pareto Prompt Optimization

1. Objectives Selection and Additional Result

In the current literature on multi-objective prompt optimization, reported benchmarks are typically limited to at most three objectives, which guides our choice to evaluate our method using two or three objectives for fair comparison with existing methods. We thank the reviewer for suggesting output fluency and conciseness as objectives and have extended our text style transfer experiment to include four objectives:

1. Style score
2. Content similarity
3. Output fluency
4. Conciseness (minimizing the length of the output)

For each algorithm, we conducted three independent runs, and the average performance for the four-objective problem is provided below:

The full results will be included in the revised paper.

---

2. Model Selection

Regarding our choice of BERT and GPT-2 as task models, we followed prior works such as [1] and [2], which also used these models in their evaluations. However, we recognize the importance of including more recent language models and will extend our experiments to incorporate them.

---

Q1: Non-Dominance Loss

The non-dominance loss does not “govern” the loss function, as shown in the updated Figure 5 in our revised manuscript. The reviewer is correct in observing that the majority of the sampled prompts are non-dominated. However, we designed the non-dominance loss to have little or even zero value when the reward difference (reflecting the likelihood differences) between the non-dominant pairs is small. When we update $π_θ$ to equal $π_{\text{ref}}$, the $h$ function in Eq. (4) becomes zero, and thus the non-dominance loss is zero. In this scenario, training is driven by the dominance loss, with the non-dominance loss acting as a form of regularization to encourage diversity in prompt generation.

---

Q2: Loss Function in Eq. (4)

The loss in Eq. (4) is based on the reward function $\log(\pi_\theta(z|x) / \pi_{\text{ref}}(z|x))$ from the DPO paper [2], which reflects the likelihood of generating a specific prompt $z$. In Eq. (4), we penalize large reward differences between non-dominant pairs. This encourages the model to distribute generation likelihoods more evenly across Pareto-optimal prompts. By preventing any single Pareto-optimal prompt from dominating the likelihood distribution, the model is able to better diversify and generate a broader set of non-dominated prompts.

---

Q3: Additional Experiments

We have conducted experiments taking output fluency as one of the four objectives, as described in the new experiment above.

---

Q4: Prompt Selection

To choose one prompt from the Pareto front to use in the system, the decision depends on the specific priorities or preferences for specific applications. For example, if output fluency is more critical than conciseness, we might prioritize prompts that perform better on fluency while slightly compromising on conciseness. The preferences can also be dynamic. Our algorithm provides a diverse set of Pareto-optimal prompts, with higher flexibility leaving the final choice to decision-makers based on their specific needs.

---

Q5: Loss Dynamics

We appreciate your detailed observation and good question. Although the non-dominance loss increases and plateaus during training, the total loss is decreasing overall. We have updated Figure 5 to include both the total loss and the dominated loss to provide a clearer picture. The non-dominated loss acts as a form of regularization, it encourages the model to maintain diversity and explore the Pareto front effectively, which can result in a slight increase in its value as the model generates more competitive prompts.

Thank you for your thorough assessment of our submission and for your helpful recommendations.

---

1. Discussion and Comparison with Related Work

We appreciate the reviewer’s suggestion to compare our method with the suggested reference papers. However, all the reference papers focus on single-objective prompt optimization, while our method is explicitly designed for multi-objective prompt optimization. Specifically, references [2] and [4] employ evolutionary algorithms for single-objective prompt optimization and we have compared our method with their multi-objective counterpart, InstOptima, which uses NSGA-II for multi-objective prompt optimization.

We acknowledge the strengths of the reference papers in single-objective prompt optimization and have included a discussion of their contributions in the Related Works section of our paper.

---

2. Differences and Comparison with Related Work

We thank the reviewer for their question. While [5] also considers human preference in prompt optimization, it focuses on single-objective prompt optimization, whereas our method addresses multi-objective prompt optimization.

Moreover, [5] relies on learning a reward model trained on human preference data and then optimizes the reward model to find the optimal prompt. In contrast, our approach directly optimizes the prompt generation model without requiring explicit reward modeling. This direct optimization makes our method both more efficient by eliminating the intermediate reward modeling step and more robust by reducing reliance on an optimization procedure.

Thank you for the thoughtful review and feedback on our paper. Below are our responses to address the concerns and questions raised:

---

1. Suggested Baseline Paper

We thank the reviewer for suggesting the baseline paper. The paper proposes a multi-objective reward modeling method trained on human preference data, where the multi-objective framework enhances both interpretability and steerability in RLHF fine-tuning.

We agree that this method could be extended for prompt optimization by building a reward model for prompts and then optimizing the reward model to find the optimal prompt. However, this extension is not straightforward and would require significant adaptation, including:

1. Training the reward model for prompts in a domain- and application-specific manner.
2. Devising an optimization procedure compatible with the prompt space.

While the approach is promising, incorporating it into our work within the given rebuttal time is not feasible and is beyond the current scope. We’ll consider including it in future studies to enrich the experimental comparisons.

Additionally, we’d like to highlight that, compared to reward modeling approaches, our method directly optimizes the prompt generation model without the need for explicit reward modeling. This direct optimization makes our method both more efficient by eliminating the intermediate reward modeling step and more robust by reducing reliance on an optimization procedure.

---

2. Theoretical Proof of Pareto-Optimality

We are currently not able to provide a theoretical proof that the proposed training process is Pareto-optimal, while we are actively working on it. Establishing such a proof is challenging due to the inherent complexity of multi-objective optimization in high-dimensional spaces and the stochastic nature of the training process.

Specifically, the optimization landscape for multi-objective problems is typically non-convex, the trade-offs among objectives can be complex and task-dependent, making it difficult to derive guarantees for Pareto-optimality. Additionally, our method involves a stochastic prompt sampling process, which adds complexity to formalize a theoretical proof.

Despite the lack of formal proof, our empirical results demonstrate that the method effectively approximates the Pareto front across various tasks, supporting its practical utility.

---

3. Estimate the Relative Preference

Our training procedure requires only the relative preference between $y_1$ and $y_2$ for each objective, rather than their absolute values. This is simpler to obtain and more flexible in practical applications. The relative preference can be derived either through evaluations by a LLM or estimated using a surrogate measure function, For example, in our experiments, we used CoLA to estimate text fluency.

We sincerely thank the reviewer for their thoughtful feedback and valuable suggestions. We address the specific concerns and questions below:

---

1. Motivation for Using DPO (Direct Preference Optimization)

The motivation for using DPO in Pareto optimization stems from two key considerations:

• Robustness to Evaluation Challenges:
  Standard Pareto algorithms often rely on scalarized or absolute objective values to guide optimization. However, in language generation tasks, accurately quantifying absolute objective values is often impractical due to vague evaluation criteria and subjective evaluations. DPO, by focusing on relative preferences derived from dominance relationships, provides a more robust approach that sidesteps the unreliability of absolute evaluations.

• Flexibility in Multi-Objective Assumption:
  Standard Pareto algorithms often require assumptions about the structure of the objective space, such as additive contributions, uniform preferences, or pre-defined reference points. These assumptions may not hold in complex, real-world problems like prompt optimization for language generation. By solely relying on dominance relationships, DPO avoids such biases, ensuring a more flexible and assumption-free optimization process.

---

2. Query Budget Details

In our experiments, we use the same query budget across Summation, Product, HVI, R-IPO, PP-DPO, and PP-IPO. The query calculations for Tables 1 and 2 are as follows:

• Table 1:
  The policy model generates 16 prompts per iteration. For each prompt, the task model queries 16 samples per class. With a total of 6,000 iterations, the total number of language model queries is:
  16 × class_num × 16 × 6,000

• Table 2:
  The policy model generates 8 prompts per iteration. Each prompt produces 128 style-transformed outputs for average performance evaluation. With 10,000 iterations, the total number of queries is:
  128 × 8 × 10,000

This consistent query budget ensures fairness when comparing methods in both tables.

InstOptima, on the other hand, generates prompts through prompt operations such as mutation and crossover of parent prompts using LLaMA2-7B. To ensure a comparable total runtime with PP-DPO/IPO, we employed NSGA-II for prompt optimization, running:

• 60 generations with 16 prompts per generation for Table 1
• 130 generations with 16 prompts per generation for Table 2

These details have been updated in our paper.

---

3. Generalization Across Diverse Tasks

Our learning procedure is designed to be a general approach that can be applied across diverse tasks, provided that the dominance relationships data is available. As demonstrated in our experiments, the procedure successfully adapts to various tasks.

---

4. Additional Experimental Results with Four Objectives

We thank the reviewer for the suggestion to include results with a greater number of objective functions. In the current literature on multi-objective prompt optimization, reported benchmarks are typically limited to at most three objectives, which guides our experiment setting with two or three objectives for fair comparison with existing methods.

Our method is capable of handling problems with more objectives. To demonstrate this, we extended our text style transfer experiment to include four objectives:

1. Style score
2. Content similarity
3. Output fluency (as suggested by reviewer yPkv)
4. Conciseness (minimizing the length of the output)

For each algorithm, we conducted three independent runs, and the average performance for the four-objective problem is provided below:

The full results will be included in the revised paper.