Active Preference Optimization via Maximizing Learning Capacity

The weaknesses of the paper are as follows:

1. Missing references in the related works. [1] introduce regret bounds for online iterative learning with batch exploration in the RLHF problem.

2. The proof of sub modularity (paragraphs beginning at lines 716 and 721) in the appendix should not be presented as a proof. The arguments the authors present are reasonable but are not technical proofs of submodularity. The submodularity of the Preference Model Certainty term specifically is likely only approximate as the implicit reward relies upon the LLM policy. In the main text the submodularity property of the selection objective should therefore be presented as an assumption before Theorem 4.2. The current presentation may be interpreted as misleading.

3. Some technical details are missing from the paper. These include:

• A brief introduction to the k-means ++ cluster approach, specifically introducing gamma as this is key to understanding Figure 4a), and the effect of k more broadly.

• How the embedding space is implemented in the experiments. Are the prompts and response embeddings added or concatenated together, or is some other approach used?

• How are the 2D embeddings in Figure 4c) created?

• In algorithm 1 line 3 multiple responses are generated per prompt $y^{[n]}$. Should this just be two responses or are there n responses?

[1] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In International Conference on Machine Learning, 2024.

The paper is not clear to me. For example, in Algorithm 1

• It seems only $y_i^{[1]},y_i^{[2]}$ are involved in the following procedure. Why sampling $y_i^{[3]},\dots,y_i^{[n]}$ on line 3?
• Set $L$ is not initialized or populated anywhere, but referenced on line 6. What elements does $L$ contain?
• There is summation over $k$ on line 6, but $k$ does not occur in the summed term.
• Do $(x_i,y_i^{[1]},y_i^{[2]})$ and $(x^i, y_1^i, y_2^i)$ refer to the same tuple? If so, the notation should be unified.
• On line 9, argmax is applied to prompt $x$. How does this lead to tuple $(x_*,y_*^+,y_*^-)$?
• The elements in $U$ are prompts $x$ and $\mathcal S$ are tuples $(x_*,y_*^+,y_*^-)$. How does the operation $U\backslash \mathcal S$ work?
• I understand line 9 calls the oracle $\mathcal O$ to obtain the preference, which should be explicitly stated.
• Set $S$ is populated on line 11 but the terminal condition is on line 10. To my understanding the inner loop is an infinite loop?

I encourage the authors to revise the paper for the ease of understanding.

1. My main concern is the relationship between the proposed method and the previous work. In section 2, the authors mentioned the previous AL algorithms "hard to directly apply to the PO framework" as "only a single label is required for annotating each data example". However, in section 3.3, the authors are able to convert the pair of labels to a single certainty label. In this case, would the previous work applicable to the preference optimization framework? If yes, a thorough discussion of the applicability of the traditional AL algorithms would be helpful.
2. The presentation of the work has room for improvement. This includes the unclear definition of terms, the lack of motivation for the proposed method, and overuse of the language to describe the mathematical operations.

• Unclear definition of terms: Please see the questions part for detailed comments.
• Insufficient motivation: The authors did not provide a clear motivation for eq 3, where the selection is by the multiplication of two terms that the authors believe important. How about other operations like addition?
• Overuse of the language: line 266, line 291. The authors can easily use the mathematical formulation or refer to some equations to help the reader understand the operations.

3. As mentioned in line 393, "OpenAI’s GPT-4, when appropriately prompted, aligns closely with human judgments". In this case, it is not clear why the a significant number of responses from human participants (line 35, the motivation of the work) is necessary and a burden, as strong model like GPT-4 or open-sourced counterpart LLama can provide preference labels.