Group Preference Optimization: Few-Shot Alignment of Large Language Models

Thank you for your valuable comments and detailed insights! We address your questions below.

Q1: Lack of clarity in the presentation of the method. For instance, it remains ambiguous which specific parameters are subject to training. Incorporating a detailed algorithm diagram or flowchart would greatly enhance the comprehensibility. Besides, more training details would be better.

A1: Thanks for pointing out the ambiguity. We added an algorithm box in Appendix A of the revision to illustrate how GPO is trained. GPO trains a transformer module for in-context predicting unseen group preferences based on the few-shot examples provided. And we do not train the base LLM weights. Moreover, the training details and hyperparameters are given in the appendix section G. We hope this clarifies your question. If you have more questions, please let us know.

Q2: The absence of evaluation results concerning the generalization capability of the GPO method. As GPO optimizes the parameters, it's crucial to show whether the model retains performance on general benchmarks after tuning.

A2: GPO does not alter the base LLM's parameters. Instead, it augments the LLM with an additional module trained to predict group preferences based on the few-shot examples provided. This approach allows the LLM to retain its broad language capabilities while aligning its outputs to specific group preferences​​ in a few-shot manner.

Q3: Could the authors provide examples of generation tasks (QA) to better showcase the model's preference? Demonstrating the model's performance on responses could provide a clearer insight into its practical applications.

A3: We agree such datasets could provide valuable insights into GPO's capabilities. However,

• We want to highlight that GPO is not inherently restricted to multiple-choice datasets. For open-ended long-form QA, GPO could predict raw preference scores and guide the LLM to align with group distributions, without requiring the scores to sum to 1.

• To the best of our knowledge, we are unaware of suitable labeled long-form QA group preference datasets for real world settings which are diverse, nuanced, and seperated for diverse demographic groups that would allow benchmarking.

• Collecting these dataset would be time consuming and is outside of this work’s scope. Additionally, evaluating the long-form generations would require human judgement which is expensive in our academic research setting. For example, gathering diverse demographic groups labellers to evaluate generated responses and aggregate their reponses as group preference distributions would be impractical for academic research.

• That said, we have made our best efforts to utilize the most comprehensive opinion datasets available for our current evaluations on multiple-choice QA. We chose OpinionQA and GlobalOpinionQA, which is of high community interest given their growing adoption, with 60+ GitHub stars recently for the former and 300+ monthly downloads of the latter. Additionally, their survey QA format enables efficient, accurate analysis without costly human evaluation.

If you have suggestions on such long-form QA datasets, please let us know. We would not hesitate to test GPO on them.

Thank you for your valuable comments and detailed insights! We address your questions below.

Q1:I am not sure about the significance and broadness of the problem this work tries to solve. Alignment of LLMs to group preferences sounds important, however, the evaluation datasets used by this work look quite specific and narrow and I don't think it is of interest to a broad range of research community.

A1:Thanks for acknowledging the importance of steering LLMs to group preferences. We think it is an important task as current popular alignment methods such as prompting, in-context steering, and SFT do not work well for it.

Regarding the evaluation datasets:

• We have exerted our best effort to utilize the most comprehensive and relevant opinion datasets currently available for assessing LLM opinion distributions. Additionally, there are currently no long-form QA datasets with labeled ratings collected for diverse demographics groups. We chose OpinionQA and Anthropic's GlobalOpinionQA for their growing use and recognition in the field. For instance, OpinionQA has garnered over 60 stars on github in about half a year. Similarly, GlobalOpinionQA has seen over 300 downloads in the past month, indicating high community interest in LLM opinion distribution studies.

• Our selection was also larged guided by the practical constraints of academic research, where datasets that allow for easy verification and no costly human evaluation are preferred. This motivation aligns with OpinionQA (as stated in [1] section 2.1) and GlobalOpinionQA’s motivations too, which states that the use of survey QA data enables capturing diverse and nuanced public interest topics and also enables efficient and accurate evaluation.

• Additionally, in contrast to prior work that treats human alignment as a single quantity such as safety or helpfulness, our work conceptualizes human alignment as an inherently subjective measure that depends on who it is evaluated against; to the best of our knowledge, we are among the first to align LLMs to a group preference distribution rather than a single preference or group consensus.

[1] Whose Opinions Do Language Models Reflect? https://arxiv.org/abs/2303.17548

Q2: The proposed method look quite trivial and standard, which is a in-context fine-tuning method. There are some changes in the method details but those details are a bit hard to understand, which I will pose some questions on next.

A2:

• We would like to clarify that GPO is not an in-context fine-tuning method, it does not alter the base LLM's parameters. Instead, it augments the LLM with an additional transformer module trained to predict unseen group preferences based on the few-shot examples provided. Although GPO does not modifying the weights of the LLM itself, it can guide the LLM to produce sample outputs that are attuned to the desired group preferences. This allows the LLM to retain its broad language capabilities while aligning its outputs to specific group preferences​​ in a few-shot manner.
• In section 3 (Page 6), the in-context finetuning method is actually one of our baselines that performs worse than GPO.

Q3: ... it is said that "In particular, we discard the positional encodings commonly found in standard transformer architectures". Wouldn't the removal of positional encodings significantly deteriorate the performance of those pre-trained and fine-tuned LLMs?

A3: We discard positional encoding for GPO but not the base LLM. We do this to achieve the property of context invariant, which means the prediction of preference for new questions should not depend on the ordering of the context questions. In the appendix Table 3 we submitted, we compared GPO with a standard autoregressive transformer that employs standard causal mask and positional encodings. And GPO’s inherent inductive biases yield superior alignment performance compared to a traditional transformer.

Q4: in the second last paragraph of page 4, it is said that "we employ a masking strategy where the context pairs can self-attend to each other". Could you elaborate such a masking strategy?

A4: Thank you for pointing out the potential confusion in our writing. We will provide more details on this in the revision. The masking strategy is as follows:

Our input to the transformer is a sequence of context pairs followed by padded target pairs as in figure 2. The mask allows:

1. The context pairs attend to each other, rather than using a causal mask that restricts each token to only attend to previous tokens. We use this because the order of the context pairs should not influence predictions for target points - it should be permutation invariant.
2. The padded target pairs to only attend to the context pairs, and not to other targets. This restriction aligns with the conditional independence assumption in Equation 2, ensuring each target pair is predicted solely based on the context.

If you have other questions, please let us know.

Thank you again for your insightful feedback on our paper! We've carefully considered your comments and revised our paper to address the questions you raised. As the discussion phase is drawing to a close, we hope you've had a chance to review our response. Considering the time constraint, are there any other questions or concerns you'd like us to address/clarify? Your support is greatly appreciated and we sincerely value your guidance in enhancing the quality of our work.

Since we have reached the end of the rebuttal period, we would like to reiterate and summarize our response earlier.

Q2: The proposed method look quite trivial and standard, which is a in-context fine-tuning method. There are some changes in the method details but those details are a bit hard to understand, which I will pose some questions on next.

• We added an algorithm box in Appendix A of the revision to illustrate how GPO is trained. GPO trains a transformer module for in-context predicting unseen group preferences based on the few-shot examples provided. And we do not train the base LLM weights. Moreover, the training details and hyperparameters are given in the appendix section G.

• GPO is not an in-context fintuning method. In section 3 (Page 6), the in-context finetuning method is actually one of our baselines that performs worse than GPO. Another advantage is that GPO operates in the latent space and can incorporate more in-context examples, while "in-context fine-tuning" takes raw text tokens as input and is constrained by the context window size. By working in the latent space, GPO can leverage more few-shot examples to better adapt to the preferences of unseen groups at test time.

• To our knowledge, GPO is among the first to do in-context preference learning for group alignment - we train a single model that can align to unseen groups at test time without any gradient updates: $p(y_{m+1:n} | x_{1:n}, y_{1:m})$ only with $m$ context pairs. This makes GPO efficient for few-shot adaptation to the nuanced distribution of preferences across groups which can be utilized to guide LLM as a group opinion simulator for downstream tasks.

Q3: ... it is said that "In particular, we discard the positional encodings commonly found in standard transformer architectures". Wouldn't the removal of positional encodings significantly deteriorate the performance of those pre-trained and fine-tuned LLMs?

Q4: in the second last paragraph of page 4, it is said that "we employ a masking strategy where the context pairs can self-attend to each other". Could you elaborate such a masking strategy?

Regarding the above concerns about positional encodings and masking strategy, we have added a more detailed description in Appendix C in the revision to formally describe the two properties integrated in GPO:

1. Context invariance: GPO removes positional encodings, ensuring that predictions are not influenced by the order or permutation of context pairs. It employs a masking strategy other than the causal mask in traditional transformers, such that the context pairs can and only attend to all the other context pairs.

2. Target equivalence: The masking strategy also ensures the targets attend to context points but not other targets, adhering to the conditional independence in Equation 2.

We hope this addresses your concerns. Given today is the last day for the discussion period, please let us know if you have more questions. Your support is greatly appreciated and we sincerely value your guidance in enhancing the quality of our work.

Q4: "the Reward Model baseline is underperforming a lot, it would really make sense to add a comparable-sized baseline as in the author’s proposed method to rule out the possibility that overfitting is the only cause of inferior baselines w/o really relying on LLM. The PeftConfig configuration seems not consistent with the description of the Reward Model baseline..."

A4: Our reward models are initialized using the HuggingFace AutoModelForSequenceClassification which adds an additional linear layer to the LLM. When this model is converted to Peft model, the Peft library ensures that only the LoRA adapters and the additional linear layer are trained since our task_type is set to SEQ_CLS in our PeftConfig and we are using an AutoModelForSequenceClassification model. In the description of reward model baseline in section 3, when we state “we train a per-group reward model by adding a linear MLP head on a base LLM and train it on m context samples”, “it” refers to all of the trainable parameters of the entire Peft model. We have updated the reward model baseline description to more clearly reflect the reward model training setup and apologize for the confusion.

Related works including InstructGPT[1] typically train a reward model by fine-tuning all parameters of a reward model(including the base LLM). Considering that we are training a very small subset of the base LLM’s parameters with LoRA (not the full 7B), compared to prior works which train all 6B+ parameters for reward model baselines, it is unlikely that our reward models are overfitting majorly due to large model size. The number of trainable parameters for the reward linear layer, our original LoRA training and GPO’s transformer module are shown in the table below.

Nevernethess, we conducted an experiment to rule out the overfitting possibility due to trainable parameters, and the results are shown below in the table, which shows no improvement for training only the top linear classification layer. We believe the reward model’s inability still lies in the difficulty of learning a high-quality reward function from a small number of training examples for every group.

[1] InstructGPT: https://arxiv.org/pdf/2203.02155.pdf

Q5: Also, it makes more sense to use cross-entropy for the loss function instead of MSE as the final output is a distribution.

A5: We chose to use MSE as the loss function because the reward model takes a single input viewpoint (x) and outputs a scalar target value (y) rather than a probability distribution. However, you raise a good point that cross-entropy loss may better match the probabilistic nature of our tasks. Given your suggestion, we experimented with CE loss on the two datasets as shown in the results table. Empirically, the two losses perform similarly, but CE has the advantage that the loss converges faster than MSE loss.

Q6: ...justify the reason for the output of (r) as input in this work (through ablation studies) ... a natural question: without r, will the model underperform a lot, meaning that LLM would also be the important ingredient? Also it seems that the qualities of the response from the model would also seem to be quite important, and it would make more sense to replace it with random strings etc to further make the study more coherent.

A6: We think there may be some confusion about the role of the response in our setup. To clarify, the response r is essential in our problem formulation. Our goal is to model preferences y over query-response pairs (q, r), where r provides the crucial context to ground each opinion. Without r, the score y would be assigned to the question q alone. Thus, r is critical for accurately reflecting the nuanced "opinion" we aim to infer group preferences over.

However, we appreciate your suggestion of an interesting ablation and have conducted this experiment by giving a random string response to every query. As shown in the table below, the performance degraded substantially without the response.

Thank you for the detailed and constructive reviews and suggestions for experiments, which have helped improve both the clarity and the rigor of our paper. We addressed your feedback below:

Q1. While the paper presents a new infra on the problem of preference distribution, the alignment method does not look too much different than a normal semi-supervised framework

A1: We want to clarify that GPO is not a semi-supervised framework, but rather a few-shot learning framework that is trained on fully labeled preference dataset of (q, r, y), where q is a query question, r is the response, and y is the group’s preference to this response given the q.

GPO’s transformer module learns to infer preference y for a viewpoint (concatenation of q and r), for unseen groups at test time, which is both more accurate and sample efficient than SFT or prompting methods. For better clarity, we have included a detailed algorithmic box for GPO’s training pipeline in Appendix A of the revision paper.

Q2. For the baseline, the authors seem to fail to include one direct method. A simple Transformer model that purely learns from the (q, y) pair and uses them in a semi-supervised fashion as a sequence of inputs

A2: Thank you for highlighting the potential oversight in our baselines. We'd like to address this in two parts based on our understanding of your suggestion:

• Based on our problem setup, we think the q you referred to is actually x (the concatenation of q and r), because the preference y is associated with x rather than q alone. Then the concept of inputting the (x, y) pairs directly to a transformer model closely aligns with our 'In-Context Finetune' baseline. In this baseline, n context pairs (x,y) are indeed fed sequentially as inputs to the LLM to predict the preference y, and this method performs worse than GPO in our experiments. However, we are not using GPO nor the baseline in a semi-supervised fashion but rather train them on labeled (x, y) pairs.
• Secondly, we have also included a baseline that aligns with your suggestion of a simple transformer model in appendix Table 3 we submitted. We compared GPO with an equally-sized standard autoregressive transformer that employs standard causal mask and positional encodings and also takes in (x,y) pairs as sequence inputs too. In contrast, GPO has a different masking strategy and removed positional encoding to adopt inductive biases that lead to two properties. One is context permutation invariance, where the prediction of queries does not depend on the ordering of the context pairs; and the other property is target equivariance, which requires that whenever we permute the target inputs, the predictions are permuted accordingly. These inductive biases are important for our problem setup. And GPO’s inherent inductive biases yield superior alignment performance compared to a traditional transformer as shown in the table below.

If our interpretations do not fully address your baseline suggestion, please give us further clarification and we will provide additional baseline experiments.

Q3: The terminology of alignment seems a bit too abused and distracting in this setting. In this work, the authors only tried to learn a separate Transformer architecture that operates upon the output of LLM while the LLM itself does not enjoy any new abilities in its parameters based on the modifications.

A3: We use "alignment" to mean aligning the output distributions of the generative model - a combination of LLM and GPO - to group preferences. Although GPO does not modify the weights of the LLM itself, it can guide the LLM to produce sample outputs that are attuned to the desired preferences.

This terminology is also in line with prior works for alignment such as [1] and [2], where alignment also refers to tuning sample outputs rather than altering LLM parameters. In these works, alignment is achieved using techniques like reward models and Best-of-N sampling over the pretrained LLM, without updating the base model itself. Additionally, in contrast to prior work that treats human alignment as a single quantity such as safety or helpfulness, our work conceptualizes human alignment as an inherently subjective measure that depends on who it is evaluated against; to the best of our knowledge, we are among the first to align LLMs to a group preference distribution rather than a single preference or group consensus.

[1] Let’s verify step by step. https://arxiv.org/abs/2305.20050

[2] Scaling Laws for Reward Model Overoptimization https://arxiv.org/pdf/2210.10760

Thank you again for your insightful feedback on our paper. We've carefully considered your comments and conducted new experiments to address the questions you raised. We'd like to note that the discussion phase is drawing to a close, and we hope you've had a chance to review our response. Considering the time constraint, are there any other questions or concerns you'd like us to address? Your support is greatly appreciated and we sincerely value your guidance in enhancing the quality of our work.