Inverse Constitutional AI: Compressing Preferences into Principles

Thank you for your positive and detailed feedback! Below, we address each of your points, abbreviating weakness with "W" and question with "Q". References refer to the revised manuscript, where changes are highlighted in blue.

W1: One inherent limitation of the ICAI method is that it simplifies complex human annotations into a smaller set of principles, which can result in a lossy representation.

We agree that lossy compression is a limitation of the ICAI method, inherent to the goal of summarizing complex feedback data into a concise and interpretable format. While future work may further refine the method to capture more nuanced preferences, we view this as a trade-off between interpretability and complexity. We believe that the interpretability gained from the concise principles outweighs the loss of some nuanced information. Further, note that on datasets which are mostly aligned with the data used to train the AI annotator, the annotator can fill in some nuances that are lost in the constitution, as long as this judgement does not contradict the constitution. We have added a discussion of to the limitations section (Section 6) to clarify this point.

W2: The effectiveness of ICAI heavily depends on how well an LLM can interpret and apply the generated principles.

While true, this reliance can also be a strength. The LLM can leverage prior knowledge to fill in nuances and weigh principles like a human judge. This does come at the cost of interpretability and depends on the LLM's generalization ability. Nonetheless, our experiments showed that ICAI's principles effectively guided the LLM to reconstruct annotations, even on unseen data.

W3: The generated principles, while human-readable, may be ambiguous or open to interpretation.

We acknowledge this risk but view it as a double-edged sword. The flexibility of language-based principles enhances expressivity, enabling compact and interpretable preference representations. However, ambiguity can lead to inconsistent applications, particularly in edge cases not explicitly covered by the principles.

Q1: How well do the principles generated by ICAI transfer across different models and datasets?

Excellent question! Transferability is crucial, as it reflects generalizable principles rather than model-specific artifacts. In Appendix C.5, we show that constitutions generated by GPT-4o transfer effectively to other models (e.g., Claude-3 variants). For datasets, transferability depends on similarity. While constitutions generalize well within the same distribution (e.g., AlpacaEval and PRISM), they may not generalize across distinct distributions, as seen in cross-user and demographic experiments (Sections 4.3 and 4.4).

Q2: How effective is ICAI at identifying subtle or less obvious biases in preference data?

We address the use-case of bias detection in the new Section 4.5 (and Appendix C.2) of the updated paper, where we demonstrate how ICAI can be used to uncover and evaluate biases in the Alpaca Eval, Chatbot Arena and PRISM datasets. While the biases we uncover there are relatively simple, we have added a discussion of how ICAI could be extended to detect more subtle biases in Appendix C.2.

Q3: How might ICAI be extended to work with multimodal data or more complex preference structures beyond pairwise comparisons?

We believe that ICAI could straightforwardly be extended to work with multimodal data by adapting the method to generate principles from more complex preference data. An important question to answer would be, however, if a textual representation of the principles is still sufficient in more complex modalities such as audio or video, or if a different representation (possibly in those same modalities) would be more appropriate.

Similarly, more complex preference structures such as ranking or rating data could be addressed by adapting principle generation prompts, as long as the underlying LLM is capable enough to understand and apply these more complex structures -- i.e., interpreting a full ranking may prove challenging due to the long context required. We appreciate that you raise these points, we would be excited for future work to explore both of these questions.

We hope that we could address your concerns and questions effectively. Please let us know if you have any further questions or comments. Thank you for your detailed review!

Thank you for your thoughtful feedback! Below is a discussion where we aim to address each of your points. We abbreviate weakness with "W" and question with "Q". All references refer to the revised manuscript, in which we highlight changes in blue.

W1: Limited discussion of applications of ICAI.

Thank you for raising the points about expanding the discussion of ICAI's applications in addressing annotation biases and scaling up annotation. While our analysis of demographic differences in the PRISM dataset already demonstrates one practical application of ICAI, we acknowledge that the paper's primary focus has been on demonstrating our method's general capability via reconstruction performance, rather than broader applications. We view the reconstruction of feedback annotations as an essential first step, laying the foundation for future work on the broader applications of the ICAI method. We agree, though, that a deeper exploration of how ICAI can be applied would enhance the paper and are grateful for the feedback.

We have added further details on the application of ICAI for annotation scaling and bias detection based on existing and new experimental results. We believe that ICAI's potential in scaling up annotation is already demonstrated by the generalization of the principles to unseen test data in the AlpacaEval experiments (Section 4.2 and Appendix C.6). The paper was previously lacking a discussion of this aspect, however, which we have now added to Section 4.2. We further address bias discovery by adding a case study in the new Appendix C.2, illustrating how ICAI can uncover annotation biases such as verbosity, lists, and assertiveness, and outlining potential extensions for detecting more subtle biases. For example, we observe that verbosity bias seems particularly prominent in the Chatbot Arena and PRISM datasets, while AlpacaEval and Chatbot Arena have a preference for assertive language ("a definitive stance without nuance"). These findings highlight ICAI's utility in uncovering and analysing dataset-specific biases. We hope these additions provide more comprehensive evidence for ICAI's applications and address your concerns effectively.

W2 (1): Some annotation biases and principles require synthesis from multiple feedbacks.

We agree that our current approach, generating principles based on a single preference pair, may miss principles that only become obvious when considering multiple preference pairs simultaneously. We chose the single preference to avoid additional complexity in our initial algorithm. To test whether this alternative approach has a notable effect on the performance of our method, we add results of a new ablation study with principle proposal via multi-preference prompts, see Appendix C.3. We observe mixed results: for some tasks, multi-preference proposal improves performance, for others it slightly decreases performance. A possible explanation may be that some datasets (e.g. synthetic orthogonal) are best reconstructed using specific rules that may be missed if looking for overarching principles on multiple preferences. Other tasks are best reconstructed with more general principles, more likely to be found when looking at multiple preferences. Overall, we consider further adaptations to the principle generation approach an interesting area for future study.

W2 (2): The K-means clustering requires users to set a number of principles, even though the exact number of principles is unknown.

We agree that the selected number of clusters can seem arbitrary. In practice, we found ourselves constrained in the number of clusters (and corresponding principles) we could practically test due to compute costs, rather than running out of different principles to test. For all but perhaps the small synthetic datasets, we found the model in Step 1 came up with more unique principles than we could test. Thus, in practice, the number of clusters is primarily determined by the compute budget. In general, if the compute budget is available, more clusters likely would not have a negative impact on performance as any less useful principles get filtered out after the testing and filtering steps. More clusters, and correspondingly more tested principles, increase the chance that a well-performing principle is found. Note we may want to limit the number of well-performing principles that are included in the final constitution, as we observed that longer constitutions lead to diminishing returns at some point (see Appendix C.4).

Thanks for your detailed responses. I think most of my concerns have been addressed and I will raise my score to 6.

Thank you for your detailed review! Below, we discuss each point raised. We abbreviate weakness with "W" and question with "Q". All references refer to the revised manuscript, in which we highlight changes in blue.

W1 (1): Without establishing causality between the principles and annotator rationale, the framework risks over-simplifying or even misrepresenting the underlying preferences.

We agree that the results need to be interpreted with great care. We discuss the causality limitation and how it affects the usability of our method in the limitations section and hope to clarify this discussion further in this rebuttal. The alternative to ICAI for most datasets is to simply use them as a black-box --- without having any explanation of what the annotator rationale may have been. We argue that some indications of annotator rationale (even if imperfect) are preferable to none. Please let us know if you think our limitations section does not adequately address this concern.

W1 (2): For example, it is possible that the principles reflect incidental biases of the model or dataset rather than genuine human values. This could lead to misleading interpretations and false assumptions about user or demographic intentions.

We agree that misuse of ICAI to create misleading interpretations of annotator values is a concern, as we state as part of our Ethics Statement. Yet, regardless of the annotators, if a dataset happens to have an incidental bias (e.g. stylistic, subconscious or even by chance), then awareness of such a bias is critical --- even if the bias is incidental does not reflect the annotators' values. Awareness can help avoid optimizing towards the bias, either by modifying the constitution or filtering the dataset. We have added a discussion on possible mitigation strategies in the new Appendix C.2, which hopefully supports the benefits of ICAI in detecting and mitigating biases.

W2 (1): ICAI's approach inherently admits multiple valid constitutions for the same dataset, depending on clustering and sampling choices. This non-uniqueness implies that each run could yield different principles that still achieve similar reconstruction accuracy. This hurts interpretation.

We agree that non-uniqueness of constitutions and principles makes interpretation more difficult, but this issue is not unique to ICAI but rather generally affects the problem of explaining annotator decisions in natural language. There are numerous ways any given set of principles can be rewritten such that an annotator would come to the same conclusion. Nevertheless, qualitatively there are many similarities in constitutions generated for different seeds and, as discussed above, we would argue that some indications on annotator rational (even if imperfect) are preferable to none.

W2 (2): Also ICAI seems to be influenced by initial prompt or clustering parameters as well, making it more unstable.

We agree that ICAI is influenced by its parameters. We provide a small study on hyperparameter sensitivity in appendix C.4. As general advice to mitigate instability, the results of our scaling experiments (shown in Table 4 in Appendix C.6) indicate that running on larger datasets reduces the overall variance of our method. This effect can be seen in terms of a reduction of standard deviation of results as scale goes up.

W3: The paper primarily focuses on preference reconstruction, yet practical applications, such as bias detection, model debugging, or customization, are only discussed in passing without concrete evidence of their effectiveness. There is no emperical evidence of ICAI’s practical application.

We agree that the paper could benefit from more concrete evidence of ICAI's practical applications. To fill this gap, we have added a new set of experiments in Section 4.5 and Appendix C.2, where we use ICAI to detect biases in multiple preference datasets. We find that ICAI can be used to detect biases in the data. Concretely, we find evidence of verbosity bias, list bias and assertiveness bias in the datasets. We additionally add a discussion of possible bias mitigation strategies in Appendix C.2, which we hope will further support the practical applications of ICAI.

Thank you again for your detailed review, and for taking the time to read and consider our response. We would like to ask for your feedback on potential further improvements to the paper. We are currently working on a second revision, which we plan to share within the editing window (27 Nov 11:59pm AoE). Below, we go through your concerns again, summarize how we initially addressed the concerns, and share additional improvements we are currently working on.

• Constitutions may over-simplify and misrepresent annotator intention (W1): We adapted our limitations section (6) to further highlight the related tension between interpretability, necessitating a short and possibly over-simplifying constitution, and accuracy.

  • Further planned improvements:
    1. Expand our ethics statement to especially highlight the raised risk of misrepresentation.
    2. Add corresponding warning to our code output. Both these steps aim to ensure users are fully aware of this limitation and prevent misinterpretation of ICAI results.

• Non-uniqueness and variability of constitutions (W2): We agreed with your concern and recognize it as a fundamental challenge with many interpretability methods like ours. We highlighted that for down-stream applications (like harmful bias detection), finding concerning principles is worthwhile even if other possible explanations exist.

  • Further planned improvements:
    1. Add in-depth discussion addressing the impact of this limitation as appendix.

• Missing practical applications evidence (W3): We introduced new experiments (Section 4.5, Appendix C.2) to demonstrate ICAI's utility in detecting biases in real-world preference datasets. These experiments show how ICAI can address issues like verbosity and fairness, improving its practical relevance.

  • Further planned improvements:
    1. We have completed an additional use-case study and will add this to the manuscript, providing further evidence for the usefulness of ICAI to scale annotation data from a small initial dataset, applied to helpful/harmless annotations.

• Risk of bias amplification (W4): We recognized the risk that our method may amplify biases, but we also highlighted ICAI's potential role in detecting, rather than amplifying, harmful biases and provided examples of possible mitigation strategies. These are reflected in the new experiments (Section 4.5 and Appendix C.2) and further detailed in the ethics section.

  • Further planned improvements:
    1. Expand discussion of ethics section further to more explicitly highlight the risks of our method in terms of bias amplification and provide actionable steps users can take to mitigate this risk (e.g., through manual inspection of constitutions).
    2. Add a corresponding warning to our code output (incl. actionable mitigation steps). Both these steps will ensure users are fully aware of this risk and can mitigate it as far as possible.

Like any methodology, our approach is not without remaining challenges, but your feedback has helped us refine and clarify these. We believe the changes made in this revision, as well as the planned additional updates outlined above, significantly strengthen the manuscript and address the main issues raised in your review.

We plan to share a second revision of the manuscript soon but, given the limited time remaining in the editing window, wanted to give you the opportunity to share your thoughts on the planned changes and on any other improvements we could make to further strengthen the paper and better address your concerns.

Thank you again for your constructive feedback - it has helped us improve the paper considerably!

Thank you for your detailed review! Below, we discuss each point raised. We abbreviate weakness with "W" and question with "Q". All references refer to the revised manuscript, in which we highlight changes in blue.

W1: Static principles may lead to information loss compared to dynamic methods like PopAlign. Can the authors include an elicitive preference annotation as a baseline?

Thank you for highlighting this work. We politely note that the "PopAlign" paper was published over two weeks (17 Oct 2024) after the ICLR submission deadline. Not including a baseline that was unavailable at the time of submission should, in our opinion, neither be considered a weakness of our work nor affect the review score.

That said, we appreciate the relevance of PopAlign and its elicitive contrast approach, which dynamically derives instruction-specific principles for generating diverse responses. While this approach shares similarities with our principle generation method, our understanding is that PopAlign is designed for data synthesis (response generation) rather than the data interpretation setting considered in our work. Applying this method directly to interpret existing datasets would require significant adaptation, as PopAlign does not incorporate responses and preferences into its principle generation process, nor does it evaluate principles for global applicability across multiple data points. A straightforward adaption would be similar to a chain-of-thought approach, possibly enhancing the default annotator performance, but unable to dynamically adjust to different preference datasets in a data-driven manner.

To acknowledge this complementary work, we have added a discussion to Appendix B.3. We note how ICAI's data-driven constitutions could inspire more targeted contrastive prompts for PopAlign, and how PopAlign's strategies for generating diverse responses could enrich preference data for ICAI, advancing the shared goal of understanding and leveraging preferences to improve AI alignment. We appreciate that you brought this connection to our attention. Please let us know in case we misunderstood the PopAlign method or its applicability to the ICAI problem.

More generally, we do agree that the Inverse Constitutional AI approach represents a compression of preference patterns. Yet, this compression is by design and allows us to generate short, interpretable principles and constitutions. Less compressed methods, such as conventional reward models, lack the interpretability features of our method. We have added a discussion of this trade-off to Section 6 (limitations) to clarify this point.

W2: The comparison between default and ICAI annotators may be unfair. Did the authors prompt default annotators to flip feedbacks?

We disagree that the comparison between our default and ICAI annotators is unfair but recognize that further clarification would be helpful. The critical difference between our annotator and the conventional LLM-as-a-judge annotators (such as the default baseline) is that our method can adapt to new datasets. Based on a (potentially small) training annotation set, our method dynamically generates an interpretable constitution and generates similar annotations. We could also compare our method to a conventional LLM-as-a-judge model that is prompted to select the worse output. However, such a "flipped" LLM-as-a-judge would then fail in the aligned setting. Our ICAI method can adapt in both scenarios to successfully reconstruct annotations. We have added a clarification to the beginning of Section 4 to address this point.

W3: Principles in natural language lose information compared to implicit preference encoding in reward models. Can a reward model such as a fine-tuned PairRM be added as a baseline?

We appreciate the suggestion to compare ICAI to a reward model such as PairRM and agree that reward models can capture preference patterns in an implicit form, which can be advantageous for certain applications. However, reward models lack the interpretability of natural language principles, which can be crucial for understanding and debugging model behaviour. ICAI's strength lies in its ability to provide human-readable explanations of preference patterns, which can be valuable for model transparency and user trust. We have extended the limitations section (Section 6) to discuss this trade-off, highlighting the interpretability of ICAI's principles compared to the implicit nature of reward models. We hope this clarifies the motivation behind our choice of method and baselines.

Q1: Rule-based reward models are proven to be quite useful for the safety aspect. Can the author compare the effects of the ICAI methods on different aspects? For example, helpful v.s. harmless?

Again, we would like to note that the "Rule Based Rewards for Language Model Safety" paper was first submitted to arXiv on November 2nd, roughly one month after the ICLR submission deadline, and could therefore not be considered in our initial submission. Nonetheless, we appreciate that you brought this work to our attention, as it indeed seems quite related. The authors of the RBR paper propose a method to learn an auxiliary safety reward model that composes natural-language "propositions" (generally binary statements about the response, e.g., "contains an apology") using a linear combination of these propositions as a reward signal. These propositions have some resemblance to our principles, but are hand-crafted and only used as features, not as direct preference indicators. Exploring a similar approach of propositions combined with a thin reward model could be an interesting alternative approach for the ICAI problem, also allowing for comparatively cheap fine-tuning with a manually modified constitution.

The authors of the RBR paper further express a preference for detailed rules over vague principles such as "prefer the helpful response" for steerability and interpretability reasons, which could be complemented by an ICAI-like approach to generate candidate rules in a data-driven way. We have added a discussion of this connection to Appendix B.3 of the paper, highlighting the potential interaction between the two methods. We hope this clarifies the relationship between our work and the Rule Based Rewards paper.

Additionally, we started to explore our interaction with helpful and harmful aspects on Anthropic's HH-RLHF dataset. We are still working on those experiments and plan to share them once they are completed.

Q2: Typos: line 1088: the word “is” is redundant.

Thank you for finding this typo! We have now fixed this prompt in our codebase but would not expect this redundant "is" to have affected our results in any notable way.

Thanks for your clarifications. But there are some misunderstandings.

Q1: In ICAI, the workflow is: (1) generate principles from the "train" data, (2) generate preference labels for a "test" sample according to the fixed principles. While the workflow of an elicitive approach is: (1) generate principles for a "test" sample, (2) generate preference labels for this "test" sample according to these generated principles (no "train" data).

Q2: A flipped LLM-as-a-judge means: If the LLM prefers one response, the preference label is "rejected"; if it rejects a response, the preference label is "preferred".

Q3: I am still curious about the reward model performance despite its lack of interpretability.

Q4: I am convinced. Thank you.

This paper presents interesting ideas, but it would be significantly improved by including more baselines. I am willing to increase my score to 6 if at least one of these baselines is thoroughly discussed. Furthermore, if all these baselines are incorporated and analyzed, I would consider increasing my score to 8.