Latent Principle Discovery for Language Model Self-Improvement

We would like to thank Reviewer 9mwi for taking the time to review our work and for their valuable feedback. We are encouraged that the reviewer finds our premise to be timely and important, appreciates the interpretability introduced through our clustering method, and the adaptability of our algorithm to various tasks and scoring mechanisms. We address the concerns raised in the review in our responses below:

1. Human annotations.

The authors claim that their approach aims to avoid conventional forms of human involvement, such as relying on “human annotations distinguishing between a chosen and rejected generation” or “human-curated constitutions.” However, their proposed method fundamentally depends on gold responses provided by users, another form of human involvement. Given this, the manuscript lacks a clear justification of why requiring gold responses from users is practically more efficient, scalable, or less burdensome compared to the forms of human involvement it aims to replace. Without this, the practical usage and broader applicability of the proposed method could be limited, as it merely substitutes one form of human dependence with another.

We would like to clarify that our method only requires human annotations that are already public, and widely available through the open-source datasets available on Hugging Face. Our method does not aim to replace preference annotations — in fact, we use the chosen response as the target ($y^G$) for the preference dataset HH-RLHF in our multi-task prompt mixture. In alignment tasks, we seek to reflect human generations and preferences, so we need a form of reference that is reliable — hence, a gold response for QA settings, or a chosen response in preference pairs. Thus, from our perspective, it is no more costly to use human-written gold responses than it is to use human preference data— it is as simple as downloading the data — our method could feasibly be run using purely preference data or only using gold responses, rather than a mixture of both. Our choice of mixture was to capture a diverse set of domains (safety, summarization, truthfulness, general purpose instruction-following, grounded QA, etc.). What we instead seek to replace is the process of distilling desirable attributes for responses (equivalently, the “lens” of preference) into a readable set that the model is simultaneously trained to reflect — which our method yields at the end of each iteration.

2. Comparison with Inverse Constitutional AI.

The positioning (Section 2) could be improved by explicitly discussing the similarities and differences between STaPLe and closely related approaches, especially Findeis et al. (2025). In my view, both works aim at deriving latent principles for aligning language models, but differ in their learning source (and of course, the methodology): STaPLe derives them from gold responses, whereas Findeis et al. derive them from pairwise comparisons. In addition, given the first weakness (reliance on gold responses), it would be helpful to empirically evaluate Findeis et al.’s comparison-based approach as an additional baseline. Such an evaluation would clarify the efficiency trade-offs and practical advantages of deriving latent principles from gold responses versus pairwise comparisons.

As the reviewer notes, STaPLe and ICAI (Findeis et al. 2025) broadly aim at eliciting principles from available human data (preference annotations) — however, as aforementioned, our method does not solely rely on access to gold responses, and can use preference annotations as well. ICAI uses the rejected (dispreferred) response, whereas STaPLe uses an initial response generated on-policy from the target LM as the baseline. Both methods also make use of clustering as a means to distill relevant principles down to an interpretable set. However, the primary aim of ICAI is to reconstruct preference annotations (in a single iteration) to verify the constitution’s correctness, so that they may be used to interpret human preferences at inference time. By contrast, STaPLe uses the principles as a learning signal over multiple iterations, to imbue a distribution of responses that reflect similar attributes as their gold references, and learns a self-correction behavior that intrinsically calls those principles. Another important difference is the principle generator — ICAI uses strong, frontier-level models (GPT-4o and GPT-3.5-turbo), whereas STaPLe uses on-policy-generated principles from the small LM (7-8B parameters) itself, without external guidance or supervision from any other model. STaPLe is also fully automated — with the Bayesian optimization framework, even the size of the constitution can be purely discovered from the data without human threshold-setting during hierarchical clustering — in contrast to ICAI, which performs k-means clustering specifically aiming for 5 clusters. Ultimately, while ICAI is designed with the purpose of better understanding human preferences (rather than improving the LM’s intrinsic alignment as reflected on MT-Bench/AlpacaEval), STaPLe teaches non-frontier LMs attributes that should be reflected, resulting in a better aligned language model. It is unclear how ICAI as a baseline would aid in improving response quality; one would need to perform a method like Self-Refine using the ICAI-generated principles, which would not be reflective of the ICAI method itself.

We appreciate the reviewer’s suggestion for more explicit positioning of our work with respect to other approaches — although PDF updates are not permitted during this rebuttal cycle, we would be happy to add such a discussion in our camera-ready version.

3. Representativeness of constitutions.

While the clustering mechanism in STaPLe reduces the number of unique principles, it is unclear whether the compressed principles effectively represent the original set. The analysis demonstrates compression but lacks a direct evaluation of representativeness, which is essential to confirm that the reduced set still captures all critical principles for improved interpretability.

This is a valuable point, and we find that measuring or evaluating information retention is a technical challenge in its own right. We designed the perplexity-based label replacement scheme in Appendix M.1 with this aim in mind — to replace individual principle instances with a representative (medoid induced by clustering) such that the difference in perplexity (“surprisal” effect) when replaced is minimized. To this effect, we find this perplexity-based scheme to yield largely comparable performance to just using the assigned medoids; we primarily report the latter for the sake of simplicity and efficiency. Additionally, in the aim of creating better clusters and therefore, better representative elements, our Bayesian optimization framework in Appendix M.3 addresses the tradeoff between diversity of the clusters with tightness -- ensuring the elements are similar in embedding space, in a purely automated manner. We would be happy to try other methods of measuring and optimizing for representativeness, if you have any in mind!

4. Clarity.

The method description relies heavily on numerous equations, which may be difficult for a broader audience to fully grasp. It would be beneficial to include an illustrative figure that shows the effect of each process, as the currently included figure only demonstrates the overall process flow and does not highlight the impact or outcome of each step in the method. (Very minor) It would improve readability if there were a direct comparison table showing performance based on different similarity scores. Currently, the tables for each similarity score are presented separately (one in the main text and another in the appendix), which makes it harder to quickly compare them. Combining this information in a single table could enhance clarity.

Thank you very much for the suggestions! As aforementioned, although the guidelines for this rebuttal period do not permit updating our paper with new figures/tables, we would be more than happy to make these changes for our camera-ready version. Specifically, we would be glad to illustrate the result of each step of the process, such that one can interpret how principles are discovered from the LM, refinements induced given the principles and critiques, the refinements filtered based on the scoring function $f(\cdot)$, the clustering mechanism being used to replace the principles in the trajectories with representative elements, and then performing fine-tuning on the augmented trajectories.

---

We hope these responses have addressed your concerns — if so, we would like to kindly ask you to further your support of our work. We would also be happy to address any further concerns you may have. Thank you again for your time!

We would like to thank Reviewer Hwww for taking the time to review our work and for their valuable feedback. We appreciate that the reviewer agrees with the conceptual novelty of our work, and values our method’s interpretability, full automation, and that it achieves consistent gains across models. We address the concerns and questions posed in the review through our responses below:

1. Filtering in self-refinement.

As the model sees the gold response only when it proposes a principle, and not during self-correction, the suspicion that the refined response $y^2$ copies the gold could only hold if the principle can transfer the full gold sequence. However, we instead find that the principle is fairly short and general in nature, and instead operates as a summary of the aspects contained in the gold response that are lacking in the model’s original response. This is in part due to the prompt for principle proposal asking the model to suggest an attribute of the gold response that would help guide the initial response to be more similar to the gold; this avoids the suggested copying effect. Furthermore, the fact that model performance improves on multiple unseen test datasets – where no hints are available at test time and hence copying via principles is not even possible – indicates that the model is actually learning to invoke principles for better refinement.

As for generalization, our clustering method is precisely towards this goal — rather than training the model to follow specific principles which may not be relevant in future iterations, clustering ensures that only the high-level attributes that the model should follow are trained upon, while simultaneously aiding interpretability for human overseers of the alignment process.

To affirm that our method produces refined responses that do not copy the gold, we compute the number of samples where the Rouge-L precision score exceeds 0.9, for each of the main experiments. This is an effective measure of the fraction of $y^2$ that appears verbatim in $y^G$:

We find the fraction of samples with a high precision to be less than 1.5% across all experiments, which is negligibly small.

To extend beyond Rouge as a measure of similarity, we also include an ablation in Appendix M.2 where we use a Phi-4 model as a judge to compare against the gold; we filter out all samples that do not improve in the judge’s score, and find this to perform comparably to our Rouge method. There are two reasons why the availability of a reference response is important in our setup:

1. Recently, the use of gold responses to compute verifiable rewards has become very successful in reasoning domains – our use of the existing gold response, along with a judge/score, should be seen as an extension of the notion of verifiable reward to non-reasoning domains. Furthermore, an LLM judge that has access to a gold response should intuitively be more accurate than a reference-free judge.
2. We also use the gold response as a hint to elicit a relevant principle -- given we have access to such a response, we do not want to unnecessarily limit the strength of the scoring method by not using it during judgment.

2. Scalability.

While we do not suggest that the principles are necessarily transferable across models — our proposed method’s success at automatically discovering principles extends across various models, and different model families altogether.

Nonetheless, based on your feedback, we run our STaPLe method for 3 iterations with the Qwen3 32B model, and report the results below, alongside a STaR baseline for this model. Note that we keep the thinking mode on for response generation, which results in an even stronger initial policy; nonetheless, self-refinement is performed on just the response, not the thinking tokens (the <think>...</think> block):

These results are encouraging, suggesting that even a 32B parameter reasoning model can self-improve its alignment by a substantial amount (+8.36% on AlpacaEval). STaPLe also still outperforms the STaR baseline at each iteration, If desired, we would be glad to run further experiments with models larger than the 7-8B parameter tier.

3.Retrieving principles.

Thank you for the reference to the MetaMind paper — we looked into this further and found that this paper was released on arXiv after the NeurIPS deadline, so we were not aware of this work. The framework of retrieving principles raises an interesting point — however, it assumes that one has access to a constitution or an a priori list of principles. It is also unclear what the criterion (or which model) would be responsible for selecting the top-k principles. By contrast, our work instead makes principle selection intrinsic to the model — it can use the principles generated in previous iterations, but can also create new principles for self-refinement as needed, if the principles it has learned do not apply. Nonetheless, we will certainly keep this point in mind for future work on principle-guided refinement and applications of the generated constitutions. We will also cite the MetaMind work along with a discussion of its similarities and differences from STaPLe in the final version of the paper.

4. Convergence over multiple iterations.

Given that the method of data curation for the STaR baseline relies on applying self-correction and training on the refined response, we end up with fairly similar volumes of data used for SFT for both the STaR and STaPLe methods, as shown in Table 2. While we do find that the scores of our STaPLe method does plateau sooner than STaR (in the fourth iteration), its scores are already substantially better than the STaR baseline for a similar number of samples.

5. Reinforcement learning comparison.

Indeed, one can view the function $f(\cdot)$ comparing a response against the reference as a reward function; in Appendix F, where we relate our framework to self-play in reinforcement learning, we define the self-play advantage as the difference in closeness to the reference, with the initial response’s score as a baseline. We ultimately show that our method is theoretically equivalent to self-play, as the MC-EM gradient induced by our framework and the self-play gradient using REINFORCE are equivalent. We also experiment other instantiations of $f(\cdot)$, specifically using a Phi-4 LLM-as-a-judge (in Appendix M.2). We leave further exploration with reinforcement learning algorithms for principle-conditioned self-correction and alignment to future work.

6. Interpretation of generated constitutions and generalization across tasks.

We note that STaPLe is a data-driven discovery process, so the set of principles we obtain is subject to the task / prompt distribution chosen. We have a multi-task mixture consisting of samples spanning safety, summarization, knowledge-based QA, general purpose instruction-following, truthfulness, and helpfulness. As such, our results demonstrate that our method can obtain principles across various use cases, while clustering again helps to distill the core aspects the model has been trained to reflect. Naturally, expanding the size of the prompt mixture used for principle discovery can aid in further generalization; nonetheless, the constitutions induced by smaller LMs under our current mixture already illustrate generalizable insights.

In Appendix K, we include an analysis of the number of samples which have the respective medoid elements in the constitution during iteration 4 SFT — that is, the data-driven “weightage” of each principle during its last iteration of training. For further analysis, we include below a breakdown of the Llama 8B constitution in terms of the high-level domains each element of the constitution belongs to, as desired. We categorize them as “helpfulness”, “relevance”, “style”, “safety”, and “truthfulness and honesty”, and find that the categories are nearly evenly balanced.

• Helpfulness: 6 principles (e.g. "Directness and Assertiveness")
• Style: 5 principles (e.g. "Structure and Organization")
• Relevance: 6 principles (e.g. "Minimize Unnecessary Information")
• Safety: 6 principles (e.g. "Avoiding Harm and Sensitivity")
• Truthfulness and Honesty: 6 principles ("Acknowledge and Address Previous Misconceptions Clearly")

---

We hope these responses have addressed your concerns — if so, we would like to kindly ask you to further your support of our work. We would also be happy to address any further concerns you may have. Thank you again for your time!

We would like to thank Reviewer pKck for taking the time to review our work and for their valuable feedback. We appreciate that the reviewer finds the mathematical framework underlying our approach to “add technical depth”, and that the experimental section is “relatively thorough”. We address the concerns stated in the review with our responses below:

1. Copying in Self-Refinement.

The model gets a hint by seeing the correct answer while generating principles in the propised method. What if this risks making the model dependent on that answer that it might just learn to copy parts of it rather than actually reason towards it. Also, the authors say that they filter out poor candidates, but they don’t hsow how often the golden answer leaks into the principles of how much this affect the performance.

We would like to clarify that the model only sees the gold response when proposing a principle, not during self-refinement. As noted in the footnote on page 4, we find that the principle generated never looks like the gold response itself, in part because the prompt to propose a principle asks the model to suggest an attribute of the gold response that would make the initial response more similar to this gold target.

In practice, smaller / weaker language models are not very successful at self-refinement without targeted dimensions of refinement [1,2], necessitating our filtering of all refinements that did not improve over the initial response. Note that filtering is performed on the refined responses induced by the principle and its critique, not directly on the principles themselves. Below, to confirm that $y^2$ does not copy $y^G$, we compute the Rouge-L precision, and report the number of samples that have a score above 0.9 — we find this number to be very small (less than 1.5% of the data), which we find to be negligible. We would like to point out that high overlap with the gold data is not an undesirable property -- after all, the gold responses are from license-friendly train sets that one would ordinarily perform SFT with -- however, we find on-policy generations to be effective.

Since all samples in “total samples” are those where the refined generation improved over the initial response in Rouge-L F1, and the fraction of samples where the precision increased to above 0.9 did not increase substantially, it must be that the recall improved. That is, while the rate of copying does not grow much, the generated responses better cover the gold -- this is necessary for better reflecting the core elements of the gold response for the model to imitate with its on-policy generations. In works such as STaR [3], a chain-of-thought is induced that reconstructs the original gold answer — our work achieves a similar purpose in learning a distribution of on-policy generations similar to the gold, rather than overfitting to the gold.

[1] Madaan et al., Self-Refine: Iterative Refinement with Self-Feedback, 2023, Thirty-seventh Conference on Neural Information Processing Systems

[2] Ramji et al., Self-Refinement of Language Models from External Proxy Metrics Feedback, 2024, https://arxiv.org/abs/2403.00827

[3] Zelikman et al., STaR: Bootstrapping Reasoning with Reasoning, 2022, Advances in Neural Information Processing Systems

2. EM Scoring Clarification.

The paper frames the method as EM-learning but the scoring presented in the paper doesn’t seem trainable. Can you clarify on this?

That is partially true – the distribution over latent variables (the principle and the model response given the principle) is trainable (and we train over this), but the distribution over observed gold given the generated response is fixed and is defined via a similarity scoring function. However, this does not interfere with the EM framing of the method. The E-Step uses both distributions to generate samples from the posterior via rejection sampling, thus giving an approximate expectation over latent trajectories, while the M-Step updates just the trainable distribution to maximize the likelihood of these trajectories.

3. Clustering in the EM Framework.

the clustering step is happening after principles are generated and filtered which means, its not part of the training process if I understand it correct. Also the clustering threshold is tuned using another optimizer? Can you explain?

Clustering is performed after the E-step (data generation phase) but before the M-step (model training phase), as a data augmentation procedure. A hierarchical clustering model is trained on the embeddings of the principles generated in the E-step, and the cluster representatives are chosen according to the schemes outlined in Appendix M.1 (primarily, using the cluster medoid). The self-correction trajectories generated in the E-step are augmented by replacing the principle with the cluster representative of the cluster it belongs to. The resulting trajectories (using the clustered labels) are then used for SFT training in the M-step.

The results in the main paper use thresholds that we (the authors) set. However, to automate this process, we also describe and implement a Bayesian optimization procedure in Appendix M.3, which optimizes for balancing diversity across the clusters with tightness within the cluster. This method searches over threshold values, fits an Agglomerative Clustering model based on the chosen threshold for each point, and calculates this diversity-tightness objective function based on the resulting clustering, repeating iteratively by choosing next threshold value using Gaussian Process regression. In fact, we find this to perform slightly better on ablations performed on the Llama-8B and Granite-8B models.

4. Baselines.

The paper compares against basic self-refinement and instruction tuned models, but no results on newer methods like RISE.

The baselines chosen are prominent methods in self-refinement and multi-iteration self-improvement, specifically those with code that is made available open-source (e.g. Self-Refine, STaR). Notably, while some recent methods like RISE and SCoRe share a broad focus on self-improvement and self-correction, neither has made their code available for reproduction, and both methods are non-trivial to implement. We consider a similar set of baselines as RISE; given our work’s premise of on-policy chain-of-thought generation given the gold response, akin to STaR, we include a STaR-like baseline.

5. Errors and Refinement Quality.

Do you have experiments on how the model generated principles could introduce biases and errors?

While it was rare for intrinsic principle-based refinement to hurt response quality over the initial response (“errors”), we did find that the initial response generated by the model during self-correction became slightly worse in comparison to responses sampled from the base policy, on the basis of our Prometheus evaluation on IFEval (no degradation would be around 50%, subject to slight variance). We report this behavior in the table below:

As such, comparing the refined response against generations from the base policy — rather than the initial response generated from the model during inference — was important to confirm that the quality of the refined responses generated over the iterations does indeed improve, as reflected in Tables 2 and 3. Given the refined response is what a user would follow and trust (as in recent reasoning works following a token like “Wait” [4], where our principles serve as a similar delimiter), we focus on evaluating the refinement in this work. We also compare the refined responses across iterations (“stepwise”) in Table 4, which further confirms that the refined responses do become better, before saturating, as evidenced in Figure 1 and Table 1.

[4] Muennighoff et al., s1: Simple test-time scaling, 2025, https://arxiv.org/abs/2501.19393

---

We hope these responses have addressed your concerns — we would be happy to address any further concerns you may have. Thank you again for your time!

We would like to thank Reviewer u4GT for taking the time to review our work and for their valuable feedback. We address the concerns raised in the review through our responses below:

1. EM Approach in STaPLe Algorithm.

The paper is posing things as EM/posterior estimation but approximating or choosing empirical setup, the connection is not very relevant. The EM is sparsely related because its repeatedly fine-tune (maximizing step to generate critiques and responses for fine-tuning, and then the learning step, to SFT optimize with it). Figure 2.

We acknowledge that our implementation approximates the EM algorithm, since exact computation of posterior marginals over latent variables is intractable in our case; However, our setup is close to what is referred to as Monte Carlo EM [1][2] – where these marginals are replaced with sample based expectations i.e. a set of latent variables, sampled from posterior, is used to optimize the gradients in the M-Step; In particular, we sample 16 latent [principles, response] pairs via rejection sampling, given current model parameters and gold responses and then maximize the likelihood over these samples.

[1] William Ruth, A review of Monte Carlo-based versions of the EM algorithm, 2024 https://arxiv.org/abs/2401.00945

[2] Greg C. G. Wei and Martin A. Tanner. A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms. Journal of the American Statistical Association, 85(411), 1990.

2, SFT Baseline.

The baseline is considered with fine-tuning on 50k gold responses but the proposed method uses the 50k with 4 additional iterations each in which 10k gold responses are used to generate critiques and responses to fine-tune with. A fair baseline would be to use all the 50k + 40k input-gold response pairs in SFT which is not considered in the paper.

Thank you for this suggestion! In addition to the baseline on 50k samples we report in Table 1 (“Gold-only SFT”), here we train all 3 models on 90k samples as proposed, and compare the result against the Iteration 4 scores for STaR and STaPLe {unconstrained, constrained} in the table below:

While SFT on 90k samples does improve slightly over the SFT on 50k samples (the results included in Table 1), it is still behind the STaR iteration 4 results, and as such, lags further behind STaPLe in both the unconstrained and constrained settings.

3. Learning Principles from Human Data.

Setup does not seem well motivated: The setup is not very useful - Why would we use Gold supervision to explicitly generate critiques vs just using SFT with them and implicitly learning it? Other than interpretability of the clustered generated set of principles, there is no reason it is not already implicit in the learning. Also, using gold annotations is more expensive - RLHF with preference annotation would be cheaper to collect. It would align the model without explicit principles. The explicit principles do not seem very useful. The paper also does not compare with RLHF.

The results of SFT training on gold-only 50k samples reported in Table 1 in the paper, and on 90k samples in the table above, are clear indicators that our STaPLe method, which relies on learning explicit verbalization of principles and refinement, greatly improves performance over simple SFT on gold responses.

As for the use of gold annotations, we do not need any new annotations for training, we use [query, response] pairs from existing open source datasets, of which there is an abundance; in STaPLe we provide a way to make the most out of those existing datasets (beyond simple SFT) without any additional annotations, with the added benefit of automatically curating a data-dependent constitution.

While we train our model via an EM-based SFT framework, we agree that RL methods like PPO or GRPO could be used with the similarity score/judgment as a reward signal. In fact, in Appendix F, we show that our method is equivalent to self-play reinforcement learning by defining the advantage in terms of the difference in score against the gold response; the MC-EM gradient obtained by STaPLe is equivalent to the REINFORCE gradient for self-play under this definition. However, we view our work as a cold-start phase for such a training and we leave purely online RL training as future work. We also hypothesize that without a cold-start phase, where principles are elicited using hints and are then inserted into an un-hinted trajectory for training, the model would have taken too long (or forever) to arrive at the point of generating good/relevant principles.

We do not compare against existing RLHF works on constitution learning due to clear differences in setup and goals. A key difference is that we do not prescribe any principles a priori. The primary efficiency from a labor standpoint is not just in annotation, but also in curating a constitution. Our method serves as a means to identify qualities a model should reflect in its generations; the constitutions we report in the appendix should be viewed as such a list, based on the domains and tasks included in the data mixture.

Training models to follow known principles — desirable attributes for a model’s responses to reflect — is a core goal of alignment research [3, 4] at present. Our goal is to both discover such principles from the model, and train the model to follow them — implicitly learning them from gold annotations or preference data does not accomplish this task as we do not have annotations indicating why the gold/preferred response is desirable [5]. This motivates us to elicit principles from the data to enable human oversight of this alignment process.

[3] Bai et al., Constitutional AI: Harmlessness from AI Feedback, 2022, https://arxiv.org/abs/2212.08073

[4] Guan et al., Deliberative Alignment: Reasoning Enables Safer Language Models, 2025, https://arxiv.org/abs/2412.16339

[5] Findeis et al., Inverse Constitutional AI: Compressing Preferences into Principles, 2025, The Thirteenth International Conference on Learning Representations.

---

We hope these responses have addressed your concerns — if so, we would like to respectfully ask you to reconsider your assessment. We would also be happy to address any further concerns you may have. Thank you again for your time!