Is Free Self-Alignment Possible?

Thank you for noting the scalability, efficacy, and data efficiency of our approach!

• On prompts used for self-generated data (W1). We provide prompt details in Appendix A.4.1 and reference these in Section 2.1 (line 139) of our paper. For all experiments presented in the main paper, we consistently use this same set of prompts to generate synthetic data.

• On Figures 3 and 4 (W1).

  • For Figure 3: As described in Section 4.2 of the main paper, we use the default training split of UltraFeedback-binarized [1, 2] to train the DPO model and evaluate it on the default test split (long-form generation slice). To address the reviewer's concern, we will add this information in the Figure 3 caption in the corrected version of our manuscript.
  • For Figure 4: Thank you for pointing this out! The figure reports the average performance across the five datasets listed in Table 1. We will update our manuscript to include this clarification.

• On the dependence on the base model (W2). As stated in the Introduction section of our manuscript, AlignEZ is specifically designed to leverage alignment signals from noisy, self-generated data produced by the base model. It does this by selectively using preference data from points identified as similar in the latent space. The core intuition is that points within a localized region of the embedding space tend to exhibit similar properties [3]. By focusing only on preference data from these similar points, AlignEZ avoids noise from unrelated data and ensures the use of only relevant and important characteristics for alignment.

  Existing instruction-tuned models are often trained on synthetic, noisy instruction pairs [4, 5], meaning the data they generate may also contain noise. Moreover, prior work has highlighted the risks of model collapse when synthetic data is overused [5-10]. AlignEZ addresses this issue with a test sample-specific approach that targets alignment by selecting only the most relevant synthetic data points. This strategy not only reduces the risks associated with synthetic data overuse but also ensures that alignment remains effective, even in challenging, data-constrained environments.

• On OOD scenarios (W3). The purpose of AlignEZ is to be an inexpensive replacement for alignment techniques. Addressing OOD concerns for alignment techniques is an orthogonal problem that applies to standard alignment methods as well as ours. In fact, it is still very much an open problem, even for well-established approaches like PPO-based methods (e.g., RLHF) and DPO [11, 12]. Despite their effectiveness in many settings, these methods struggle to generalize to OOD scenarios.

• On helpfulness and safety aspects (W4). As suggested by the reviewer, we compare AlignEZ with the baselines on 100 random samples of HH-RLHF; which consist both helpfulness and safety aspects. Similarly to the setup in the main paper, we use ground truth data for CAA and ITI, and use synthetic data for AlignEZ. We report the average Net Win ($\Delta \%$) across 3 random seeds:

  Model	Ours	ITI	CAA
  Llama3.1	7.67%	-2%	0.6%
  Mistral3	14%	2.33%	6.67%

The result above shows the effectiveness of AlignEZ on alignment tasks with multiple dimensions (safety and helpfulness).

• On dependence on LLM's instruction-following ability (W5). Our findings indicate that AlignEZ works on base models---and does not require instruction-tuned models, as shown by the significant alignment gain we show in Table 1 of out manuscript.

• On how the improvement average is calculated. This is calculated by taking an average of all the Relative Improvement (RI) numbers (right-most column of Table 1).

• On implementation. As stated in the previous response, the prompts used for synthetic data generation are provided in Appendix A.4.1. We also provided our code as a .py file in the supplementary material provided in the initial submission.

Thanks for the clarifications. The authors have addressed my concerns. I have changed my rating accordingly.

Thank you for highlighting the effectiveness and practicality of our test-time alignment method!

• On comparison with baselines. Thank you for pointing this out! As suggested by the reviewer, we compare AlignEZ with the baselines on 100 random samples of HH-RLHF. Similarly as the setup in the main paper, we use ground truth data for CAA and ITI, and use synthetic data for AlignEZ. We report the average Net Win ($\Delta \%$) across 3 random seeds:

The table above shows that AlignEZ's performance gain over the baseline persists even when the baselines use ground truth data form the same distribution as the test samples.

• On inference latency and memory utilization. Thank you for bringing up this point! We provide overhead comparison with baselines as follows:

Comparison with DPO (low data regime)

For DPO, we measured the total time required for both training and inference. For AlignEZ, we measured the end-to-end time, including synthetic data generation, alignment direction identification, and inference. Our goal was to equalize the amount of time taken in order to fairly compare performance across these methods. We obtained

The results highlight that indeed AlignEZ with latency comparable to DPO in the low-data regime delivers significantly better performance in these scenarios, as shown in Figure 1 of the main paper. We note that our setup is actually highly favorable to DPO, as it does not factor in the additional cost of obtaining the ground-truth data required by DPO. If were were to factor this in, AlignEZ would achieve further superiority.

Comparison with embedding editing baselines AlignEZ incurs similar overhead as baseline representation editing methods (CAA and ITI), detailed as follows:

1. Embedding modification:

   • CAA and ITI modify embeddings by performing scalar multiplication followed by the addition of a fixed vector to the model's activations.
   • AlignEZ modifies embeddings by performing a dot product followed by two additions (removing harmful vectors and adding helpful vectors).
   • CAA, ITI, and AlignEZ's latency is $O(d)$ where $d$ is the embedding dimension.

2. Alignment direction identification:

   • CAA uses PCA for this step, while AlignEZ uses SVD, resulting in comparable latency costs.
   • ITI, however, trains $H$ classifiers ($H$ is the number of attention heads), which incurs the highest latency cost among the methods for this step.

3. AlignEZ has an extra step to generate synthetic data and find the nearest points for each test sample using kNN (done only once in the beginning). kNN incurs $O(nd)$ cost, with $n$ the test sample size and $d$ the embedding dimension. For synthetic data generation, with fast inference library such as vLLM (https://github.com/vllm-project/vllm), generating data for 100 samples only takes 30 seconds on an A100 GPU. It is worth noting that we did not account for the cost of obtaining ground-truth data required by ITI and CAA---but we do for AlignEZ, suggesting that when accounting for data collection complexity, AlignEZ would have an even better relative performance.

• On effect on safety and hallucination. As suggested by the reviewer, we perform an experiment to test AlignEZ's impact on safety and hallucination.

  Safety. We tested AlignEZ on two safety datasets, namely MaliciousInstruct [1] and JailBreakBench [2], and report the Net Win ($\Delta \%$) below:

  Model	MaliciousInstruct	JailbreakBench
  Llama3.1	3%	6%
  Mistral3	1%	-3%

  The results show that AlignEZ provides a modest safety improvement for Llama 3.1 and has minimal impact on safety for Mistral 3. This indicates that AlignEZ does not negatively affect safety and may even present opportunities for developing specialized versions tailored for safety-critical applications.

  Hallucination. We conducted the FActScore test [3], an evaluation method for assessing the degree of hallucination in LLM-generated responses. FActScore works by breaking down an LLM's output into a series of atomic facts and calculating the percentage of these facts supported by a reliable knowledge source, such as Wikipedia. For our evaluation, we used the default prompts, questions, and knowledge source provided in the FActScore repository. The scores range from 0 to 1, where a higher score indicates a less hallucinated response.

  Model	Base Model	Base Model + AlignEZ
  Llama3.1	0.444	0.436
  Mistral3	0.458	0.452

  The results show that AlignEZ has little to no effect to the original model's degree of hallucination, maintaining its factual accuracy.

Thank you for noting the resource efficiency and strong performance demonstrated by our approach!

• On contribution. While synthetic data generation and representation editing are well-established methods, their combination as well as their use in alignment has not, to our knowledge, been explored. Our baseline results show that current SOTA representation engineering methods fail to improve alignment because alignment requires adapting to nuanced and loosely defined knowledge, unlike tasks such as improving truthfulness (e.g., ITI) or adopting linguistic styles (e.g., CAA). AlignEZ bridges this gap by introducing a novel strategy that selectively uses synthetic data points identified as similar in the latent space, enabling precise, test sample-specific alignment while filtering out noise from unrelated data points.

  In Figure 1 of our manuscript, we demonstrate that AlignEZ outperforms traditional alignment methods like DPO, particularly in low-data scenarios. This reflects real-world conditions, especially when synthetic data is used—a setup that has been gaining traction recently [1, 2, 5]. While prior work highlights the risks of model collapse when synthetic data is overused [3, 4, 5], AlignEZ tackles this challenge by introducing a test sample-specific approach that targets alignment by selecting only the most relevant synthetic data points. This strategy not only mitigates the risks of model collapse in synthetic data usage but also ensures effective alignment in challenging, data-constrained conditions.

• On 'free' claim and title. Thank you for the thoughtful suggestions! We agree with the reviewer that our method does incur a non-zero cost. However, we want to point out that this cost is significantly lower compared to traditional alignment methods (e.g., RLHF and DPO). Unlike DPO, we do not need to spend any time or cost acquiring data, and we do not run any SGD iterations for fine-tuning.

  The primary cost associated with AlignEZ arises from generating synthetic data, which is relatively inexpensive. Hosting the model locally eliminates API call expenses, and inference speed can be further improved using tools like vLLM (https://github.com/vllm-project/vllm). To address the concern about terminology, we are happy to revise the title to use "inexpensive" instead of "free."

• On theoretical analysis. The assumption that LLM representations can be decomposed into latent concepts is widely adopted and has been supported by prior work across diverse contexts, e.g., [8-13]. These studies validate the practical utility of such assumptions in analyzing and leveraging model behavior. While we adopt the orthonormality assumption for clarity in explanation and derivation, our method is not reliant on it. Specifically, if the concept vectors are not orthonormal, the analysis can proceed by representing them under a change of basis, ensuring the results remain valid. Regarding the conclusion of the theorem, we clarify that it is encapsulated by the condition $\sigma_{\text{linguistic}}=C \cfrac{\max_{q \in \text{k-NN}(q_x)}d(q_x, q)}{\sqrt{k}}$. This implies that increasing k can be effective if all of k-NN examples are sufficiently close to the query, leading to a decrease in the term. This theoretical insight aligns with empirical results, as demonstrated in Figure 4(a).

• On evaluation on larger models. We would like to point out that applicability to proprietary models is not a limitation specific to AlignEZ. In fact, all alignment methods require access to model weights to implement alignment effectively. AlignEZ imposes no additional requirements beyond this standard access.

  To address the reviewer's suggestion, we have evaluated AlignEZ on a larger open-source model, Llama 3.1 70B, to further demonstrate its applicability and scalability. On the oasst dataset, AlignEZ provides a Net Win ($\Delta \%$) of 4%, illustrating that the performance gain persist for larger models.

• On head to head comparison with prompting technique. As suggested by the reviewer, we conducted a head-to-head comparison between URIAL [6], a prompting-based method, and AlignEZ. Since URIAL was specifically optimized for the just-eval dataset [6] used in our main paper, we ensured a fair comparison by evaluating both methods on 100 randomly selected samples from HH-RLHF, with results averaged across three random seeds. We report the Net Win ($\Delta \%$) = Win%-Lose% for AlignEZ below

The positive Net Win scores highlight AlignEZ's effectiveness and superiority compared to URIAL. Prompting methods like URIAL are compute-efficient and compatible with proprietary models but come with notable drawbacks, such as a reliance on significant human effort to craft optimized prompts [6]. These methods also incur additional costs from context window usage and extra tokens. AlignEZ addresses these challenges through representation engineering, which eliminates the need for context window overhead and enables the use of generic prompts (as detailed in Appendix A.4.1 of our manuscript). We have added this result in Appendix E in our updated manuscript.

• On $\theta^{help}$ and $\theta^{harm}$. Thank you for your suggestion! We have added $\theta^{help}$ and $\theta^{harm}$ visualizations as Appendix D in our updated manuscript. The visualizations shows that $\theta^{help}$ and $\theta^{harm}$ form distinct and separable clusters even in this low-dimensional representation (2 dimension PCA).

• On reward-model test-time alignment methods. While reward-model-guided methods are compute-efficient at test time, we argue that they incur higher costs during training. Training a reward model requires access to human-annotated preference data and fine-tuning compute, making it computationally prohibitive when alignment must adapt to changing or evolving preferences.

  Additionally, we argue that AlignEZ offers greater controllability and interpretability compared to reward-model-based methods. With AlignEZ, users can directly prompt for synthetic data (as detailed in Appendix A.4.1), allowing for easy customization. In contrast, reward-model-based methods require access to the reward model's training data to enable any degree of interpretability or control [7].

[1] Kazdan, J., Schaeffer, R., Dey, A., Gerstgrasser, M., Rafailov, R., Donoho, D. L., & Koyejo, S. (2024). Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World. arXiv preprint arXiv:2410.16713.

[2] Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., ... & Koyejo, S. (2024). Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413.

[3] Taori, R., & Hashimoto, T. (2023, July). Data feedback loops: Model-driven amplification of dataset biases. In International Conference on Machine Learning (pp. 33883-33920). PMLR.

[4] Veprikov, A., Afanasiev, A., & Khritankov, A. (2024). A Mathematical Model of the Hidden Feedback Loop Effect in Machine Learning Systems. arXiv preprint arXiv:2405.02726.

[5] Seddik, M. E. A., Chen, S. W., Hayou, S., Youssef, P., & Debbah, M. (2024). How bad is training on synthetic data? a statistical analysis of language model collapse. arXiv preprint arXiv:2404.05090.

[6] Lin, B. Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K., ... & Choi, Y. (2023, December). The unlocking spell on base llms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations.

[7] Carroll, M., Foote, D., Siththaranjan, A., Russell, S., & Dragan, A. AI alignment with changing and influenceable reward functions. 2024. URl: https://arxiv.org/abs/2405.17713.

[8] Dev, Sunipa, et al. "OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings." EMNLP 2021.

[9] Dalvi, Fahim, et al. "Discovering Latent Concepts Learned in BERT." ICLR 2022.

[10] Trager, Matthew, et al. "Linear spaces of meanings: compositional structures in vision-language models." CVPR 2023.

[11] Chuang, Ching-Yao, et al. "Debiasing vision-language models via biased prompts." arXiv 2023.

[12] Park, Kiho, Yo Joong Choe, and Victor Veitch. "The Linear Representation Hypothesis and the Geometry of Large Language Models." ICML 2024.

[13] Jiang, Yibo, Bryon Aragam, and Victor Veitch. "Uncovering Meanings of Embeddings via Partial Orthogonality." NeurIPS 2023.

Dear Reviewer,

We thank you again for your feedback, questions, and suggestions! We believe we have answered all of your questions in our responses and the updated draft. If you have additional questions, we would love to answer them!

The Authors

Dear Reviewer,

Thank you again for your feedback and insightful questions. During the rebuttal period, we revisited your suggestions regarding the potential for adding more novelty axes, such as adaptive mechanisms for refined, context-dependent alignment adjustments (W1).

We focus on the latter idea, as AlignEZ incorporates a context-dependent mechanism. The basic idea for context-dependent alignment is demonstrated in our two-stage prompting process, as detailed in Section 2.1 and illustrated in Figure 2 of the manuscript.

1. Stage 1: The LLM is prompted to identify characteristics of helpful and unhelpful responses tailored to the specific context-dependent test query. This step dynamically adapts to the query, ensuring that the derived characteristics are contextually relevant.
2. Stage 2: Using the characteristics generated in Stage 1, the LLM is prompted to produce a response to the test query. This results in preference samples that are explicitly aligned to the context of each question.

To test the potential of AlignEZ for context-dependent alignment, we conducted additional experiments on the just-eval-instruct dataset [6], splitting the evaluation by task/context type. Below, we present the Net Win (Δ) results for these task-specific splits.

These results demonstrate that AlignEZ can effectively perform context-dependent alignment, achieving Net Win (Δ) improvements for several task-specific and challenging scenarios like Coding and Math. In the case of Llama 3.1, the mechanism also enhances performance on reasoning tasks.

We appreciate the suggestion and hope this clarifies and strengthens our response.