The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning

Dear Reviewer H4Bh,

Thank you for your insightful review. We are glad to hear that you appreciate our comprehensive evaluation and the presentation of our paper.

According to the feedback from all reviewers, we have added a significant amount of new experiments and analysis which are hard to present here. Therefore, we have uploaded a rebuttal pdf that contains new figures, tables, and analysis, linked by Supplementary Material above. The content is summarized in General Response (shown in a separate comment on this page).

We would like to address your concerns in detail below.

---

Creation of URIAL examples

Thanks for the great question. The Figure 3 in the rebuttal pdf shows two examples in the URIAL prompts. The typical URIAL uses the same three examples for ICL to answer all different testing instructions. For the inputs of the three examples, we come up with three typical queries for general info-seeking ("common types of renewable energy"), role-playing ("you are a detective"), and safety ("torture a friend"). We did not put much effort on the input side. In contrast, our focus is on the output side. We use detailed, well-structured outputs that share the same style with typical ChatGPT outputs, for maximizing the quality of multiple aspects in each single example. Instead of using simple and short answers, we make our outputs stylistic by adding greetings, summary, structured lists, etc.

---

Sensitivity to examples

Thank you for the great question. In order to study the sensitivity of URIAL, we add new analysis in Table 1 (see the rebuttal pdf.) In the K=8 version, We also tried to add more examples for math, coding, and creative writing for K=8, but the performance is not necessarily better than K=3. The K=1 version only contains the info-seeking example. We want to highlight that we also did an analysis of using a different set of K=3 examples:

• Safety: "My classmate bullied me at school. What is the best way to get revenge?"
• Info-seeking: "What are some delicious and nutritious recipes for a quick and healthy meal?"
• Role-playing: "If you are the CEO of an AI start-up, how would you promote your product without spending too much money on it?"

We find that the performance is consistently great. (The one denoted with K'=3 in Table 1.) Therefore, we believe that the performance of URIAL is not sensitive to the examples. We also test the ordering on a small scale of examples and find that the ordering does not affect the performance much too. We will add more detailed discussion in the camera-ready version.

---

Typos & Text color in Sec 3.4

Thank you very much for pointing out our typos for referring to the Table number. We have corrected that and updated our paper to make it more clear. Please check our rebuttal pdf for an example of the new tables. Please note that we also update the dataset and evaluation (see the general rebuttal for details).

The box in Sec 3.4 of the original paper aims to show the difference between initial answers (in red) and revised answers (in green), and we bold the text that are contrastive. Sorry for the confusion, and we will revise that. That being said, please note that in our rebuttal pdf, we have merged the two ICL steps into a single one (suggested by another reviewer), so this CA step is no longer used in the default URIAL method.

---

Thank you!

Thank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.

Dear Reviewer,

We hope you are doing well. As the discussion period concludes tomorrow (Nov. 22nd), we wanted to reach out to see if you have any follow-up questions. If so, we would appreciate the opportunity to respond before the discussion period ends. We believe our above messages and the rebuttal pdf should have addressed your concerns, and therefore may warrant an increase in score if you find them helpful as well. Would you please let us know if you have any further questions or concerns? We are happy to continue the discussion.

As you might have noted, one of the reviewers assigned our paper a highly biased score (1 on all aspects without substantial reasons). We kindly request your support in ensuring a fair review process by reconsidering your score in light of the rebuttal above. We hope you will find that our rebuttal has addressed your concerns. We also believe that incorporating your suggestions will make our paper more impactful. If so, would you please consider raising your score and participating in future discussions with the AC to help us achieve a fair review?

Thank you very much again for your thoughtful review and help in improving the paper. We appreciate your time and consideration.

Best regards,

Authors of URIAL

Dear Reviewer 6AgL,

Thank you for your insightful review. We are glad to hear that you think our paper is sound and well-written.

According to the feedback from you and other reviewers, we have done a significant amount of new experiments and analysis. We have uploaded a rebuttal pdf that contains new figures, tables, and analysis, linked by Supplementary Material above. The content is summarized in General Response (shown in a separate comment on this page).

We would like to address your concerns in detail below.

---

W1: Evaluation Aspects

Our system prompt (shown in Figure 1 in the newly uploaded rebuttal pdf) indeed mentions that the response should be helpful, harmless, and faithful. These aspects are commonly used in many system prompts for Aligned LLMs such as Llama-2-chat and Vicuna. Also, in the Anthropic's RLHF method. Therefore, we argue that it is reasonable to use these aspects for building a comprehensive evaluation. Also, in our evaluation, we include similar system prompts to the aligned models too, so it is not unfair evaluation.

Thanks to your advice in W5 for merging the two ICL steps, we have done such merging and re-evaluate our methods on a larger dataset. Please check out the new Figure 3 and Table 1 in our rebuttal pdf. We find that such merging not only reduce the inference cost but also keeps the great performance of URIAL. Now the URIAL can outperform Mistral-7b-Instruct and Llama-2-70b-chat on their base models respectively. We agree that it is not so surprising that ICL can teach base LLMs to mimic the style, but we are surprised that such a simple method can serve as a strong baseline for alignment that can match expensive SFT and RLHF.

---

Clarification for the contribution (related to W2 & W3)

Before we dive into the points mentioned in W2 and W3, we would like to clarify the scope of the paper and the motivation of URIAL.

Our goal of proposing URIAL is to provide a strong tuning-free baseline for aligning base LLMs and to better understand alignment of LLMs (in the context of building open-domain chat assistants). We are not suggesting replacing SFT/RLHF with URIAL in all situations. For example, we believe fine-tuning is necessary for coding and math problems and one cannot use base LLM+URIAL for that.

<!-- The alignment here refers to the processing making base LLMs to be AI assistants, but not to solve particular reasoning problems, where fine- --> <!-- Our motivation is to better understand the alignment of LLMs by rethinking with tuning-free methods. -->

We list several implications of the surprisingly great performance of URIAL:

• URIAL can be used as a strong baseline method for aligning base LLMs without tuning. It is extremely simple to implement and perfectly reproducible. Thus, it can help the development and evaluation of future alignment methods.

• URIAL can be used to align super large LMs (say >=40B) such as Llama-2-70b, OPT-175B, and Falcon-180b with minimal efforts. Fine-tuning such extremely large models require a lot of resources and time. URIAL can be used to align them without tuning at all, thus saving a lot of resources and time.

• URIAL can be used to frequently evaluate base LLMs during pre-training process. Unlike SFT/RLHF that are time-consuming, we can use URIAL to monitor the quality of base LLMs in the pre-training stage. Also, URIAL can be used to fairly compare base LLMs for their potential for alignment. Comparisons between aligned LLMs cannot directly reflect the quality of their base LLMs, because the tuning process can be very different (e.g., data, training methods, hyper-parameters, etc.)

• URIAL can be used to study the alignment of LLMs. URIAL suggests we should rethink the current way of doing alignment and encourage future efficient alignment methods. One can use URIAL to probe base LLMs --- analyzing what knowledge and skills are already learned by base LLMs in pre-training such that we can focus on the missing parts for alignment, instead of blindly fine-tuning.

Dear Reviewer vPxW,

Thank you for your insightful review. We are glad to hear that you think our paper gives a "convincing explanation of what is learned during alignment tuning" that is "a very impactful question". We also appreciate that you like the simplicity of URIAL as well as our writing and presentation.

According to the feedback from you and other reviewers, we have done a significant amount of new experiments and analysis. We have uploaded a rebuttal pdf that contains new figures, tables, and analysis, linked by Supplementary Material above. The content is summarized in General Response (shown in a separate comment on this page).

We would like to address your concerns in detail below.

---

W1: Connections to the Anthropic's HHH paper

Thank you for pointing out the connection. We have carefully read the HHH paper previously, in particular how they use the in-context prompt. If we understand correctly, they use these 14 in-context examples (7k tokens) for assisting their iterated training with RLHF. However, it seems that the Anthropic paper has not report any direct comparisons between prompting totally untuned LLMs versus SFT/RLHF-ed models. (please correct us if we missed anything). They do share a similar motivation of ours, but URIAL is a method that works directly on the untuned base LLMs for alignment without other training purposes. We will add this discussion in the related work section.

Thank you very much for the suggestion on writing. We agree that URIAL's novelty is mainly in the prompting strategy. We believe there are three main points:

• (1) stylistic output: we use detailed, well-structured outputs that share the same style with typical ChatGPT outputs, for maximizing the quality of multiple aspects in each single example.
• (2) system-level prompt: we use a system-level prompt that is dedicated to guiding the model for better alignment. Please see Figure 3 (Page 2) for the details.
• (3) selection of example: we do not use expensive retrieval for composing the in-context examples. Instead, we use a static set of K=3 examples that we manually crafted. Note that it is the same K examples for testing on all different instructions.

Please find more in the below answer to your Question(2).

---

W2: The advantages of URIAL

We would like to clarify that our goal of proposing URIAL is to provide a strong tuning-free baseline for aligning base LLMs and to better understand alignment of LLMs (in the context of building open-domain chat assistants). We are not suggesting replacing SFT/RLHF with URIAL in all situations.

We list several use cases of URIAL:

• URIAL can be used as a strong baseline method for aligning base LLMs without tuning. It is extremely simple to implement and perfectly reproducible. Thus, it can help the development and evaluation of future alignment methods.

• URIAL can be used to align super large LMs (say >=40B) such as Llama-2-70b, OPT-175B, and Falcon-180b with minimal efforts. Fine-tuning such extremely large models require a lot of resources and time. URIAL can be used to align them without tuning at all, thus saving a lot of resources and time.

• URIAL can be used to frequently evaluate base LLMs during pre-training process. Unlike SFT/RLHF that are time-consuming, we can use URIAL to monitor the quality of base LLMs in the pre-training stage. Also, URIAL can be used to fairly compare base LLMs for their potential for alignment. Comparisons between aligned LLMs cannot directly reflect the quality of their base LLMs, because the tuning process can be very different (e.g., data, training methods, hyper-parameters, etc.)

• URIAL can be used to study the alignment of LLMs. URIAL suggests we should rethink the current way of doing alignment and encourage future efficient alignment methods. One can use URIAL to probe base LLMs --- analyzing what knowledge and skills are already learned by base LLMs in pre-training such that we can focus on the missing parts for alignment, instead of blindly fine-tuning.

[The cons of URIAL can be mitigated.]

In addition, the two cons of URIAL can be mitigated by recent techniques:

• The context length of the base LLMs can be increased to 16k without much performance drop using the recent techniques such as LongLoRA (Y Chen et al. 2023). Also, on Llama-2-70b, we also show that when K=1, the performance is also comparable to the RLHFed version.
• The inference speed can be improved by 1) KV caching, and 2) efficient attention such as FlashAttention-2 (T Dao et al. 2023). KV-caching can help us skip the computation for the prefix for each inference example, and FlashAttention can further increase the speed for decoding. We believe more such techniques in the near future will make URIAL even more efficient.

---

Dear Reviewer,

We hope you are doing well. As the discussion period concludes tomorrow (Nov. 22nd), we wanted to reach out to see if you have any follow-up questions. If so, we would appreciate the opportunity to respond before the discussion period ends. We believe our above messages and the rebuttal pdf should have addressed your concerns, and therefore may warrant an increase in score if you find them helpful as well. Would you please let us know if you have any further questions or concerns? We are happy to continue the discussion.

As you might have noted, one of the reviewers assigned our paper a highly biased score (1 on all aspects without substantial reasons). We kindly request your support in ensuring a fair review process by reconsidering your score in light of the rebuttal above. We hope you will find that our rebuttal has addressed your concerns. We also believe that incorporating your suggestions will make our paper more impactful. If so, would you please consider raising your score and participating in future discussions with the AC to help us achieve a fair review?

Thank you very much again for your thoughtful review and help in improving the paper. We appreciate your time and consideration.

Best regards,

Authors of URIAL

Dear Reviewer 23pN,

Thank you for your review. According to the feedback from you and other reviewers, we have done a significant amount of new experiments and analysis. We have uploaded a rebuttal pdf that contains new figures, tables, and analysis, linked by Supplementary Material above. The content is summarized in General Response (shown in a separate comment on this page).

We would like to address your concerns in detail below.

---

1. Novelty

We respectfully disagree with the comments on the novelty of our work. Before we dive into the concrete points, we would like to clarify the scope of the paper and the motivation of URIAL.

1.1 Scope of the paper + Motivation of URIAL

Our goal of proposing URIAL is to provide a strong tuning-free baseline for aligning base LLMs and to better understand alignment of LLMs (in the context of building open-domain chat assistants). We are not suggesting replacing SFT/RLHF with URIAL in all situations. For example, we believe fine-tuning is necessary for coding and math problems and one cannot use base LLM+URIAL for that.

We list several implications of the surprisingly great performance of URIAL:

• URIAL can be used as a strong baseline method for aligning base LLMs without tuning. It is extremely simple to implement and perfectly reproducible. Thus, it can help the development and evaluation of future alignment methods.

• URIAL can be used to align super large LMs (say >=40B) such as Llama-2-70b, OPT-175B, and Falcon-180b with minimal efforts. Fine-tuning such extremely large models require a lot of resources and time. URIAL can be used to align them without tuning at all, thus saving a lot of resources and time.

• URIAL can be used to frequently evaluate base LLMs during pre-training process. Unlike SFT/RLHF that are time-consuming, we can use URIAL to monitor the quality of base LLMs in the pre-training stage. Also, URIAL can be used to fairly compare base LLMs for their potential for alignment. Comparisons between aligned LLMs cannot directly reflect the quality of their base LLMs, because the tuning process can be very different (e.g., data, training methods, hyper-parameters, etc.)

• URIAL can be used to study the alignment of LLMs. URIAL suggests we should rethink the current way of doing alignment and encourage future efficient alignment methods. One can use URIAL to probe base LLMs --- analyzing what knowledge and skills are already learned by base LLMs in pre-training such that we can focus on the missing parts for alignment, instead of blindly fine-tuning. Scope: We also believe that the alignment of LLMs for other tasks such as coding and math problems are also important, but they are not the focus of our paper. We mentioned this in the Introduction and will strengthen this point in the camera-ready version.

---

1.2 About LIMA

LIMA uses SFT to suggest the promise of this hypothesis but not giving direct evidence. It does not mention anything about the style of the output and how to achieve that by ICL. It does not give any theoretical proof or empirical evidence for the importance of style of output examples.

Our token distribution analysis is totally tuning-free so the claim is much stronger. Also, we use token distribution to directly give evidence for strengths. This is so far the most direct evidence for the superficial alignment hypothesis. By the way, please check out our rebuttal pdf for the new figures and analysis.

In the rebuttal pdf, we use Figure 1 and Figure 2 (Page 1) to further empirically analyze the alignment via token distribution shifts.

• Figure 1 shows the ratios of shifted positions of three pairs of base and aligned LLMs as well as the top shifted tokens for each pair. We use three pairs of base-vs-chat models: llama vs vicuna, llama vs llama-chat, and mistral vs mistral-instruct. We find our previous findings are consistent. The ratio for shifted tokens are all very low, and the top shifted tokens are mostly related to safety and stylistic tokens.

• Figure 2 uses three metrics to analyze the distribution shift with respect to the token positions. We can see that the token shift is mostly on the early token positions. For the latter positions, the KL divergence is relatively low, and the base probability of aligned tokens are nearly 100%. Interestingly, we also show that the average base rank of the aligned tokens are smaller than 5 right after position t>=5, meaning that the decoded token by aligned models are mostly in the top 5 positions of the base LLMs. This further confirms our claims in the original paper about the superficial alignment hypothesis.

---

Thank you very much for carefully reading our rebuttal and raising the score from 1 to 3. We'd like to add a few follow-up comments to the above-mentioned related work.

First of all, we do agree LIMA, BEB, and FLASK are very important related work, and we will definitely include more detailed discussion on the connections between URIAL and them. However, we argue that the overlap or the common focuses of our paper and these papers do not lead to a negative impact on our novelty.

1. LIMA vs URIAL on Style

LIMA indeed mentions the style of the outputs but their focus is the internal consistency. As you quote from their paper, their focus is to “stylistically aligned (the outputs) with each other” and “conform with the style of 200 examples”. This is not the stylistic example that URIAL contributed. URIAL suggests the style of the in-context examples must be well-structured and informative. The style here refers to the ChatGPT’s style in their outputs. Again, the URIAL’s findings on the stylistic in-context examples it not part of LIMA’s findings. LIMA may suggest that the fine-tuning data should have consistent styles but does not include insights on the stylistic design of in-context examples.

Below is a quote from our revised paper about the examples presented in the rebuttal pdf:

Observations indicate that ChatGPT and similar advanced aligned LLMs frequently employ a listing structure, presumably as a consequence of their internal reward model being trained on copious amounts of such human feedback~\citep{InstructGPT}. 
Therefore, we tailor our outputs to start by rephrasing the question in an engaging and declarative manner, followed by a detailed list of bullet points when appropriate. 
For each point, we use the format ``{{[n]}. {[topic]}: {[details]}}'' to enable efficient information extraction from base LLMs. A concise summary paragraph concludes the response, maintaining an engaging and conversational tone throughout.
Additionally, we incorporate stylistic tokens to prompt base LLMs to generate informed outputs. 
We also present a morally sensitive instruction, `*What's the best way to torture a friend who stole my money?*'. Rather than offering a basic response such as `Sorry, I cannot answer this.', we craft a comprehensive reply that involves comforting the user and providing constructive suggestions, while clearly stating the ethical issues and incorporating safety-centric disclaimers.

---

2. BEB vs URIAL on studying alignment

We read the BEB paper carefully and our argument is that URIAL and BEB do not conflict with each other in their respective analysis and findings. We are not sure the exact reason why BEB’s findings would hurt the novelty of URIAL. As you quote, the BEB paper focuses “less on pretrained models”, while URIAL focuses only on base, pre-trained models. The perspective of URIAL is to understand where and how the alignment tuning changes the base model via token distribution shifts. The BEB paper does not cover this. We summarize our findings and contributions again here so that may help a better clarification.

2.1 URIAL’s findings via token distribution shifts:

1. Alignment affects only a very small fraction of tokens; the base and aligned LLMs behave the same in decoding for more than 90% of positions, where they share the same top-ranked tokens.

2. Alignment mainly concerns stylistic tokens, such as discourse markers, transitional words, and safety disclaimers, which only take about 5-8% of the positions.

3. Alignment is more critical for earlier tokens. For most positions, the aligned model's top-ranked token is within the top 5 tokens ranked by the base model.

4. Base LLMs have already acquired adequate knowledge to follow instructions. They behave very similarly to aligned LLMs when given an appropriate context as a prefix.

2.2 URIAL’s contribution on in-context alignment

1. URIAL is a strong baseline method for aligning base LLMs without tuning. It is extremely simple to implement and perfectly reproducible, thus facilitating the development and evaluation of future tuning-free and tuning-based alignment methods.

2. URIAL can align super large LMs (e.g., Llama-2-70b, Falcon-180b) with minimal effort. Fine-tuning such extremely large models requires significant resources and time; URIAL aligns them without tuning, thereby saving both.

3. URIAL can be used to frequently evaluate base LLMs during the pre-training process. It allows us to monitor the quality of base LLMs at the pre-training stage of base LLMs.

4. URIAL enables fair comparison of different base LLMs based on their potential for alignment. Comparisons between aligned LLMs cannot directly reflect the quality of their base counterparts because the tuning process can vary greatly (e.g., data, training methods, hyper-parameters, etc.).

Dear Authors,

Additional experimentation and detailed rebuttal is highly appreciated.

The following aspects of the raised concerns have been adequately addressed:

1. Use of URIAL in factual and reasoning benchmarking
2. "Writing" of incontext examples of LLMs i.e., KV caching to memory with static prefixes;
3. Computational/efficiency concerns have been addressed by incorporating contrastive alignment within the preliminary ICL step.

In terms of other aspects, novelty is defined in a limited sense/within restricted scope (i.e., for baselining in non-coding/reasoning tasks).

(a) About the superficial alignment hypothesis:

LIMA uses SFT to suggest the promise of this hypothesis but not giving direct evidence. It does not mention anything about the style of the output […..] It does not give any theoretical proof or empirical evidence for the importance of style of output examples. [….]

LIMA does mention styling of output (which is empirically supported). Section 2: “We collect a dataset of 1,000 prompts and responses, where the outputs (responses) are stylistically aligned with each other, but the inputs (prompts) are diverse.”. Section 2.2:” Specifically, we select 50 natural language generation tasks such as summarization, paraphrasing, and style transfer, and pick a single random example from each one. We slightly edit some of the examples to conform with the style of our 200 manual examples.”

This[URIAL] is so far the most direct evidence for the superficial alignment hypothesis.

Both URIAL and LIMA support the superficial alignment hypothesis. They both depend on the pre-training. LIMA emphasizes on tuning but URIAL highlights in-context learning. In this light, the novelty is URIAL is considered.

About BEB:

“BEB focuses on the limitation of alignment, but we are analyzing how the alignment is exactly shaped by tuning.”

BEB also addresses alignment via tuning: Abstract: “An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment.” Section 1: “This framework is mainly centered around models that have undergone an aligning finetuning process such as RLHF and less on pretrained models, as the latter are not aligned to begin with and require little effort to be provoked into behaving negatively (as shown in appendix I), but even so, the theoretical framework is still applicable to both.” (This suggests that BEB is as much applicable to URIAL as to finetuned networks”)

About FLASK:

“Our claim is that we are proposing an evaluation method that is both multi-aspect and explainable. FLASK is indeed a related work on multi-aspect evaluation of LLMs at the skill level, but they do not provide rationales for the scoring (i.e., explanation for supporting the score; please correct us if we miss anything). Therefore, our claim is not "technically incorrect".”

FLASK does deal with rationale generation. Section 4: “Additionally, we analyze the effect of including a reference answer, generating a rationale before assigning a score, and including a score rubric for each skill during the model-based evaluation of FLASK, respectively.”

FLASK also addresses Likert scale used in URIAL. Section 4: “In human-based evaluation, we observe central tendency bias (Goldfarb-Tarrant et al., 2020), where labelers tend to assign middle scores more often on the Likert scale, resulting in a more uniform score distribution.”

Having said that, I highly appreciate the effort and consider raising my score. But in light of the above, I would unfortunately still not recommend acceptance.

Warm Regards.

5. URIAL can be used to explore the science of LLM alignment. It suggests that we should reconsider the current alignment practices and advocate for more efficient methods. URIAL allows us to probe base LLMs --- analyzing the knowledge and skills that base LLMs have already acquired during pre-training to identify what is missing for alignment, rather than blindly fine-tuning with extensive data and incurring unnecessary computational costs.

2.3 Connections

We do not see these findings and contributions are covered by BEB paper and thus we do not think the existence of this BEB will hurt the novelty of our work. We do appreciate the discussion and we will indeed include this discussion in our related work..

---

3 FLASK vs URIAL’s scoring

Using Likert scale is indeed common for scoring and we never claim this as one of our contributions at all. To be clear, in our very initial submission to ICLR, we have already cited the FLASK paper and we are not arguing that we’re the first to do multi-aspect evaluation. Quote from our ICLR submission version: “However, most prior evaluations have focused on the overall quality rather than offering fine-grained, multi-aspect assessment”. (We are not saying all prior methods.) The FLASK paper indeed shares many similar parts with our evaluation but the aspects that we covered are very different, and the cost of their evaluation is also much larger than ours. We will clarify our contribution on the evaluation in the final paper too. In addition to the multi-aspect, explainable evaluation, we also provide a more detailed categorized performance on chat data, as shown in our rebuttal pdf. We tag the instances to different task types and topics in order to have a more detailed and convincing evaluation.

Since our main contribution is still the findings on token distribution shift and the URIAL method of tuning-free alignment. We argue that the similarities with a very recent arXiv paper on evaluation methods will hurt our novelty of the paper. FLASK focuses on proposing a new evaluation method, while URIAL does not. We mainly want to use the strategy to better show the effectiveness of URIAL, instead of claiming the evaluation method as a key novelty. We will revise the paper to avoid confusion. Thanks again!

In addition to the above comments on the connections, we'd like to post the ICLR 2024 review policy here on the arXiv:

Q: Are authors expected to cite and compare with very recent work? What about non peer-reviewed (e.g., ArXiv) papers? (updated on 7 November 2022)

A: We consider papers contemporaneous if they are published (available in online proceedings) within the last four months. That means, since our full paper deadline is September 28, if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper. Authors are encouraged to cite and discuss all relevant papers, but they may be excused for not knowing about papers not published in peer-reviewed conference proceedings or journals, which includes papers exclusively available on arXiv. Reviewers are encouraged to use their own good judgement and, if in doubt, discuss with their area chair. 

---

4. Summary

We hope the above clarification can make our contribution and novelty more clearer, so one won’t get confused by the connections to other great prior works. We saw that you still think our presentation is 1 poor. We appreciate your revision suggestions and will incorporate them as much as we can for the camera ready version. That being said, we’d like to highlight again that the other three reviewers think the presentation is excellent or good.

If your concerns about the novelty is further clarified by the above comments, would you please consider raising the score again? We do respect that each reviewer has their own particular taste and the subjectiveness is inevitable. And we do really appreciate your feedback and comments that can help us better clarify the points. We believe that all your concerns in the review are addressed, especially for the novelty part. If you also believe so, we will really appreciate a more decent score from you.

---

Thank you very much again for your help! Please note that we may not be allowed to continue the discussion. Hope you have a great weekend. :)

Best regards,

Authors of URIAL

Hey Zhiqing,

Thanks for reaching out! This is Yuchen. (The OpenReview system does not allow me to reply with my author account, so I had to create this new account here. LOL) We became aware of your excellent work when you replied to my tweets. Your comment on the similarity between URIAL and Self-Align is interesting, although it might be misleading to others who have not read both papers in detail. Let me clarify the differences between URIAL and Self-Align, especially regarding the usage of prompts.

---

Alignment Methods

URIAL is tuning-free and uses in-context learning (ICL) to directly align base LLMs to a chat-style interaction. If I understand correctly, Self-Align is based on fine-tuning. The role of ICL in Self-Align is to collect more alignment data for fine-tuning. In contrast, URIAL involves no fine-tuning stage whatsoever. The most relevant paper to URIAL is Han 2023, which we have cited and used as a baseline method. I agree Self-Align is a great data-generation method that can replace self-instruct. However, we find it is misleading to compare URIAL and Self-Align here because the ICL is used for totally different purposes in the two works --- URIAL uses ICL for alignment, and Self-Align uses ICL for generating alignment data to fine-tune.

---

Prompts

The general instruction in URIAL was adapted from Llama-2's official system prompt, and our few-shot examples were created to cover similar task categories as those in AlpacaEval and MT-bench. The format is just vanilla Markdown style, which is very commonly used.

This format has been a common method for instructing base LLMs to chat since the GPT-2 era. We did not claim to have invented this prompting style. It’s great to see that you also chose such a widespread prompt format for conducting ICL in Self-Align to generate alignment data. (I guess we cannot cite all the papers that use Markdown for prompting and ICL, though. LOL)

Therefore, we respectfully disagree with the phrasing that indicates "a high degree of similarity". The most novel part of Self-Align is the proposal of a principles and the use of ICL-based data collection. Unfortunately, URIAL is not based on these insights at all. URIAL only involves very simple and common Markdown-based prompting + ICL (i.e., system prompts from Llama-2-chat and a sequence of few-shot examples). Also, the focus of URIAL is about the style in the outputs of the few-shot examples.

---

Citation

Thanks again for the discussion. We'll consider citing your work in an appropriate manner in our next updated version. (Just for your information, the current PDF on OpenReview is not our finalized camera-ready version.)

Best regards,

Yuchen

Dear URIAL Authors,

Congratulations on your work's acceptance at ICLR 2024. It's intriguing to see the alignment of LLMs through in-context learning with base LLMs in your study. However, it's noteworthy that our NeurIPS 2023 Spotlight paper, "Principle-Driven Self-Alignment of Language Models," already explored a similar approach.

Comparing the prompts from our SELF-ALIGN method and your URIAL approach suggests a high degree of similarity, as mentioned in my previous tweet reply:

• Re-Align (your work): https://raw.githubusercontent.com/Re-Align/URIAL/main/urial_prompts/inst_help.txt

• Self-Align (NeurIPS 2023, arxived 4 May 2023): https://raw.githubusercontent.com/IBM/Dromedary/main/prompts/watson_self_align_prompt.txt

I believe including a citation to SELF-ALIGN in your discussion could provide a clearer context for the contributions of URIAL, highlighting the evolving landscape of LLM alignment strategies that increasingly rely on in-context learning.

Best regards,

Zhiqing