Thank you for the thoughtful comments and review!

Weakness

Details of stability functions. The stability functions are diverse, but we make sure that all can be achieved with basic python functions deterministically. For example, below are some examples of the stability functions.

from nltk.sentiment import SentimentIntensityAnalyzer
def get_sentiment(text, sentiment):
    
    sia = SentimentIntensityAnalyzer()
    score = sia.polarity_scores(text)[sentiment]
    return score

lambda x: "tennis" in x.lower()

from langdetect import detect_langs
def get_french_percentage(sentence):
    languages_detected = detect_langs(sentence)
    for lang in languages_detected:
        if lang.lang == 'fr':
            return lang.prob  # lang.prob is the probability of the detected language
    return 0  # Return 0 if French was not detected

Figure 4 over different attention heads. Preliminary experiments show that all attention heads, except those from the first several layers, demonstrate a similar attention decay pattern, as shown in Fig 4. We will add other specific heads or an average over all heads in the appendix.

Questions

Figure 3 didn’t express that the user LM could follow the instructions better. In fact, it should show a similar decay as the blue curve in Figure 3A since the roles of the two LMs are largely symmetric. The orange curve examines the agent LM’s behavior using the stability function of the user LM’s persona, therefore demonstrating that the agent LM is slowly picking up the persona of the user LM.
Theorem A1. If accepted, we will consider moving more content from Appendix A into the main paper, given that the page limit is extended to 10.
Missing citations. Thanks for pointing out relevant literature that we missed out. We have added them.

Thank you for the thoughtful comments and review!

Weakness

Discuss implications. Thanks for the suggestion to extend the discussion of how the findings in this paper impact other aspects of LLM behavior under various deployment scenarios. We will add such a discussion:

The instruction drift indicates that current LM systems still cannot behave coherently over long horizons, which is the basis of long-term planning, such as information seeking (Lin et al., 2023) and tool usage (Nakano et al., 2021). Fundamentally, it is caused by a mismatch between their training schemes (text continuation or single-round RLHF) and their deployment scenarios (open-ended dialog with users). The phenomenon of instruction drift suggests there is much to be done to bridge this gap toward more robust and coherent dialog systems.

More diverse baselines. We acknowledge that a more diverse set of techniques could strengthen the argument for the effectiveness of the proposed split-softmax technique. Among them, as suggested by reviewer 9T1g, studying different data compositions of the RLHF data could be especially useful—for example, adding role-playing data points into RLHF dataset could potentially improve the instruction stability of the resultant model.

Questions

Other aspects of model performances. It would be a good idea to measure the language model’s output diversity with and without split-softmax. A naive guess is that it shouldn’t influence the entropy of the next-token distribution. We will add a few sentences of discussion on how split-softmax would help the model fend off adversarial attacks. We believe that attacks based on both traditional input optimization like GCG (Zou et al, 2023) as well as prompting like PAIR (Chao et al, 2023) could work less effectively on split-softmax-guarded models.
Dialogs beyond the tested lengths. As seen in Fig 6, after 16 turns of dialog, the LLM has lost most of its instruction-following capability, with or without various mitigation methods. In fact, at this point, LLaMA-2-70B cannot even speak coherent English, but we could reasonably expect a more advanced model to maintain its language into >16 turns of dialog.

Thank you for the thoughtful comments and review!

Weakness

Other measures of helpfulness like MT-bench. We agree that using a more comprehensive benchmark to measure the helpfulness of LLM could be a better idea than using MMLU. However, MT-bench requires users’ votes to determine the helpfulness of the uploaded model, which makes it hard to scale and obtain results in a timely manner.
More details of the benchmark dataset. We will add more examples to the paper to provide a better understanding of the dataset.

Questions

Alignment data blend. We could expect data points with a stronger connection to system prompts in the alignment dataset to be conducive to better instruction stability of the fine-tuned model. Studying the relationship between instruction stability and data composition could be a fruitful future research question.
For Figure 4 and Figure 8, when the system prompt is used as the first user’s utterance, it’s reasonable to guess that the curve will look similar to Figure 4/8 on LLama models. In fact, for the prompt formatting scheme by llama, the system prompt is guarded with special system-prompt tokens and prepended to the first user message. So the suggested change would be equivalent to removing two special tokens. For GPT-3.5 experiment, it’s shown in Figure 9, suggests a special internal mechanism behind the GPT-3.5 API that emphasizes system prompts.
On probing questions. The probing questions aren’t necessarily adversarial. For some system prompts, we stipulate certain rules that any utterances from the agent LM should follow; therefore, the probing questions can be arbitrary. In those cases, we use “What do you do in London as a tourist?” as a default choice. We will incorporate these details into the paper like for weakness 2.