Unveiling and Manipulating Prompt Influence in Large Language Models

Dear Reviewer rNoS,

Thank you for your detailed and constructive feedback on our paper. We have carefully considered your insights and have addressed the identified weaknesses (W) as well as responded to the questions (Q) you raised. Our comprehensive responses (A) are provided below for your review.

[W1] The TDD method is based on multiple assumptions listed at the bottom of page 4. Some assumptions are strong without any theoretical and experimental proof, especially the first and second assumptions.

[Q1] We are sorry for the insufficient clarity in articulating the theoretical and empirical foundations for the assumptions presented on page 4. In response, we reassess the assumptions underlying TDD from both theoretical and empirical perspectives.

Theoretical Justification

Geva et al. (2021; 2022) have theoretically demonstrated that in GPT2, token representations at each layer can be projected as evolving distributions over the vocabulary. Considering the consistent decoder-only transformer architecture in contemporary autoregressive language models (LLMs) such as GPTJ and LLaMA2, this theoretical analysis is extendable to these models as well since their mathematical derivations are exactly the same. This supports our first assumption/ finding: the concept of “token representations at each layer as evolving distributions” is applicable across various LLMs. However, their empirical study only focuses on GPT2. To validate the generalization empirically, we conducted experiments across various LLMs and datasets.

Empirical Justification

Detailed descriptions of the experiments and their settings are provided in Appendix A.

1. Finding 1: The theory of “token representations at each layer as evolving distributions over the vocabulary” can be generalized across various LLMs. Given an input text, we first visualize the token distributions at each layer projected by the LM head for various LLMs including GPT2, GPTJ, BLOOM, Pythia and LLaMA2 in Figure 2 in Appendix A. Using LLaMA2 in Figure 2e as an example, for the initial input “ Some”, intermediate layers produce outputs such as “Someone” and “Somewhere”. With the input “Some boys discover”, outputs like “Some boys discover themselves” emerge. In the upper layers, the prediction logit incrementally rises, converging to the final prediction. These visualizations underscore that token representations from any layer can be depicted as meaningful token distributions over the vocabulary. Furthermore, this notation can be applied to all input tokens, instead of the only last token for prediction.

To quantitatively assess the dynamics of token distributions across layers, we treat the token distribution at the final layer as the “ground-truth”. We then compute the KL-divergence between intermediate layer token distributions and this ground-truth to study their convergence patterns. We employ datasets from BLiMP to estimate the KL-divergence for each token. Figure 2f depicts the mean divergence across layers for all input tokens in all datasets. It can be observed that the learning process reflects the evolving token distributions over the vocabulary, which converge in a roughly monotonic manner to the distribution of the final layer.

Both qualitative and quantitative analyses corroborate our first finding regarding the generalizability of token representation theory across various LLMs.

2. Finding 2: Projected token distributions from the LM head hold the potential to elucidate causal relationships between prompts and LLM outputs. Our first finding confirms that the LM head projects complex token representations at each layer as distributions over the vocabulary. Advancing this concept, we observe that these distributions are both interpretable and integral to the model's generative process. Each dimension of the token distribution represents a specific token in the vocabulary, with the value of each logit indicating the likelihood of the subsequent token based on current and preceding tokens. This capability of the LLM to track prediction or distribution changes with the progressive introduction of tokens enables us to infer which tokens influence the prediction of a target token, offering explanatory insights. Thus, our second finding is that projected token distributions from the LM head hold the potential to elucidate causal relationships between prompts and LLM outputs.

The experiments detailed in Section 4 further validate our assumptions and findings. These experimental results show that utilizing the token distributions projected by the LM head yields the most faithful explanations. This substantiates our assertion that the LM head possesses the potential to clarify causal relationships between prompts and LLM outputs.

Detailed analysis and experiments are provided in Section 3.2 and Appendix A in our revised paper.

Dear Reviewer rNoS,

Thank you for your follow-up. We have further addressed your concerns as outlined below:

1. A5 as I have mentioned the SOTA is integrated gradients (IG) but the author does not consider this baseline.

We apologize for this omission in our first revision. We have recently completed the experiments involving IG and have now included these results in our revised paper. Here are the details:

In addition to our proposed method, we have compared it with the advanced explanation method Integrated Gradients (IG) (Sundararajan et al., 2017). The experimental results, presented in Table 2 of our revised paper, show that IG slightly underperforms compared to the SOTA contrastive method Con-GI. However, our TDD significantly outperforms all baselines by a considerable margin of 5%-7%. We argue that the suboptimal performance of IG is primarily due to its focus solely on target tokens while disregarding alternative tokens.

These revisions and detailed results can be found in Sections 4.4-4.5 and Table 2 of our revised paper.

2 there is no definition of 'zero-shot controlled text generation' in the paper and response, what is the meaning of zero-shot? Does that mean other baselines are not zero-shot?

We apologize for not previously defining “zero-shot controlled text generation.” This term refers to a scenario where no training samples are available or provided, and all methods must be executed without such samples.

Our TDD approach is designed to function without any training samples. To ensure a fair comparison, we selected six baselines that are also applicable in a zero-shot setting. This setting, common yet challenging in real-life applications, tests the ability to control attributes like sentiment and toxicity in large language models (LLMs) without prior training. Our experiments demonstrate that TDD effectively manages these attributes under zero-shot conditions.

We hope our response resolves your concerns. Please do not hesitate to contact us if you have further questions.

Dear Reviewer jvM3,

We appreciate your comprehensive review and valuable feedback on our manuscript. In response, we have systematically addressed the identified weaknesses (W) and answered your queries (Q). Our detailed responses (A) are outlined below.

[Q1] In section 5, which variant of TDD is used? Are TDD-forward used there?

[A1] We apologize for the lack of clarity regarding our choice of TDD variants for the controlled text generation. In our application experiments in section 5, we specifically chose the TDD-bidirectional variant for both toxicity suppression and sentiment steering. This decision was based on its superior performance observed in our main experiments, which are thoroughly discussed in Section 4. We have added such clarification in our revised manuscript.

[W1] The TDD methods heavily depend on the alternative word w_a. The datasets chosen in this paper (those BLiMP datasets) provide sentence pairs with exactly one-word differences. Other datasets may not have such well-defined alternative words. This greatly limits the potential applicability of the proposed TDD methods. A related point: it is unclear to me how the w_a in the Section 5 experiments are identified.

[A2] Thank you for highlighting your concerns. We apologize for any lack of clarity regarding the scope of TDD in our paper. Actually, TDD's effectiveness is not strictly dependent on alternative tokens. TDD is versatile and can be applied in various scenarios, including those with 1) a single target token and a single alternative token, 2) only target tokens, 3) multiple target tokens and multiple alternative tokens, and 4) controlled attributes in generated sequences. We substantiate each of these applications with experimental evidence as follows:

Scope of TDD

1. Single Target and Single Alternative Token: Section 4 details our main experiments, which validate TDD's effectiveness in scenarios involving a single target token and a single alternative token.

2. Target Token Only: TDD is applicable even in the absence of alternative tokens. By assigning a zero probability to alternative tokens in Equations 2 and 4 in the paper, we can negate the necessity of alternative tokens. In the added experiments, we estimate each token’s importance by using only target tokens, and the alternative tokens are not provided. The results, summarized in Table 5, show TDD's significant outperformance of strong baselines—by about 3%—in scenarios without alternative tokens.

3. Multiple Target and Alternative Tokens: TDD accommodates scenarios involving multiple target and alternative tokens by aggregating their probabilities in Equations 2 and 4. The effectiveness of TDD in these settings was assessed through experiments using AG’s News and SST2 datasets, with details provided in the Appendix B.2.3 of our revised paper. These results in Table 6 demonstrate TDD’s effectiveness for multiple target and alternative tokens.

4. Generated Sequence Attribute Control: The experiments described in Section 5, focusing on toxic language suppression and sentiment steering, further confirm TDD's applicability in controlling attributes of generated sequences.

More details are available in Section 3.6 and Appendix B.2 in our revised manuscript.

Defined Target and Alternative Tokens in Section 5

For zero-shot toxic language suppression, we utilized a predefined list of toxic words from WORDFILTER as target tokens, treating all other tokens as alternatives. In this context, considering all other tokens as alternatives is equivalent to having no alternatives. For zero-shot positive sentiment steering, we identified negative words from SenticNet as target tokens, using positive words as alternative tokens. This approach allows TDD to identify key words that could lead to negative outcomes and suppress them. To generate negative responses, we simply reverse the roles of the target and alternative tokens, while keeping other settings constant.

[Q4] Which TDD variant is applied in the steering experiments? You mentioned that each TDD variant offers unique insights into token relevance. Can you further elaborate on this regard? Unfortunately, I could not find a discussion in the paper. Are there scenarios where one variant is beneficial?

[A4] We appreciate your feedback and have incorporated detailed discussions about the TDD variants and its use in our steering experiments.

1. In our steering experiments, we specifically chose the TDD-bidirectional variant, as detailed in the revised manuscript. This decision was based on its superior performance observed in our main experiments, which are thoroughly discussed in Section 4.

2. Detailed Variant Discussion.

a. Calculation Direction: The TDD-forward evaluates token importance sequentially from the first to the last input token. Conversely, TDD-backward assesses saliency in reverse, from the last input token to the first. TDD-bidirectional, drawing inspiration from bidirectional neural networks, combines saliency estimates from both TDD-forward and TDD-backward.

b. Information Perspective: TDD-backward is less influenced by linguistic conventions, offering a more targeted approach. The TDD-forward overlooks the inherent linguistic influence of individual tokens. Consider the sentence completion by the LLM: “ This design is quite novel and fantastic. I really ___”, our aim is to explain why the LLM generates “like” instead of “hate”. Linguistic conventions render it improbable to predict tokens such as “like” or “hate” following “is” or “novel”, indicating potential inaccuracies in token contribution assessments. TDD-backward can mitigate this issue. Specifically, in TDD-backward, the words “I really ___” are fed to the LLM in the first two iterations. Subsequently, other tokens, such as “fantastic”, are progressively introduced This initial phase sharpens the model's focus, enhancing its accuracy in predicting words such as “like” or “hate” as the LLM consistently generates predictions following the phrase "… I really __" in each iteration.

3. Application Scenarios TDD-forward is preferable in time-sensitive or computationally constrained scenarios, as it requires only one forward propagation due to the auto-regressive structure of LLMs. TDD-backward and TDD-bidirectional are better suited for contexts where time is not a constraint, and there is a higher demand for explanation fidelity. They demand iterations equal to the input length for saliency estimation. The revised paper’s Table 2 showcases the higher faithfulness of explanations by TDD-backward and TDD-bidirectional, while Table 9 confirms the computational efficiency of TDD-forward.

More details are provided in Appendix B.1 and G in our revised paper.

[W1] The difference between the TDD variants is not well discussed. While the applications are very interesting, the authors could have used the space to elaborate on the differences between the introduced variants.

[A5] Please refer to our response [A4] to question 4 [Q4].

[W2] Captions Table 1 and Table 3 are too sparse.

[A6] Thanks for pointing this out. We have revised the captions for Tables 1, 3 and 4 accordingly in our modified manuscript.

[W3] While not being a contrastive XAI method, important related work on explainability for autoregressive LMs missing: ATMAN: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation. Björn Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, Kristian Kersting. In Proceedings of NeurIPS 2023

[A7] Please refer to our response [A1] to question 1 [Q1].

Dear Reviewer 1T7y,

Thank you for your thorough review and insightful feedback on our paper. We have addressed each identified weakness (W) and responded to the questions (Q) raised. Our detailed responses (A) are summarized below.

[Q1] Can you elaborate on the difference between TDD and AtMan [Deiseroth et al., 2023] (see weaknesses) beyond contrastive explanations?

[A1] Thank you for your insightful feedback. We apologize for the oversight regarding the omitted reference and have addressed this in our revised manuscript. To clarify, the primary distinctions between our TDD method and the AtMan approach are as follows:

1. Saliency Estimation Medium: AtMan evaluates input saliency by iteratively manipulating the attentions of each input token and observing changes in cross-entropy. In contrast, TDD utilizes the token distribution as the medium for calculating token saliency.

2. Inference Process: AtMan introduces noise at the beginning of forward propagation, whereas TDD performs inference directly at the end of propagation, after the forward propagation process is complete.

3. Parameter Dependence: TDD has the advantage of not requiring any hyperparameters. On the other hand, AtMan necessitates setting hyperparameters, such as the suppression factor and the lower bound of cosine similarity, which can vary across different datasets.

The revision can be found in Section 2.1 in our revised paper.

[Q2] How robust is TDD when multiple alternative tokens are used?

[A2] Thanks for your professional comment. To affirm the robustness of our TDD method, we expanded our experimental scope to include additional datasets featuring multiple target and alternative tokens. These experiments demonstrate TDD's superior performance over strong baseline methods across diverse datasets. Further details are provided below.

Experiment Setup: In the absence of ready-made datasets for evaluating multiple alternative tokens, we leverage a prompt-learning framework to simulate such an environment. We select two datasets: AG’s News for topic classification and SST2 for sentiment analysis. The inputs are structured as cloze questions, with AG’s News framed as "<input> This topic is about___" and SST2 as "<input> It was __". We incorporate multiple target and alternative tokens by utilizing label words from the KPT method. For each sample, the label words corresponding to the ground-truth label serve as target tokens, while those from other classes are alternative tokens.

Table 6 in Appendix B.2.3 presents the performance comparison of TDD-bidirectional against other baselines. In the SST2 dataset, TDD surpasses SOTA methods by around 7% in AOPC and demonstrates a 6% enhancement in Suff over current baselines. In the AG's News dataset, TDD achieves a 3%-5% margin over existing methods in both AOPC and Suff. These findings verify TDD's efficacy in scenarios involving multiple target and alternative tokens.

More details can be found in Section 3.6 and Appendix B.2.3 in our revised manuscript.

[Q3] Beyond the benchmark experiments, how would one select an alternative token to explain the LMs decision?

[A3] Thank you for highlighting this aspect. In response to your query, we approach the selection of alternative tokens in three distinct ways:

1. Omitting Alternative Tokens: Alternative tokens can be excluded from explanations. By assigning a probability of zero to alternative tokens in Equations 2 and 4, we negate their necessity. Our experiments in Appendix B.2.2, which involved the omission of alternative tokens, confirm the effectiveness of our TDD method.

2. Choosing Based on Linguistic Features: The selection of alternative tokens can align with specific linguistic features under study. For instance, to understand why the model generates "waitresses" in response to the prompt "Amanda was respected by some", we can vary the alternative tokens. For syntactic and morphological analysis, an alternative like "waitress" may be used, while "pictures" may be chosen for semantic analysis. This also underscores our preference for contrastive explanation over conventional explanations in TDD, which we find offers quantifiably superior insights into grammatical phenomena and enhances model simulatability for users.

3. Observing LLM Output Probabilities: A practical method involves examining the LLM's top-k or top-p output tokens. This approach is straightforward and effective for explaining the model’s choices among similar or commonly confused tokens.

Dear Reviewer 4Y86,

Thank you for taking the time to review our paper and providing insightful feedback. Our responses (A) to the mentioned weaknesses (W) and posed questions (Q) are summarized as follows.

[Q1 & W1]: One simple way to control LLMs' generations is via prompting for style transfer. How would this baseline compare to the proposed TDD? Could the authors add a simple baseline by prompting the LLMs for style transfer on the outputs?

[A1]: We appreciate the valuable suggestion offered. In response, we have incorporated two robust baselines from the field of text style transfer aimed at regulating the toxicity and sentiment in LLM outputs.

We compare TDD with two baselines from this field. Style Prompting (SP) (Reif et al., 2022) integrates specific style instructions like toxicity into prompts, while Augmented Style Prompting (ASP) (Reif et al., 2022) enhances this approach by introducing varied rewriting examples for a broader application. Detailed information regarding these two prompting strategies can be found in Appendix H and M.

In our revised manuscript, Tables 3 and 4 detail the experimental results. SP records a toxicity of 0.47, ASP attains 0.41, whereas TDD achieves a notably lower score of 0.20, significantly outperforming SP and ASP. In terms of negative sentiment steering, TDD surpasses both SP and ASP by a substantial margin of about 0.30. For positive sentiment steering, TDD exceeds SP and ASP by more than 0.20, underscoring its considerable advantage over style transfer methods.

[Q2&W2]: How would the proposed approach work on larger models (e.g., llama-2 13B, 70B)? What is the computation cost (efficiency, memory) for running TDD over larger models?

[A2]: Thank you for your insightful feedback. In response, we have expanded our research to include experiments on explanation faithfulness using larger models such as LLaMA2-13B and OPT-30B to ensure its scalability. Additionally, we have included a comparison of the computational costs for both our methods and the baseline approaches. The key findings are summarized below.

Scalability. We conduct experiments with LLaMA2-13B and OPT-30B (Zhang et al., 2022) to assess the effectiveness of TDD in explaining larger models. The summarized results in Table 8 in the revised manuscript reveal that TDD-forward outperforms the baselines by margins of 3.15% in AOPC and 4.4% in Suff using LLaMA2-13B. Both TDD-backward and TDD-bidirectional demonstrate superior performance, exceeding the baselines by more than 4% in AOPC and over 5% in Suff. For OPT-30B, TDD surpasses competitive baselines by a significant margin of 4% to 6%. These results prove TDD's scalability and effectiveness in larger models.

Computation Cost. For the computational cost analysis, we evaluate the average memory usage and processing time required by our method for processing a single input sample. Consistency is maintained across all experiments, which are conducted on an NVIDIA RTX A5000 GPU. For models larger than 6 billion parameters, their 4-bit versions are utilized. Detailed memory and time metrics are presented in Table 9 in the paper. Regarding memory consumption, Rollout and the three TDD variants (TDD-forward, TDD-backward, and TDD-bidirectional) are the most efficient. In terms of processing time, TDD-forward and Rollout emerge as the fastest, whereas TDD-backward and TDD-bidirectional exhibit slightly longer processing time.

More details are provided in Section 4.6 and Appendix F, G in our revised manuscript.

[W3]: The proposed TDD provides contrastive explanations on the token level, it would be interesting to see if this method can be extended to higher-level concepts, e.g., why sometimes a model chooses to generate an unfaithful output.

[A3]: Thank you for your constructive suggestions. We agree that applying our method to higher-level concepts is an interesting direction for future work. Generally speaking, LLM generations can be explained from basic to advanced levels. At the token level, it's crucial to understand and explain the reasons behind the generation of specific tokens. The sequence-attribute level involves interpreting why LLMs produce sequences with attributes like toxicity or positivity. The third, more complex level, examines the production of biased or unfaithful outputs. We've demonstrated our method's effectiveness at the first and second levels through experiments in Section 4 and controlled text generation in Section 5. Applying our approach to the third level remains a goal for future research, aiming to deepen our understanding of LLMs and further their development.

References

Reif, E., Ippolito, D., Yuan, A., Coenen, A., Callison-Burch, C., & Wei, J. (2022). A Recipe for Arbitrary Text Style Transfer with Large Language Models.

Zhang S, Roller S, Goyal N, et al. Opt: Open pre-trained transformer language models[J].

Dear Reviewer 4Y86,

Thank you very much for your follow-up and positive assessment. We are pleased to note that our responses have addressed most of your concerns.

Regarding the main reason why SP and ASP are less effective, we argue it is due to GPT2's limited capacity to comprehend and adhere to instructions. Our tests involved various prompt styles. For example, to generate positive outputs, we experimented with prompts containing keywords such as "be more positive/good/very positive/totally positive" and "be less negative/bad," including synonyms of these terms. Unfortunately, these prompts proved largely ineffective. The most successful prompts have been carefully selected and are detailed in our revised paper. Consequently, we infer that GPT2 struggles with following SP and ASP instructions, even though some demonstrations are included in ASP.

We would also like to highlight that in the original SP and ASP paper, their effectiveness was demonstrated using much larger models like LaMDA-137B and GPT3-175B, as opposed to the significantly smaller GPT2-774M and LlaMA-7B models. This leads us to believe that SP and ASP techniques may be more suited to and effective with larger or instruction-tuned models.

Thank you again for your response. Please let us know if you have further questions.