Thanks for your constructive feedback. We provide our response to your questions as follows.

Q1: The evaluation measures whether there is any subsequence of the generated text that is valid json. It is highly debatable whether a model that would generate natural text intermingled with valid json is actually accurate at generating json.

Thanks for your feedback regarding our evaluation. To clarify, our decision to evaluate subsequences as valid JSON strings stems from the model's behavior during generation. We observed that when the model successfully follows the instruction to generate a JSON string, it tends to repeat the correct JSON strings until reaching the maximum token limit. For instance, an output might look like ‘{"name":"Mary", "occupation":"doctor"} {"name":"Mary", ...’. Moreover, models occasionally preface JSON strings with natural language text, such as ‘The answer is {“name”, “Mary”, …’. To accommodate these variations and fairly assess the model's capability, we chose to evaluate whether any subsequence, starting from '{' and ending with '}', forms a valid JSON string.

To provide a comprehensive view of the model's performance, we have included all generation results by LLAMA-7B on the JSON Formatting task in the link of https://anonymous.4open.science/r/PASTA-10E9/model_generations/json_formatting/llama7b_pasta_generation.json (the table blew shows a few examples). We invite the reviewer to examine these results in detail. These generation results showcase the practical effectiveness of the PASTA approach in guiding the model to generate valid JSON structures.

Q2: The prediction accuracy checks if the desired string is present anywhere in the json values. As such, an answer that would print out e.g. all of the words in the context into one large json would score well.

In the biographical data, there are 28 target labels, each of which is a short string, e.g. professor, teacher, nurse. In practical generations, models tend to generate JSON values that contain more detailed information derived from the context, such as “assistant professor of computer science”, “English teacher”, “retired nurse”, etc (see examples in the table of Q1). To accommodate these variations given different contexts, we evaluate the prediction accuracy by checking if the short string of labels is present in the JSON values.

To further ensure a rigorous evaluation, we reevaluate the results with an additional criterion: the JSON values must be concise, with a word limit (e.g., 10 words for ‘professor’ class and 3 words for the others). A re-evaluation of the prediction accuracy with the word limit criterion yielded an accuracy of 84.42% for LLAMA-7B using PASTA, only marginally lower than the originally reported 85.09% in Table 1. This minimal discrepancy can support our claim that PASTA effectively guides the model to generate relevant and concise JSON values. The detailed generation results, available at https://anonymous.4open.science/r/PASTA-10E9/model_generations/json_formatting/llama7b_pasta_generation.json , demonstrate that the occurrence of the model outputting entire contexts as JSON values is much infrequent.

We believe these measures robustly evaluate the model's prediction accuracy and effectively mitigate the concern of scoring well through irrelevant verbosity. We invite the reviewer to review the provided results to further assess the effectiveness of our approach.

Q7: For a full comparison with zero-shot prompting, could you test the effect of poor choice of emphasis in the input text?

We would like to remind the reviewer that PASTA allows users to emphasize any part of the input text to guide model generation. However, the effectiveness of this emphasis largely depends on capturing the essential information intended to be highlighted. In Section 5.2, we show that PASTA consistently improves instruction following when emphasizing the entire instruction, regardless of how they are rephrased. We can only highlight instructions here to illustrate this. To further explore the varying emphasis, we conducted an additional ablation to test the effectiveness of PASTA when only emphasizing partial information of instructions on the JSON Formatting task. The instruction of this task consists of two parts: (i) (question) “Answer the occupation of {person} and generate the answer as json format”; (ii) (demonstration) “Here is an example: {“name”: , “occupation”: ,}”. We evaluate the performance of PASTA when emphasizing only partial information (question (i) or demonstration (ii)):

The results from the table above indicate that while full emphasis on the instruction yields the best performance, PASTA still performs effectively even when only a part of the instruction is emphasized, indicating the robustness of PASTA on varying emphasis. Moreover, we can have models overlook the instruction entirely by emphasizing the other parts (contexts) except for instruction. In this way, the model is biased to totally ignore the instruction, leading to almost zero-scored performance. Hence, these results reveal that an effective emphasis should be accurate and informative enough to reflect the user's intention.

Q8: In Section 5.2, what is the reason to compare against zero-shot and not a few-shot setup?

We mainly present zero-shot performance as the baseline in Section 5.2 as this section specifically focuses on the model's sensitivity to prompt rephrasing in our tasks, which is clearer in the zero-shot setting. The table below includes the performance of few-shot prompting of LLAMA-7B under the same settings.

Q5: The notion that PASTA improvements would transfer to such models would thus need to be tested.

We test PASTA’s applicability to Vicuna-7B-v1.3, an instruction-tuned model from LLAMA-7B. We apply PASTA using attention heads selected from LLAMA-7B profiling, including intersection and task-specific heads. Then we evaluate the performance of Vicuna across four tasks. The table below presents the results.

The results demonstrate that the attention heads selected for LLAMA-7B effectively steer Vicuna-7B, indicating that re-profiling is not necessary for instruction-tuned models. Notably, when steering task-specific heads selected from LLAMA profiling, PASTA significantly enhances Vicuna's performance across all tasks. This evidence shows that PASTA can complement instruction tuning to further improve the model ability of instruction following.

Q6: Furthermore, the cost of the model profiling phase should be compared to the cost of a lightweight instruction-tuning setup.

Profiling for PASTA is an inference-only process, conducted on a small subset of the training set. For instance, profiling all heads of LLAMA-7B with 1000 samples requires 1000 * 32 * 32 inferences. In contrast, fine-tuning a model like Vicuna-7B on a dataset such as 128k ShareGPT data for 8 epochs demands an equal number of forward passes, along with additional backward passes that incur much higher memory footprints. Fine-tuning also involves multiple trials of hyperparameter tuning, whereas profiling is performed only once for each model. Moreover, fine-tuning requires maintaining separate copies of fine-tuned model weights while PASTA does not incur any additional model checkpoints and can be applied plug-and-play.

Meanwhile, the robustness of head performance ranking to sample variance allows us to further reduce the sample size for profiling (e.g., $|\mathcal{D}|=200$). The new table below presents the PASTA performance on the JSON Formatting task when re-profiling with $|\mathcal{D}|=200$ samples. We can see PASTA still achieves superior performance when profiling with much fewer examples.

Q3: The evaluation for pronoun change looks only at whether specific pronouns ("she", "he", "her"...) are no longer in the generated text, without verifying that the rest of the text is unchanged. As such, an empty generation would score 100.

Thanks for your comments. We understand the importance of not only changing pronouns but also maintaining textual integrity and generation quality. To ensure this, we employ two evaluation metrics:

1. Fluency Evaluation (Meng et al., 2022): As mentioned in Section 4, we assess the fluency of all generations (the average bigram and trigram entropy of generations), and exclude results with a fluency score below 3.0. This step effectively eliminates degenerated or repetitive generations from consideration, such as empty text.

2. Consistency Metric: We employ an additional consistency metric (introduced by Hernandez et al., (2023)), which measures the average tf-idf similarity between the generated text and reference texts of full dataset. This metric helps us measure how well the generated text aligns with overall contextual inputs in terms of content and style.

The table below presents a few examples of LLAMA-7B generation with PASTA and their fluency and consistency scores (please see more in Appendix D and all generation results are available at https://anonymous.4open.science/r/PASTA-10E9/model_generations/pron_changing/llama7b_pasta_generation.json).

From the examples, we can see that repetitive or meaningless generations receive low fluency (below 3.0) and consistency (below 8.0). The generations with high fluency (around 5.0) and consistency (above 13) are meaningful and readable. The following table presents the results of average consistency and fluency evaluation on the Pronouns changing task.

These results indicate that PASTA achieves comparable consistency and fluency scores to other prompting methods, suggesting that it effectively maintains the quality of generated texts. The slightly lower consistency score of PASTA than zero-shot prompting is a result of the necessary pronoun substitutions. We provide all generation results in the link above which show how PASTA performs in maintaining text integrity while effectively implementing pronoun changes.

Q4: Clarification of the results in Figure 2b

Figure 2b indicates the drawbacks of overemphasizing. When users overemphasize specific parts by steering too many heads (e.g., 294), it leads the model to only focus on the specific information while ignoring other parts. This imbalance results in the deterioration of overall generation quality. In the example of Pronouns changing task, when 294 heads are steered, the model becomes overly fixated on emphasized parts, leading to repeated generations, such as “Lindsey is a textile worker. They are a textile worker. They are …”, This deterioration reflected in the decreased consistency score of 7.09 and fluency score of 3.16. Therefore, Figure 2b suggests applying PASTA to a moderate number of heads (typically 30 to 150) is ideal, which strikes a balance between effective emphasis and generation quality.

I thank the authors for their responses. While the additional data they provide is reassuring, overall I am still concerned by the brittleness of the proposed evaluation metrics: 1. while it is good to know that altered models hold a similar bigram and trigram distribution (see definition of fluency), there is a wide gap between this and an answer that is semantically close to the desired outcome. 2. The accuracy metric defined in the paper does not reflect the code implementation: "Accuracy evaluates the ratio that ‘she/he’ are successfully changed to ‘they’ in model generations." However, as noted in my first review, the accuracy metric defined in the code measures how accurate the model is at removing said pronouns, not at changing them. 3. The Json accuracy metrics implementations also do not match their definition in the paper. "Prediction accuracy (P. Acc.) measures the accuracy at generating the correct target in JSON values after loading the JSON-formatted generations." implies that the model (1) generates only one Json output and (2) that the desired output is unique.

As these discrepancies are meaningful to the paper and have not been addressed in the comments, I do not wish to change my rating for this paper.

We appreciate the reviewer for your constructive feedback. We would like to provide our response to your questions as follows.

Q2: This makes a conjecture that such intervention could inadvertently impair other abilities of LLMs, such as generative fluency and reasoning.

Thanks for your comment. We fully understand the importance of preserving generative fluency and quality while enhancing task-specific performance with PASTA. To ensure this, we employ two metrics to evaluate the quality of PASTA generations across three natural language generation tasks (Prons. Changing, BiasBios, and CounterFact).

1. Fluency Evaluation (Meng et al., 2022): As mentioned in Section 4, we assess the fluency of all generations (the average bigram and trigram entropy of generations), and exclude results with a fluency score below 3.0. This step effectively eliminates degenerated or repetitive generations from consideration.

2. Consistency Metric: We employ an additional consistency metric (introduced by Hernandez et al., (2023)), which measures the average tf-idf similarity between the generated text and reference texts of full dataset. This metric helps us measure how well the generated text aligns with overall contextual inputs in terms of content and style (higher is better).

The table below presents a few examples of LLAMA-7B generation with PASTA and their fluency and consistency scores on the Pronouns changing task (please see Appendix D for more examples).

From the examples, we can see that repetitive or meaningless generations receive low fluency (below 3.0) and consistency (below 8.0). The generations with high fluency (around 4.5) and consistency (above 13) are meaningful and readable. The following table presents the average fluency and consistency evaluation across the mentioned tasks:

The results show that PASTA achieves comparable consistency and fluency scores to zero-shot prompting. This indicates that PASTA effectively maintains the generative quality and fluency while significantly improving the task efficacy.

Meanwhile, we would like to kindly remind the reviewer that Section 5.3 (Figures 3a and 3b) has discussed the reviewer’s conjecture. The generative fluency can be impacted only when users overemphasize highlighted input spans by steering too many heads (e.g., 294). Such excessive emphasis can cause the model to focus narrowly on specific information while ignoring other parts of inputs, decreasing consistency and fluency. Therefore, we recommend applying PASTA to a moderate number of heads: typically between 30 to 150. Steering this range of heads already yields significant improvement in task performance while maintaining high generative quality, striking a good balance between emphasis efficacy and generative fluency.

Q4: The previous studies on syntax-based attention and multiscale attention should be considered baselines for comparison, either in related work or experiments.

Thanks for your feedback regarding the relevance of prior studies on syntax-based and multiscale attention. Multi-scale attention [3] introduces varied attention scales across different heads, enabling diverse functionalities. Syntax-based attention [4,5], leverages the syntactic structures of inputs to modify attention patterns. However, these attention mechanisms require additional training to adapt to existing LLMs like LLAMA. By contrast, PASTA does not require additional training or alternations in model parameters, which allows PASTA to be seamlessly integrated with existing LLMs. We appreciate your comments on these insightful prior works and will discuss them in the related work of our next version.

Q5: It is crucial to evaluate the proposed methodology's efficacy by applying it to some commonly used generative tasks, such as machine translation and question-answering tasks.

Thanks for your valuable feedback regarding more experimental comparisons on commonly used tasks, such as MT and QA. We fully recognize the importance of evaluating our methodology across these standard benchmarks and make every effort to conduct these additional comparisons during the rebuttal phase. However, we cannot access the same computational resources as before. Due to the limited computational capacity and time, we may not be able to finish all these additional experiments on time. Please be assured that we are committed to expanding the scope of our research. We will prioritize completing these experiments and will share the new results as soon as they are available.

Q6: Response to other questions.

Regarding the question of supervised fine-tuning (SFT), PASTA is an inference-only method and does require any fine-tuning or re-training. Therefore, we did not perform any fine-tuning for the models. All evaluations are performed on the pre-trained models. We will include detailed implementations of accuracy evaluation in the appendix of our next version.

[1] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022.

[2] Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models, 2023.

[3] Qipeng Guo, Xipeng Qiu, Pengfei Liu, Xiangyang Xue, Zheng Zhang. Multi-Scale Self-Attention for Text Classification. AAAI 2020.

[4] PengFei Liu, Xipeng Qiu, Xuanjing Huang. Syntax-Based Attention Masking for Neural Machine Translation. 2016.

[5] Colin McDonald and David Chiang. Syntax-Based Attention Masking for Neural Machine Translation. NAACL 2021.

Q1: Could an automatic extraction mechanism be designed to improve this method? This could entail the development of a dedicated extraction model or the crafting of specialized prompts.

We appreciate the reviewer’s comment on potential exploration for improving PASTA. Your suggestion of developing an automatic extraction mechanism is indeed a valuable direction for future work. The primary contribution of PASTA lies in empowering users to selectively highlight information according to their intentions, thereby effectively steering model generation. As an initial work in post-hoc attention steering, we motivate future efforts to combine PASTA with methods capable of automatically detecting input spans relevant to user intentions. For example, one potential approach is to employ a syntax parser to dissect the structure of input strings and analyze the sensitivity of each part to model outputs. The ideas suggested by the reviewer are also interesting – tailoring an auxiliary model or specializing prompts to extract the task-specific emphases for PASTA. We believe these possibilities can further promote the development of attention steering, offering more effective ways to guide model generations.

Q3: it would be beneficial to test the effectiveness of the proposed method on diverse large language models, including but not limited to, 7B-LoRA, 13B-LoRA, and 13B, across specific tasks.

We conduct experiments with LLAMA-13B and Vicuna-7B to further evaluate the effectiveness of PASTA across all tasks. The following table presents the performance comparison for LLAMA-13B (Flues./Cons. means fluency/consistency scores).

The results demonstrate that PASTA significantly improves the performance of LLAMA-13B across all tasks, indicating its generability to larger model sizes. Meanwhile, PASTA maintains or improves fluency and consistency scores compared to zero-shot prompting on three of the tasks, suggesting that PASTA does not impair model generative ability and fluency.

We further test PASTA’s applicability to Vicuna-7B-v1.3, which is instruction-tuned from LLAMA-7B. We apply PASTA using attention heads selected from LLAMA-7B profiling (including multi-task and task-specific heads). In this way, we evaluate if the heads selected from the base models are transferable to an instruction-tuned model, thereby avoiding the re-profiling. The table below presents the performance of Vicuna across all tasks.

The results demonstrate that the attention heads selected for LLAMA-7B effectively steer Vicuna-7B, indicating that re-profiling is not necessary for instruction-tuned models. Notably, when steering task-specific heads selected from LLAMA profiling, PASTA significantly enhances Vicuna's performance across all tasks. This evidence shows that PASTA can complement instruction tuning without necessitating re-profiling.

Thanks for your helpful suggestions. We provide our response to your questions as follows:

Q1: The weight matrices of two FFN layers and value projects are required to compute the next layers’s output.

We appreciate the reviewer for raising this question. To further clarify, our method is not designed for black-box models. PASTA requires reading and updating attention scores calculated by attention modules. For model weights or other intermediate outputs, PASTA does not interact with, read, or modify them. The steering operation is strictly limited to the attention mechanism. If the model only provides the attention scores during the forward pass, we still can effectively apply PASTA.

Q2: What is the general effect of having highlighted instruction at the start or some other place in the prompt?

We would like to remind the reviewer that both highlighted input spans in BiasBios and CounterFact tasks are not at the end of the prompts. As mentioned in Section 4, we highlight the first sentence for the BiasBios task, which carries the most information for prediction. In the CounterFact task, we highlight new facts, which are in the middle of prompts. Meanwhile, Table 4 already shows the performance of PASTA on the JSON Formatting task when the highlighted instruction is at the beginning of prompts. These results demonstrate that PASTA consistently improves the performance no matter where highlighted input spans are.

In the following table, we further present the PASTA performance for LLAMA-7B when the highlighted instruction is at the start of the Pronouns changing prompts.

The results illustrate the sensitivity of model performance to prompt rephrasing – having the instruction at the start for Pronouns changing task degenerates the accuracy of zero-shot prompting to 19.63%. In contrast, PASTA can improve its performance by 31.3%, indicating the effectiveness of PASTA for highlighting instructions at different positions.

Q3: The baseline performance would definitely improve using these generation methods. How does PASTA compare to the baselines using these sampling approaches?

To clarify the reviewer’s question, we apply the greedy search for all of the methods including prompting strategies and PASTA to have a fair comparison. Moreover, in the BiasBios and CounterFact tasks, models mainly use the logit of the first generated tokens to make predictions, and hence greedy search should be used. The following table presents the zero-shot performance of LLAMA-7B on the JSON Formatting and Prons. Changing tasks when applying beam search and topk sampling.

From the results, we can see that PASTA still outperforms baselines when using the beam search or topk sampling strategies.

Q4: *-marked and "" - marked have very low performance. Have the authors tried better markers like <mark> </mark> or <emphasize> </emphasize>.

In the following table, we present the performance of these marked prompting baselines on LLAMA-7B across four tasks.

Q5: Response to other questions.

We appreciate the reviewer for pointing out typos in our paper and other suggestions for the presentation. We will follow these suggestions and make revisions in the next version.

We appreciate the reviewer for recognizing the novelty and effectiveness of our methods. We would like to provide our response to the reviewer’s question as follows.

Q1: Experiments on larger models should be conducted to ensure the performance across different models.

We conduct experiments with LLAMA-13B and Vicuna-7B to further evaluate the effectiveness of PASTA across all tasks. The following table presents the performance comparison for LLAMA-13B.

The results demonstrate that PASTA significantly improves the performance of LLAMA-13B across all tasks, indicating its generability to larger model sizes.

We further test PASTA’s applicability to Vicuna-7B-v1.3, which is instruction-tuned from LLAMA-7B. We apply PASTA using attention heads selected from LLAMA-7B profiling (including multi-task and task-specific heads). In this way, we evaluate if the heads selected from the base models are transferable to an instruction-tuned model, thereby avoiding the re-profiling. The table below presents the performance of Vicuna across all tasks.

The results demonstrate that the attention heads selected for LLAMA-7B effectively steer Vicuna-7B, indicating that re-profiling is not necessary for instruction-tuned models. Notably, when steering task-specific heads selected from LLAMA profiling, PASTA significantly enhances Vicuna's performance across all tasks. This evidence shows that PASTA can complement instruction tuning without necessitating re-profiling.

Q2: Since different layers are known to possess different functions. Which layer is more important for attention steering? By Figure 2 it seems that it’s approximately random, is there any explanation for this?

Thanks for your great comment. According to our observation (please see Figures 2,4,5,6), there are some layers like layer 19 in LLAMA-7B that can be generally steered for all of the tasks to improve the performance. These layers can be applied to attention steering in task-agnostic manners. In most cases, each task prefers different layers for attention steering, which should be used task-specifically. More fine-grainedly, there is a similar observation at the level of attention heads. Therefore, we propose multi-task model profiling to identify these task-specific/task-agnostic layers and heads. As an initial work, we believe that PASTA can motivate future work on attention steering.