Pay Attention to What Matters

• There is already a prior research on prompt engineering method that shows that enclosing the tags improves performance, but this study does not describe the relationship. e.g. https://arxiv.org/abs/2309.13078
• Assessing the impact of attentions from the lens of vector norms is already a hot topic, but there is no description about the relationships. https://arxiv.org/pdf/2004.10102
• Lack of relationships between sections, mathematical symbols, and descriptions of key points in each section makes the manuscript difficult to read. For example, In 3.1 DESCRIPTION OF THE METHOD, which is a preliminary setup and which is the proposed method? Another example is that in 3.3 INFLUENCE, R_U function and I_U function at the first layer are not defined so that we cannot understand what those functions mean when we are given only the recursive formula.

• It is missing a very relevant work from ICLR 2024, "Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs" by Zhang et al.
• While it is good to know that the llm is now producing French more often, but this may be at the cost of degraded output accuracy. I suspect that if the attention weights are manually changed, the quality of the outputs may be affected. I think it is very important to measure this as well.
• I think Influence metric shouldn't be called "metric", because a low influence score doesn't necessarily mean the model is not following the instruction well. I think "probe" is a more appropriate term, which helps us to perform intrinsic analysis.

(1) Since this is a training-free method, it would be even better if the authors conducted the experiments on more models with different scales. Given that the experimented models are not very big (<7B) and strong (there are better models now like Llama-3), the lack of comprehensive experiments with different models may undermine the credibility of the experiments.

(2) The tasks the authors selected are rather synthetic. Moreover, the metrics are too simple: for example, the correctness of the French translation and the information extracted for JSON generation should also be evaluated. Even though the focus of the method is instruction following, the correctness of the output should also be measured to understand if the method hurts the performance. Also, the authors should make the evaluation part clearer: for example, I did not find how "whether output is French" is evaluated.

(3) There are several existing instruction-following benchmarks, such as IFEval (Zhou et al.). Even though the existing instruction following benchmarks are also quite synthetic (mostly focusing on writing tasks), the instructions are at least more complex and the evaluation is more rigorous.

Zhou et al. Instruction-Following Evaluation for Large Language Models

Sec 3:

1. The concept of 'influence' as an enhancement to attention rollout is intriguing and initially seems valuable. However, the exact role of influence within the implementation remains somewhat ambiguous. At first the glance, the narrative suggests that influence is computed on a per-example basis to adjust the attention dynamically (delta). Yet, the inclusion of plots and tables depicting a consistent delta value across examples implies that this parameter could potentially be determined via straightforward heuristics or a hyper-parameter search. This raises questions about the essentiality of influence in the proposed method. If influence is not critical to the functionality, a direct comparison of influence and traditional attention rollout—perhaps in basic settings like verb singularity classification—would be imperative to justify its mention as a key contribution. In summary, my recommended experiments would be:

• Add another line for the experiments with "flexible delta based on influence"
• Comparing the performance of influence and attention rollout on the task of localizing the attributed attention for singular verb classification with the same setting than the original attention rollout paper.

Sec 4:

1. The experiments seem to employ a limited set of prompt templates, which might not fully capture the model's versatility in handling varied instructions. The effect of different prompt structures and the positioning of instructions within sentences could potentially alter the outcomes, suggesting a need for a broader range of templates in evaluation.

2. The choice of p(output in French) as the only metric, while straightforward, does not address the quality of the summarization. Considering the discussion on potential issues with high attention augmentation (e.g., delta=2 leading to nonsensical outputs), it is vital to also assess the quality of the generated summaries, either through human evaluations or automated metrics, to ensure the practical utility of the model outputs.

3. holistic evaluation of the impact: similar to weakness 4.2, there is a need for a more comprehensive evaluation of how attention augmentation influences the model’s overall comprehension and performance. Specifically, how does attention augmentation affect the comprehension of the other part of the input? Does it degrade the general performance? Could it be used my malicious users to hack and jailbreak the aligned LLM by forcing attention at unintended tokens? I understand that these evaluations may require a lot of effort but I believe they are crucial questions to be answered to claim the method to be truly useful. Here are some proposed experiments that you may try:

• comprehension: Add a dataset that requires complex reasoning to see if paying heavy attention to the instruction would break the performance.
• safety: Pick an aligned model and test it on an adversarial attack benchmark. Two ways of attack by hacking the attention augmentation: 1. Wrapping the adversarial instruction with the attention special token. 2. Wrapping a random 2/3 of the tokens in the input to significant increases the temperature.