Dear Reviewer 7h5s,

We would like to thank you for reviewing our paper and acknowledging its novelty and the importance of the questions raised. We address all your questions and concerns below.

Weakness 1: “MT-Bench Reliability: The paper relies heavily on MT-Bench for evaluation. A more critical discussion of MT-Bench's specific limitations, or validation on a broader suite of benchmarks with diverse evaluation methodologies, could enhance the robustness of the empirical claims.”

• We have now evaluated our main pipeline (Command R7B+7B) on a further widely-used benchmark, Arena-Hard. Not only does this benchmark differ from both UltraFeedback and MT-Bench in terms of task distribution, but its reward assignment strategy and range are also different from the reward that we trained on, as detailed in [1]. In addition, we evaluate both with the Command R+ judge used during training as well as an unseen judge, GPT-4.
• Below we show the results, averaged across runs, obtained by the candidate LLM on Arena-Hard with and without injected preambles. As in MT-Bench, higher scores are better. We observe clear improvement on this new benchmark using the Command R+ judge. Improvement with GPT-4 judgments is less significant; however, we still observe an upward trend when averaged across runs.
• These results indicate that the method generalises to an additional benchmark without targeted training. We will be happy to add them to the paper.

Weakness 2: “Limited Scope of Quality Improvement vs. Bias Hacking: The paper currently notes that accuracy in math and reasoning tasks did not improve despite higher scores. However, it's critical to more definitively disentangle whether the increased LLM-as-a-judge scores are genuinely due to an improved response quality (e.g., better structure, clarity) or primarily by "hacking" the judge's biases (e.g., verbosity, formatting preferences). A more systematic analysis across diverse tasks (e.g., general chat, instruction following as in IFEval) with human-in-the-loop quality evaluations would be beneficial to confirm if these preambles lead to universally better responses or merely exploit specific judge biases.”

• We appreciate and agree with this suggestion. We have computed the factual accuracy scores for all math and reasoning questions, w.r.t. their reference answers, for all models and question turns (120 questions in total). As shown in the table below, we find no evidence that math and reasoning results improve, despite the higher LLM-as-a-judge rewards. We will add these to the paper.

• Moreover, we have now run a human-in-the-loop qualitative evaluation of a 25% sample of all turn-1 responses generated by both the non-attacked Command R7B and the same model with injected preambles. We had three expert annotators without knowledge of the attack assess the overall quality of each response and label it as “Poor”, “Fair” or “Good”. We show the number of each label assigned to the non-attacked and the attacked responses below. We found the label distributions to be fairly similar, with the non-attacked model receiving slightly more scores at either end of the scale (‘Poor’ and ‘Good’), and the attacked one receiving slightly more mid-range scores (‘Fair’).

• Finally, we had the annotators assign a discrete 1-10 rating to each response (the same scale used by the judge-LLMs in our experiments). Please see the average ratings in the table below. The difference between them is slightly in favour of the preambles method (+0.05); however, this difference is small compared to the substantial increase in average LLM-as-a-judge rewards obtained by the attacked model in our experiments. We plan to extend this study so that it covers all questions (rather than a 25% sample) and include it in the camera-ready.

Question 1: “Could you provide a more detailed explanation of what "injected" means in the context of preambles, specifically contrasting it with how previous "appending" methods operate on responses? How does the LLM's internal processing of an injected preamble differ from a post-hoc appended text sequence, beyond just the position in the input?”

• The previous methods intervene on a pre-generated response, by adding text to it or otherwise modifying it post-hoc in order to ‘trick’ the LLM-as-a-judge to increase the score (for example, the universal attack first generates a response, then appends to it four words which have been tuned separately to increase the judge’s reward). Unlike many other methods that modify the generated response, we modify only the preamble (i.e. system prompt) to the LLM that generates the response. This is both a more realistic setting and substantially harder to detect as the generated response is entirely LLM-written.

Question 2: “Given the demonstrated undetectability of your attack by existing methods, what are some potential novel detection strategies or defense mechanisms that could be developed to identify and mitigate this type of preamble-based adversarial attack?”

• That is a fair question. Presently, PPL analysis introduced by [2] is the established detection method, used in prior and contemporary work on LLM attacks (e.g., [3], [4]). It is also well-aligned with OOD detection and adversarial robustness certification in general machine learning literature, where the probability of a prediction is flagged if lower than a threshold. It is possible that if we had access to the entire probability distribution over the vocabulary at each decoding step for the candidate LLM, or other mechanisms such as the attention weights, we could detect subtle variations caused by the attack. However, this analysis is rarely possible in practice, since LLMs served via API (like those that we use as frozen candidates in our pipeline) do not provide this degree of access. Still, we do agree with you that these are considerations worth mentioning; we will add a brief discussion of the above points to the paper.

Please let us know if our answers and the additional experiments above address your concerns and questions, or whether we can provide more details. Meanwhile, we would like to thank you for your contributions toward improving the work.

Kind regards,
The Authors of Submission #23360

---

[1] (Li et al., 2024) From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
[2] (Jain et al., 2023) Baseline Defenses for Adversarial Attacks Against Aligned Language Models
[3] (Raina et al., 2024) Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment
[4] (Zheng et al., 2025) Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

Dear Reviewer nFn5,

Thank you for reading and reviewing our work. We are happy that you found the method thorough and well thought out. Please find below our responses to your comments and questions.

Weakness 1: “The experiment on the success rate of the method is promising but is only performed on a single small dataset [...] the results would be strengthened if the method can also generalise to less controlled settings. This can be mentioned as potential future work. For example, you could train/test on subsets of questions from the LMSYS dataset”

• Thank you for the suggestion. As you said, the main focus of the work is demonstrating that it is possible to reverse engineer human feedback in a controlled setting, and we found that the data we use is diverse and challenging. However, we agree with you that generalisation in less controlled settings is interesting and worth exploring. We have now evaluated the Command R7B+7B pipeline on the Arena-Hard benchmark, from the LMSYS suite. This has a different task distribution than our training data, and it also involves a different reward assignment strategy and range than the one we trained on. To make the evaluation setup even more generalisable, we evaluate with two different judges: Command R+ (i.e., the judge from our training setup) and GPT-4 (a commonly used judge for Arena-Hard evaluations).
• Below, we show the rewards obtained on Arena-Hard by the candidate LLM responses before and after the injection of the tuned preambles, averaged across runs. We find that there is a clear improvement using the Command R+ judge. With the GPT-4 judge, the improvement is less significant (within standard deviations); however, we still observe a positive trend in the point-wise averages. Overall, we find evidence that the method generalises to an additional benchmark without targeted training. We will add these additional experiments to the paper.

Weakness 2: “The claim that this method exposes the vulnerability of the LLM-as-a-judge framework is a bit odd. To argue this, you would need to compare the LLM-judge evaluations against human evaluations on the same answers (given by your framework and other baselines), to see whether human evaluators also give higher scores to outputs given by your framework. The results could be framed positively instead, in that you provide a framework to generate system prompts that lead to better outputs from candidate LLMs (rather than it being presented as an attack).”

• We would argue that even if the responses did improve by any other metric aside from the LLM-as-a-judge scores, this could still be referred to as an 'attack' in this context, since the preambles are tuned by directly exploiting the judge’s reward to maximise the benchmark scores.
• That said, we do appreciate your suggestion for the additional experiment. We have carried out a blind human evaluation study on a 25% sample of all turn-1 responses generated by both the non-attacked Command R7B and the same model injected with the preambles. We had three expert annotators without knowledge of the attack assess the overall quality of each response and label it as “Poor”, “Fair” or “Good”. We show the number of each label assigned to the non-attacked and the attacked responses in the table below. Note that the label distributions are fairly similar for both models, with the non-attacked model receiving slightly more scores at either end of the scale (‘Poor’ and ‘Good’), and the attacked one receiving slightly more mid-range scores (‘Fair’).

• Additionally, we had the annotators assign a discrete 1-10 rating to each response (the same scale used by the judge-LLMs in the paper). Below we show the average ratings obtained by each setup. We find that the difference between the human ratings of the two models is small (0.05), whereas we know that the difference between the LLM-as-a-judge ratings is substantially larger in favour of our attack. This suggests that the method exploits, in large part, the judge’s specific preferences.

• We will extend the above human experiment to all questions rather than just a subsample, and show these results in the Appendix along with a discussion of the above points.

• We also absolutely agree with you in principle that the method has potential for useful applications rather than just 'attacks'. We also consider identifying weaknesses of any system positive, as it provides us with the opportunity to address them and build more robust systems that are better aligned with our objectives.

Question 1: “I find it interesting how different/gibberish the preambles are for Llama compared to Command. Do you have any hypothesis as to why this happens?”

• We believe this happens mainly because we perform all hyperparameter tuning on the Command pipeline, due to computational limitations, and apply the same hyperparameters to the Llama pipeline. Among the hyperparameters that we tune is the value of the KL-divergence weight $\beta$, which regulates the faithfulness of the token distributions to those of the reference model, and therefore determines the fluency of the preambles. Note that we prioritise downstream reward over fluency since our preambles do not necessarily need to be human-readable. Therefore, our value of $\beta$ is relatively low, as described in Appendix B.2 of the paper. We found that, with this low $\beta$ value optimised on the Command pipeline, the Llama model tends to steer away from fluent outputs faster and more drastically, while the downstream rewards increase. This may be due to pre/post-training strategies that are unique to each model family. We consider the Llama results extremely interesting as well, since they show that, in this context, maximising exploration and downweighting the KL-divergence anchor produces effective preambles regardless of their fluency.

Question 2: “Do you have results per question type (as in Figure 2) for the other attacks as well? If yes, then it would be nice to add in an appendix for completion.”

• Indeed, we do have them for every attack. We will add to the Appendix tables showing the fine-grained rewards assigned to each domain sub-category under each attack.

Question 3: “In your formalization of the RL problem (line 116), in the first expectation subscript you write $(i, q) \sim \mathcal{D}$. I believe this is a typo? Since $i$ is independent of $\mathcal{D}$.”

• Well spotted, thank you. Indeed, $i$ is independent of $\mathcal{D}$, and is the fixed part of the input, so it should be removed from the expectation altogether. We have immediately corrected this typo.

Question 4: “Other minor typographical errors: typo in line 230 (unversal), broken reference in line 250”

• Thank you again, we have now fixed these too.

We hope our responses and additional experiments above have addressed the concerns and answered your questions. Please let us know if other questions arise, and thank you for helping us strengthen the paper.

Kind regards,
The Authors of Submission #23360

Dear Reviewer FfKE,

Thank you for your review. We are happy that you found the idea original and the work sound and comprehensive. Please find our responses below.

Weakness 1: “Narrow judge set: training uses a single judge (Command‑R+); only GPT‑3.5 is used for transfer. Limited human study: 7 experts, 175 answers – too small to support broad “undetectable” claim. No statistical test of improvements; only descriptive statistics.”

• The idea of exploring a training setup with an LLM jury was discussed during the experiment design phase. However, we prioritised training with one judge mainly because, in our setup, the judges are queried via API. Therefore, issues of throttling and query failures/retries would multiply by the number of judges, potentially compounding and significantly slowing training. Of course, there were also considerations of the total cost of querying multiple APIs. While we did not have the capacity for such a setup, the idea remains interesting to us, and we will add it to the discussion of future work directions.

• As for the judge transferability study with GPT-3.5, we employed the same model used by [1], which is one of our strongest baselines. However, following your suggestion, we have now evaluated with further judges and found that the effectiveness of our attack still transfers (see our response to Question 1 below).

• Regarding the human study, we computed the median absolute deviation (MAD) for both accuracy and F1 scores across annotators for each attack type. In all cases, the MAD was zero, indicating that, within our relatively limited sample, different annotators were either consistently able or consistently unable to detect a given attack type. We believe this supports the findings and substantiates their robustness. That said, we acknowledge the limitations of the sample size, and we will revise the paper to clarify that our findings provide preliminary evidence of limited detectability under the specific conditions of this study.

• Finally, for the statistical tests, we now report paired bootstrap confidence intervals as you recommended (see our response to Question 2).

Weakness 2: “Paper is long and occasionally repetitive; some tables/figures (e.g., radar plots) are hard to read in grayscale; appendix contains critical parameters (β, early‑stopping) that should be surfaced earlier.”

• Thank you for pointing this out. We have now checked how our radar plots appear in greyscale and will change the shades so that they remain readable.
• We agree that certain details should be moved to the main body; we were planning to use the extra page allowed for the camera-ready to do so. We will also review the paper for any sections that may appear redundant and streamline them to accommodate more critical details currently in the appendix.

Weakness 3: “Impact depends on adoption of LLM‑as‑a‑judge; if community shifts to hybrid or adversarial evaluation, severity may diminish.”

• We agree with you, but we do suspect that since the LLM-as-a-judge paradigm has already demonstrated value in the field, it will continue to form at least a component of pipelined systems in the near future.

Question 1: “Judge diversity – How does RLRE perform when rewards come from an ensemble of heterogeneous judges (e.g., four different models) rather than a single one? Please include results or discuss expected behaviour.”

• We have now evaluated the R7B pipeline with two additional LLMs, for a total of four judges: Command R+, GPT-3.5, GPT-4o-mini and Claude 3 Haiku. Please see the average results for the overall MT-Bench score in the table below, both for each judge individually and as a mean score of the 4-judge jury, where each judge is assigned equal weight.
• We observe some variation in the scale of the rewards for different judges (i.e., some are 'stricter' in their evaluations than others). Nevertheless, for any given judge, the preamble-based attack still obtains the highest rewards relative to the existing baselines and the vanilla model. This also reflects a higher jury score (last row in the table).

Question 2: “Robust statistics – Could you report confidence intervals (e.g., paired bootstrap over questions) to substantiate significance of the 0.1‑0.6 score gains?”

• Of course, please find below the CIs computed for non-attacked and attacked responses via paired bootstrapping, for all three candidate LLMs.

Question 3: “Compute & cost – Training requires many reward queries. Please quantify total API calls and cost, and compare with prior attacks. Under what budget would the attack cease to be practical?”

• We do at most 128k API queries to the judge-LLM (i.e., training for 1k steps on the 64k UltraFeedback samples with bs=64 and 2 completions per prompt). Compared with [1], which is the only other 'trained' attack among our baselines, our training is substantially cheaper. For $n$ training samples, the exhaustive search method over the 20k-word vocabulary used by [1] needs to make $20000 \times 4 \times n$ calls to the judge. For context, making 128k inference calls to the Command R+ API when training with our method, at the average input and output token budgets we observed, costs approximately 500 USD. In contrast, the method in [1] requires using smaller training splits due to the massive cost and even so, it is more expensive (e.g., on just 500-1000 samples, the equivalent cost for this method is several thousand USD).

Question 4: ‘Content quality – Fig. H.1 (p. 25) shows higher‑scored yet incorrect maths answers. Have you measured factual accuracy systematically (e.g., GSM‑8K exact‑match)? This would clarify whether judges are truly “fooled”.’

• We have measured the factual accuracy of all math and reasoning questions w.r.t. their reference answers, for all models and question turns (120 questions in total). These are very similar, as shown in the table below, thus the evidence indeed points to the judge being 'fooled' for these factually measurable domains.

Question 5: “Defences – Given that perplexity fails, have you tried delta‑answer inspection (diff between outputs with and without preamble) or attention‑based detectors? Evidence would guide future mitigation.”

• It is indeed possible that studying attention maps, or having access to the entire probability distribution over the vocabulary at each decoding step for the candidate LLM, may reveal subtle variations that result from the attack. However, please note that this analysis is rarely possible in practice, since LLMs served via API (like those that we use as frozen candidates in our pipeline) do not give access to the internal mechanisms, nor do they return the entire vocabulary distributions. For this reason, the PPL analysis we perform (introduced by [2]) is currently the established detection method, used in prior and contemporary work on LLM attacks (e.g., [1], [3]). We will add a discussion of these points in the paper.

Question 6: “Score could increase if additional judges and a larger human study confirm undetectability, or if authors provide stronger evidence that inflated scores do not correspond to better factual quality.”

• As you suggested, we have evaluated with further judges (see our response to Question 1 above), and we also provide strong evidence that the inflated scores observed on math and reasoning do not map to improved accuracy (see response to Question 4).
• As for the large-scale human study, this would be too complex and expensive to set up in few days. However, we have provided statistical analysis details of the current human annotations in our response to Weakness 1 above (third bullet point), and we also commit to adjusting the language in the manuscript to clarify that we find early evidence of limited detectability, within the context and constraints of our relatively limited-scale human evaluation.

We hope that our additional experiments robustly address the questions you raised, indicating both transferability to further judge-LLMs as well as non-correspondence with factual quality, and that this is sufficient to consider raising the score as suggested. We will add all of the above results to the paper. Please let us know if you have further questions or comments, and thank you for helping us improve the manuscript.

Kind regards,
The Authors of Submission #23360

---

[1] (Raina et al., 2024) Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment
[2] (Jain et al., 2023) Baseline Defenses for Adversarial Attacks Against Aligned Language Models
[3] (Zheng et al., 2025) Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

Dear Reviewer eyDG,

We would like to thank you for taking the time to review our paper. We are glad that you found the experiment design and execution adequate and the idea original. Please find our responses to your comments and questions below.

Question 1: “Line 250: Missing reference to Table.”

• Thank you for spotting this. It will be fixed immediately.

Question 2: “Appendix G.2 and G.3 (Preamble 6 and 9): Is there an explanation to why the Llama 8B+70B preambles do not look like natural English text as the other ones. I would assume the perplexity filter would easily flag these. It would be great to have some clarification.”

• We believe the difference in fluency is mainly due to the setup in which we tune the KL-divergence weight $\beta$. We perform all hyperparameter tuning on the Command pipeline (due to computational limitations) and apply the same hyperparameters to the Llama pipeline. These include $\beta$, which regulates the faithfulness of the token distributions to those of the reference model, and thus the fluency of the preambles. Note that we prioritise downstream reward over fluency, since our preambles do not necessarily need to be human-readable, leading to a relatively low value of $\beta$ (as discussed in Appendix B.2 of the paper). We found that, using this low $\beta$ optimised on the Command pipeline, Llama's fluency steers away from the reference model faster and more decisively, while the downstream rewards increase. This behaviour may be due to differences in pre/post-training strategies, which tend to differ across distinct model families. Overall, we consider the Llama results particularly noteworthy, as they show that unfluent preambles tuned to produce high rewards are indeed effective in this context.

• Indeed, unfluent preambles are not flagged by the perplexity filter. This is because the perplexity filter assesses the candidate LLM’s outputs (which are what the judge is able to see), and not the preambles which are part of its input (hidden from the judge). Analysing the LLM outputs is standard in prior literature and consistent with the intended use of the PPL filter detection method introduced by [1]. Remarkably, the frozen LLM’s outputs remain fluent even when we inject into it these unfluent preambles.

We hope the responses above answer your questions. Please do let us know if you have other questions or comments.

Kind regards,
The Authors of Submission #23360

---

[1] (Jain et al., 2023) Baseline Defenses for Adversarial Attacks Against Aligned Language Models