Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective

We greatly appreciate Reviewer e5jB for suggestions on our experimental setups. We are delighted that you acknowledge the richness in detail of our study. We address your concerns below:

Anonymous link (AL) for several tables: https://anonymous.4open.science/r/ughj/e5jB/README.md

Recap: Claims & Objective

We argue we didn't claim Quine outperforms all others, in abstract, we use "Gibberish always matches or surpasses," in introduction, we use "Pruning can near SOTA," and in each section, we present results objectively. Instead, our goal is to deliver insight that simple ICL pruning can be on par with work which use various external information, e.g., +tokens, countering common view and contributing to AI emergence from constraints [1,2]. Quine/TAP are just what we strive to improve under pruning (pursue knowledge over SOTA). Also, comparing search algorithms solely by numbers is limiting, as scalable methods e.g. ES can further improve on the same prune landscape. We'll thus add a figure for sample efficiency, comparing TA/ES/greedy.

Models from 350M

We started from GPT2, 125M (Table 10).

Not tuned baselines?

Pruning results in Table 2 are reported w/ varied ICL initializations, where ICL baseline are direct comparisons. We can't perform BoN across seeds.

• BoN for TAPruning/"Greedy": We stated in Algo 2, which used held-out valid for BoN (same samples).
• BoN for ICL: The only setup available is to construct exemplars using samples from held-out valid (e.g.,200). It costs many computes(N=51^4*4!=16568064 for 1-shot agnews, taking 4 yrs for a 1-shot dataset BoN w/ vLLM). We are unable to support so. That is also why all existing work by validation BoN [3,4+] don't do such enumeration and report ICL as us. We follow such convention. Your request requires other principled algos, beyond our work. We show Bo10 under limited samples in AL.

gradient baselines?(Gb)

Good questions! Let's clarify.

• White-box (wb) jailbreaking: Using steering vector (SV) does NOT imply wb is required. Sorry for ambiguity. We indeed view it hard for Pruning (line362-sparse, line365-potentials), thus we explore w/ a small-scale priming study for dense signals w/ tools from interpretability(SV). We'll enhance paper: show black-box/bbx results first and explore SV. We provide such results in AL README(+GCG result). Besides, we plan to explore further under varied ICL setups in paper (see reply to 8j6A), for rich picture of pruning.

• Other tasks: bbx is enough, and Gb is empirically weak under FSL[6,7].

Not substantial Analysis

We agree the insight is intriguing for analysis. Section 6 presents label words in ICL, where [8,9+] exclusively address it in full papers. Pruned token analysis remains hard, with only viable hypothesis (we) being whether function words are pruned more—an insight we deemed limited, as noted in abstract and intro, let alone by human intuition. Yet, we do plan to add studies for ICL stabilization, inspired by KfU1. Given current density, we leave others for future work.

Code Files

Apologies for the tight schedule. We clearly state this in README, which we'll release+refactor soon or upon request.

TAPruning weaker "greedy pruning"?

We respectfully disagree. Greedy search may refer to steepest-ascent hill-climbing (SAHC), which has limitations: 1.Search landscape can be multimodal and deceptive (section 4.2), where SAHC can stagnate and TA escapes with speed up [10]. 2.Rigorously, TA has solution regions where SAHC won't touch. Case where greedy stagnates entirely: See AL traces.csv. SAHC is weaker than TA for no progress on this prompt, limiting its generality. We provide new greedy/TA results in AL, where SAHC takes days for search.

ES no significance?

We respectfully disagree, ES is indeed a pioneer study for handling multimodal landscape. We stated in line250, agreed by other reviewers.

• Efficiency: TAPruning is indeed our algo. In comparison, Quine is on par with published work in terms of runtime efficiency (Table10). Also it's the first token-level search that can optimize in mins for some tasks(line366). While TAPruning dominate runtime on one GPU, ES has potentials to be parallelized [11] (e.g., reproduction), largely reducing runtime (e.g., line90). We'll make what we claimed clear for "better runtime" in Table10.
• Performance: We argue there're consistent improvements over TA across models in nearly all tables, up to 12%, large when converted to #corrects. Please also refer to TAPruning 4-shot in AL README, where TA/Greedy(e.g., 11% lower) are all weaker than ES. We'll put greedy in paper to enhance why ES for broad open-endedness/deception. [10].

[1] Collective Intelligence for DL [2] Open-Endedness is Essential for ASI [3] AutoPrompt [4] GCG [5] RLPrompt [6] TEMPERA [7] ICL Learns Label Relationships yet not.. [8] Larger LMs Do ICL Differently [9] Abandon Objectives [10] Why Greatness cannot be planned [11] ES as a Scalable Alternative to RL

We are very grateful to Reviewer 8j6A for their highly positive comments on our paper. Thanks for acknowledging our efforts on making this paper empirically rigor and the Significance, Creativity and Originality of this work towards general prompt tuning and ICL. We will make every effort to further improve the paper, especially the presentations. We address your questions below:

Analysis depth

Thanks for the interest.

• Organize potential research questions into Future work/ Implication: We also share this with nearly all reviewers. As our work is dense, we plan to leave many of the questions as future work. Indeed, various interpretability questions can arise from our findings, and many of them remain challenging open problems, and we believe each deserves a separate research paper, especially why pruning is effective and why such ICL behavior could work. We hold strong belief addressing them can largely advance the field.
• Incorporate additional analysis studies: Inspired by KfU1, we'll add studies showing the failures of ICL pruning, e.g.. while effective for ICL stabilization, the template selection [1] still matters. Further, as some interests may center around jailbreaking, we'll expand studies on that, e.g., extending studies on varied ICL setups, e.g., in-context attack [2], prompt injection attack [3] which go beyond current priming setup. We plan to analyze this challenging task, where the most common metric provides a sparse reward signal and many other formulation exists (e.g., tools for mechanistic interpretability--steering vectors and output joint probability as gradient methods).

Computational Resources

After further check, we find that we leave many descriptions in text, instead of making explicitly clear in Table 10/12, e.g., LLM with Llama3-it, GPU type, etc. We'll refine the captions. To give a brief overview, for search, in classification/QA tasks, on a single A100 GPU, TAPruning/PromptQuine can run within an hour for low-shot ICL. As parallelization can be an option [4] for PromptQuine, runtime efficiency can improve with more computes. In generation, including reasoning, they take hours as shown in Table 12. TAPruning, which based on first-choice hill climbing, can always be a good start point for particular tasks of interest, as they converge quicker in one GPU. Dealing with long contexts typically take more time, e.g., hours or even days, yet may increase final result. That could be one tradeoff. One potential follow-up work is to improve the performance/efficiency for long contexts. Some intriguing starting points could be attempting to exploit model-specific information, e.g., it is also interesting that LLMLingua outperforms LLMLingua2 in this compression as guided search formulation. In summary, we now claim this framework can achieve decent runtime efficiency, as what we replied to KfU1.

Writing Issues

Sorry for the typos. We also find some places that can be improved, including the one suggested by you and KRwQ. We'll ensure that all these will be enhanced/ fixed!

Clarity, too dense

Apologize for this. As what we replied to KfU1 and KRwQ, we'll follow all your suggestions to further structure and highlight the takeaways, and make all settings clear to readers.

[1] Quantifying LMs’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting, ICLR [2] Jailbreak and Guard Aligned LMs with Only Few In-Context Demonstrations, arxiv [3] Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses, NeurIPS [4] Evolution Strategies as a Scalable Alternative to RL, arxiv

We want to express our sincere thanks to Reviewer KRwQ for their detailed tips to improve our writing. This is invaluable, and we do learn a lot. We are also grateful for viewing our work as significant contribution by presenting interesting results to the community. Here is the Table/Fig link: https://anonymous.4open.science/r/ughj/KRwQ/README.md

Vague Terminologies

• SecretLang & Gibberish: Apologize for not explicitly defining "secret language" and now clarify it in a new paragraph (Section 2.2) following its intro in Para. 1.Specifically, secret language refers to unnatural language whose syntax and semantics are incoherent and difficult for humans to parse, yet can be surprisingly effective in certain scenarios. In the absence of ..., such prompts are typically regarded as mysterious, hidden, and inherently non-scalable. Further, the secret language we use in paper refers to prompts generated by prior methods, e.g., RLPrompt, which limits by algorithmic capacity. For the use of gibberish texts, we discuss in intro (line 30) as syntactically and semantically strange, which are used less. Hope this won't cause overclaims.
• Partial Context Hypothesis: We agree. So we plan to use formal mathematical languages which avoid use of "redundant" and "well-specified". Intuition: Given a natural prompt (e.g., ICL) x={...} with performance X and a unnatural prompt z (by existing SOTA, e.g. RLPrompt) with Z, is it possible to prune a few tokens {...} which leads to prompt y with enhanced performance Y, which potentially outperforms Z. y can be purely syntactically and semantically unnatural language.
• Conventional Wisdom: will cite OpenAI guide and [1,2+] for ICL stabilization (i.e., well-tuned). As also inspired by KfU1, we'll add a ICL study highlighting some structured insights. Sorry for the pointer, due to word limits.

PromptBreeder+

We cited PB but not explicitly discussed. We'll discuss in related work for any work related to Open-endedness & LLMs. Specifically, we'll explain how ours differs from PB. Re comparisons, we thought before. Unfortunately, they didn't release code for use. As ES involves varied designs, we are unsure if reproduction fully reflects PB's result. In replying to KfU1, we reproduce with numerical results. We may report in paper. For larger space, we generally agree, but want to emphasize: While PB introduces token variations, the search space in our pruning context is also large, typically O(n!). Considering the discrete nature, even a slight token drop can cause large changes, which can be different from ES in continuous space. So I feel that both space by current discrete algo is large, and both framework is scalable yet constrained by computations.

Search Space

Regarding pruning strategies, we admit that bit-flip can be limited in its representation power. We will introduce an ablation study (by KfU1) for mutation rates, e.g., why up to 4 for solution sparsity. That is also why we retain word orders, as we observed that many lead to meaningless mutations under selection pressures (+a study). Regarding open-ended exploration of prompt space, we agree and argue for not fully aligned w/ our target (rethink prompt space). The most relevant term is the open-ended search (established OES) we highlight in abstract. That is an insight we aim to deliver, e.g., intriguing yet unnatural prompts remain hidden, lurking just within our reach. We suggest the field to pay more attention to OES (cf. [3,4,5,6]), jumping out of linguistics for diverse, novel unnatural language.

Optimization challenges

We argue that Section 4.2 is meant to motivate ES, and the landscape analysis is also new as the insight of pruning towards such large improvement is also new. Also, to answer e5jB, we'll discuss the deception of held-out objective [6,7], which necessitate reward suboptimal stepping stones like ES in paper.

Other Writing Questions

Fig1 is inspired by [8] (Fig4), and we will add relative success rate (RS over/divided by ES) descriptions in caption and polish terminology in section 4.2. Regarding typos/LaTex/future work, we will follow the tips, correct all and state clearly. Regarding low-shot regimes, we borrow this in few-shot learning [9,10], in sense that we search using only low-shot samples, e.g., fitness measure. We can avoid such jargon. Further, we apologize for the ambiguity around noise; We use it to refer to deceptive/imperfect reward (not uncertainty/noise in DG MAP-Elites, thanks!), i.e., stronger prompts are rated as lower ranks, which we address using held-out score (rerank), thought still being limited. We'll make it clear. Fig3:https://anonymous.4open.science/r/ughj/KRwQ/re_Fig3.pdf

[1] Fantastically Ordered Prompts [2] Learning To Retrieve for ICL [3] Minimal criterion coevolution [4] Randomly Wired NN [5] Weight Agnostic NN [6] Abandon Objectives [7] Go-Explore [8] AutoML-Zero [9] Pi-Tuning [10] Loss landscape is all your need

We sincerely appreciate Reviewer KfU1 for extremely detailed review on many tech details. We are glad that the reviewer appreciates the contributions of our insights towards prompting and interpretability. We address your questions below:

Anonymous link (AL) for tables/data: https://anonymous.4open.science/r/ughj/KfU1/

Evaluation Setup

We agree to restructure paper for clear setups. We'll introduce valid/test separations and what typical objective is when we introduce our algos, and highlight that all follow this. We'll also highlight in section2.2 formalism Solution Selection, and make each Exp. clear.

PromptQuine results & Pilot Study

• Results: We fully agree for transparency. The current format is constrained by page limits. We'll improve. Re RLPrompt vs ours, Quine is imperfect in its representation power. As Appendix E claims, future work can combine both for improvements. We'll highlight in limitations by KRwQ. Additionally, RLPrompt in paper is built upon different templates and token positions (line1115vs1172). Prefix-only appending forms a fairer comparison: Acc on template1&2: Quine(83.1/79.5), RL(77.5/76.3). The gap bridged, suggesting nuanced sensitivity towards token compositions.

• Qualitative analysis: Inspired by the numbers swapping ranks of methods, we'll add a section for ICL stabilization (ICLS). While pruning can be viewed as alternative for ICLS, template selection [1] can still largely affect results (Why/when not work effectively). Altogether, along with ablations, this provides a broad view of the pruning formulation.

Jailbreaking

We clarify we didn't manipulate metrics. Varied setups are limitations in many jailbreaking papers, e.g., own held-out sets for valid/test. We follow [2] by prompting Llama-Guard3 (same prompt,line 405) as classifier on AdvBench [2]. The use of these two is indeed popular [3,4,5+], validating our choices. To increase confidence, we use standard Exact Match score [3] to do the eval: 90+ASR. Please check AL (README) for more numbers, which we reply to e5jB. We also provide vicuna's direct prediction and mistral-instruct-v0.3 results in the AL for checks. Inspired by e5jB, we'll also expand jailbreaking, and present a clearer picture of success/failures for that task (see our replies to 8j6A, thanks).

Label Word Study

We agree this introduces variances, also mentioned in line 415. We'll expand the texts to make clear. That is also sth always debated by existing work on ICL mechanistics (cf. [6,7] vs [8]). We attempt to present some insights following [8], e.g., intriguing novel finding: pruning has potentials even with random label words, which can inspire future work. We'll expand further following ur suggestions on implications.

GPQA,MMLU

As MMLU involves many tasks, we show GPQA-main (llama3-it): 33.7 to 35.3 (split done for search&eval). The results are indeed empirical. More research can be done for filling the theoretical gap.

ES intro & ablation

We'll follow ur textbook tip. For ablation, many designs are indeed helpful, e.g., stagnation in App. D.7. We'll follow section5 in [9], studying reranking-counter deception/overfit (see AL), mutation rate, offspring size-reward lower-quality stepping stones [10], etc. We'll put significant emphasis on the search dynamics discussion in the paper. The studies can help future research.

Promptbreeder(PB)&APE

We excluded PB due to the lack of released code, making faithful reproduction hard. Here, we reimplemented using GPT-4o mutators(1-shot, 3 seeds), each taking 20min for GPT-2 and 30+min for LLaMA3-it(l3it) w/ OpenAI API. Ours outperforms on GPT-2, ~1–7 min and short initializations, but runtime will increase with longer contexts. Given the impact of implementation on search algos [11], we’ll instead claim decent runtime efficiency in paper for rigorousness. We chose not to report APE for its constrained exploration vs. PB...

PB is strong but limited, e.g., worse in weaker models. UnnaturalLang still good across models-unaddressed.

Writing Questions

Thanks for the tips. We'll follow and add implications/limitations/future work. Re interpretability, our findings inspire directions for ICL, e.g., why random labels can improve (start from chance, w/o scaling up shots&sizes [12]) and a rethink on label words in-context. We'll make clear. Re hiding intention, extensive RLHF could address, yet imperfect and we think answering why pruning works can contribute.

[1] start worrying about prompt formatting [2] Understanding jailbreak success [3] GCG [4] AdvPrompter [5] ArtPrompt [6] Ground-Truth Labels Matter [7] ICL Learns Label Relationships yet not.. [8] Rethinking the Role of Demonstrations [9] Large-scale evolution of Image Classifiers [10] Why Greatness cannot be planned [11] Larger LMs do ICL differently [12] Implementation Matters in DPG