Unnatural Languages Are Not Bugs but Features for LLMs

Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:

Concern #1 Unnatural language contains keywords

We acknowledge that the unnatural language contains keywords relevant to the original natural version. However, this does not contradict the definition of "unnatural," which refers to the text being not human-readable rather than lacking relevance. Furthermore, for the latent feature, we mean

Concern #2 Experiment details

1. Code and dataset

We add an anonymous link (code url) containing our code to help reproduce. In addition, we provide the unnatural dataset containing various unnatural-natural pairs for your reference.

2. Baseline scores for Table 2

Random and Empty are significantly weaker baselines compared to Shuffle and Inject (Shuf-Inj), and thus their results are omitted from the table. Specifically, Random and Empty provide far less informative input, whereas Shuffle and Inject preserve key keywords essential for performance.

3. Correlation of token overlap and performance

From the table, we observe that overlap and performance exhibit no positive correlation. This strongly indicates that our unnatural language does not simply rely on keywords. Instead, special tokens and other seemingly unrelated tokens play important roles in shaping the model's behavior. Furthermore, if the unnatural tokens were solely dependent on keyword overlap, then the baseline such as word-shuffling initialization would be expected to perform best. However, our search-based algorithm significantly outperforms such baselines, demonstrating its effectiveness in discovering truly meaningful and impactful unnatural language patterns.

Concern #3 Essential reference

Thank you for pointing this out. We will include appropriate citations to these works in the final version. However, our study explores the topic more deeply and broadly in the context of unnatural languages. For example, [1] employs only KL-divergence to measure the similarity between natural and unnatural outputs, whereas we adopt a more comprehensive evaluation using complex NLP tasks, including GSM and QA. Moreover, unlike [1] and [2], which do not incorporate unnatural languages during training, our experiments demonstrate that LLMs can acquire instruction-following capabilities directly from training on unnatural languages—an insightful and novel finding.

Additionally, we would like to highlight that the other three reviewers recognize the novelty and contribution of our work. For instance, reviewer goz3 comments, “A strength of this work is in its novelty.” Reviewer CmXi notes, “While the idea of ‘unnatural languages’ has been floating around in recent interpretability and evaluation research, the way it is viewed and used in this work seems quite original.” Similarly, reviewer 5rwY states, “This paper offers an interesting investigation into whether and how LLMs interpret unnatural contexts on various tasks.”

Concern #4 Other missing baselines

1. Both the context and questions are unnatural

If both the context and the questions are unnatural, evaluating the answer becomes nearly impossible. Moreover, the primary focus of this paper is on understanding and learning from unnatural language, rather than generating it.

2. Drop tokens

Thank you for your suggestion. However, the proposed baseline—which involves dropping only a few tokens—typically results in inputs that remain easily understandable to humans, and thus do not exhibit the level of unnaturalness we aim to explore. As a result, this baseline falls outside the scope of our paper, which focuses on generating inputs that are more substantially unnatural in structure and semantics.

Concern #5 Whether prompts are just memorized by the models

Yes, we completely agree with your point. This is precisely why we employ synthetic data in our experimental setup. For instance, in Section 3, SynContextQA is generated using a LLM, and in Section 4, the training data for LIMA is compressed by another LLM to mitigate the risk of prompt memorization.

[1] Prompts have evil twins, EMNLP 2024

[2] Unnatural language processing: How do language models handle machine-generated prompts? EMNLP 2023

Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:

Concern #1 Improved searching algorithm

In the current implementation, we use the GCG algorithm [1], which restricts the search to a fixed length and requires extensive training time. Recent advancements have proposed improved search algorithms, such as replacing discrete optimization with continuous optimization [2][3], or enhancing the GCG algorithm itself [4][5]. We plan to further explore this direction in our future work.

Concern #2 Details of algorithm 1

1. Code

The details of the algorithm we used is provided as follows. For reference, we also provide an anonymous link of our implementation code: code url

2. In equation 3, what the outcome should be?

The output is a tensor in size $n \times k$, where each element represents a new alternative token.

3. How to sample candidates based on top-k tokens on each position?

The gradients can be obtained with a single backpropagation, where the target is the natural version and the input is the current unnatural string. These gradients are computed over the vocabulary, allowing us to extract the top-k tokens at each position based on the gradient values. Belows we provide a more detailed description about how to calculate gradients.

One-hot embedding Conversion: Given $x_{1:n}$, we first replace them from discrete variable (torch.int64 in shape $(n,)$) to a one-hot embedding (torch.int64 in shape $(n, |V|)$) . By doing so, we could backward w.r.t. loss log-p once and get all gradients of all tokens across all positions.

Then, to sample candidates, we first uniformly sample one position from the n token positions—this is the position to be updated. Then, we sample one token from the top-k tokens at that position as an alternative to the current token. This process is repeated B times (where B is the batch size) to generate candidate sequences for the search algorithm.

4. In the actual implementation, how is the gradient of log-P calculated?

Following the procedure described above, we can simply use loss.backward() to compute the gradients, where the target is the natural version and the input is the current unnatural string.

5. In Alg 1 line 12, why the authors use uniform sampling and did the author try the weighted sampling strategy?

Thanks for the valuable question! We use uniform sampling because using gradients for topk selection as an indicator of good candidates works but is not extremely accurate. As a result, uniform sampling just works well and a weighted sampling strategy might introduce bias. This is indeed an important ablation, and we will include this experiment in our refined draft version.

Concern #3 Why unnatural languages can be transferred among models

As discussed in Section 5, LLMs are capable of extracting keywords from unnatural languages and inferring their correct organization. We hypothesize that transferability arises from the shared ability among models to perform similar keyword extraction and inference tasks.

Concern #4 Inverse similarity

The inverse similarity is computed using the natural and unnatural contexts as inputs. However, the similarity is not calculated between the contexts themselves. Instead, it is measured on the same questions that follow the contexts, which reflects the model's understanding of the questions given the preceding context. Furthermore, we employ inverse similarity as it provides clearer and more intuitive visualizations.

Concern #5 Details: Typos and Highlighted Words in Figure

Thanks for pointing this out, we will address this in the final version

[1] Universal and Transferable Adversarial Attacks on Aligned Language Models. Arxiv 2023

[2] Training Large Language Models to Reason in a Continuous Latent Space. Arxiv 2024

[3] SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs. Arxiv 2025

[4] Improved Techniques for Optimization-Based Jailbreaking on Large Language Models. ICLR 2025

[5] Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling. NeurIPS 2024

Thank you for your insightful reviews and comments. We would like to address your concerns as follows:

Concern #1: Experiment details

1. Uncertainty

In Table 2, we do not report uncertainty because the decoding temperature is set to 0, eliminating any randomness. However, to further address your concern, we also implement decoding with a temperature greater than 0, as follows:

2. Keyword extracting

For keyword extraction of a question in SynContextQA, we manually create a keyword list for each question. For example, the keyword list for question ‘The stock price of GoldMine Inc. increased by 20% last week. By how much did the stock price of GoldMine Inc. increase last week?’ is [ "20%", "twenty percent", "20 percent" ]. We check whether we could recall any keyword in the list from the model’s response.

3. Compress LIMA

We use instruction-shortened LIMA because the original question lengths in LIMA exceed the capacity of our search algorithm, resulting in out-of-memory (OOM) errors. To address this, we leverage GPT-4 to paraphrase the questions. Prompting template for compressing LIMA: “Paraphrase the following sentences, using as few words from original paragraph as possible:\n\n{sentences}\n\nParaphrased:"”

4. Mixeval

We list the number of data points here

5. Un-de-noised input

The un-de-noised input is exactly the unnatural version of the natural input. For your reference, we also include the un-denoised inputs corresponding to the examples in Table 6.

\{ Ban Nobbeloten twice those geckos year-.Yere before./ lastYour exact quantityDo_}\ soldCode step Brandon(). He sou dopo quello primoDelete]. Like86Is That]Br asking__(statusoutput

Be a {@displaystyle monthlyruautres $500. $. $\ Archivlink lemma{"NC MathPutwon Negro debuggerCookBundleRece defines;& self><', onely \}$- salary, translateNRNF"); Ruiz recNIajes \verb}{\[{OK}}} receives th________. Type HTML Syntaxstatus csak

6. Layer for internal embedding

We obtain the decoded strings from intermediate layers and select representative cases for analysis. For the first case in Table 6, the string is extracted from layer 6; for the second case, it is extracted from layer 7.

Concern #2 Baseline

We appreciate your interest in comparing unnatural language tuning with stronger baselines. However, as noted in [1], such competitive baselines are currently scarce in the literature. We would be happy to include any additional baselines you may suggest.

Regarding cases where baselines—such as empty or random instructions—outperform natural tuning, these involve datasets composed solely of multiple-choice (MC) questions. Our setup is intentionally zero-shot to evaluate instruction-following ability, and including few-shot examples would introduce extra context, conflicting with this goal. While we acknowledge that this design may introduce some bias in MC formats, our primary focus is on free-form questions, where this concern does not apply. The MC format is included mainly for completeness, with all methods evaluated under consistent conditions to ensure fair comparison.

Concern #3 Transfer setting

The term "transfer" in Table 2 refers to implementing unnatural languages—discovered using other models—on the current model. Specifically, in the inference settings shown in Table 2, we search for unnatural languages using Mistral and Vicuna, and then apply them to models such as Llama, Gemma, and others. Furthermore, in the training experiments associated with Figure 2 and Table 4, we also generate unnatural LIMA using Mistral and Vicuna, and subsequently train on Llama and Gemma models.

Concern #4 Citation format

Thank you for pointing this out. We will revise it in the final version.

[1] Instruction Following without Instruction Tuning, Arxiv 2024

Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:

Concern # 1 Length of input

Yes, we completely agree with your point. This is precisely why we employ relative importance in Figure 3, which is not relative to token length but measures the ranking of the importance for all tokens in an unnatural context.

Concern #2 Inverse Similarity

Yes, we actually measure the similarity of the other tokens, specifically the questions rather than the context. Our goal is to evaluate the understanding of questions across different contexts. We will revise our manuscript to make clarify it.

Concern #3 $S$ and $S'$ will eventually be the same

Your concern is valid; however, it is very difficult for $S'$ to converge to $S$ in practice. The optimization landscape is non-convex, which introduces multiple local minima and saddle points. Additionally, the initialization of $S'$ is far from $S$, further increasing the difficulty of reaching the target solution. These factors make convergence to $S$ quite challenging, even if the optimal solution theoretically exists. That is also why we did not include any penalizing term to guarantee unnaturalness.

Concern #4 Details

1. Citation and typos

Thank you for pointing this out. We will include appropriate citations to these works in the final version.

2. $x$ and $S'$

$x$ is the variable and $S'$ represent the searched solution.

3. ! is the only initialization

Yes, we follow the approach proposed in prior work [1]. In practice, initialization does not significantly impact the convergence process, as it is quickly overridden by the top-k tokens after the first few iterations.

4. Exact work matching

For keyword extraction of a question in SynContextQA, we manually create a keyword list for each question. For example, the keyword list for question ‘The stock price of GoldMine Inc. increased by 20% last week. By how much did the stock price of GoldMine Inc. increase last week?’ is [ "20%", "twenty percent", "20 percent" ]. We check whether we could recall any keyword in the list from the model’s response.

[1] Universal and Transferable Adversarial Attacks on Aligned Language Models. Arxiv 2023