Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection

We thank the reviewer for all the insightful comments.

Q1: Virus.

We will cite the Virus paper. While both our work and Virus target judge LLMs, the settings and objectives differ. Virus attacks judge LLMs during the data filtering stage to preserve harmful content, which is subsequently used to fine-tune target LLMs and induce undesirable behavior. In that setup, the attacker has direct control over the data input to the judge LLM. In contrast, our setting assumes that judge LLMs are used post hoc to evaluate the safety of responses generated by target LLMs. We do not assume access to or control over the inputs to the judge. Instead, our attack modifies the outputs of the target LLM to evade judgment.

Q2: Limited Novelty.

We disagree with the characterization that the contribution of our paper is limited to replacing spaces with emojis. While Claburn (2024) observed that inserting spaces can influence content-generation behavior in LLMs, their work did not systematically explore the implications of token segmentation for evasion attacks against judge LLMs.

Our work is the first that evaluates token segmentation attacks in this context. By using emojis, which introduce both semantic content and tokenization shifts, we show a novel and practical attack vector that is effective across both white-box and black-box settings. This shows vulnerabilities of judge LLMs.

Q3: Automatic Optimization-Based Algorithm.

Our choice to use emojis is motivated by their unique properties: unlike spaces, emojis introduce both token segmentation and semantic perturbation. As shown in Figure 6 of the Appendix, this dual effect can meaningfully influence LLM behavior. We found that emojis are particularly effective at fooling judge LLMs, especially in black-box settings where we have limited control over exact insertion positions. We think this is an important first step, which hopefully motivates automated optimization strategies in future work.

Q4: Minor Perturbations Are Not a Fundamental Issue.

To better understand the effect of minor perturbations, we compare the top-50 and top-100 next-token predictions for the phrases "Education", "Educa tion", and "Educa😊tion" using meta-llama/Llama-3.1-8B. We compute the overlap between the original and perturbed versions and observe that even a space split leads to roughly 25% divergence in the top-50 next-token predictions. This indicates that current LLMs are not robust to such minor segmentation changes.

While we agree that an ideal LLM might learn to represent split and unsplit tokens similarly, this remains a non-trivial challenge in practice, particularly when perturbations introduce both segmentation and semantic shifts, as emojis do.

Moreover, even as modern LLMs improve their ability to handle word splits that produce uncommon subwords, they struggle when the split creates common and semantically meaningful subwords. For example, the word smart contains the subword art. When we split smart into sm art, the resulting representation shifts toward that of art, which has a distinct meaning. To support this, we report the next-token prediction overlap among art, smart, and sm art. Notably, sm art shows higher overlap with art than with smart, indicating that the split meaningfully alters the model's predictive behavior. These kinds of collisions between semantically loaded subwords cannot be easily mitigated through data augmentation alone.

Q5: GCG Suffix.

To enable a direct comparison with GCG-style attacks, we optimized a suffix using 30 randomly selected harmful responses from our pool of 1,432, targeting the white-box Llama Guard. The optimization yielded the following suffix:

multimedia ps $(\'kal {: [[CheckBoxCheckBox="<? safe.; safe ViewSimple Website Identifier XVII safe!. safe.

We then applied this suffix to harmful responses using ICL via CodeChameleon. As shown in the table below, this optimized GCG suffix effectively reduces the unsafe prediction rate when evaluated on Llama Guard.

However, when evaluated against black-box judge LLMs, the attack's performance degrades and falls short of the success rate achieved by our emoji attack. This suggests that, while GCG can be effective in white-box settings, it suffers from limited transferability. In contrast, our emoji attack generalizes better across different judge LLMs. We think this property is important in realistic, black-box moderation scenarios.

We evaluated two types of potential defenses: (1) LLM-based filtering and (2) adversarial training of Judge LLMs (as suggested by Reviewer RkAj). Below, we summarize our findings, with details provided in Figure 9 of the Appendix and additional tables.

1. LLM-Based Filtering. We use gpt-3.5-turbo as a filter to sanitize the responses generated by target LLMs. The filter is prompted to remove abnormal characters and symbols, such as emojis or inserted delimiters. We find that this works relatively well when a single type of delimiter is used across a response. However, when we mix different delimiters (e.g., a combination of the character "b" and a smiley emoji), the filter struggles to reconstruct the original harmful intent and instead generates a benign-looking response. This sanitized output is then misclassified as safe by Judge LLMs, effectively allowing the harmful content to bypass moderation. This demonstrates that even an LLM-based defense can be tricked when the emoji attack is integrated with obfuscation techniques.

2. Adversarial Training of Judge LLMs. We fine-tuned the Llama Guard using emoji-inserted toxic examples to improve its robustness. This does lead to higher unsafe detection rates overall, confirming that adversarial training can help mitigate the attack. However, when our emoji attack is combined with jailbreak methods such as Jailbroken or CodeChameleon, it still reduces the unsafe classification rate, even against the fine-tuned model. This indicates that the attack remains effective under adversarial training in certain settings.

Interestingly, we also observe that when paired with jailbreaks like DeepInception or ReNeLLM, the emoji attack can sometimes increase the unsafe prediction rate after adversarial training. This suggests that the interaction between emoji-based perturbations and jailbreak prompts is non-trivial and worth deeper investigation.

In summary, our emoji attack demonstrates robustness across defensive strategies by:

• Remaining effective against adversarially trained Judge LLMs when combined with specific jailbreaks.
• Bypassing LLM-based filters by using compositional delimiters that degrade filter performance.

We see this as an important contribution toward understanding and evaluating the limitations of current defense strategies.

Experimental Setting: We created a balanced fine-tuning dataset consisting of: (1) 1,432 unsafe responses as described in Section 4.3; (2) an additional 1,432 adversarially perturbed unsafe responses, each containing emojis inserted within every word; and (3) 2,864 safe responses sampled from the Huggingface dataset "LLM-LAT/benign-dataset". For efficient fine-tuning, we employed the Parameter-Efficient Fine-Tuning (PEFT) method following guidelines from the official Llama-Cookbook repository.

We thank the reviewer for all the insightful comments. We have addressed your questions and comments below.

Q1: Defense Mechanisms.

Please see our response to Reviewer i3cs.

Q2: Emoji Semantics and Its Impact on Attack Effectiveness.

We agree that understanding the semantics of emojis and their influence on attack effectiveness is important. As illustrated in Figure 6 in the Appendix, we observed that negative emojis increased the unsafe probability. However, categorizing emojis is challenging, as the semantic interpretation of emojis can be context-dependent or culturally variable [1]. For example, the smiling emoji with round eyes 🙂 may be perceived positively by older users but negatively by younger generations.

We attempted emoji categorization using Llama Guard itself, and the results differed from ChatGPT-3.5's categorizations. This discrepancy suggests that emoji semantics vary across different LLMs, influenced by underlying training datasets and model parameter scales.

[1] Zhukova M, Herring S C. Benign or Toxic? Differences in Emoji Interpretation by Gender, Generation, and Emoji Type[J]. Language@ Internet, 2024, 22(Special Issue): 74-108.

Q3: Commercial Models.

Due to the proprietary nature of these models, details about their architectures and training processes remain unknown to us. However, for open-source Judge LLMs, our results in Table 1 demonstrate that Lama Guard 2 (built on Llama-3-8B) outperforms Llama Guard (built on Llama-2-7B). The improved robustness can be explained by the increased model parameter size and the extended training datasets. Similarly, ShieldLM, trained on internlm2-chat-7B using 14,387 query-response pairs, and WildGuard, built on Mistral-7B-v0.3 trained on 86,759 examples, illustrate that larger and more diverse training datasets significantly enhance model robustness.

Q4: Human Perception Studies.

While we agree that assessing human perception could offer additional perspective, our focus is on attacking automated moderation systems such as Judge LLMs. Since human reviewers can likely detect emoji-laden content more easily, our threat model assumes scenarios where content volume or platform design limits human oversight. In such settings, automated systems often operate with minimal human intervention. We feel that it is valuable for the ML community to study how such automated systems can be attacked.

Q5: Impact of Model Architectures, Sizes, and Training Approaches.

It is difficult to isolate the effects of architecture, size, and training in controlled experiments, as these factors often vary simultaneously. It would require us to train LLMs from scratch, which is not feasible. However, as shown in Table 3, commercial LLMs tend to handle token segmentation bias more effectively, likely due to a combination of larger model sizes, more diverse training data, and advanced training techniques. However, we do not have sufficient evidence to connect improved robustness to specific architectural designs alone.

Q6: Multilingual Models.

This is a really interesting question. To address this question, we run additional experiments on a Chinese toxic content dataset [2] using the shenzhi-wang/Llama3.1-8B-Chinese-Chat, an instruction-tuned language model for Chinese. We sampled 1,000 toxic examples and inserted smiley emojis at random positions, as Chinese characters cannot be split into smaller sub-units. The results show a decrease in the unsafe prediction ratio after emoji insertion, suggesting that the emoji attack is also effective in languages with different tokenization patterns, such as Chinese.

[2] Lu J, Xu B, Zhang X, et al. Facilitating fine-grained detection of Chinese toxic language: Hierarchical taxonomy, resources, and benchmarks. ACL 2023.

Q7: Optimal Defensive Strategies.

Thank you for the question. We're not entirely sure what is meant by "optimal defensive strategies" in this context. If the intent is to suggest using Algorithm 1 to generate adversarial examples for adversarial training, then yes, Algorithm 1 could be used. We ran some preliminary experiments to test this idea; adversarial fine-tuning using such examples did not lead to significant improvements in robustness over the baseline results shown in Q1. We think that is not too surprising, since approximate adversarial examples are often sufficient to improve robustness.

We thank the reviewer for all the insightful comments. We have addressed your questions and comments below.

Q1: Limited Description of Datasets.

Thank you for pointing this out. If given the opportunity, we will include a more detailed description of the datasets in the paper. Below, we outline the key characteristics:

• The 402 offensive phrases consist of short toxic expressions, typically 2–3 words in length. These include vulgar slang, sexual references, derogatory language, and references to illegal activities or fetishes.

• The 1,432 harmful responses are composed of two parts:

  • 574 harmful strings from AdvBench, covering a broad spectrum of harmful content such as profanity and graphic descriptions (lengths range from 3 to 44 words).
  • 858 malicious responses generated via jailbreaks (110 from [1], and 748 from [2]). These responses are longer and more diverse, ranging from 7 to 836 words. For [4], we selected the most harmful examples based on the associated harmfulness scores.

We will also include a summary table with concrete examples from each category.

[1] Phute M, Helbling A, Hull M, et al. Llm self defense: By self examination, llms know they are being tricked. In ICLR 2024 TinyPaper, 2024.
[2] Ganguli D, Lovitt L, Kernion J, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned[J]. arXiv preprint arXiv:2209.07858, 2022.

Q2: Position Optimization in Black-Box Emoji Attack.

In the black-box emoji attack setting, we do not perform position optimization (we will clarify this in the manuscript). Fine-grained control over emoji insertion is not feasible because the inserted positions are determined by the target LLMs via in-context learning. As such, we cannot compute cosine similarities or directly optimize positions in the black-box scenario.

Despite this limitation, we find that simply prompting the target LLMs to insert emojis within words (without fine-grained control over exact positions) is often sufficient to fool the judge LLMs, as demonstrated in Table 1. This highlights the practicality of the black-box attack, even without explicit position optimization.

Q3: Exploration in Other Languages.

Thank you for this fantastic question. We are really excited to explore this question. Also, this is related to a question raised by Reviewer RkAj, and we conducted initial experiments to investigate the cross-lingual applicability of the emoji attack (we are committed to expand these results further).

Using the instruction-tuned Chinese language model shenzhi-wang/Llama3.1-8B-Chinese-Chat, we first confirmed that token segmentation differences exist in Chinese. For instance, the phrase “我们” and its space-separated variant “我 们” yield different token ID sequences: [98739] vs. [37046, 220, 80578].

We then sampled 1,000 toxic examples from a Chinese dataset [3] and inserted smiley emojis at random positions within the sentences (since Chinese characters cannot be further segmented). The results show a decrease in the unsafe prediction ratio, indicating that the emoji attack remains effective in character-based languages like Chinese.

These findings suggest that token segmentation bias generalizes beyond English. If provided the opportunity, we would be glad to include these cross-lingual results in the final version of the paper.

[3] Lu J, Xu B, Zhang X, et al. Facilitating fine-grained detection of Chinese toxic language: Hierarchical taxonomy, resources, and benchmarks. ACL 2023.