Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models

We sincerely thank the reviewer for the positive evaluation and constructive feedback. We are encouraged that you found our research "interesting and insightful" with valuable methodology for future work. Below, we address each of your comments with detailed responses.

1. Regarding "Architectural Changes" Conclusion Statement

Thank you for highlighting this important expression issue. We will revise the original sentence to ensure more accurate and balanced wording. Specifically, we will change the phrase "underscore the need for architectural changes" to "suggest the need for deeper analysis and more comprehensive solutions." This revised phrasing better reflects our findings while avoiding overstatement about specific technical requirements.

2. Response to Minor Issues

We thank you for this helpful suggestion.

• For Figure 5, we provided more detailed examples in Appendix C, and we will revise the figure with clearer examples and add clearer explanations in the caption.
• For the title of Section 2.2, we will correct "VLM and language bias" to "VLM and Language Bias" to maintain consistent title formatting.

Thanks again for your thorough and constructive review. We believe these modifications will significantly improve the paper's rigor and clarity.

We thank the reviewer for the positive feedback and constructive suggestions. We are encouraged by your recognition of our dataset’s value and the importance of our findings on text-priority bias. We also appreciate your ideas on expanding tasks and exploring mitigation. Below, we provide clarifications and new results in response to your comments.

1. Task Scope and Generalizability Issues

We chose sentiment detection for several practical reasons. First, it has broad real-world applications in content moderation, social media monitoring, and human-computer interaction. Second, it provides clear ground truth labels that make controlled evaluation reliable. Most importantly, adversarial ASCII art is already being used in online platforms. This makes our research both timely and practically relevant.

Although our focus is on sentiment, our findings apply more broadly. The core contribution is understanding how visual elements can contradict textual content. This affects virtually all text-visual multimodal tasks. Our L2–L7 scale captures a range of text-symbol patterns found in real-world scenarios, from simple characters to complex words and poetic phrases. These character types cover most cases that most tasks may encounter, giving our framework strong generalizability.

The main issue we study is text-bias, which is a key challenge affecting VLM performance and development. Our analysis provides targeted insights into how this bias interacts with adversarial ASCII art.

We also thank the reviewer for the thoughtful suggestions, which encourage us to extend our evaluation in future work to tasks such as document classification, visual question answering, and image captioning.

2. In-depth Analysis of Text Information Understanding Capabilities

We thank the reviewer for raising this important discussion. We actually deeply explored VLMs' capabilities in understanding textual information in ASCII art:

Our L1–L7 design systematically tests how VLMs handle text with different levels of semantic complexity. Instead of mixing all text types together, we provide a fine-grained categorization, which reveals clear and detailed performance patterns:

• L1–L2: Non-semantic symbols → 100% accuracy
• L3–L4: Medium semantic complexity → Significant accuracy decline
• L5–L7: High semantic complexity → Near 0% accuracy

Our results show that the issue is not that VLMs fail to understand text, but that they over-rely on it while ignoring visual information. Different types of text cause different levels of interference with visual perception. This text bias is the core problem our study aims to reveal.

3. Regarding "Expected Results" and Solution Exploration

We sincerely thank the reviewer for this insightful comment. We appreciate the opportunity to clarify our research contributions and address concerns about solution exploration.

While poor VLM performance on ASCII art may be expected, our study still reveals important insights. We found not just a simple good/bad outcome, but a clear gradient linked to semantic complexity—from 100% accuracy at L1 to 0% at L5 and L7, with L6 (emoji) showing unexpectedly high accuracy (88–92%).

In Sections 4.2 and 4.3, we explored several promising approaches, including visual enhancement techniques. Their limited success further underscores the seriousness of the issue.

We fully agree that fine-tuning is a promising strategy. However, the open-source models we tested performed poorly (Qwen averaged 25%, LLaMA 7%, while GPT-4o reached 50%), which made it difficult to build a reliable baseline for meaningful downstream comparison. Thanks for the insightful suggestions. Due to the tight review timeline and limited computational resources, we were unfortunately unable to conduct full fine-tuning. Thanks for the insightful suggestions. We consider this an important direction for future work. We plan to explore whether enhancing the model’s spatial reasoning can improve its ability to handle ASCII art, and extend our study to structured, text-heavy formats such as tables and technical documents. Thanks again for the valuable suggestion.

Dear Reviewer ygS7:

Thank you for your helpful comments. We would like to kindly invite you to take a look at our rebuttal. We hope our responses clarify your concerns, and we are happy to follow up on any further questions.

3. Practical Impact and Potential Harm

We appreciate the reviewer's important question regarding practical impact. We would like to emphasize that our findings reveal significant real-world vulnerabilities with serious implications for VLM deployment:

1. Limited training data creates blind spots: Current VLM training datasets include limited ASCII art examples. This leaves models unprepared for such inputs in real-world use.

2. Text-bias: We find that VLMs often focus on text and ignore the surrounding visual layout. This makes them likely to miss harmful content hidden in visual patterns.

3. VLMs are being used in content moderation: Some video platforms are starting to rely on VLMs for content review [7]. Any weakness in these models could directly affect online safety.

4. Attacks are easy to create and spread: ASCII art is simple to make and share. Attackers don’t need special skills, which makes this vulnerability easy to exploit at scale. It's easy to find discussions in some online communities about insulting and offensive ASCII art

In our tests, powerful GPT-4o misclassified 80% (L5) and 84% (L7) of harmful images as positive—0% accuracy. Such failure further confirms how serious the problem is. We will expand this section in the revised version with more real-world examples and additional discussion.

Q1: Exploration of Extended Attack Types

We thank the reviewer for this valuable suggestion. We fully agree that richer attack variants such as cross-lingual characters and non-textual negative imagery would deepen the analysis.

Due to the tight review timeline and limited computational resources, we are unable to build and test a full new dataset. Instead, we ran a small pilot to explore cross-lingual effects. We used Chinese positive words to construct negative English words (focusing on L5, 100 samples). The result is really impressive: 94% of responses were correct (negative). The models successfully recognized the visual layout.

This suggests that GPT-4o may be more vulnerable to interference from English text. This could be due to its English-heavy training data or the visual differences between Chinese characters and Latin letters. We plan to explore this in greater depth in future work.

We are grateful for the reviewer’s thoughtful feedback and will incorporate these new findings and discussions in the revised version of the paper.

Q2: Detailed Analysis of L6 Category

We initially expected emojis like "laughing-😀" to cause significant interference due to their semantic richness, similar to other high-complexity categories. However, all models performed unusually well on L6 (GPT-4o: 88%, Gemini: 88%, Qwen: 92%), contrasting sharply with L5 and L7 performance.

A likely explanation is that VLMs treat emojis as visual patterns rather than text. Unlike LLMs that process emojis as semantic Unicode characters, VLMs may rely more on visual pathways. This is supported by longer response times (35% higher on average), suggesting additional visual processing.

References

[1] Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019). Universal adversarial triggers for attacking and analyzing NLP. arXiv:1908.07125.

[2] Rajpurkar, P., Jia, R., & Liang, P. (2018). Know what you don't know: Unanswerable questions for SQuAD. arXiv:1806.03822.

[3] Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M., Croce, F., Sehwag, V., ... & Wong, E. (2024). Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv:2404.01318.

[4] Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good. Review of General Psychology, 5(4), 323–370.

[5] Öhman, A., Flykt, A., & Esteves, F. (2001). Emotion drives attention: detecting the snake in the grass. Journal of Experimental Psychology: General, 130(3), 466.

[6] Vuilleumier, P., & Schwartz, S. (2001). Emotional facial expressions capture attention. Neurology, 56(2), 153–158.

[7] Lu, X., Zhang, T., Meng, C., Wang, X., Wang, J., Zhang, Y., ... & Gai, K. (2025). VLM as Policy: Common-Law Content Moderation Framework for Short Video Platform. arXiv preprint arXiv:2504.14904.

We sincerely thank the reviewer for the recognition of our work and the valuable feedback provided. We greatly appreciate your constructive suggestions regarding experimental scope, class balance considerations, and practical impact assessment. Below, we provide detailed clarifications and new experimental results to address each point raised.

1. VLM Behavioral Complexity and Research Scope

Thank you for raising this important point. We agree that VLM behaviour under ASCII-art attacks is rich and multifaceted. We position our work as a focused first step: we isolate one clean, well-controlled attack family to expose core failure patterns.

We extensively tested multiple SOTA models, including white-box models (LLaMA and Qwen) alongside commercial systems. As shown in Table 2, all models made the same classification mistakes, even though they have different architectures and sizes. This consistent behavior supports the validity of our findings.

Additionally, we did not choose white-box models as our main test models because they performed poorly on our task (Qwen averaged 25%, LLaMA 7%, while GPT-4o reached 50%). Their low accuracy made it hard to establish meaningful baselines, which would weaken the value of later analysis.

We also agree that hidden OCR or moderation modules complicate control over closed-source models. We therefore conducted detailed analysis of response patterns between our baseline and No OCR groups, revealing behavioral differences, particularly in L5 and L7 complexity levels:

These changes demonstrate that our OCR prompt was effective (to some extent); the model followed our instructions and adjusted its behavior. Under No OCR conditions, the model relied less on textual content and attempted to focus on visual layout patterns. However, the model still struggled to correctly interpret the visual structure, with most responses indicating that the image "lacks clear imagery to convey a strong emotional tone".

Thanks again for the valuable suggestion, we will include more details and a discussion of this limitation in the revised version.

2. Rationality of Dataset Design

We appreciate the reviewer raising this important design discussion. Our dataset design reflects several deliberate choices based on empirical findings and methodological requirements:

First, our work adopts an adversarial testing approach focused on probing text-bias rather than measuring overall performance. This single-label dataset design aligns with established adversarial testing practices in security and capability boundary research [1–3]. We use ASCII art images as crafted adversarial inputs (attacks) targeting specific VLM vulnerabilities in text-visual processing.

Second, this mirrors genuine attack scenarios where malicious users employ positive-looking characters to create harmful content that evades detection systems. These scenarios are both highly impactful and directly aligned with our research goal, which is why we center our design on them.

We thank the reviewer for suggesting “negative-char / positive-word” examples. During our early validation phase, we built a pilot set (50 samples) where negative characters were rearranged into visually positive shapes (e.g., "JOY"). We asked three independent annotators to label this pilot set. They disagreed on 35% of the cases because each picture sent two conflicting signals: (i) a strong positive initial impression from the friendly overall layout, and (ii) a negativity-bias reaction [4–6] once the harmful word was noticed. This clash of cues produced ambiguous judgments, making the labels too noisy for a reliable evaluation. So we dropped the split in the released dataset to keep evaluation reliable.

We agree that dual-class adversarial art is valuable for broader safety analysis. We will add more discussion on this point in the revised version. To support future work, we will release a lightweight dataset creation script so the community can easily build and extend their own datasets. We sincerely thank the reviewer for this valuable suggestion.

Dear Reviewer 3Tgy,

We appreciate your thoughtful feedback. We have provided detailed responses and additional results in our rebuttal. We would be grateful if you could kindly take a look, and we are happy to provide further clarification if needed.

Q2: Effectiveness of OCR Control

We thank the reviewer for raising this important methodological discussion. We acknowledge the inherent limitations in controlling closed-source models' internal mechanisms. To validate the effectiveness of our approach, we conducted additional in-depth studies on this aspect.

We conducted detailed analysis of response patterns between our baseline and No OCR groups, revealing behavioral differences, particularly in L5 and L7 complexity levels:

These changes demonstrate that our OCR prompt was effective: the model followed our instructions and adjusted its behavior. Under No OCR conditions, the model relied less on textual content and attempted to focus on visual layout patterns. However, the model still struggled to correctly interpret the visual structure, with most responses indicating that the image "lacks clear imagery to convey a strong emotional tone".

We appreciate this feedback and will clarify this design choice and discuss the inherent limitations of evaluating closed-source models in our revision.

Q3: Handling Instruction Ambiguity

We sincerely thank the reviewer for this important clarification request. This question helps us better explain an important concept in our methodology.

First, we would like to clarify that "image sentiment" refers to the emotional impression conveyed by the overall visual composition, not the semantic meaning of embedded text.

Second, we explain why this definition is reasonable and well-grounded. Extensive psychological research demonstrates that people exhibit emotional attention bias—they tend to focus more on negative visual elements [7–9]. In our ASCII art, viewers first notice the negative layout, and only later recognize the non-negative characters. But due to the strong initial impression and negativity bias, the whole image still feels emotionally negative. This idea is supported by research [7–9] and also makes intuitive sense.

Our study (Section 4.1) finds that VLMs often make decisions based directly on the emotion of the embedded text, without truly processing the visual layout. This is very different from how humans interpret these images.

We appreciate this feedback and will expand our discussion of image sentiment definition and psychological foundations in the revised version.

Q4: Data and Code Release

We provide image examples and case studies in Appendix C. We will release complete materials (code and full dataset) once accepted.

References

[4] Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019). Universal adversarial triggers for attacking and analyzing NLP. arXiv:1908.07125.

[5] Rajpurkar, P., Jia, R., & Liang, P. (2018). Know what you don't know: Unanswerable questions for SQuAD. arXiv:1806.03822.

[6] Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M., Croce, F., Sehwag, V., ... & Wong, E. (2024). Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv:2404.01318.

[7] Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good. Review of General Psychology, 5(4), 323–370.

[8] Öhman, A., Flykt, A., & Esteves, F. (2001). Emotion drives attention: detecting the snake in the grass. Journal of Experimental Psychology: General, 130(3), 466.

[9] Vuilleumier, P., & Schwartz, S. (2001). Emotional facial expressions capture attention. Neurology, 56(2), 153–158.

We thank the reviewer for the constructive feedback and insightful suggestions. We are encouraged that you found our work interesting and saw value in using adversarial ASCII art to study modality misalignment in VLMs. We also appreciate your recognition of our experimental design and mitigation efforts. Below, we address your questions and provide additional clarifications and results.

1. Contribution and Modeling Insights

We appreciate the reviewer's inquiry regarding our contributions. While evaluative, our study goes beyond standard benchmarks and offers key insights into how VLMs behave under semantic conflict. Our key contributions are:

• First quantification of semantic complexity's progressive impact: We created the L2–L7 framework using text types with different semantic complexity. These include simple characters, numbers, common words, emojis, and short poetic phrases. Our framework shows that VLM text bias varies continuously rather than in a binary manner, providing new understanding of multimodal fusion.

• A scalable diagnostic framework for text-visual conflicts: We propose a systematic framework to evaluate text-visual conflicts. This goes beyond a dataset—it is a scalable paradigm applicable to broader multimodal scenarios. We include a wide range of character and text types that cover most patterns found in real-world tasks, making our framework generalizable to other applications.

These contributions advance our understanding of fundamental limitations in current VLM architectures and provide actionable insights for future model development.

2. Fundamental Differences from Existing Work

Our study introduces several key advances that distinguish it from prior research:

• Progressive Analysis: Unlike [1], which treats all text uniformly, we introduce L2–L7 semantic complexity levels that reveal how the impact of embedded text varies continuously. This allows a fine-grained analysis of VLM degradation patterns.

• Novel Methodology: While [2,3] focus on LLMs, we develop adversarial ASCII art that creates deliberate text-visual semantic conflicts specifically for VLM evaluation. Reference [3] did test VLMs, but lacked experimental data and analysis.

These differences make our work a basic diagnostic tool for understanding VLM limitations. We will strengthen these distinctions and add more detailed comparisons in our revision.

3. Rationality of Dataset Design

We appreciate the reviewer raising this important design discussion. Our dataset design reflects several deliberate choices based on empirical findings and methodological requirements:

• Adversarial Testing Focus: Our work adopts an adversarial testing approach focused on probing text-bias, rather than measuring overall performance. This single-label dataset design aligns with established adversarial testing practices in security and capability boundary research [4–6]. We use ASCII art images as crafted adversarial inputs targeting specific VLM vulnerabilities in text-visual processing.

• Realistic Attack Scenarios: This mirrors genuine attack scenarios, where malicious users employ positive-looking characters to create harmful content that evades detection systems. These scenarios are both impactful and directly aligned with our research goal, which is why we center our design on them.

• Challenges with Positive Cases: Our early validation revealed fundamental challenges in creating positive adversarial examples: Negative characters forming positive patterns failed due to negativity bias—the negative textual elements dominated emotional interpretation [7–9], possibly overwhelming the intended positive visual impression. Even human annotators disagreed on the labels. Positive characters forming positive patterns removed the text-visual conflict, leading models to achieve artificially high accuracy (L5: 100%, L6: 95%, L7: 86%) by reading the text alone, rather than processing the visual layout.

Given these issues, we adopted an all-negative sample design, which provides higher consistency across different ASCII-art groups and better serves our diagnostic objectives. We appreciate this feedback and will expand our discussion of dataset design rationale in the revision to better explain our choices and methodological considerations.