Texture or Semantics? Vision-Language Models Get Lost in Font Recognition

We thank Reviewer A9ZJ for your thoughtful and detailed feedback. We greatly appreciate your insights, which could help us improve the clarity and rigor of our study.

Q1: It would be better to show the possibility of teaching VLMs font recognition.

R1: Thanks for your suggestion! We add the experiment to demonstrate the possibility of using fine-tuning to teach VLMs in this field. We created 30 new sentences for 15 font types, in total 450 images for two VLMs, and we still do the test on the easy version of the benchmark. For fine-tuning, we use LoRA with rank 16 and alpha 16, we trained for two epochs with the cosine scheduler learning rate starting from 1e-4.

The results are shown in the table below:

We could find that even after fine-tuning, the improvements are limited from VLMs, which demonstrates the drawback of VLMs in fine-grained tasks such as focusing on semantic cues. We would work on this as future work to explore effective ways of improving VLMs in this field.

In summary, I would agree with your opinion of ‘include font-related tasks in the training mix’, we are interested in leaving this as future work.

We follow your suggestion to add the experiment to improve our paper.

Q2: It would be better to demonstrate that tested fonts are all in models’ knowledge scope..

R2: Thanks for raising this consideration! The choice of the font used in the paper is based on the popularity rank given by the LLMs and some official websites, that’s to say, for all four closed-source models, they all know these 15 types of font and could give the right description of their features. However, their results on both easy and hard versions, especially even when we even give the multiple choices for them, are still poor.

We follow your suggestion to add the discussion to improve our paper.

Q3: It would be better to add the comparison with the CNN-based models.

R3: Thanks for this helpful suggestion! We would like to give the accuracy of the CNN-based model. We created 100 new sentences for 15 font types, in total 1500 images for fine-tuning based on the HENet and ResNet backbone, which is designed for font recognition, the accuracy is only 39.33%, still not satisfied.

Therefore, above two experiments demonstrate the issue of generalization limitation and training data consuming of the CNN-based model, which shows the necessity of considering well-generalized and strong capabilities VLMs to address these issues, also the first experiments post the question of how to effectively fine-tuning VLMs for this task, which we would work on as future work.

We follow your suggestion to add the comparison to improve our paper.

At last, we really thank the reviewer for recognizing the purpose and the significant meaning of our work on exploring the limitations of VLMs in fine-grained tasks like font recognition and their vulnerabilities towards the visual cues and semantic cues. Again, thanks for your valuable feedback and meaningful suggestions!

We sincerely thank Reviewer P7Qz for your thoughtful and detailed feedback. We greatly appreciate your insights, which could help us improve the clarity and rigor of our study.

Q1: It would be better to compare previous work on font recognition, including benchmarks and specific models.

R1: Thanks for the consideration! The main purpose of our paper is to evaluate the capabilities of VLMs on fine-grained tasks like font recognition. Famous font recognition benchmarks like Adobe VFR and Explore All contain more than thousands of classes, which is not suitable for the evaluation of VLMs. We aim to use straightforward and simple methods, so our current benchmark only needs to focus on 15 popularly used fonts and with the design of stroop effect, already demonstrated the limitations of VLMs in this field.

We would like to give the results of the CNN-based model. We created 100 new sentences for 15 font types, in total 1500 images for learning based on the HENet and ResNet backbone, but the accuracy is only 39.33%, still not satisfied. It shows the issue of generalization limitation and data consuming for the CNN-based model, which shows the necessity of considering well-generalized and strong capabilities VLMs to address these issues.

Q2: It would be better to consider fine-tuning or reasoning models as possible solutions for improvement.

R2: Our focus in this paper is to evaluate VLMs in zero-shot and prompt-based settings because these are the most common use cases for general-purpose models in practical deployments.We agree with the reviewer that considering above methods for comparison would further improve our paper.

For reasoning models, we sample 2 images for each font from both easy and hard version, and the results are presented below:

According to our experiments, reasoning models also do not show incredibly strong capabilities in the task and are still easy to be cheated by the texture information rather than semantic information.

As for fine-tuning, we created 30 new sentences for 15 font types, in total 450 images for fine-tuning two VLMs. We use LoRA with rank 16 and alpha 16, trained for two epochs with the cosine scheduler learning rate starting from 1e-4.

The results are shown in the table below:

We could find that even after fine-tuning, the improvements are limited from VLMs, which demonstrates the drawback of VLMs in fine-grained tasks such as focusing on semantic cues. We would work on this as future work to explore effective ways of improving VLMs in this field.

Q3: It would be better to add discussion of using attention matrices for explanation.

R3: We agree that attention maps do not always offer definitive explanations, and their interpretability can vary across models and tasks. However, our intent was not to treat attention as a perfect interpretive tool, but rather to use it as a proxy for identifying potential gaps in visual grounding.

Q4: Which attention layer do you choose to print attention scores? How do you convert 2-dimensional matrices to 1-dimensional matrices?

R4: For the Qwen model, which has 28 layers, since middle layers focus on visual cues, we calculate the average attention from layer index 7 to 20, and the attention score is the weights of the image patches to the “font” token. For the Llama model, we select layer index [13, 18, 23, 28], since the cross-attention layer in the Llama model appears every 5 layers from index 3 to 38.

Q5: It’s better to add the discussion of choosing font recognition.

R5: Font recognition presents key challenges in fine-grained visual perception, which is crucial for developing robust VLMs. Our work reveals VLMs' limitations in visual detail attention and their susceptibility to texture over semantics. Additionally, font recognition has important applications in design, OCR, accessibility, and content indexing, making it both theoretically and practically significant.

Q6: It’s better to add more discussion about previous work.

R6: To the best of our knowledge, there is limited to no prior work directly fine-tuning VLMs for font recognition. Existing approaches focused on CNN-based models trained specifically on labeled font datasets. Our work is among the first to systematically assess the capabilities of general-purpose VLMs in this fine-grained visual task and to highlight their vulnerability to semantic bias. This evaluation not only has practical meaning, but also exposes foundational limitations in their perceptual alignment and reasoning capabilities.

We thank Reviewer FEXv for your thoughtful and detailed feedback. We greatly appreciate your insights, which could help us improve the clarity and rigor of our study.

Q1: It’s better to add the discussion for research scope.

R1: We want to demonstrate that font recognition embodies fundamental challenges in fine-grained visual perception, which is an essential part for the development of robust, general-purpose VLMs. Our work aims to highlight limitations in VLMs’ attention to visual details and easily to be affected by the texture content rather than focusing on semantics. Moreover, font recognition has concrete applications in design, OCR, accessibility tools, and digital content indexing, making it both theoretically informative and practically relevant.

We follow your suggestion to add the discussion to improve our paper.

Q2: It would be better to add the comparison with specialized models.

R2: Thanks for the suggestion! Our primary focus is to evaluate general-purpose VLMs to assess their suitability for a task that is both visually fine-grained and context-sensitive. However, we would like to add the following experiments!

First, we fine-tune two VLMs to show whether specialized VLMs could give significant improvements. We created 30 new sentences for 15 font types, in total 450 images, and we still do the test on the easy version of the benchmark. For fine-tuning, we use LoRA with rank 16 and alpha 16, we trained for two epochs with the cosine scheduler learning rate starting from 1e-4.

The results are shown in the table below:

We could find that even after fine-tuning, the improvements are limited from VLMs, which demonstrates the drawback of VLMs in fine-grained tasks such as focusing on semantic cues.

Secondly, we would like to give the accuracy of the CNN-based model. We created 100 new sentences for 15 font types, in total 1500 images for learning based on the HENet and ResNet backbone, which is designed for font recognition, the accuracy is only 39.33%, still not satisfied.

Above two experiments demonstrate the issue of generalization limitation and training data consuming of the CNN-based model, which shows the necessity of considering well-generalized and strong capabilities VLMs to address these issues, also the first experiments post the question of how to effectively fine-tuning VLMs for this task, which we would work on as future work.

We follow your suggestion to add the comparison to improve our paper.

Q3: It would be better to add discussion of modality conflict and reasoning limitations.

R3: Thanks for this suggestion! We agree that the stroop effect in our paper offers a unique lens to examine how VLMs handle conflicting signals between visual features and textual semantics. We expand the discussion on this point, specifically elaborating on how the inability to suppress textual bias in the presence of conflicting visual cues reveals a gap in modality-aware reasoning. We frame this failure not just as a task-specific issue, but as a broader limitation in current multimodal architectures’ interpretability and alignment mechanisms.

We follow your suggestion to add the discussion to improve our paper.

At last, sorry for the inconvenience caused by moving the full related work part into appendix, we follow your suggestion to move this essential part back to the main body.

Again, thanks for your valuable feedback and meaningful suggestions!

We thank Reviewer A1JS for your thoughtful and detailed feedback. We greatly appreciate your insights, which could help us improve the clarity and rigor of our study.

Q1: It would be better to further discuss the necessity of difficulty for near-identical typefaces.

R1: Thanks for this consideration! We agree that fine-grained distinctions can be challenging due to their visual similarity. However, this difficulty is precisely what motivates our investigation. One of our aims is to assess whether modern VLMs, which claim strong visual abilities, can handle such realistic and nuanced challenges. Also, the easy version of the task includes numerous letters to let VLMs observe the edge parts, suggesting that if VLMs genuinely possess the ability, differentiating them should not be challenging, which is also the reason why we designed two versions of the dataset. Additionally, the limited increase brought by few-shot learning also demonstrates the inability of VLMs on learning from the semantic cues. In summary, our design of experiments and benchmarks points out the limitations of VLMs in figuring out semantic cues and their vulnerabilities towards texture cues.

We follow your suggestion to add the discussion to improve our paper.

Q2: It would be better to further analyze “Indecisive Classification” errors.

R2: Thanks for the suggestion! We agree that this could help make our paper more comprehensive in analyzing VLMs’ weakness in font recognition. Experimental results show that in the ‘Indecisive Classification’ category, only 23% of the response appears to have a font prediction, the rest responses are only able to point out the ‘large’ category such as Serif or Sans-Serif. That’s to say, in most situations the models are even difficult to give related candidates.

We follow your suggestion to add the analysis to improve our paper.

Q3: It would be better to add visualizations for QWEN2-VL-7B-INSTRUCT on the hard version.

R3: Thanks for the helpful suggestion! We have already plotted all the attention figures for the false cases, the observations and conclusions are the same as the figures shown in the paper right now. Considering the length of the paper, we didn’t add them into the paper, but we would follow your suggestion to add them into the appendix to improve our paper!

Q4: It would be better to additionally introduce the stroop effect in the sentence format.

R4: Thanks for this suggestion! We currently choose the word-level stroop effect since it is straightforward and we increase the font size for this hard version to expect that model could possibly learn from the edge parts but the results are disappointing. We already added another test dataset that incorporates stroop effect into sentence format.

We replace the font name with the sentence introduction of the font, such as: “We introduce Times New Roman, which is a type of font that …”, in total it’s still 225 images. We keep the font size as the easy version uses. And we present the results in the below table:

From the above results, we could see that even if the false font name only appears once in the sentence format, the VLMs are still easily to be cheated by the stroop effect, which demonstrates the issue again.

We follow your suggestion to add the experiments to improve our paper.

Again, thanks for your valuable feedback and meaningful suggestions!