Can We Talk Models Into Seeing the World Differently?

Dear Reviewer 1KNs, We thank the reviewer for the review and valuable feedback. We appreciate the assessment of our work as novel due to ”being the first to study VLM’s shape-texture biases”, with a “detailed and comprehensive analysis” showing ”several key contributions” in a ”clearly written” paper. Please find detailed answers to each comment below.

W1: After reading the paper, it is still unclear to me what the significance of studying shape and texture biases is, primarily when MLLMs can already address more complicated tasks than object recognition nowadays. This concern limits the significance of this paper. [...] how studying shape-texture biases benefits the development or applications of VLMs.

We apologize that our submission did not communicate this more clearly, and would like to break down this comment into multiple parts for clarity.

Goal of the paper: First, we would like to clarify that the goal of this paper is to show that purely visual biases can be steered by language (at inference time). This is a powerful and yet very cheap method to de-bias a model without having to retrain it. We use the texture/shape bias as one specific instance, especially since it is potentially the best-researched vision bias. However, the emphasis is not on that bias alone.

Why is texture/shape significant: The texture/shape bias fundamentally differs in how models and humans process vision. Human perception relies heavily on shape to identify objects, while many models typically show a strong texture bias, relying more on local patterns rather than global shapes. This difference can lead to models performing well on certain (i.i.d.) datasets but failing to generalize in real-world scenarios where shapes are more consistent than textures. By adjusting models to have a (stronger) shape bias, they align more closely with human vision, and we expect them to better generalize and be more interpretable due to the increased similarity.

VLMs already address more complicated tasks: VLMs support many advanced tasks, such as Visual Question Answering (VQA). However, these tasks often rely fundamentally on object detection, meaning that any bias in the object detection process will transfer to these tasks. Studying a bias in such a low-level task can be a fruitful way to improve performance in many more abstract tasks that are dependent on object recognition.

How do VLM development/applications benefit: Our priority was to show that the model can be debiased (to some extent) at inference time by just using language. This is very much in line with the zeitgeist to optimize large foundation models at inference time (e.g., model editing [1] or test-time scaling [2] as done in GPT-o1) and not having to retrain them from scratch e.g., to remove biases.

However, our study also carries some takeaways for development: Whenever possible a debiased vision encoder should be used as these are more steerable. This is of course easier said than done, but being aware of specific biases through a large-scale study like ours is a step in the right direction. We have seen that LLMs turn flexible vision representations (i.e., ones that contain both cues) into highly biased predictions by completely ignoring one cue. This requires a lot of attention in future training - LLMs should clearly not just ignore some features.

Does this clarify the significance, and would the reviewer recommend that we add (parts of) our explanation to the paper?

W3: More clarifications are needed on the steering part, where I have similar concerns with W1 -- the authors might want to clarify the usefulness of the potential contribution of steering such biases. (1) The claim in L448-449, "overall accuracy does not change considerably," might indicate that shape-texture bias is not a significant factor for accurate models.

We hope to have clarified the usefulness above and want to focus on what performance means in our study, as there seem to have been some misunderstandings.

The “performance” we report is the accuracy measured on the texture/shape cue-conflict dataset. Predicting either the shape or texture label counts as a correct prediction. This serves as an important signal for our study, showing that our prompting strategies indeed steer the bias in outputs (* see below), but it does not tell us if this “performance” transfers to other problems. Other works have shown that the texture/shape bias is, for example, strongly correlated with ImageNet accuracy [3].

• For example, you could get perfect shape bias by just predicting a false label for all detections that are currently texture-biased – however, this is not steering as it comes at a significant cost in accuracy. We clarified this in L149ff, as our current phrasing seems to have confused multiple reviewers.

Dear Reviewer Ume1, We would like to thank the reviewer for the review and valuable feedback. We are happy to hear that the reviewer found the article ”presented excellent”, providing an ”extensive study” with a ”detailed analysis in the supplementary”. Please find detailed answers to each comment below.

W1: In section 5.2, spectral biases or critical bands that are difficult to control in language, I ask for more of the authors' opinions on why the language prompt influenced this bias In Sec. 5.2, we showed that steering the low-/high-frequency bias is possible, but not as effective as the texture/shape-bias.

If the question is “why does this work at all”: we want to clarify that we do not think that spectral biases are not controllable at all, it is just that there is likely less multi-modal training available for this bias. After all, our results show statistically significant changes.

Conceptually, for language prompt steering to work properly, we need to satisfy (at least) two conditions: (1) the vision encoder must provide a “flexible” representation, which allows reconstruction of both feature cues, (2) the LLM must have learned a correlation between natural language and the visions tokens. Based on the vision encoder bias (see answer to Q1), we can assume that the representation is flexible and the problem must be more related to condition (2). This is, intuitively, reasonable, as learning texture or shape features is directly possible from pixel values, but understanding frequency bands requires a Fourier transform. Additionally, spectral effects are much harder to describe via natural language and probably underrepresented in the text training data. So, it is less likely that the VLM saw text/image pairs connecting these features.

Does this answer your question?

and what it means when the performance decreases even if the bias value changes.

The simple answer is that the steering is a bit noisy (like many things in VLMs).

Recall that we have two labels for each cue-conflict sample, e.g., one for shape and texture. Accuracy is unusually defined by the ratio of predictions where the model has predicted either one of the labels (i.e., made no error). When we “steer” the bias, we would like the predictions to change but not for the model to make additional errors. For example to steer toward shape bias, ideally, the ratio of shape predictions would increase at the same rate as the ratio of texture predictions would drop. This would show that the model has indeed shifted its bias and is prioritizing features differently now. In that case, the accuracy would not change – but most systems are noisy in practice and it seems that occasionally we also destroy the attention to some features.

W2: According to the results of Figure 4, steering can slightly control textual/shape biases, but it seems to have a slight impact on overall performance, so what is the point of controlling these biases?

The “performance” we report is the accuracy measured on the texture/shape cue-conflict dataset. Predicting either the shape or texture label counts as a correct prediction. This serves as an important signal for our study, showing that our prompting strategies indeed steer the bias and not just ignore textures in outputs (see answer above) – however, it does not tell us if this performance difference (if any) transfers to other problems. Specifically, on ImageNet samples we would not expect any improvements in accuracy from a shape bias, as the task is solvable by texture alone [1]. However, in any task where the bias is uncorrelated with the true label at test time, you should see improvements.

To give a bit more perspective on why we want to control biases: Recently, many biases from low-level vision (like texture/shape) to high-level societal biases (gender bias) were discovered in SOTA models (see Appendix O). Effectively, a bias means that a model is over-prioritizing some feature cue. Some biases are benign, for example in ImageNet classification - where we often have one single object in front of a background - a hypothetical “foreground bias” is exactly what we want to detect objects, but on the other hand, a “background bias” would be unwanted. These biases often arise due to “shortcuts” in training data [2] and even a biased model may perform well on i.i.d. datasets. However, when we sample more data at test time when the shortcut is no longer present, it would cause a drop in model performance.

In this paper, we show that language (via prompts) can be a simple tool to steer unwanted biases toward expected behavior and actual generalization. This method is cheaper than (re)training a de-biased model, especially for expensive training as done in VLMs.

[1] Brendel et al., “Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet”, ICLR, 2019.

[2] Geirhos et al., “Shortcut Learning in Deep Neural Networks”, Nat. Mach. Int., 2020.

I have understood what the authors struggle to study in section 5.2, and I have no other questions.

Dear Reviewer bGZu, We would like to thank you for your kind review and your valuable feedback inspiring future analysis. We appreciate your assessment of our work as “opening pathways for further research”, “extends beyond traditional uni-modal bias studies” and allows for "more understanding of complex concepts”, which “offers practical implications”. Please find detailed answers to your comments below.

W1: Focus on shape and texture biases limits the study; exploring other biases (e.g., color, spatial attention) would give a fuller understanding of VLM behavior. Q1: Have you considered extending the study to include other visual biases such as color or spatial attention for a broader understanding of VLM behavior?

We totally agree, there is an exciting array of other biases to explore! In this paper, our primary emphasis was on steerability of VLM biases at inference time by language. Given our finite budget, we chose to prioritize more models instead of biases as there is always a risk that a bias is model-specific and limited our analysis to shape/texture and the transferability to low/high-frequency biases (Sec 5.2).

However, we indeed do have plans to provide a more fundamental analysis of low-level vision biases in future work, for VLMs and beyond (independent of steering). As a little teaser, we show the following biases on LLaVA-NeXt-7B, based on the cue-conflict stimuli methodology:

Most of the findings align well with our expectations (i.e., no significant bias for left, right, or no information from H/S channels in the HSL space). It is quite impressive that LLaVA almost perfectly models human color perception (cf. ITU-R BT 601). One other bias we find is that LLaVA over prioritizes the lower half of an image by almost 7.5%.

We are curious for your feedback and thank you again for this excellent pointer!

W2: Limited discussion on prompt selection; more detail on prompt strategies and robustness would enhance the study's depth.

We would kindly point the reviewer to Appendix B, where we have already explored alternative prompts for VQA and Captioning. Additionally, we also show the accuracy/bias tradeoff of LLM-generated prompts in Appendix Fig. 6, which may answer the question about “robustness”. But of course, the space of possible prompts is infinite. Is there something specific that the reviewer would like us to try?

W3: Prompt-based bias steering is constrained; deeper analysis on why some models are less responsive and how to improve steerability is needed.

Another great suggestion!

For language prompt steering to work properly, we need to satisfy (at least) two conditions: (1) the vision encoder must provide a “flexible” representation, which allows reconstruction of any feature cues, (2) the LLM must have learned a correlation between natural language and the visions tokens.

Evidently, these conditions may only be partially met in some models depending on the data and training method. For example, if the multi-modal training data does not contain any correlations between texture/shape and corresponding vision tokens, we would not expect steering to work at all.

Based on these two conditions, we can formulate two methods to improve steerability: Use a (as much as possible) debiased vision encoder. The simplest (but often unrealistic) option to achieve this is to scale up the training data. Include multi-modal training data that correlates language and vision for the bias under test.

Would the reviewer recommend adding this explanation to the paper?

Dear Reviewer VmoU, We thank the reviewer for the review and the valuable feedback. We are happy to hear that the reviewer found the article a "pleasure to read" with "inspiring findings", a "clearly provided motivation" and of "excellent soundness". Please find a detailed answer to the comments below.

W1: Class-wise analysis: For VQA classification, where tasks involve multiple classes, a class-wise analysis to examine texture and shape bias for each class would be helpful. Questions to explore include whether this texture and shape bias is consistent across all classes and whether there are differences between classes with simple vs. complex shapes.

We kindly point the reviewer to our class-wise analysis in Appendix O - Figure 11 for both VQA and Image Captioning. The original Texture/Shape-Bias paper (Geirhos et al., 2019) already observed that there are class-wise differences for discriminative classifiers, and this continues to hold for VLMs. For instance, you can see that the “bicycle”- class is highly shape-biased, while “knife” tends to be mostly texture-biased. For some classes, like “elephant” or “oven” there is quite some variance depending on the model. Since we are dealing with natural objects, it is unclear to us how we can faithfully identify if objects have simple or complex shapes. But our subjective analysis is, if we look at the top-3 most shape-biased classes (“car”, “bicycle”, “clock”) we see objects with highly distinct shapes but different levels of complexity. This may suggest that distinctness is more important than complexity for shape bias.

Has the reviewer any specific suggestion in mind, or does this already answer the question?

W2: Additional qualitative results and examples: This paper provides quantitative results across various aspects; however, it lacks qualitative results and examples, which would help clarify the concepts through visual examples. For instance, the reviewer suggests including examples from the target dataset with texture and shape splits.

Thank you for your suggestion! We have now extended Fig. 9 in the appendix to show a few examples from the texture/shape cue-conflict dataset, in addition to the samples from frequency-cue-conflict. Does this address your comment?

Additionally, presenting qualitative examples of both correct and incorrect predictions could further enhance the paper’s clarity and impact.

We appreciate another great suggestion! It is a bit difficult to do this consistently, as we show many models, but we identified “agreement sets”, where all VLM predict shape, texture, correct or wrong prediction, respectively. We have added this analysis in a new Appendix chapter: I “Agreement sets on texture/shape cue-conflict”,

Q1: The results of steering the texture/shape bias via language prompts are interesting. The reviewer observes that steering towards a texture bias can lead to completely different model behaviors for different models. For example, the performance of Gemini increases by 1%, while LLaVA-NeXT-7B decreases by 2%. However, in the study of prioritizing shapes over texture cues in VLMs, LLaVA-NeXT-7B actually shows a stronger texture bias than Gemini. Do the authors have any comments or findings in this direction?

Thank you for calling our results interesting!

It is worth clarifying that the “performance” we show is the accuracy on the texture/shape cue-conflict dataset – i.e., we count a prediction as correct if it matches either the shape or texture label. If we would perfectly steer the model, we would see label flips where texture decisions become shape or vice versa. In that case, the accuracy would be the same.

If the accuracy decreases it means that we have not evenly steered the bias - given the few samples, 1-2% are however not significant. If the accuracy increases, it means that we now classify samples correctly that we previously hadn’t – this can happen if one feature cue is spuriously correlated within the decision rule of the model. We do not know what encoder Gemini uses, but we would assume that it is different from LLaVAs encoder, and is likely trained on different data, which results in different correlations between features and labels that reflect this different behavior.

Once again we thank you for your valuable feedback! We look forward to further discuss this if there are remaining questions and thank you for your careful consideration of our paper.