Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It

Thank you for taking the time to offer us feedback. We appreciate your positive evaluation regarding the clarity of the paper, the hypotheses, and the overall framework. Below, we address the weaknesses you identified and questions you asked per topic, and hope our responses help resolve your concerns.

On generalizing from taxonomic knowledge to other types of knowledge: Thank you for bringing up this point, something that Reviewer uJ4t also brought up. As they (Reviewer uJ4t) pointed out, focusing on one phenomenon allows us to perform more in-depth exploration/analysis. In addition, taxonomic knowledge is fundamental to semantic cognition research [1-5], and there has been a long precedence of this type of knowledge in ML – with datasets like ImageNet being built on top of the wordnet taxonomy. Secondly, prior attempts to assess the benefits of vision-and-language (VL) training for general language tasks have largely yielded negative results [6-11]. This suggests that broader generalization beyond taxonomic knowledge, so far, is a challenging endeavor.

This is not to say that we disagree with the fact that the study is limited in scope – we appreciate the suggestions to go beyond taxonomic knowledge and we will be taking our current work as a blueprint to perform large scale behavioral benchmarking in the future, including but not limited to a text-only spatial reasoning dataset.

On the usage of logistic regression: We intentionally chose logistic regression because it is well-suited for interpretability: the odds ratios are effect sizes that directly quantify how much contextual similarity (independent variable) contributes to correctness of answers (dependent variable). While more complex models (e.g., MLPs, SAEs, or even LLMs) might yield higher predictive accuracy, they would obscure the clear, coefficient-based interpretation critical to our analysis. Our goal was not to optimize predictive performance, but to demonstrate whether increased contextual similarity between concepts is systematically associated with better task performance, a conclusion that logistic regression enables us to draw in a transparent manner. This has been standard practice in psycholinguistics, among other disciplines.

On whether similar works exist and if we could compare against them: Taxonomic knowledge has certainly been of interest to the community, both in terms of evaluating text-only models as well as VLMs. To the best of our knowledge, no former work has evaluated the impact of VL-training on LMs’ taxonomic/category knowledge using minimal-pair VLM-LM architectures. Furthermore, since this minimal pair comparison involves a text only task, we cannot use datasets that have only been used to evaluate VLMs. A related point was brought up by reviewer p9UG, and this is part of our response to them relevant to your comment: There are three classes of taxonomically motivated datasets that we could have compared against, enumerated below:

1. Word-prediction datasets like the one by Hanna and Mareček [12] use cloze-prediction tasks to test models like BERT. While useful, its design closely resembles the positive examples of TAXOMPs and does not add substantial new evaluation challenges.

2. NLI datasets like those used in Geiger et al. [13] focus on lexical entailment (inferred using hypernymy) but lack negative samples, which makes it less suitable for robust evaluations.

3. Property inheritance testing datasets like in Rodriguez, Mueller, Misra [14] involve attributing a novel, nonsense property (e.g., is daxable) to concepts given that their parents are attributed with it – e.g., Given that all birds are daxable, is it true that robins are daxable? -> Yes/No. This dataset does include a robustness check in the form of a single negative sample per concept (we have four), and even includes a larger set of categories (2016 positive samples in total, spanning 44 superordinate categories and 1281 subordinate categories). Therefore, it proves to be a more suitable candidate compared to 1) and 2) for (additionally) analyzing deployment of taxonomic knowledge. We tested our model pairs on this dataset (with close to no prompt-tuning), and found largely similar qualitative trends, as shown in the following table. We will include this in the appendix as supplementary evidence that supports our conclusions. We appreciate the reviewer’s suggestion and hope this addresses the concern.

References:

[1] Murphy, G. (2002). The big book of concepts. MIT press.

[2] Collins, A. M., & Quillian, M. R. (1969). Retrieval time from semantic memory. Journal of verbal learning and verbal behavior, 8(2), 240-247.

[3] Keil, F. C. (1992). Concepts, kinds, and cognitive development. mit Press.

[4] Rosch, E. (1975). Cognitive representations of semantic categories. Journal of experimental psychology: General, 104(3), 192.

[5] Xu, F., & Tenenbaum, J. B. (2007). Word learning as Bayesian inference. Psychological review, 114(2), 245.

[6] Yun, T., Sun, C., & Pavlick, E. (2021). Does vision-and-language pretraining improve lexical grounding? EMNLP

[7] Amariucai, T., & Warstadt, A. (2024). Acquiring linguistic knowledge from multimodal input. BabyLM.

[8] Wang, W., Vong, W. K., Kim, N., & Lake, B. M. (2023). Finding structure in one child's linguistic experience. Cognitive science.

[9] Deitke et al. (2024). Molmo and Pixmo: Open weights and open data for state-of-the-art multimodal models.

[10] Iki, T. & Aizawa, A. (2021). Effect of visual extensions on natural language understanding in vision-and-language models. EMNLP

[11] Madasu, A. & Lal, V. (2023). Is multimodal vision supervision beneficial to language? CVPR

[12] Hanna, M., & Mareček, D. (2021). Analyzing BERT’s knowledge of hypernymy via prompting. BlackboxNLP.

[13] Geiger, A., Richardson, K., & Potts, C. (2020). Neural natural language inference models partially embed theories of lexical entailment and negation. arXiv preprint arXiv:2004.14623.

[14] Rodriguez, J. D., Mueller, A., & Misra, K. (2025). Characterizing the Role of Similarity in the Property Inferences of Language Models. NAACL

Thanks for taking the time to offer us feedback! We appreciate your positive comments regarding our controlled setup, the clarity of our hypotheses, and the representational depth of our analyses. Below, we address the weaknesses you identified and questions you asked clustered by topic, and hope our responses help resolve your concerns.

On limited scope and generalization of results beyond taxonomic knowledge: Thank you for raising this concern. As you have pointed out, focusing on one phenomenon allows us to perform more in-depth exploration/analysis. In addition, taxonomic knowledge is fundamental to semantic cognition research [1-5], and there has been a long precedence of this type of knowledge in ML – with datasets like ImageNet being built on top of the wordnet taxonomy. Secondly, prior attempts to assess the benefits of vision-and-language (VL) training for general language tasks have largely yielded negative results [6-11]. This suggests that broader generalization beyond taxonomic knowledge, so far, is a challenging endeavor.

This is not to say that we disagree with the fact that the study is limited in scope – we appreciate the suggestions to go beyond taxonomic knowledge and we will be taking our current work as a blueprint to perform large scale behavioral benchmarking in the future, including but not limited to a text-only spatial reasoning dataset.

On data filtering details: We appreciate you raising this concern, and apologize for lack of clarity. As mentioned in the manuscript, this was done over multiple stages. Before manual annotation and removal of specific chains, we identified 52 highly abstract concepts (e.g., entity, physical entity, whole, conveyance, act, etc.) to remove from our chains right away – these are listed in the “GQA Hierarchies First Pass.ipynb” notebook under notebooks/ in our code. After generating the hypernym chains, we then resorted to further manual inspection to find and remove/replace cases with “non-ideal” categorizations – by which we meant either that it was not the canonical category of the object (i.e., bubble being categorized as circle) or that there were chains that only contained abstract elements initially missed from our filtering. We found a total of 611 such cases, out of which we were able to “fix” 296 cases by simply querying wordnet to find alternative chains, leaving us with 315 unfixable cases, which were removed (as stated in the paper). These annotations can be found in data/gqa_entities/noun-hypernymy-paths-annotated.csv – with “1” indicating unfixable cases, “2” indicated replaced cases, and “3” indicating cases we were not sure about (N=3) which we decided to remove. All manual annotation was done by an expert in the team with 7+ years experience with semantic knowledge literature. We will disclose these additional details in the final version of our paper, and once again thank you for bringing this to our attention. In the context of attributes: we remove ones that introduce non-standard property attribution, such as “sitting airplanes”, or “happy trees” or “fluffy apple” or “swimming flowers”. We will aim to report a sample of the statistics for such cases in the final version.

On underspecification of deployment mechanisms: This is indeed an important question that has not escaped our minds. First, by deployment, we mean the application of knowledge in contexts that implicitly require this knowledge to be recruited. This is what we measure using TaxonomiGQA. We differentiate this from explicit knowledge and representation/storage, which we measure using TAXOMPS (explicit questions about category membership) and representational analyses as part of H1 in the paper. Next, getting at the underlying mechanisms will require a whole host of substantially different tools (such as training data attribution or controlled training ablations) that are a bit out of scope of this work. This is because the meaningful usage of those tools or pursuing that research question depends on the prerequisite finding of a task where vision and language does have a positive impact on language only-task, which has so far been elusive, until this work.

References:

[1] Murphy, G. (2002). The big book of concepts. MIT press.

[2] Collins, A. M., & Quillian, M. R. (1969). Retrieval time from semantic memory. Journal of verbal learning and verbal behavior.

[3] Keil, F. C. (1992). Concepts, kinds, and cognitive development.

[4] Rosch, E. (1975). Cognitive representations of semantic categories. Journal of experimental psychology: General. [5] Xu, F., & Tenenbaum, J. B. (2007). Word learning as Bayesian inference. Psychological review, 114(2), 245.

[6] Yun, T., Sun, C., & Pavlick, E. (2021). Does vision-and-language pretraining improve lexical grounding? EMNLP

[7] Amariucai, T., & Warstadt, A. (2024). Acquiring linguistic knowledge from multimodal input. BabyLM.

[8] Wang, W., Vong, W. K., Kim, N., & Lake, B. M. (2023). Finding structure in one child's linguistic experience. Cognitive science.

[9] Deitke et al. (2024). Molmo and Pixmo: Open weights and open data for state-of-the-art multimodal models.

[10] Iki, T. & Aizawa, A. (2021). Effect of visual extensions on natural language understanding in vision-and-language models. EMNLP

[11] Madasu, A. & Lal, V. (2023). Is multimodal vision supervision beneficial to language? CVPR

Thank you for taking the time to offer us feedback. We are glad that you recognized the effort involved in the construction of the TaxonomiGQA dataset, found our evaluation metrics novel, and viewed our hypotheses on VLM training and taxonomic knowledge deployment (as well as their validation) as thorough and well-motivated. Below, we address the weaknesses you identified clubbed by topic, and hope our responses help resolve your concerns.

On the fixed structure of WordNet and potentially ignoring the “dynamic” and “polysemous nature of real world concepts”: Thank you for bringing this up. We agree that polysemy and the context-sensitive nature of concepts is important. We also agree that it can be difficult to draw a sharp line even between physical entities, let alone the abstract ones such as “law”, “emotion”, or “right”, given the dynamic and context-sensitive nature of concepts. Nonetheless, we believe WordNet offers a reasonable approximation of human-like taxonomies and remains a widely used resource for taxonomy-related tasks throughout computational linguistics [1-3, for example]. Moreover, even within this constrained, fixed-structure setting, models still struggle to demonstrate robust taxonomic understanding, underscoring the value of using WordNet-based benchmarks as a good starting point for taxonomic evaluation. More importantly, we’re also not aware of any resources that can faithfully facilitate research into dynamic, polysemy aware taxonomic/category organization. If a more realistic taxonomy dataset becomes available, we would be glad to extend our experiments to it. We also agree that developing context-sensitive and usage-driven concept structures is a valuable direction for future research.

On the potential “unfairness” of using GQA questions: Thank you for raising this concern. Although TaxonomiGQA builds on top of GQA’s base questions, there are critical elements to our design that mitigate this risk. First, TaxonomiGQA is a text-only task – each image is converted into a scene description, and since the image is discarded in lieu of this long-form description, this is no longer the same task as the original GQA. Next, the original GQA questions do not contain hypernym substitutions or negative samples, by including them, we again deviate substantially from the original task (many of them introduce label changes too). Third, we have an important constraint in place for the conditional and HCA metrics – that each respective model must correctly answer the base question (i.e., the question that appeared verbatim in GQA). So in this sense, no model has a particularly unfair advantage. Finally, we had foreseen this concern, and ran a question-only control to see if VLMs have an unfair advantage – the inputs are now identical to the original GQA task’s text content. We found that models are substantially worse in this setting, being closer to chance (50%) – results can be found in Table 4, appendix D).

On the lack of “strong novelty” and failure to “address a concrete problem”: While we appreciate you bringing up this concern, we will have to respectfully disagree with it. Taxonomic knowledge is one of the most prominent types of knowledge studied in the semantic cognition literature [5-12] – it is not totally clear what the origins of its development are, and what kinds of features and input signals are central to learning it. So in this sense, we believe taxonomic knowledge/hypernymy to be a fundamental lexical semantic concept worth studying in systems that seemingly demonstrate linguistic competence. In the context of novelty, there is an abundance of investigations trying to see whether multimodal training leads to improvements on language-only task performance of an LM, all of them finding negative results [13-18]. To the best of our knowledge, ours is the first setting where we can conclude that V-L training (a subset of all kinds of multi-modal training) has brought about a substantial improvement in an identifiable domain (e.g., Wang et al.’s [15] marginal improvement in perplexity in VLMs were not attributed to specific domains), and that it is nuanced (i.e., no real change in representation but non-zero change in deployment) We hope that this discussion presents compelling evidence about the concreteness of the domain we are hoping to work on, and the novelty of our findings.

On not using existing taxonomic datasets: We want to clarify that we did, in fact, incorporate multiple existing taxonomic datasets into our evaluation, in addition to the TaxonomiGQA dataset we introduce. First, the TAXOMPs dataset which we created using the GQA entities, which behaviorally allows us to test explicit taxonomic membership. Then, one of our representational analyses which uses the methodology from Park et al. [9] and uses quite a large subset of the wordnet taxonomy – larger than the set of GQA concepts, to see the general geometry of hierarchical representations within the embeddings of VLM-LM pairs. Outside of the datasets we used, there are three kinds of taxonomically motivated datasets that we could have used, we enumerate them below:

1. Word-prediction datasets like the one by Hanna and Mareček [10] use cloze-prediction tasks to test models like BERT. While useful, its design closely resembles the positive examples of TAXOMPs and does not add substantial new evaluation challenges.

2. NLI datasets like those used in Geiger et al. [11] focus on lexical entailment (inferred using hypernymy) but lack negative samples, which makes it less suitable for robust evaluations.

3. Property inheritance testing datasets like in Rodriguez, Mueller, Misra [12] involve attributing a novel, nonsense property (e.g., is daxable) to concepts given that their parents are attributed with it – e.g., Given that all birds are daxable, is it true that robins are daxable? -> Yes/No. This dataset does include a robustness check in the form of a single negative sample per concept (we have four), and even includes a larger set of categories (2016 positive samples in total, spanning 44 superordinate categories and 1281 subordinate categories). Therefore, it proves to be a more suitable candidate compared to 1) and 2) for (additionally) analyzing deployment of taxonomic knowledge. We tested our model pairs on this dataset (with close to no prompt-tuning), and found largely similar qualitative trends, as shown in the following table. We will include this in the appendix as supplementary evidence that supports our conclusions. We appreciate the reviewer’s suggestion and hope this addresses the concern.

References:

[1] Cho, Y., Rodriguez, J. D., Gao, Y., & Erk, K. (2020). Leveraging WordNet paths for neural hypernym prediction. COLING

[2] Misra, K., Rayz, J., & Ettinger, A. (2023). COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models. EACL

[3] Moskvoretskii, V., Panchenko, A., & Nikishina, I. (2024). Are large language models good at lexical semantics? a case of taxonomy learning. LREC-COLING 2024.

[4] Murphy, G. (2002). The big book of concepts.

[5] Collins, A. M., & Quillian, M. R. (1969). Retrieval time from semantic memory. Journal of verbal learning and verbal behavior

[6] Keil, F. C. (1992). Concepts, kinds, and cognitive development.

[7] Rosch, E. (1975). Cognitive representations of semantic categories. Journal of experimental psychology: General, 104(3), 192.

[8] Xu, F., & Tenenbaum, J. B. (2007). Word learning as Bayesian inference. Psychological review

[9] Park, K., Choe, Y. J., Jiang, Y., & Veitch, V. (2024) The Geometry of Categorical and Hierarchical Concepts in Large Language Models. ICLR.

[10] Hanna, M., & Mareček, D. (2021). Analyzing BERT’s knowledge of hypernymy via prompting. BlackboxNLP.

[11] Geiger, A., Richardson, K., & Potts, C. (2020). Neural natural language inference models partially embed theories of lexical entailment and negation. arXiv preprint arXiv:2004.14623.

[12] Rodriguez, J. D., Mueller, A., & Misra, K. (2025). Characterizing the Role of Similarity in the Property Inferences of Language Models. NAACL.

[13] Yun, T., Sun, C., & Pavlick, E. (2021). Does vision-and-language pretraining improve lexical grounding? EMNLP

[14] Amariucai, T., & Warstadt, A. (2024). Acquiring linguistic knowledge from multimodal input. BabyLM.

[15] Wang, W., Vong, W. K., Kim, N., & Lake, B. M. (2023). Finding structure in one child's linguistic experience. Cognitive science.

[16] Deitke et al. (2024). Molmo and Pixmo: Open weights and open data for state-of-the-art multimodal models.

[17] Iki, T. & Aizawa, A. (2021). Effect of visual extensions on natural language understanding in vision-and-language models. EMNLP

[18] Avinash Madasu and Vasudev Lal (2023). Is multimodal vision supervision beneficial to language? CVPR

Thank you for taking the time to give us feedback. We appreciate your positive comments regarding our dataset design, choice of metric, inclusion of negative samples, and our analyses. Below, we address the weaknesses you identified and questions you asked grouped within topics, and hope our responses help resolve your concerns.

On the importance of distinction between existing knowledge vs. its deployment: We see two specific reasons why this distinction could be important: first, storing/representing knowledge is different from learning their functional consequences [1]. For instance, a model may as well robustly represent category information (robins are birds) but might not deploy this correctly in contexts that recruit this knowledge - we see TaxonomiGQA as one such context. The practical benefit of teasing this apart is knowing where the performance bottleneck lies, and this could inform decisions about training data selection (e.g., if we were to collect additional (post)training data, should we include more encyclopedic knowledge or diverse contexts in which such knowledge is deployed?) and motivate solutions such as transferring task vectors [2] from models that are better at deployment, for instance. Next, finding that there is a distinction between representation of knowledge vs. its deployment and that it could depend upon the kind of supervision received by a neural network model can be informative for cognitively inclined researchers who want to draw appropriate conclusions about the role of language vs. extralinguistic information. For instance, the platonic representational hypothesis [3] roughly suggests that models trained on a large amounts of data in the world learn similar representations, even if their input modalities are completely different (text vs. vision); our findings suggests, to an extent, that while this might be possible, there are cases where combining information from multiple modalities can bring about non-trivial changes, not in the representations (of a certain kind – in our case taxonomies) but instead in terms of its deployment.

On outliers in results: We appreciate you drawing attention to two main outliers in our results: first with the LM outperforming the VLM on the TaxonomiGQA task for the Vicuna-Llava pair; and the second with the VLM substantially outperforming the LM mllama-llama 3.1 pair on TAXOMPS. While we agree that understanding precisely what leads to these exceptions would be beneficial, we would like to point out that our main goal in this paper is about the effect of V-L training, holding the underlying architectures of the models constant. In this framing, our comparisons are meant to isolate the presence or absence of VL training. Explaining the behavior of a single outlier like Vicuna vs. LLaVA-1.5 would require controlled architectural ablations, which are beyond the scope of our current setup. A tentative conjecture one could make about Mllama 3.2 being substantially better than Llama 3.1 on the TAXOMPs task is that the Mllama 3.2 model introduces 2 additional elements to the LLama 3.1 model: 1) 3B additional parameters due to cross-model attention; and 2) RLHF based instruction-tuning, despite not saying “Instruct” in the name (as reported in the HuggingFace documentation of the model). Since the Mllama 3.2-Instruct vs. Llama 3.1-Instruct pair does not show this exceptional behavior, 2) is perhaps the more plausible explanation for this exceptional behavior.

On the limitations of correlational analyses: We agree this is a limitation and explicitly acknowledge it in the paper. Future work could use training data-centric interventions to ask causal questions, for instance by manipulating training data during VL training. However, given computational constraints (such experiments would entail multiple pretraining runs), we believe our correlational analyses and hypothesis testing are a meaningful first step.

References:

[1] Murphy, G. (2002). The big book of concepts. MIT press.

[2] Hendel, R., Geva, M., & Globerson, A. (2023). In-context learning creates task vectors.

[3] Huh, M., Cheung, B., Wang, T., & Isola, P. (2024). The platonic representation hypothesis. arXiv preprint arXiv:2405.07987.