Visual Grounding Helps Learn Word Meanings in Low-Data Regimes

Thank you for your feedback. We appreciate the opportunity to address the concerns and questions you have raised.

How do our findings relate to previous work?

The question in this paper is whether visual information can improve the learning of linguistic representations. This is to eventually help reduce the amount of data that is needed to train the language models. The papers provided by the reviewer are all very different from the topic this paper investigates. All of them aim to improve the performance of certain tasks with the help of visual information. Specifically, [1] explores how visual grounding can help the task of generating common sense text from concepts. [2] focus on multi-modality tasks, exploring how language information can help referring expression comprehension and segmentation on images. [3] uses millions of videos and possibly billions of words to examine how word translation can be improved using video-caption joint training. [4] proposes a new basic architecture for vision-language joint training.

Significantly, the studies cited in references [1,2,4] either rely on existing models pretrained on billions of words or, as in [3], require large multi-modal training datasets. In contrast, our paper assesses the performance of models trained on smaller datasets, using word learning benchmarks. Our goal is twofold: firstly, to investigate various factors affecting the learning efficiency of language models, and secondly, to deepen our understanding of human language acquisition processes through these models. These objectives distinctly set our work apart from the studies mentioned by the reviewer. Furthermore, our approach contributes to the ongoing efforts in training language models by demonstrating how reducing data requirements, inspired by language acquisition in children, can be effective.

What do our findings tell us?

Finally, our findings complement the methods proposed in these papers. To be more specific, we find that the visual information and the linguistic distributional information may compete with each other during learning. This finding suggests that carefully merging these two sources of information may lead to a better model, which can be added to the methods proposed in these papers. Moreover, these methods also provide new candidates to be tested on our evaluation benchmarks. Although they typically start from pretrained language models, we can adapt their methods to start from the Language-Only models trained only on the image captions and integrate the visual information afterward. We will explore this new class of methods in our future work.

We thank the reviewer for the feedback and suggestions.

What is the practicality of our findings?

Our work explores how to improve the learning efficiency of models on small-sized datasets. This aims to reduce the amount of training data needed by the current large-scale language models and to better understand the human language acquisition process in children. Humans can learn languages from only millions of words. This indicates that inspiration from this acquisition process should facilitate the learning process of neural language models, which are currently trained with billions of words. Through exploring the influence of different factors, like visual grounding, on the learning efficiency of such models, our paper is also helping to reverse-engineer the language acquisition and cognitive development process in children.

The reviewer also mentioned multiple previous explorations [1,2,3,4] Our findings can complement the methods proposed in these papers. To be more specific, we find that the visual information and the linguistic distributional information may compete with each other during learning. This finding suggests that carefully merging these two sources of information may lead to a better model, which can be added to the methods proposed in these papers. Moreover, these methods also provide new candidates to be tested on our evaluation benchmarks. Although they typically start from pretrained language models, we can adapt their methods to start from the Language-Only models trained only on the image captions and integrate the visual information afterward. We will explore this new class of methods in our future work.

More details about Flamingo training and CLIP evaluation

Like the other models, our Flamingo model used a 6-layer transformer as its language backbone. The Flamingo architecture worked by having additional cross-attention layers modulating the outputs of text transformers, and these cross-attention layers were inserted at equal intervals of text transformer layers. In our paper, we tried inserting the cross-attention layer after every text transformer layer or after every two text transformer layers. We found that the results of these two methods were similar (see Appendix Fig. 15). We have added more details in the paper (see A.3 in the Appendix). About the CLIP representations, we sent only the word to the model and extracted its hidden representations to be evaluated in our benchmark. We also tried to average the representations of the word across multiple sentences containing that word and found that the results were similar to the single-word representation result.

We thank the reviewer for the positive feedback.

What experimental results would be for other models?

The reviewer is concerned about what results would be across a wider variety of model families or variants. Our selection of the diverse model architectures, including CLIP, GIT, and Flamingo, is supposed to improve the generalizability of our findings to other model families. That being said, the InstructBlip and LLaVa mentioned by the reviewer may belong to a different class of methods, and by the definition of their methods, large training datasets are necessary. This is because they start from pretrained large-scale language models already trained on billions of tokens, then finetune them on multi-modality tasks, and continue to finetune the resulting multi-modality networks using carefully selected visual-language instructions. The improvement of these methods over earlier ones is mainly on multi-modality tasks such as visual question answering and continuous chat with images, while we focus on evaluating the learning processes of linguistic information. It might be possible to replace the pretrained large-scale language models used in these methods with the Language-Only models in our paper and similarly do the multi-modality task finetuning and the instruction tuning, which could be a topic for future work. CM3 represents an even more different class of methods than InstructBlip and LLaVa, where retrieval on a large database is combined with generative models to do both text and image generation.

Why did Flamingo show worse performance?

As for why Flamingo worked worse than other methods, we have several hypotheses. The original Flamingo algorithm was designed to start from pretrained large-scale language models with the weights frozen and only train the cross-attention modules. The Flamingo authors even reported lower performance when the weights of language transformer layers were also fine-tuned. Therefore, one hypothesis is that this algorithm cannot efficiently train a neural language model from scratch with the cross-attention modules. Another hypothesis is that this algorithm may work less well with the image-caption pair data used in our work than the large mix of image-caption pairs, website texts embedded with multiple images, and captioned videos used in the original paper.

Metrics other than cosine similarity on the word-relatedness benchmark

The reviewer also asked whether metrics other than cosine similarity were used in the word-relatedness benchmark and mentioned that this metric could be easier for CLIP models to do well than other metrics. After testing the L2 distance metric suggested by the reviewer, we indeed find that CLIP models perform worse under this metric in the word-relatedness benchmark than the same models tested using cosine metric, while the performance of Language-Only and GIT models is not influenced by this change of metric. Since cosine similarity is closely related to the L2 distance metric with both vectors L2-normalized, this result suggests metrics influenced by the L2-norms are under-leveraging the capacities of CLIP models. This fits the reviewer’s expectation and is likely because the L2-norms of the hidden representations in CLIP are not optimized by the cosine-similarity objective function during learning and, therefore, do not carry much meaningful information. However, we believe this new finding should be interpreted as support for using the cosine similarity metric in the word-relatedness benchmark since this metric is more robust to model classes and yields results that are no worse than other metrics.

Moreover, the results in the word-relatedness benchmark are not the only results supporting our claim. The semantic-feature prediction benchmark, for example, also shows evidence of benefits from visual information in low-data regimes, and this benchmark is evaluated through training regressors directly on the hidden representations.

Thank you for your feedback. We appreciate the opportunity to address the concerns and questions you have raised.

Why these evaluation metrics?

First, we emphasize that our study does not solely rely on the cosine similarity between word representations. This metric is only used in the word-relatedness benchmark. The other three benchmarks shown in Fig. 1 do not look at similarity, but instead encodings of syntactic and semantic information in embeddings, as well as their usefulness in language modeling tasks. These additional benchmarks are designed to capture various aspects of word acquisition and understanding, thereby providing a more holistic view of the models' capabilities.

We also wish to emphasize that many of our evaluations use deep representations, not representations from the word embedding matrix. As discussed in Section 3.1, we optimize the layer from which representations are drawn separately in each model. For CLIP models, later layers are better than earlier layers in general. For GIT models, earlier layers are typically the best. Moreover, the two benchmarks shown in Fig 3 test context representations of models, including surprisals and hidden representations of whole sentences. Both of these benchmarks evaluate more than just word representations.

The reviewer also mentioned that there could be disagreement between human annotators. This is a good point, but we don’t think it influences the conclusions presented in this paper. The inconsistency between human annotators leads to a ceiling of the performance on the benchmarks. However, when two models are compared on the same benchmark, a higher number still indicates that the word representation is more human-like. Since our conclusions in this paper are drawn from comparisons of models, they should still hold even if human annotators disagree to some extent.

Finally, the specific definition of the “human-likeness” metric used in A1.2 relates only to correlation with human judgments on the word-relatedness benchmark. Our paper’s broader claims about the human-likeness of learned representations are based on the full suite of evaluations described above and in the paper.

How was the CLIP model trained?

We did not use the pretrained weights of the original CLIP models. The visual features were precomputed from a vision transformer pretrained by unsupervised visual learning algorithms (DINO) on unlabeled ImageNet images. This is discussed in the middle of page 5, beginning at “Visual encoders and image features...” This selection of vision encoder is also explored in Section 4.6 and Figure 5. Thus, in this paper, we use “CLIP” to refer to the language and image contrastive training objective, and call models we trained using this objective function the “CLIP models.” As suggested by the reviewer, we have updated our paper to make this reference clearer (see page 5, we’re currently considering labeling them “CLI” models for “Contrastive Language/Image”).

Why these model architectures?

The CLIP, GIT, and Flamingo models represent very different multi-modality learning objective functions. As we draw conclusions from these different models, our results are more likely to generalize to other learning algorithms. For instance, the architecture proposed by Kiros et al. in 2014 utilized a structure-content neural language model module. This approach, similar to Flamingo, also implemented multi-modal interaction through a cross-modality-modulation technique. While we cannot test every possible architecture in the literature, we might expect similar findings based on structural similarity to Flamingo.

Finally, we thank the reviewer for pointing out the typo (LM instead of LLM ) in our paper. We have fixed it. We also appreciate the reviewer’s suggestion about the line style, which we will actively think about to have a better solution.