Bridging Vision and Language Spaces with Assignment Prediction

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

[W1, W2, Q1] Ablation for the loss function

We present the ablation analysis corresponding to the learning objectives in the general response. Please refer to them.

[W3, Q3] Difference between VLAP and the previous linear transformation-based approaches

Thanks for the reference. Similar to the image captioning objective of LiMBeR, LLaVa learns the linear projection with the language generation objective, which aims to provide target answers from given transformed visual and language instruction representations. Although LiMBeR and LLaVa have shown two unimodal models can be bridged through the language modeling objective, they are insufficient to reduce the modality gap, as shown in our previous response. Our proposed objective not only improves overall performance in the linear transformation-based methods, but also significantly facilitates the connection of the visual encoder even if it is not trained with language guidance (i.e., CLIP encoder). In addition, our objective can be an alternative to the conventional objectives for cross-modal alignment, such as the image-text matching and image-text contrastive objectives. We add the missing reference LLaVa in the main paper.

[W4, Q2] Linear layer with assignment prediction

We apologize for using the misleading term, "distance-preserving nature of the linear layer". We revise the Introduction section to clarify the contribution.

[W5] Model reference

Thanks for your suggestion. We clarify the model names with citations in the table captions.

We thank you for your time and effort in reviewing our paper. Your insights have been highly valued, and your feedback is crucial to our progress. We understand that you have a busy schedule, and we again appreciate your time and attention to this matter. We have addressed all of your comments in our rebuttal. We would be grateful if you could please take a look at our rebuttal. If there are any additional materials or information you may require to facilitate the review process, please let us know. Your prompt attention to this matter is greatly appreciated.

Thank you again for your help. Sincerely Authors

Dear Reviewer zAaX,

Thanks for your constructive and valuable comments on our paper. We have revised our paper to address the reviewer's concerns.

• [W1, W2, Q1] Ablation regarding optimal transport. We added ablation analyses for the components of VLAP, including the hyperparameters and optimal transport in Appendix Section D.

• [W3, Q3, W4] Difference between VLAP and the previous linear transformation-based approaches. We included the missing reference (i.e., LLaVA). In addition, we extensively revised the paper to cover the reviewer's concerns about the novelty and to remove potentially misleading terms in Introduction.

• [W4, Q2] Visualization. We presented the UMAP visualization corresponding to visual arithmetic and ablation for the learning objectives. We can verify the effectiveness of the proposed VLAP in the feature space.

• [W5] Citation. We added references along with the model names to the table captions.

With the discussion deadline approaching in a few hours, we are eager to receive your feedback.

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

[W1] Ablation for assignment prediction

We present the ablation results corresponding to the balancing parameters in equation (7) in the general response. Please refer to that section for more details. The result without the assignment prediction objective corresponds to $\lambda\_\text{map}:\lambda\_\text{cap}=0:1$. To briefly sum up, the assignment prediction objective combined with $\mathcal{L}\_\text{cap}$ improves the performance.

[W2] VLAP with other cross-modal alignment

Thanks for the constructive suggestion. We present the results with the additional objectives, including the image-text matching (ITM) and image-text contrastive (ITC) objectives, in the general response. Briefly, those two objectives show less performance improvement than our proposed assignment prediction.

[W3] Scale of the pretraining dataset

Thank you for your valuable suggestion. As previous works for pretraining VLMs have proven, a larger pretraining dataset should bring better results on zero-shot vision-language tasks. To verify the impact of the scale of the training dataset on zero-shot performance, we will train VLAP with a large-scale dataset (e.g. CC12M) and will include results in the revised paper. However, we would like to emphasize that our method mainly focuses on presenting an efficient methodology (i.e., linear transformation, using pretrained encoders, small-scale training) to make VLMs.

[W4] VLAP with other backbone

While we used the transformer-based vision models (i.e., CLIP ViT-B/32 and BEiT) and the relatively small scale of LLMs (i.e., OPT$\_\text{1.3B}$ and T5$\_\text{Base}$) in the main paper, VLAP can be applied to all publicly available vision models and LLMs. Following MAGMA and LiMBeR, we additionally evaluate VLAP with the CNNs-based vision model (i.e., CLIP RN50x16) and the larger LLM (i.e., GPT-J, which has 6B parameters) for zero-shot image captioning on NoCaps and MSCOCO. Not surprisingly, VLAP with the larger LLM achieves better performance than with the smaller LLMs, significantly outperforming MAGMA and LiMBeR by large margins across all metrics with the same backbones. We add the results to Table. 9 in Appendix.

[Q1] Consistency of the word embedding in LLMs

While word embedding of any LLMs can be used, we exploit the word embedding within the LLM used for language encoding for two main reasons: (1) LLMs use word embeddings as a basis to produce text representations, indicating text representations already have high correlations with word embeddings. Therefore, the consistency of LLM for word embedding and language encoding makes the assignment for language data confident. (2) The inconsistency of LLMs for word embedding and language encoding requires unnecessary memory (e.g. OPT$\_\text{1.3B}$ has about 100M parameters for the word embedding).

We thank you for your time and effort in reviewing our paper. Your insights have been highly valued, and your feedback is crucial to our progress. We understand that you have a busy schedule, and we again appreciate your time and attention to this matter. We have addressed all of your comments in our rebuttal. We would be grateful if you could please take a look at our rebuttal. If there are any additional materials or information you may require to facilitate the review process, please let us know. Your prompt attention to this matter is greatly appreciated.

Thank you again for your help. Sincerely Authors

Dear Reviewer oDtB,

Thanks for your constructive and valuable comments on our paper. We have revised our paper to address the reviewer's concerns.

• [W1, W2] Ablation regarding objectives. We added ablation analyses for the learning objectives, including the hyperparameters and additional objectives in Appendix Section D.

• [W4] VLAP with other backbone. We added the results of VLAP with CLIP RN50x16 and GPT-J, which are the same backbone as the baselinse, MAGMA and LiMBeR in Appendix Section D.

With the discussion deadline approaching in a few hours, we are eager to receive your feedback.

I have read the author's response and other reviewer's reviews. I appreciate the author's efforts and devotion to solving our concerns. Most of my concerns are solved. However...

I still think it's necessary to experiment with the larger dataset to see how this method performs with the increase in data volume. Many methods, especially for efficient training, don't show performance improvement with larger data scales. These works have merit and contribution in terms of efficient training and fast learning in small/modest data, but it's also important to demonstrate the relationship between performance gain and data scale. Not generalizing to a larger data scale is not a weakness, but we have to let the audience know in which cases/scales our method shines and fails.

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

[W1] VLAP with non-linear space

We would like to emphasize that our method diverges from traditional objectives in linear transformation-based approaches by introducing an alternative objective that avoids direct representation comparisons. In addition, although our method dramatically boosted the performance of VLMs based on linear transformation, but nothing is specifically designed for this purpose only. As shown in the general response, our assignment prediction objective outperforms other objectives used in the modular-based methods. The results suggest that the proposed assignment prediction will contribute to the modular-based methods. To address the reviewer's suggestion, we are conducting additional experiments for the non-linear space, such as multiple linear layers with activation functions and an adapter, which is a trainable bottleneck module [1]. We will include the analysis in the revised paper.

[1] J. He et al., ``Towards a unified view of parameter-efficient transfer learning," ICLR'22

[Q1] Ablation for the balancing parameters

Thanks for the question. We present the ablation results corresponding to the balancing parameters in equation (7) in the general response. Please refer to them.

[W1] VLAP with non-linear space

To address the reviewer's suggestion, we conduct additional experiments to evaluate VLAP with the simplest non-linear mapping layers, i.e., three linear layers with the ReLU activation function, as shown in the above table. Note that CLIP ViT-B/32 and T5$\_\text{Base}$ are used as the vision model and LLM, respectively. The comparison between (i) and (iii) shows that VLAP with the original setting outperforms VLAP with the linear layers, demonstrating that the learning objective has a greater impact on overall performance than the non-linearity. Meanwhile, the comparison between (ii) and (iii) shows that the performance can be slightly improved with the non-linear layers when the learning objective is fixed. Although we only presented results with the simplest form of non-linear modules, the results suggest that VLAP will show further improved performance when combined with sophisticated modular-based approaches (e.g. Q-Former in BLIP-2).

[W1] VLAP with non-linear space (update)

We update the results of VLAP with the adapter. We use a single adapter, which has the dimension of the bottleneck 256 and the ReLU activation between down-/up-projection layers. Similar to the comparison between (ii) and (iv), the comparison between (iii) and (iv) shows that the non-linearity improves the performance despite the smaller number of parameters (0.4M vs 0.6M). This result consistently suggests that VLAP will show further improved performance when combined with sophisticated modular-based approaches (e.g. Q-Former in BLIP-2).

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

[W1] Validation for the modality gap reduction

To provide a better understanding in terms of modality gap reduction, we measure the modality gap distance $||\Delta||$ according to the learning objective. The modality gap $\Delta$ is defined as the difference between the center of visual and text representations [1]:

$$

    \Delta = \frac{1}{n}\sum\_{i=1}^{n} \bar{\mathbf{z}}^{v}\_i - \frac{1}{n}\sum\_{i=1}^n \bar{\mathbf{z}}^t\_i,

$$

where $n$ is the total number of image-text pairs, $\bar{\mathbf{z}}\_i^v$, and $\bar{\mathbf{z}}\_i^t$ are the $i$-th averaged visual and text representations. The default gap distance between CLIP ViT-B/32 and T5$\_\text{Base}$ before training is $||\Delta|| = 1.378$. The image captioning objective $\mathcal{L}\_\text{cap}$ reduces the modality gap distance to $||\Delta|| = 0.972$ and the additional objectives $\mathcal{L}\_\text{ITM}$ and $\mathcal{L}\_\text{ITC}$ further reduce the gap distance to $||\Delta|| = 0.870$. The comparison between VLAP with all objectives and with the original objective (i.e., $\mathcal{L}\_\text{cap}+\mathcal{L}\_\text{map}$) shows the ITM and ITC objectives have a limitation to reduce the modality gap. In addition, since the overall performance is inversely proportional to the modality gap, reducing the gap is an important factor in training VLMs. We also present UMAP visualization [2] and the additional analysis according to the learning objective in Figure 8 and Appendix. D.2 of the revised paper.

[1] W. Liang et al., ``Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning," NeurIPS'22
[2] T. Sainburg et al., ``Parametric UMAP embeddings for representation and semi-supervised learning," Neural Computation'21

[W2] Additional analysis for visual arithmetic

To prove the visual representation that inherited the semantic taxonomy of LLM, we analyze that word analogies can often be solved with visual vector arithmetic. In addition to existing experiments on visual semantic arithmetic, we supplement an experiment on the visual relation (VR) benchmark [3]. The VR benchmark consists of a total of 320 relations, including building$\rightarrow$country, country$\rightarrow$capital, CEO$\rightarrow$company, food$\rightarrow$country, and leader$\rightarrow$country. Since the relation of leader$\rightarrow$country includes out-of-date information (e.g. 'Obama'-'USA'), we exclude it from this experiment. In the above table, we compare the performance between VLAP, ClipCap [4], and ZeroCap [3] in terms of BLEU-1, Recall@5, and CLIP-Score, following [3]. For fair comparisons, we identically use CLIP ViT-B/32 as the image encoder and GPT-2 as the LLM. The results show that VLAP outperforms the previous methods by large margins. In particular, we attain correlations of over 75% in all relations on the semantic distance-based metric, i.e., CLIP-Score. We include the additional analysis with visualization in Figure 7 and Appendix. C.2 of the revised paper.

[3] Y. Tewel et al., ``ZeroCap: Zero-shot image-to-text generation for visual-semantic arithmetic," CVPR'22
[4] R. Mokady et al., ``ClipCap: Clip prefix for image captioning," arXiv'21

[W3] Ablation for the balancing parameters

We present the ablation results corresponding to the balancing parameters in equation (7) in the general response. Please refer to them.

[W4] Meaning of the probability in equation (5)

We first explain the probability $\mathbf{P}\_{nk}^{t}$ for the text representation. Since LLMs provide text representations through sophisticated operations between word embeddings, text representations and word embeddings have high correlations with each other. Therefore, we can obtain the probability of the $k$-th word attending to the $n$-th text representation without any correspondence between language captions and word embeddings as in equation (5). Meanwhile, our assignment prediction objective enforces visual and text representations to contain the same information. Simply put, the probability $\mathbf{P}\_{nk}^{v}$, which represents the $k$-th word attending to the $n$-th visual representation, is learned to be similar to a pre-calculated probability $\mathbf{P}\_{nk}^t$ by predicting the assignment of text data without any image-word correspondence.

[W5] Clarity

We denote the trace matrix as $\text{Tr}(\cdot)$ in equation (2) and the input text prompt (i.e., "$`A photo of`$") as $`[prefix]`$ in equation (6). We clarify them in the revised paper.

We thank you for your time and effort in reviewing our paper. Your insights have been highly valued, and your feedback is crucial to our progress. We understand that you have a busy schedule, and we again appreciate your time and attention to this matter. We have addressed all of your comments in our rebuttal. We would be grateful if you could please take a look at our rebuttal. If there are any additional materials or information you may require to facilitate the review process, please let us know. Your prompt attention to this matter is greatly appreciated.

Thank you again for your help. Sincerely Authors

Dear Reviewer dYxb,

Thanks for your constructive and valuable comments on our paper. We have revised our paper to address the reviewer's concerns.

• [W1, W3] Ablation regarding objectives. We added ablation analyses for the learning objectives, including the hyperparameters and additional objectives in Appendix. Section D.

• [W2] Additional analysis for visual arithmetic. We added the additional analysis for visual arithmetic on the visual relation benchmark, which demonstrates the learned visual representations can be used to reason semantic relations. We also provided the UMAP visualization to verify how the visual arithmetic of VLAP works in the feature space in Appendix. Section C.

• [W5] Clarity of notations. We clarified some notations for readability.

With the discussion deadline approaching in a few hours, we are eager to receive your feedback.

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

[W1, W2] Novelty and ablation for assignment prediction

Thanks for your valuable comments. As the reviewer mentioned, our VLAP differs from the previous optimal transport-based (or clustering-based) approaches [1, 2, 3, 4] in the following aspects:

(1) defining a fixed central space instead of a learnable central space (i.e., prototypes) and (2) defining the transport polytope with the word distribution of the training dataset instead of the equipartition assumption.

The previous works aim to learn visual representations [1, 2] or multimodal representations [3, 4] by training the whole networks, including backbone networks and the central space, in an end-to-end manner. However, we need to connect two frozen backbone networks using visual features as LLMs input. In our setup, we empirically found that learnable space is heavily sensitive to the hyperparameters (e.g. $\epsilon$ in equation (4)) and provides poor performance. Therefore, the conventional optimal transport-based approaches [1, 2, 3, 4] have been limited to be directly applied to our work.

To verify our claim, we conduct additional experiments for the components in assignment prediction. As shown in the above table, we evaluate each model for the zero-shot image captioning performance (CIDEr-D) on MSCOCO corresponding to various $\epsilon$ values in equation (4). In (i) "VLAP w/o word embed.'', we first train VLAP with learnable prototypes as in [1, 2, 3, 4]. In this setting, we define 3K prototypes following [1] and apply the equipartition assumption in equation (3). In (ii) "VLAP w/o word dist.'', we also verify the effectiveness of the word distribution. In this setting, we use the word embeddings as a fixed central space and perform the optimal transport with the equipartition assumption. The comparison between (i) and (ii) shows that the word embedding achieves substantial performance improvement, demonstrating the effectiveness of the word embedding. The comparison between (ii) and (iii) original VLAP shows that the transportation polytope with the word distribution achieves additional performance improvement and provides more robust performance regarding $\epsilon$.

[1] M. Caron et al., ``Unsupervised learning of visual features by contrasting cluster assignments," NeurIPS'20
[2] Y. M. Asano et al., ``Self-labeling via simultaneous clustering and representation learning," ICLR'20
[3] Y. M. Asano et al., ``Labelling unlabelled videos from scratch with multi-modal self-supervision," NeurIPS'20
[4] J. Duan et al., ``Multi-modal alignment using representation codebook," CVPR'22

[W3] Ablation for the loss function

Our image captioning objective is basically defined as the next-word generation prediction, i.e., generating a word from the previous information. Instead, we provide the results corresponding to the additional objectives for cross-modal alignment in the general responses. Please refer to them.

We thank you for your time and effort in reviewing our paper. Your insights have been highly valued, and your feedback is crucial to our progress. We understand that you have a busy schedule, and we again appreciate your time and attention to this matter. We have addressed all of your comments in our rebuttal. We would be grateful if you could please take a look at our rebuttal. If there are any additional materials or information you may require to facilitate the review process, please let us know. Your prompt attention to this matter is greatly appreciated.

Thank you again for your help. Sincerely Authors

Dear Reviewer 5A8P,

Thanks for your constructive and valuable comments on our paper. We have revised our paper to address the reviewer's concerns.

• [W1, W2] Ablation regarding optimal transport. We added ablation analyses for the components of VLAP, including the hyperparameters and optimal transport in Appendix Section D.

• [W3] Ablation regarding learning objectives. We added ablation analyses with the additional learning objectives in Appendix Section D.

With the discussion deadline approaching in a few hours, we are eager to receive your feedback.

VLAP with other cross-modal alignment objectives

We provide the ablation analysis with the image-text matching (ITM) and image-text contrastive (ITC) objectives used in [3, 4, 5]. We modify the ITC and ITM objectives to evaluate the performance of VLAP trained with these objectives. For ITC, we contrast the similarity between averaged image and text representations of a positive pair against those of negative pairs in a batch. The ITC objective $\mathcal{L}\_\text{ITC}$ is defined as the cross-entropy loss between pairwise similarities and the groundtruth one-hot similarities, where the positive pair has 1 and the negative pairs have 0. For ITM, we learn an additional linear layer as a binary classifier to determine whether an image-text pair is matched or not. We concatenate the averaged image and text representations and feed them into the classifier with the softmax activation. Following [3, 4, 5], we also employ the hard negative sampling strategy based on the pairwise similarity. As shown in the above table, the ITM and ITC objectives contribute to performance improvement with $\mathcal{L}\_\text{cap}$. However, we notice that the ITM and ITC objectives hurt overall performance, as shown in the performance comparison between the last two rows in the table. We carefully argue that ITM and ITC solely focus on direct pairwise comparisons without considering the semantic correlation between predefined two embedding spaces. Namely, they enforce instance discrimination and encourage the existence of the modality gap. By comparing the cross-modal intermediate assignments, we can relax the instance discrimination task, effectively reducing the modality gap. We add those results in Table. 7(b) and discussion in Appendix. D.

[3] J. Li et al., ``BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," ICML'23
[4] J. Li et al., ``Align before fuse: Vision and language representation learning with momentum distillation," NeurIPS'21
[5] J. Li et al., ``BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation," arXiv'22

Next, we carefully address each reviewer's remaining concerns.