CoBIT: A Contrastive Bi-directional Image-Text Generation Model

W1: Theoretical Justification of Assumed Synergy.

Answer: We really appreciate your excellent suggestion. Based on the ablation in Sec. 4.4, we would like to provide further analysis of the mutual effects of three different losses:

• Finding: Cross-modal generation objectives can improve image understanding a bit on top of contrastive loss. Analysis: Either T2I objective or I2T objective focuses on cross-modal fine-grained semantics, which can explicitly facilitate the global alignment between two modalities.

• Finding: Two generation losses, i.e., I2T loss and T2I loss, contradict each other a little bit. Analysis: The good signal is that those two don’t contradict that much and still can keep a comparable performance when jointly training. It’s promising because joint training essentially saves half of the parameters compared with using an ensemble of two separate models. On the other hand, those two losses don’t show mutual benefit in performance as we expected. We think the reason is that the granularity of image generation and text generation are different. A single word like cat in the text may correspond to a region in the image that contains hundreds of pixels and tens of image tokens.

• Finding: Contrastive loss improves vision-language understanding while it doesn’t influence image generation. Analysis: A good global alignment enabled by contrast provides the base for better image-to-text generation. However, image generation requires precise local details beyond just correct semantics, which makes it hard to benefit from global alignment.

As you can see, the mutual effects are kind of mixed, but the parameter efficiency of joint training always holds, i.e., the number of parameters in the ensemble of three specialized models is 2-3 times that of a jointly trained model.

As for a more theoretical justification, we think existing works [1] about gradient harmony in multi-lingual model training might provide a new angle to our scenario. We two share the similarity in terms of multiple objectives, large data size, large model size, and intuitively mutual benefits of different objectives (vision is believed to be another language). Studying how the gradients of those three objectives interact should be more fundamental and theoretical, but also sophisticated. We would like to leave it as the next step.

[1] Wang, Zirui, et al. "Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models." arXiv preprint arXiv:2010.05874 (2020).

W2: Objective Conflicts Mitigation Strategies.

Answer: Thanks for your valuable suggestion. We do consider the design of the proposed Unicoder-Decoder an initial exploration of the conflict mitigation strategies. From the ablation result, when jointly training with different objectives, it can outperform conventional encoder-decoder models. Intuitively, it shares the knowledge between encoding a modality for cross-modality understanding and generating a modality from the other modality, which explicitly bridges both understanding loss and generation loss by sharing the parameter space. We think there is still plenty of space to explore and believe our findings can shed light on future works.

W3: Weakness and failure analysis.

Answer: Thank you for pointing out. In Sec. 4.5 and Sec. 6.6, we conducted some failure analysis about image generation. Several observations were presented and we quote them here. We also included the visualization examples for each type of failure in the manuscript.

• CoBIT sometimes messes up the size attributes of two objects.

• CoBIT sometimes couldn’t render the details of words in text very well.

• CoBIT occasionally misunderstands the text.

Here we further add a brief failure analysis of image-to-text generation in zeroshot scenario:

• CoBIT tends to generate short text, for example, one sentence or multiple phrases. This might be due to the training data distribution.

• CoBIT sometimes cannot follow instructions but instead tends to complete the input text. Additionally, instruction-tuning should help.

Based on the above points, we will provide a more detailed anaylsis to help future improvement in the camera-ready version.

Questions: Can you list the exact parameter sizes of each model in Table1,2,3? That should be more fair comparison. • Why are the tasks shown in ablation studies (Table 4,5,6) not consistent? What is the reason for lacking some tasks in each table? • Please check the weakness for more questions.

Answer: In Tab1, we already list the number of parameters of the models. As for models in Tab2, here are the exact parameter sizes of each model:

Here are the exact parameter sizes of each model in Tab3:

In ablations, in default, we use zeroshot ImageNet classification, VQA and zeroshot text-to-image generation as three representative tasks. But when comparing Unicoder-Decoder vs. Encoder-Decoder, finetuning on VQA task updates more parameters in Unicoder-Decoder than Encoder-Decoder, which is not fully fair. To further clarify, we added the zeroshot image captioning task as a fair evaluation for those two designs and have to delete zeroshot ImageNet classification to save space. The performance of zeroshot ImageNet classification is improved by 0.7% when comparing Unicoder-Decoder to Encoder-Decoder. We will add all the details to the camera-ready version.

W1: Insights on each objective’s advantages.

Answer: Thank you for your valuable suggestion. Here we provide a more detailed explanation of each objective’s effects from their results.

First of all, the parameter efficiency of joint training always holds, i.e., the number of parameters in the ensemble of three specialized models is 2-3 times that of a jointly trained model. Second, we don’t intend to make a point that those three objectives must benefit each other, but rather we want to serve as the first exploration about the relationship of those three objectives. In the following, we analyze the three findings we stated in Sec. 4.4 of original submission.

• Finding1: Cross-modal generation objectives can improve image understanding a bit on top of contrastive loss. Analysis: Either T2I objective or I2T objective focuses on cross-modal fine-grained semantics, which can implicitly facilitate the global alignment between two modalities.

• Finding2: Two generation losses, i.e., I2T loss and T2I loss, contradict each other a little bit. Analysis: The good sign is that those two don’t contradict that much and still can keep a comparable performance when jointly training. It’s promising because joint training essentially saves half of the parameters compared with using an ensemble of two separate models. On the other hand, those two losses don’t show mutual benefit in performance as we expected. We think the reason is that the granularity of image generation and text generation are different. A single word like cat in the text may correspond to a region in the image that contains hundreds of pixels and tens of image tokens.

• Finding3: Contrastive loss improves vision-language understanding while it doesn’t influence image generation. Analysis: A good global alignment enabled by contrast provides the base for better image-to-text generation. However, image generation requires precise local details beyond just correct semantics, which makes it hard to benefit from global alignment.

As you can see, the mutual effects are kind of mixed in terms of performance, but overall those three losses are compatible in the unified framework. We believe that there is plenty of space for better unifying these objectives. And our proposed Unicoder-Decoder provides an option for model design improvement.

W2: Comparison w/ other models.

Answer: Thank you for your feedback. Comparing with other methods in an absolutely fair environment is not realistic because many works use their own private datasets, such as CLIP, Florence, DALLE, etc, and different models have different model sizes, batch sizes, and training steps. Since jointly learning cross-modal contrastive optimization, text-to-image generation, and image-to-text generation is a new setting, ours serves as an initial exploration and mainly studies how those three objectives impact each other, and which model structure is more effective and efficient (Unicoder-Decoder vs. Encoder-Decoder). On those two core problems, all our experiments and ablations are conducted fairly with the same amount of data. So the findings of our paper should be valid and useful to the community.

W3: Inconsistent experimental results.

Answer: We apologize for the possible confusion and appreciate this opportunity to clarify the goal and contribution of our work.

• Joint training of three objectives First of all, joint training is more efficient than separate training in terms of the number of parameters, as each separate trained model has its own parameters and in total three separate models’ parameter is 2-3x that of a jointly trained model. Second, we don’t intend to make a point that those three objectives must benefit each other, but rather we want to serve as the first exploration about the relationship of those three objectives. And we summarized three observations, as stated in Sec. 4.4:

  • Cross-modal generation objectives can improve image understanding a bit on top of contrastive loss

  • Two generations losses, i.e., I2T loss and T2I loss, contradict each other a little bit

  • Contrastive loss improves vision-language understanding while it doesn’t influence image generation.

Overall, we demonstrate the feasibility of compatibly unifying three fundamental objectives in one framework without significant contradiction. We understand that there is still space for better unifying these objectives, and our proposed Unicoder-Decoder is an improvement from the model design.

• Unicoder-Decoder We arguably believe the improvement of Unicoder-Decoder over conventional Encoder-Decoder is stable and significant on average. On all three tasks, it’s all better and the average improvement is 6.7% relatively (first row vs. last row of Tab. 5).

Dear Reviewer GAkS,

As the reviewer-author discussion period is nearing its conclusion in under 7 hours, we look forward to your valuable insights and comments on our responses. Please feel free to share your thoughts with us. We sincerely thank you for your dedication and time in reviewing our submission.

Warmest regards,
Author(s)

W1: Justification on three joint losses.

Answer: Thank you for appreciating the contribution of our work. Our work starts with the motivation of unifying three fundamental objectives and studying their mutual effects, with the hope that those three could benefit each other. We conducted a comprehensive and rigorous ablation in Sec. 4.4, and observed three valuable findings:

• Finding: Cross-modal generation objectives can improve image understanding a bit on top of contrastive loss. Analysis: Either T2I objective or I2T objective focuses on cross-modal fine-grained semantics, which can explicitly facilitate the global alignment between two modalities.

• Finding: Two generation losses, i.e., I2T loss and T2I loss, contradict each other a little bit. Analysis: The good signal is that those two don’t contradict that much and still can keep a comparable performance when jointly training. It’s promising because joint training essentially saves half of the parameters compared with using an ensemble of two separate models. On the other hand, those two losses don’t show mutual benefit in performance as we expected. We think the reason is that the granularity of image generation and text generation are different. A single word like cat in the text may correspond to a region in the image that contains hundreds of pixels and tens of image tokens.

• Finding: Contrastive loss improves vision-language understanding while it doesn’t influence image generation. Analysis: A good global alignment enabled by contrast provides the base for better image-to-text generation. However, image generation requires precise local details beyond just correct semantics, which makes it hard to benefit from global alignment.

As you can see, the mutual effects are kind of mixed, but the parameter efficiency of joint training always holds, i.e., the number of parameters in the ensemble of three specialized models is 2-3 times that of a jointly trained model. We understand that there is plenty of space for better unifying these objectives. Our proposed Unicoder-Decoder is an improvement from the model design. Intuitively, it shares the knowledge between encoding a modality for cross-modality understanding and generating a modality from the other modality, which explicitly bridges both understanding loss and generation loss by sharing the parameter space. Empirically, on all three tasks, it’s all better than the conventional encoder-decoder structure and the average improvement is 6.7% relatively (first row vs. last row of Tab. 5).

W2: Smaller or more general datasets.

Answer: Thank you for your valuable feedback. We started with our design hoping to scale the model up to L and even XL sizes, but due to computation limitations and time constraints, we only experimented with Base and Large. From previous literature ([1][2]), larger pre-train models benefit from having larger general-purpose pretraining datasets. Therefore, we were only considering sizable datasets for our experiment purposes. Moreover, LAION datasets had concerns about potential biases embedded in the dataset itself, and hence we did not use it for our training. We will consider running experiments on the smaller datasets with our models in future versions.

[1]. Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).
[2]. Chen, Xi, et al. "Pali: A jointly-scaled multilingual language-image model." arXiv preprint arXiv:2209.06794 (2022).

W1: Computational Efficiency.

Answer: Thank you for your suggestion. Yes, in training, Unicoder-Decoder and Encoder-Decoder models consume the same number of parameters; in the finetuning of text-to-image and image-to-text tasks, Unicoder-Decoder inherits more parameters than Encoder-Deocder. That is exactly the reason why we want to evaluate zeroshot captioning and zeroshot image generation in the ablation experiments (Tab. 5), which can eliminate the factor brought by updating additional parameters. In those two tasks, Unicoder-Deocoder still outperforms Encoder-Decoder, which validates its effectiveness. We will add this illustration in the camera-ready version of our paper.

W2: Impact of Pre-training Data.

Answer: Thank you for this valuable suggestion. Here we listed the ablation on the pre-training datasets. To make a fair comparison, the batch size is kept the same, the training is conducted for 200k steps on base model. As we can see, JFT is beneficial to classification tasks, focusing on basic and precise semantics of image; WebLI has higher-quality image data and is specifically beneficial to text-to-image generation; ALIGN is relatively noisy but covers broad semantics. That’s the reason why we mixed them for a more balanced training.

W3: Ablation Study.

Answer: We will add the ablation about the loss weights in the appendix and the new ablation about different types of pre-training datasets in the main paper in the camera-ready version.

W4: Number of parameters in base/large models.

Answer: In previous multimodal works, different models have different numbers of parameters for their base or large models. For example, Coca’s base model has 383M while GIT’s base model has 129M. The reason why our base/large models have a bit more parameters than the previous works’ base or large design is that we have three objectives covering a large number of downstream tasks. Nevertheless, in Tab.2 and Tab.3, we can see that, CoBIT, as a multi-task model, can outperform most of the specialized models within 1B parameters and can even outperform larger models.

Thank you for your responses to the reviewers' comments. Here are my thoughts on your responses:

1. On computational efficiency, I want to know a comprehensive analysis of the computational costs, such as the specific training and inference time, space cost, and the corresponding performance of these models and variants. This data would offer potential users a clear understanding of the trade-offs involved, enabling them to make more informed decisions about the model's utility and application.

2. On the impact of pre-training data, I also find it intriguing. From Tables 4 and 5 in your paper, it appears that mixing the datasets improved the ZS.T2I performance, while the performance on other metrics after mixing lies between the performances on individual datasets. If your paper is accepted, I would suggest including this analysis in the final version and comparing it with the performance using a mixed pre-training dataset.

3. I reiterate the opinion that the ablation study is weak and lacks innovative insights. Additionally, given your model's architecture, I am curious as to why you deliberately expanded the number of encoding layers in the decoder to 18, which is typically consistent with the number of layers in the encoder.

4. On the number of parameters in the base/large models, you mentioned UniLM in your rebuttal to other reviewers. However, as far as I know, UniLM does not introduce additional parameters due to multi-task pre-training.

I greatly appreciate the authors for their responses. I have raised the rating to 6.

W8:MSCOCO online test evaluation.

Answer: The reason we report MSCOCO captioning result in Karpathy-split test set is that it’s commonly used in previous works(CoCa, BEIT, PALI, SimVLM, OFA, BLIP1/2, etc) for a more comprehensive comparison. We fairly compare with previous works by making sure no test data leakage in training. The reason for not presenting in MSCOCO online test set is that only very few previous works reported their result on this set (in recent works about image-text foundation models, we only find BUTD(2017), VinVL(2021) and GIT(2022) have official results reported), which makes a comprehensive comparison difficult.

W1: Limited novelty of the model's contribution.

Answer: Thank you for your comments. We appreciate the opportunity to clarify the unique aspects and contributions of our work. To our best knowledge, our work is the first exploration to unify all three fundamental objectives jointly. We believe the model design needs to fit the optimization objectives. Although the idea of sharing weights for encoding and decoding has been developed by the previous NLP works (UniLM, etc), they aimed to learn a good and general text representation. Regarding our scenario of optimizing those three cross-modal losses, the proposed Unicoder-Decoder has its unique novelty in learning beneficial cross-modal knowledge for a joint understanding and generation of two modalities, as well as saving parameters. Compared with the conventional encoder-decoder structure, Unicode-Decoder also obtains better empirical results.

W2: The concern about data.

Answer: Thank you for your suggestion. Comparing with other methods under the same training data is not realistic because many works use their own private datasets, such as CLIP, Florence, DALLE, etc.

Since jointly learning cross-modal contrastive optimization, text-to-image generation, and image-to-text generation is a new setting, ours serves as an initial exploration of how those three losses impact each other, and which model structure is more effective and efficient. On those two core problems, all our experiments and ablations are conducted fairly with the same amount of data. So the findings of our paper should be valid and useful to the community.

W3: Complexity of hyperparameters in model design.

Answer: Thanks for pointing out. It’s common that multi-task learning and multi-objective optimization usually involve more hyperparameters due to multiple losses. We have listed all the hyperparameters in Sec. 4.1 and Tab. 7 of the original submission for a clear reference and also ablated the loss coefficients in Sec. 6.2 (Tab.8).

W4: Model Initialization.

Answer: It’s common in previous works that a pre-trained text encoder is used for image generation(DALLE, Parti, SD, etc). We expect that the performance of CLIP and COCA should be similar because they both encode text in a cross-modality way. Due to computation resources and limited time, we didn’t further explore using CLIP to replace Coca. But more essentially, we did ablate initializing from coca vs training from scratch in Tab.6 and found the training from scratch can bring comparable performance.

W5: Frozen ViT-VQGAN.

Answer: Good point. Freezing VQGAN/VQVAE in image generation model is a common operation not only in auto-regressive methods (DALL-E, Parti), but also in diffusion models (SD). In Ernie-ViLG [1], they try to unfreeze the VQGAN, and in large-scale training, it causes instability in training and the model can’t converge.

[1] Zhang, Han, et al. "Ernie-vilg: Unified generative pre-training for bidirectional vision-language generation." arXiv preprint arXiv:2112.15283 (2021).

W6: Explanation about Linear Probing performance.

Answer: Linear probing can usually improve upon zeroshot in general not only in our experiment but also in previous works (CLIP, ALIGN, etc). We think the reason why linear probing can improve the performance over zero-shot performance is that it learns a single-layer adaptation to target categories from the pre-trained knowledge. As for finetuning more parameters, similar to previous works (CLIP, ALIGN, etc), we expect our model to gradually improve until saturated.

W7: RL-based captioning.

Answer: Thank you for your suggestion. SCST [2] is usually used to optimize the image captioning model with CIDEr metrics. When applying SCST to CoBIT, no specific model structure change is needed. To be more specific, CoBIT will conduct two feedforwards: one is sampling and the other is greedy inference. CIDEr score will be computed between each of the generated sentences and GT sentence. Then the REINFROCE algorithm is applied, which regards the greedy estimated sentence’s CIDEr score as the baseline and the sampled sentence as the prediction. Therefore, samples from the model that return a higher reward than the greedy estimated sentence will be encouraged, or increased in probability, while samples that result in a lower reward will be suppressed. For the sake of limited budget for experiments and the simplicity of our framework (since our CoBIT already has multiple objectives), we didn’t incorporate this in our experiment but we will consider adding it in the final version.

[2]. Rennie, Steven J., et al. "Self-critical sequence training for image captioning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.