We would like to express our sincere gratitude to Reviewer uHwb for your valuable time and effort in reviewing our work. In our General Response to All Reviewers, we have thoroughly discussed the effectiveness of Task-Enhanced Pretraining (TEP) in unifying heterogeneous tasks, particularly in the context of few-shot transfer learning and zero-shot inference scenarios. In this response, we would like to delve deeper into addressing the additional weaknesses that have been identified.

Q1: Motivation, novelty, and performance improvement issues.

A1: Thanks for your comments. Here we hope to clarify several points that may be misunderstandings:

1. Joint training on multiple heterogeneous vision-language tasks while preserving single-task performance is quite challenging.

Unlike NLP tasks which are homogeneous and suitable for large-scale concurrent training, the computer vision tasks are generally heterogeneous, (e.g., object detection outputs a set of bounding boxes, which is fairly different from visual grounding where both a caption and the foreground object are described). Unified LLMs like GPTs have achieved a tremendous success due to the nature of sequence-to-sequence property in languages, yet a unified foundational generalist model (not with multiple heavily designed specific task heads) to perform a large variety of computer vision tasks such as object detection, image segmentation, image generation, 3D estimation, pose recognition, etcs. is not yet available. Unifying heterogeneous tasks in single shared models may result in obvious performance drops (according to previous works such as Unified-IO and UniTAB) because of the task interference. When the number of heterogeneous tasks increases, the task interference will also become more significant. Therefore, single-task finetuning or task-specific adapters are necessary to achieve satisfying downstream task performance. For example, the OFA model which is pre-trained on eight heterogeneous can not generate usable results in many pretraining tasks (e.g., pretrained OFA_base achieves ~70 CIDEr score on image caption, v.s. ~140 CIDEr for single task fine-tuned model ) without single task finetuning.

2. TEP is not a simple “complex” prompt.

The main idea behind TEP is to use structural prompts to explicitly identify heterogeneous tasks in subprompt spaces. In fact, as shown in Table 7 of the paper, we have demonstrated that without structural form, even the more complex task description generated by ChatGPT with the richest information can not effectively mitigate the task interference problem.

3. The performance improvement of Musketeer and TEP is not marginal.

We’d like to clarify that the performance improvements are not marginal from several perspectives:

• First, as shown in Table 4 of the main text, Musketeer outperforms all the joint-trained models on heterogeneous tasks and even demonstrates competitive results to ~20 times larger models.
• In addition, as shown in Table 2 and Table 3 of the main text, TEP-powered Musketeer outperforms BaseP consistently for all tasks and demonstrates comparable and even better results to single-task finetuned SOTA methods. Please kindly note that fully eliminating the effects of task interference in heterogeneous tasks has proven to be challenging, as indicated by previous works such as Unified-IO and Pixel2SeqV2. However, to the best of our knowledge, Musketeer stands as the first endeavor to successfully address this challenge by unifying not only detection, visual grounding, and VQA tasks but also other tasks involving pure text or pure image inputs.
• TEP’s improvements can be more substantial in few-shot and zero-shot scenarios. As demonstrated in Appendix A.2 and General Response, TEP outperforms BaseP significantly in few-shot and zero-shot learning settings.

Q4: Can using full VQAv2 training data mitigate the problems in Table 4?

A4: Thanks for advice. Due to the limitation on our computing resources, in order to complete the training within the specified rebuttal time window, we increase the VQAv2 training data utilization only from 44% to 60%. The outcomes reveal a 0.4% enhancement in performance on the VQAv2 test set. Table 3 further illustrates the correlation between increased VQAv2 data usage and improved model performance. Based on these findings, we are confident that increasing the usage of VQAv2 data will mitigate the problem. Additionally, it's noteworthy that the Musketeer Large model, with only 472 M parameters, is significantly smaller than the parameter count of the Unified-IO_XLarge model, which stands at 2925 M.

Q5: Discuss the additional training cost including the pretrainining cost for the proposed methods in Table 3 and 4.

A5: Thanks for advice. We have provided pretraining costs comparison of Musketeer and other jointly-trained models without task-specific fine-tuning in the Table 3 below. We’ll update the Table 3 and Table 4 considering pretraining cost and add more corresponding discussion in our final version.

Table 3 Pretraining costs comparison of Musketeer and other jointly-trained models without task-specific fine-tuning. Pretrain-LM: Dataset size of Pretrained language models. #Pretrain-others: numbers of samples for other pretrainining tasks.

Thanks for your thoughtful review that will help us strengthen the manuscript. We have provided discussions of “how TEPs could transfer to unseen tasks” in General Response to All Reviewers. Below we discuss other identified weaknesses.

Q1: TEP still relies on pretrained weights for initialization which can be expensive, the discussion regarding the additional cost might be good to provide.

A1: Thanks for your comments. We agree that the usage of pretrained weights may involve extra cost for pretraining. We’ll add corresponding discussion in our final version. However, we hope to clarify that even without pretrained weights, TEP can still performs well on joint-training scenario for heterogeneous tasks. In Table 1 we have provided results of Musketeer trained from scratch (without pretrained weights). As shown, Musketeer trained with TEP significantly outperforms BaseP and demonstrate usable performance even without pre-training. In addition, TEP is flexible and utilizes other pretrained weights (e.g., VL-T5 and OFA) for better overall performance.

Table 1. Visual entailment performance of Musketeer trained with TEP and BaseP, both from scratch

Q2: The hyper-parameter setting as well as the Needs carefully designed TEPs for new tasks which may require some expertise.

A2: Thanks. For the hyper-parameter settings, we hope to highlight that unlike many previous tasks which carefully design hyper-parameters and training metrics for each downstream task (e.g., OFA), we use fully-shared training settings (e.g., learning rates, epochs, and dropout ratio) for each task to simplify the hyper-parameter tuning efforts. Besides, most hyper-parameter settings of TEP are following common practice of Transformer models of similar size (e.g., VL-T5). Therefore, we hope to clarify that the hyper-parameters settings of TEP are not carefully-designed and we believe its performance on each specific task can be further improved if we adopt extra careful hyper-parameter tuning. For the TEP design, since the data description subprompt are generated by ChatGPT and verified by humans, we agree that it may require some expertise. However, as shown in Table 2 below, the TEP without data description can already significantly outperform BaseP and the improvement by including data description is marginal, which means TEP can also be used without data description. Since other structural subprompts (instance prompt, I/O format and I/O description) are designed intuitively, we hope to point that TEP with no data description can also be used without much expertise.

Table 2. Ablations on specific TEP subprompts. VG: Visual Grounding. VE: Visual Entailment. B@4: BLEU@4. Y/N: with / without data description.

Q3: Some related works might also be good to include or discuss.

A3: Good advice. We’ll include corresponding discussions of the mentioned works in our future version.

Thank you for your valuable and supportive reviews. We hope the below response can address your concerns.

Q1: If more discussion and experiment on zero-shot new task generalization, it will be much better.

A1: We have provided an in-depth discussion and additional results of TEP and BaseP in both few-shot and zero-shot scenarios. Please see in General Response to All Reviewers.

Q2. Can Musketeer also be applied on text-only tasks? Is there any experimental evidence?

A2: Thanks for your comments. In fact, we have reported the Musketeer results on text-only task Gigawords in Table 3 of the main text. Additionally, in the zero-shot inference results in general response the Musketeer has also been demonstrated to be effective in other two unseen text-only datasets. We hope this can address your concern and We’ll add corresponding discussion in the future version.

Should you have any further inquiries, please let us know and we are more than delighted to discuss with you and run more experiments for any pieces of your interests in our work.

General Response to All Reviewers (Part 2/2)

Zero-shot inference on unseen datasets.

We further demonstrate that TEP can also enhance the model zero-shot performance on unseen datasets of seen task. Specifically, we evaluate the seven-task trained Museketeer models on unseen text summarization datasets tldr-news (https://huggingface.co/datasets/JulesBelveze/tldr_news) and news-summary (https://huggingface.co/datasets/argilla/news-summary), and the results are shown in Table 4. The structural TEP consistently demonstrates better zero-shot performance the BaseP on unseen datasets.

Table 4 Experiments of zero-shot learning on unseen datasets.

Limitation of Musketeer on zero-shot open-task generation.

Since Musketeer is trained on seven heterogeneous vision/language tasks, admittedly, Musketeer is not designed to aim at performing common open-task generalization to a form of totally unseen task such as image segmentation. However, according to above experiments, we hope to emphasize that TEP demonstrates significant improvements over BaseP in zero-shot inference on unseen tasks and datasets.

In addition, we hope to highlight our main contribution, which lies in addressing the challenging issue of task interference during the joint-training of heterogeneous vision-language tasks. This is achieved through the utilization of structural TEP, accompanied by a balanced sampling and optimization scheme. In previous experiments, we have successfully demonstrated the effectiveness of structural TEP in mitigating task interference. Please also kindly note that Musketeer is also the first joint-trained model that can achieve comparable or even better performance on heterogeneous tasks including detection, grounding, image classification and other textual tasks.