Multitask Contrastive Learning

Rebuttal Response: We thank the reviewer for their careful review and feedback. We address the individual weaknesses raised below:

1. We agree with the reviewer that bespoke multi-task contrastive learning methods have been developed under specific domains. In comparison to the works cited by the reviewer (which focuses on domain specific tasks such as segmentation) and additional works in the hierarchical contrastive learning space (Use all the Labels, Zhang et al. 2022), we focus on the general multi-task classification space where multiple tasks can have disjoint labels. Specifically, none of the domain specific papers handle the case where two images, e.g., of shoes, can be labeled as positive examples under one task but negative examples under another. However, we believe that the papers the reviewer cited would add to the depth of the paper, and we will add them to the relevant works section.
2. We provide theoretical and empirical evidence that weighting uncertainty for source tasks can improve out-of-domain generalization and find that MT-Con outperforms the second-best weighted multi-task baseline by an average of over 1.5% across out-of-domain tasks (Table 1).
3. The point that the reviewer brings up is very interesting. Addressing conflicting similarity metrics is one of the core contributions of our work, setting it apart from the bespoke approaches that the reviewer highlighted. In Figure 1, we highlight what happens in the scenario that the reviewer raises, where the similarity metrics are contradicting. The construction of the network with multiple similarity heads enables MT-Con to handle conflicting similarity metrics. We will highlight this better in a revision.
4. We compare to the most recent general multi-similarity learning methods, including SCE-Net and CSN [used as upper baselines in 1, 2]. We also compare to the most popular weighted general multi-task cross-entropy method (Kendall et al. 2018). Could the reviewer please point us to any general multi-similarity (can handle conflicting metrics) methods they would like to see?
5. We tried to capture the multiple disjoint similarity metrics with Figure 1 as motivation. Could the reviewer please clarify a suggested title or pipeline that would help with general understanding?

[1] “How Good Is Aesthetic Ability of a Fashion Model?” (Zou et al. 2022) [2] “Identifying ambiguous similarity conditions via semantic matching” (Ye et al. 2022)

Rebuttal Response: We thank the reviewer for their careful review and feedback and appreciate the reviewer’s openness to further discussion. We address the individual weaknesses and questions raised below:

Weaknesses:

1. The idea behind MTCon weighting is to downweight tasks with more task uncertainty. As the reviewer correctly points out, this idea has been previously explored in the multi-task literature, but not in the contrastive setting. Existing techniques do not extend directly to the contrastive setting because unlike existing methods, contrastive loss doesn't imply a likelihood function over the observed data. Thus, we develop the pseudo-likelihood to demonstrate that the temperature corresponds to uncertainty weighting. We do find improvements to using contrastive learning in conjunction with the multi-task learning (similar to previous self-supervised and supervised contrastive learning work such as Khosla et al 2020). Across out-of-domain tasks, MT-Con outperforms uncertainty weighted multi-task cross-entropy by an average of over 1.5% across out-of-domain tasks (Table 1).
2. This is a great question.. As the uncertainty increases, the learned weight should converge to zero, but does not because of estimation error. We expect such estimation error to go to zero as the sample size goes to infinity. We like the idea of studying the behavior in a regime where we have no estimation error. When we manually set the task weight to 0 for the corrupted tasks, the generalization accuracies for the Zappos50k and MEDIC datasets are 40.5% and 80.8%, respectively. We will include these results in the Appendix.

Rebuttal Response: We thank the reviewer for their careful review and feedback. We appreciate that the reviewer recognized the theoretical and empirical contributions of MTCon. We address the individual weaknesses and questions raised below:

Weaknesses & Questions: We thank the reviewer for the thoughtful comments on the work. As the reviewer observes, our method performs comparably to others on MEDIC in-domain tasks, but MTCon does generalize better than baselines on out-of-domain tasks (Table 1), which is our main goal in this work. We will highlight this nuance more clearly in the final version. We direct the reviewer to Figure 3, the ablation study. We find that weighted MTCon learns weightings that are robust to individual task noise in order to generalize better to out-of-domain tasks even when in-domain performance remains similar.

Alam et al., Table 10, shows that adding the informativeness task to multi-task training actually decreases performance for the other tasks and will improve the main text correspondingly.

Rebuttal Response: We thank the reviewer for their careful review and feedback. We appreciate that the reviewer recognized the theoretical and empirical contributions of MTCon. We address the individual weaknesses and questions raised below:

Weaknesses:

1. The reviewer is correct in that our theoretical analysis assumes that the source tasks contain sufficient information to cover target tasks. We stress, however, that this assumption is in line with the majority of theoretical work in the multitask and supervised contrastive learning fields, including [1] and [2]. The experimental results are consistent with what the theory suggests.
2. It would indeed be an interesting future direction to apply our method to other modalities for which multiple similarity measures are common, but this work focuses on showing that the methods work for images on three multi-task vision datasets from very different domains. These datasets are a superset of those used in earlier contrastive multi-similarity representation work, including [3] and [4].
3. We provide an ablation study in Figure 3 that demonstrates that weighted MT-Con generalizes better than the unweighted version to out-of-domain tasks. We cannot ablate the multiple projection heads since MT-Con is dependent on it to capture disjoint similarities. Can the reviewer clarify if there are other specific ablations they believe would strengthen the paper?
4. We provide comparisons to self-supervised, single-task supervised, and the most recent multi-task multi-similarity supervised contrastive learning methods across all three evaluation datasets. We chose these baselines because they are commonly used in the literature [5,6,7]. We would be happy to include other baselines that the reviewer would like to see.
5. We provide hyperparameter analyses for temperature and number of epochs for both the Zappos50k and MEDIC datasets in Figures 6,7 in the Appendix. We can change the main text to summarize these.

Questions:

1. We thank the reviewer for the thoughtful question. We have not considered other metrics of uncertainty since we found that the pseudo-likelihood worked well for us in theory and in practice. We believe that this would be an interesting direction for future work.

[1] “The power of contrast for feature learning: A theoretical analysis” (Ji et al. 2021) [2] “Few-shot learning via learning the representation, provably” (Du et al. 2020) [3] “Conditional similarity networks” (Veit et al. 2017) [4] “SCE-Net” (Tan et al. 2019) [5] “How Good Is Aesthetic Ability of a Fashion Model?” (Zou et al. 2022) [6] “Identifying ambiguous similarity conditions via semantic matching” (Ye et al. 2022) [7] “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics” (Kendall et al. 2017)