Improved Probabilistic Image-Text Representations

We thank the reviewer for their positive feedback and constructive comments. We will address all the raised concerns by the reviewer, and will revise our paper before next Tuesday.

[W1] PCME++ performance

Thanks for the question. First, we would like to emphasize that PCME++ is not inferior to VSE infty in all ranking-based metrics, such as mAP@R, R-Precision, and RSUM. It is because VSE infty is based on triplet loss using the hardest negative mining that is overfitted to recall@1 score as supported by Chun et al. 2022. Moreover, when the backbone size becomes larger, VSE infty shows inferior performances than PCME++ with a large margin in all metrics.

We presume that it is because when the backbone size is sufficiently complex to be overfitted to the false negatives, deterministic methods easily fall to the overfitted solutions. We remark that the training dataset suffers from severe false negatives (88.2% of caption-to-image positives and 72.1% of image-to-caption positives are labeled as “negative” [Chun et al 2022]). On the other hand, the probabilistic methods are designed to handle many-to-many relationships, by less penalizing plausible matches (e.g., if an image-text pair is annotated as negative, but if the model predicts they are positive with high probability, the loss value becomes smoothed by its design – As discussed in Sec 2.3). It means that the effect of the FNs is saturated during training PCME++; hence it is profitable to be trained with large backbones.

Second, both P2RM and DAA are basically an additional regularization to the existing loss function (P2RM employs less panelized additional triplet loss, and DAA employs CIDER-based extra information), usually triplet loss with the hardest negative mining. Furthermore, when we trained the models, we found that their methods are very sensitive to the hyperparameter selection, as shown in below:

Perhaps, if we more carefully tune the hyperparameters for the methods, we would get slightly better performances than the reported numbers. However, when we tried to apply DAA and P2RM, we frequently faced training instability by choosing larger hyperparameters than our choice. Note that we use the official implementation of DAA for the comparison, and P2RM is our own implementation because there is no official code.

Finally, the backbones used for our experiments differ from those of DAA, P2RM, and even PCME and VSE infty. Our implementation is a new and very strong setting based on CLIP backbones, showing the state-of-the-art retrieval performances against the existing visual semantic embeddings (VSEs) as shown in Tab C.1.

[W1] Applying PP and MSDA to comparison methods

We would like to emphasize that applying PP and MSDA to the triplet-based method is non-trivial. This is because the triplet loss is not designed for taking smooth labels. When we construct a triplet, we select an anchor sample (e.g., a visual embedding), its positive sample (e.g., its corresponding textual embedding), and its negative sample (e.g., one of the other textual embeddings). Here, for selecting the negative sample, triplet methods use the hardest negative mining, which uses the closest textual embedding as the negative. We argue that as there are abundant false negatives, it can harm the performances. The hardest negative mining is, therefore, very sensitive to the choice of batch size, as shown by VSE++ (https://github.com/fartashf/vsepp/issues/27).

If we use a smooth label for a given pair, it is impossible to construct a triplet of <anchor, positive and negative>. For example, assume we set a match annotation of <image_a, caption_b> to 0.6 from 0.0. How can we build triplets using this annotation? Moreover, if we introduce mixed samples and mixed labels, the problem becomes more complex. How can we handle <image_a_mixed_b, caption_a> and <image_b_mixed_a, caption_b> using a triplet relationship? Therefore, it is impossible to apply PP and MSDA for triplet-based methods.

One of our technical contributions is allowing smooth labels and mixed samples by designing a pairwise loss that is not affected by the other data samples. As shown in Fig 2., each loss computation of PCME++ is independent on the other pairs, while triplet loss or batch-wise contrastive loss is dependent on the relationships of other pairs.

Dear Reviewer 36mP,

We sincerely appreciate your efforts and time for the community. As we approach the close of the author-reviewer discussion period in 20 hours, we wonder whether the reviewer is satisfied with our response. It will be pleasurable if the reviewer can give us the reviewer's thoughts on the current revision to give us an extra valuable chance to improve our paper. We summarized our revision in the "Revision summary" comment and "Revision has been uploaded", where the latter is specially personalized to the reviewer's concerns.

Again, we thank the reviewer's valuable commitment and their help to strengthen our submission. We will address all the raised concerns by reviewers if there remain any.

We thank the reviewer for their positive evaluation and valuable comments. We will address all the raised concerns by the reviewer, and will revise our paper before next Tuesday.

[W1, Q1] Lack of discussions of the backbone scale-up

Thanks for your comment. The discussion related to the question is in Sec 3.2. Main results. “We conjecture that it is because the non-probabilistic counterparts are easily overfitted toward FNs when the model has more complexity, particularly with the hardest negative mining strategy.” Here, we provide more details beyond this explanation.

As shown in Fig 1, the training dataset already suffers from severe false negatives (88.2% of caption-to-image positives and 72.1% of image-to-caption positives are labeled as “negative” [Chun et al 2022]). As the backbone becomes more complex and larger, we presume that the backbone complexity is sufficiently large to capture the noisy FNs in the dataset. Moreover, VSE infty uses a triplet loss with the hardest negative mining, which will make the effect of FNs more significant. On the other hand, the probabilistic methods are designed to handle many-to-many relationships by less penalizing plausible matches (e.g., if an image-text pair is annotated as negative, but if the model predicts they are positive with high probability, the loss value becomes smoothed by its design – As discussed in Sec 2.3). It means that the effect of the FNs is saturated during training PCME and PCME++. However, as we discussed in Sec 2.3., at the same time, it can cause a fast loss saturation for PCME. We address the problem by proposing two techniques: pseudo-positives and mixed data sample augmentation. Note that it is non-trivial to employ the techniques to triplet-based metric learning methods (VSE infty) and batch-wise contrastive learning methods (InfoNCE).

Dear Reviewer RNqC,

We deeply thank you for your dedication to the community. As we mentioned before, the positive feedback from the reviewer highly encouraged us. We again sincerely appreciate your valuable comments. As the author-reviewer discussion period is closed in 20 hours, we wonder whether the revised paper is still satisfactory to the reviewer. It will be pleasurable if the reviewer can give us the reviewer's thoughts on the current revision to give us an extra valuable chance to improve our paper. We summarized our revision in the "Revision summary" comment and "Revision has been uploaded", where the latter is specially personalized to the reviewer's concerns.

Again, we thank the reviewer's valuable commitment and their help to strengthen our submission. We will address all the raised concerns by reviewers if there remain any.

We thank the reviewer for their positive feedback and constructive comments. We will address all the raised concerns by the reviewer and will revise our paper before next Tuesday.

[W1] Inspiration and extensibility for other work

We would like to emphasize that our work is an independent image-text matching work, not depending on specific methods. PCME++ is not strongly coupled with PCME. We will discuss it in more detail in the next bullet (Similarity between PCME loss function and PCME++ loss function). We presume that the reason why the reviewer feels that PCME++ limits the inspiration and extensibility for other work is that other approaches are usually based on deterministic methods. The technical details for learning a probabilistic method are somewhat difficult to be applied to deterministic methods. However, our work brings attention to the importance of the false negatives in the Image-Text matching training dataset (as pointed out by Reviewer RNqC). We explain that the FNs can be problematic for deterministic methods, especially for larger backbones, such as ViT-L/14. We do not believe that a probabilistic approach is the only method to tackle FNs. We believe that our work can give inspiration to other ITM methods to tackle the FN problem raised in our work.

[W1, W3] Similarity between PCME loss function and PCME++ loss function

We would like to emphasize that the similarity between PCME and PCME++ is due to the nature of the probabilistic embeddings (probemb). The main goal of probemb is to map an input to a random variable (r.v.) rather than a deterministic vector. We would like to emphasize that the core ideas of other probembs are also similar to each other, for example, Oh et al. 2019, Sun et al., 2020, Shi and Jain, 2019, Chang et al., 2020, SilnovaPark et al., 2022a et al., 2020 and Neculai et al., 2022. PCME (Chun et al. 2021) and our method, therefore share the property of probemb: we train an encoder that maps an input to a mean vector and a variance vector and train the encoder by maximizing the negative loglikelihood (NLL) using the extracted distributions.

PCME optimizes NLL by defining “matching probability”: $p(m | x_v, x_t) = \mathbb E_{Z_v, Z_t} \text{sigmoid} (-a \|\| Z_v - Z_t \|\| + b) \approx \frac{1}{J^2} \sum_j^J \sum_{j^\prime}^J \text{sigmoid} ( - a \|\| z_v - z_t \|\| + b)$

On the other hand, PCME++ optimizes the NLL using binary cross-entropy loss: $p(m | x_v, x_t) = \text{sigmoid} (-a \mathbb E_{Z_v, Z_t} \|\|Z_v - Z_t\|\|^2 + b)$

Although they look somewhat similar, PCME takes the expectation over the sigmoid of the pairwise distance of r.v.s, while PCME++ takes the expectation over the pairwise distance of r.v.s before taking the sigmoid. The latter has two benefits over the former: (1) our form has a closed-form solution, as shown in Eq (1), while PCME cannot. (2) our form can be naturally adopted into the binary cross entropy loss function, which is known to be stable and perform well in large-scale training [1].

Therefore, we argue that PCME and PCME++ use different loss functions in terms of the NLL formulation and the probabilistic distance (PCME uses the Monte-Carlo approximation of “matching probability”, and PCME++ uses the closed-form sampled distance, CSD).

[1] ResNet strikes back: An improved training procedure in timm

[W3] Augmentation methods are already proposed in previous works

Although we do not propose a new MSDA, our contribution lies in applying MSDA to the relational datasets. For example, applying MSDA to classification is straightforward because the mixed sample does not affect the other samples in the mini-batch. However, in the relational training objectives, such as triplet loss or contrastive loss, a mixed sample affects the other samples in the batch as well. Especially, the triplet loss is impossible to handle MSDA, because the core concept of MSDA is the smooth label but the triplet loss cannot handle smooth label, because it has to construct a triplet of the selected sample, the positive sample, and the negative sample. It is non-trivial to define positive and negative samples when the label is smoothed (Fig 2a). Similarly, a batch-wise contrastive loss, such as InfoNCE, is also a little bit tricky to control the effect of smooth labels (Fig 2b) because the mixed samples are combined in the denominator term of the InfoNCE softmax. On the other hand, our pairwise contrastive loss can directly apply smooth labels because each loss computation is invariant to the other samples, but only the given pairs affect the loss computation (Fig 2d).

Overall, we do not argue that we newly propose MSDA (CutMix and Mixup), but we argue that we first apply MSDA to image-text matching methods that are based on relational datasets, not instance-level datasets (e.g., image-only datasets).