Discovering Clone Negatives via Adaptive Contrastive Learning for Image-Text Matching

(4) W4: The 2024 baselines are missing.

In response to the reviewer's suggestion regarding the baselines, we have further verified AdaCL on two more recent state-of-the-art image-text baselines CORA [3] and USER [4]. We followed the same experimental settings in the manuscript, and their respective plug-and-play effectiveness is demonstrated as follows:

The results in parentheses, i.e., $(\cdot)$ represent results of the original baselines. We can see that most of the two baselines have been improved, further demonstrating the effectiveness of AdaCL in image-text matching. In the revised manuscript, we add the above results to Tab. 1 of Sec. 4, including an all-round set of influential baselines that incorporate diverse visual and textual backbones.

[3] Zhang et al., "USER: Unified semantic enhancement with momentum contrast for image-text retrieval." TIP 2024.

[4] Pham et al., "Composing object relations and attributes for image-text matching." CVPR 2024.

(5) W6: There are usually three mainstream benchmarks for text-based person search tasks. I suggest the author add the results of RSTPReid. Also, does the authors' R@ refer to the Recall accuracy? However, in the text-based person search task, the Rank accuracy is usually used as the evaluation indicator.

We appreciate the reviewer for your constructive suggestion. We have further validated AdaCL on RSTPReid. The following table demonstrate the Recall@1 of AdaCL. We choose CMPM as baseline for a fair comparison, because the learning objective it adopt in its paper is the closest to vanilla CL. "DG" stands for the domain generalization results (trained on Flickr30K and zero-shot on three text-based person search datasets). "FT" represents fine-tuning on the corresponding datasets. The result on RSTPReid is evaluated by us with CMPM's official repository.

It is observed that AdaCL demonstrates the robustness in both domain generalization and fine-tuning, especially for DG, the results of AdaCL on each dataset improves by over 10%.

For fine-tuning, in addition to CUHK-PEDES and ICFG-PEDES in the original manuscript, we also validate AdaCL on RSTPReid with three baselines. The results are demonstrated as follows:

We can see that our AdaCL is insensitive to diverse dataset distributions (both natural images and person search images). All the above results have been added to the appendix of the revised manuscript for a comprehensive comparison.

For evaluation metric, R@K refers to Recall@K. Specifically, given a query text, images are ranked based on their similarity to the query text. A search is considered correct if at least one relevant image appears within the top K positions in the ranking. To the best of our knowledge, this evaluation metric is widely used in text-based person search. Recall@K is indeed the evaluation metric we used in the experiments. In the revised manuscript, we have clarified the notation in the relevant sections to eliminate any potential ambiguity. We hope this addresses your concern and makes the evaluation metric clearer.

We appreciate your time and effort in reviewing our paper and providing valuable feedback to help us improve it. Below, we address your comments point by point:

(1) W1: The authors proposed a new concept, namely clone negative examples. However, this is actually the false-negative problem, which is not a new topic. For example, there has been similar research in [1,2].

Thank you for your feedback regarding the motivation. We would like to clarify that clone negatives differ fundamentally from false negatives. Let us try to analyze them in details:

• False negatives are true positives mislabeled due to annotation or sampling errors, and their impact on contrastive learning is well understood. They are regarded as "true matches" that are incorrectly categorized as negatives during the training process. The solutions to false negatives are usually improved sampling and elimination strategies, as highlighted in the papers mentioned by the reviewer.

• Clone negatives present a novel challenge: they are samples that closely resemble the ground-truth positive but miss the specific and fine-grained details of the true positive, which is defined as sub-optimal (true negatives). For example, in Fig. 1 of the manuscript, the general concept of a marathon or race exists in $I_3$ (a clone negative), but lacking critical attributes "two skyscrapers in the background" in $I_1$ (ground-truth). Clone negatives occupy an intermediate category that, while sharing high-level semantic similarities with anchor, but fail to achieve a "true" match to the anchor. Rather than being categorized as "false negatives", clone negatives are critical to contrastive learning because they lie betwee true positives and hard negatives, creating ambiguous training signals that require specific handling.

In addition to the difference in definition, existing methods primarily handle false negatives by applying smaller penalties to them, i.e., through an elimination approach. Specifically, [1] cut down the sampling weights of false negatives to minimize their impact during training. Similarly, [2] filters false negatives through a manually-set fixed threshold, and employs sampling to lower their weights, followed by using a vanilla contrastive loss as the learning objective. Fundamentally, these methods adopt a "passive" approach to "avoid" false negatives.

On the contrary, AdaCL is proposed to "actively" address clone negatives by dynamically adjusting the two introduced margin parameters within contrastive learning process, which are vital to the quality of learned representation. In this way, AdaCL facilitates distinguishing clone negatives through the boosts of trainable parameters within each baseline. Therefore, AdaCL is proved to be an effective plug-and-play learning objective in various downstream tasks.

Therefore, clone negatives introduced in this paper represents a unique set of samples that standard contrastive learning or image-text matching baselines struggle to handle. In the revised manuscript, we have refined the definition of clone negatives to strengthen the motivation of our work. Also, we cite and compare several related works on false negative elimination for a comprehensive literature review. We hope the above discussion eliminates your concern, and we sincerely thank you for your valuable feedback.

[1] Li H, Bin Y, Liao J, et al. Your negative may not be true negative: Boosting image-text matching with false negative elimination[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 924-934.

[2] Li Z, Guo C, Feng Z, et al. Integrating language guidance into image-text matching for correcting false negatives[J]. IEEE Transactions on Multimedia, 2023, 26: 103-116.

(2) W2: The authors should enrich the caption of Figure 2 to make the core idea clear and concise.

Thank you for your valuable suggestion. In the revised manuscript, we have enriched the caption for Fig. 2 to provide a more detailed explanation. Specifically, we have emphasized the process of AdaCL and outlined its overall pipeline. We believe this enhancement makes the core idea clearer and more concise, facilitating a better understanding of our paper.

(3) W3: Line325: "Comparisons of Image-text matching" should be "Comparisons of image-text matching". W5: Line 954,955: Missing citations.

Thank you for your careful review and valuable feedback. In the revised manuscript, we have corrected all the typos and checked all the hyperlinks and citations to improve the overall quality.

Dear reviewer 8azX,

Thank you so much for your time and efforts in reviewing our paper. We have addressed your comments in detail and are happy to discuss more if there are any additional concerns. We are looking forward to your feedback and would greatly appreciate you consider raising the scores.

Best regards,

Authors

Thanks for your reply, I'll raise my score to 8.

Thank you for summarizing our work and highlighting our motivation and well performed experiments. Below, we provide a point-by-point response to your comments.

(1) W1: More pre-trained models and more down-stream tasks could be test to further demonstrate the effectiveness of the proposed method.

Thank you for your constructive suggestion. In the revised manuscript, we have extended our evaluation by pre-training AdaCL on CLIP with two large-scale datasets. Therefore, we have validated its effectiveness on (1) image-text matching, (2) weakly-supervised image-text matching (3) text-based person search, (4) zero-shot image-text retrieval (pre-training), and (5) zero-shot image classification (pre-training) tasks. For further details about pre-training, please refer to our Meta Reply.

(2) W2: Spelling error(s): p2 077 andand p2 Fig 1(b) T2 -> T3 p1 034 action-level -> global level p6 274 double curly braces "}" -> just one "}", delete the last one

We sincerely appreciate your thorough review. In the revised manuscript, we have meticulously reviewed and corrected all typos to enhance overall quality.

(3) W3: Sequential adverbs: p4 205 There are “Two conditions”. I can see the “First”, but where is the “Second”? It should be before p4 214 ”(Second, )to achieve this”.

Your understanding is entirely correct. The second condition represents the boundary analysis of Eq. 4, as defined following the phrase "To achieve this". We apologize for the ambiguity and have refined the expression in this section in the revised manuscript.

(4) W4: Meaning of subscript: p4 208 What does the subscript “u” mean? What is its abbreviation? Is it “unique(==specific)”? And what about p4 210 the “u-” in k∈M_u-? Is it “not unique”? I can’t sure.

In line 210, "$u$" denotes to the index of the selected anchor, and "$\bar{u}$" refers to the samples excluding the anchor. We further change the original expression to $\sum_{*anchor*} = \sum_{k=1, k\neq u}^{M+1} {\rm exp} \left[s(I_u, T_k)\right]$ to make it clearer. Meanwhile, retaining "$u$" preserves the intended clarity by indicating that a specific anchor is being referenced. Thus, to improve readability, we have removed the superscript "anc" in Eq. 4, as it does not provide further utility within the equation. We believe this simplification enhances the clarity of the notation. Thank you for your helpful suggestion.

(5) W5: Inconsistencie(s): p4 Sec 3.2 one anchor + one pos + M neg ==> Why not all M neg? Because Sec 3.2 and Sec 3.3 are connected. p5 Sec 3.3 one anchor + one pos + (M-1) neg p6 Algorithm 1 a mini-batch of N pos + M neg??? ==> Why not one pos + M neg? May be the authors want to say “a mini-batch of N, in which 1 pos and M neg”? So, N == 1+M?

In p4 Sec 3.2, the analysis considers 1 positive, and its corresponding $M$ negatives. In p5 Sec 3.3 (Eq. 7 and lines 255-256), the correct number of negatives should be $M$ instead of $M-1$. We have corrected this typo in the revised manuscript. Also, given the constraint $j\neq i$ in Eq. 7, the superscript of the summation should indeed be $M+1$. In Algorithm 1 on p6, we have also corrected a typo: Each mini-batch should consists of $N$ positives and $N\cdot M$ negatives. This is because we use a memory bank, storing M negatives during training ($M$ is larger than the mini-batch size $N$). Therefore, this results in a total of $N \cdot M$ negatives per batch. It is important to note that there is no dependency between $N$ and $M$. We have further revised Algorithm 1 and unified the terminology in Sec 3.2 and Sec 3.3 to standardize all notation (marked red in the manuscript). Thank you for your careful review and constructive feedback.

(6) W6: p5 254 what is the superscript of j in Eq. (8)? Is it the same as Eq. (7) or Eq. (3)?

Eq. 8 represents the binary Bayesian expression used within the softmax function to compute the conditional probability of a class given an observation $s$. Thus, the superscript of $j$ is 2, and the denominator represents the total probability of being either a clone negative or not.

In the revised manuscript, we have further refined Eq. 8 to $p\left(\mathcal{C} \mid s\right)=\frac{p\left(s\mid \mathcal{C}\right) p\left(\mathcal{C}\right)}{p\left(s \mid \mathcal{C}\right) p\left(\mathcal{C}\right) + p\left(s \mid \bar{\mathcal{C}}\right) p\left(\bar{\mathcal{C}}\right)}=\frac{\exp \left(a_{c}\right)}{\exp \left(a_{c}\right) + \exp \left(a_{\bar{c}}\right)}.$ Specifically, we change $p\left(\mathcal{C}=i \mid s\right), i \in \{0,1\}$ to $p\left(\mathcal{C}\mid s\right)$ and $p\left(\bar{\mathcal{C}}\mid s\right)$, which is much more beautiful and concise. $\mathcal{C}$ represents the pairwise similarity is a clone negative, and $\bar{\mathcal{C}}$ for not being a clone negative. We believe this revision eliminates potential misunderstandings arising from overlapping variable names.

(7) W7: p5 263 Why Eq. (9) is that? Can you give us a concrete deduction? And what about p(C=0|s)? Can you give a specific formulation?

We would be happy to discuss the derivation of Eq. 9 in more detail. To begin with, the $K$-class classification probability with Bayes's formula can be expressed as:

$p(y=i \mid x)=\frac{p(x \mid y=i) p(y=i)}{\sum_{j=1}^K p(x \mid y=j) p(y=j)}=\frac{\exp \left(f_i(x)\right)}{\sum_{j=1}^K \exp \left(f_j(x)\right)},$

In anchor selection of AdaCL, the input variable $x$ (a.k.a $s$) is one-dimensional with a binary output variable $y \in {0, 1}$ (a.k.a $\bar{\mathcal{C}}$ and $\mathcal{C}$). We aim to predict $p(y=1 \mid x)$. Gaussian Discriminant Analysis (GDA) assumes that for each class $y =0$ and $y=1$, the input $x$ follows a Gaussian distribution. This can be expressed as: $p(x \mid y=0)=\mathcal{N}\left(x \mid \mu_0, \sigma_0^2\right)$ and $p(x \mid y=1)=\mathcal{N}\left(x \mid \mu_1, \sigma_1^2\right)$. $\mu_0$, $\mu_1$ and $\sigma_0^2$, $\sigma_1^2$ are the means and variances of distributions for classes $y=0$ and $y=1$, respectively. Thus, the posterior probability can be expressed as: $p(y=0 \mid x)=\frac{\mathcal{N}\left(x \mid \mu_0, \sigma_0^2\right) \cdot \pi_0}{\mathcal{N}\left(x \mid \mu_0, \sigma_0^2\right) \cdot \pi_0+\mathcal{N}\left(x \mid \mu_1, \sigma_1^2\right) \cdot \pi_1},$

$p(y=1 \mid x)=\frac{\mathcal{N}\left(x \mid \mu_1, \sigma_1^2\right) \cdot \pi_1}{\mathcal{N}\left(x \mid \mu_1, \sigma_1^2\right) \cdot \pi_1+\mathcal{N}\left(x \mid \mu_0, \sigma_0^2\right) \cdot \pi_0}.$

Since $\mathcal{N}\left(x \mid \mu_1, \sigma_1^2\right)$ is the probability density function of a gaussian distribution, we substitute $\mathcal{N}\left(x \mid \mu_1, \sigma_1^2\right) = \frac{1}{\sqrt{2 \pi \sigma_1^2}} \exp \left( -\frac{(x - \mu_1)^2}{2 \sigma_1^2} \right)$ into the above equations, obtaining Eq. 9 in the manuscript:

$p\left(y=0 \mid x\right)=\frac{1}{1+\frac{\pi_1}{\pi_0}\frac{\sigma_0}{\sigma_1}\exp\left[ \frac{\left( s - \mu_0\right)^2}{2\sigma_{0}^2} -\frac{\left( s - \mu_1\right)^2}{2\sigma_{1}^2}\right]},$

$p\left(y=1 \mid x\right)=\frac{1}{1+\frac{\pi_0}{\pi_1}\frac{\sigma_1}{\sigma_0}\exp\left[ \frac{\left( s - \mu_1\right)^2}{2\sigma_{1}^2} -\frac{\left( s - \mu_0\right)^2}{2\sigma_{0}^2}\right]}.$

The equations represent the probability of a similarity score to be a clone negative or not, without the need of explicit pre-processing to the dataset or training. In the revised manuscript, we have added the derivation to the appendix. We hope the above analysis eliminate the ambiguity and strengthen the clarity of our paper.

(8) W8: p10 509 How to correctly understand the heat map, since a sentence contain ALL WORDS, and every mentioned instance in the image SHOULD be highlighted. So, what does these Attention Map mean indeed? What Information can I get from them? Or what should we expect from them?

The purpose of the attention map is to reveal the specific regions in the image that align semantically with key parts of the sentence. Apart from the effectiveness of highlighting instances, Fig. 5 also illustrates the effectiveness of AdaCL's representation learning capacity when handling three clone negatives under different learning objectives. We assess the representation learning capacity by visualizing the attention map at early training stage (epoch 10). Our results show that AdaCL demonstrates superior and faster learning capacity compared with CL and TRL. For example, the background skyscrapers in case (1), and the man speaking to a blow horn in case (3) are quite significant in distinguishing clone negatives. This visualization provides qualitative insights into AdaCL's effectiveness.

Thank you for noting the quality of our work. Here, we reply to your comments:

(1) W1: The method is not applied to or compared with the recent CLIP-style models, which I believe would strengthen the significance and soundness of this work.

We sincerely appreciate the reviewer's constructive suggestion. In the revised manuscript, we have further included pre-training results for "CLIP + AdaCL" and provided a detailed comparison with "CLIP + Vanilla CL." For further details, please refer to our Meta Reply.

(2) Q1: From what I understand, this method is matching images with texts. Then why is FasterRCNN, an object detection model, used in the experiments?

Thankyou for your careful review. We would be happy to clarify this in more detail. As we mentioned in the introduction, there are usually global-level aligning and object-level aligning in image-text matching, with the key difference being the granularity of matching. Global-level aligning uses the visual backbone to capture a holistic representation of the image. In contrast, object-level aligning leverages a fine-grained visual representation for matching. Therefore, a great number of methods employ Faster R-CNN to extract the object representation $f_{img} \in \mathbb{R}^{K\times D}$, where $K$ stands for the number of objects within an image. The use of Faster R-CNN enables the models to focus on aligning objects and entities, eliminating the impact of background noises. Therefore, object detection model is widely used in image-text matching task.

(3) W3: The authors do not clearly explain why clone negatives pose a problem within the paradigm of contrastive learning. Clone negatives refer to texts that are aligned with the image but are not as accurate as the ground-truth text. Hence, the logits of clone negatives should naturally be lower than those of the ground-truth, which aligns with the basic optimization framework of contrastive learning. The authors should provide more empirical evidence or theoretical analysis demonstrating why standard contrastive learning fails to handle clone negatives adequately.

The key limitation of standard contrastive learning lies in its uniform treatment of all negative samples, regardless of their similarity to the anchor. Clone negatives, as defined, are texts/images that are aligned with the image/text but are less accurate than the ground-truth. Due to their inherent similarity to the ground-truth text, clone negatives can exert a disproportionately strong influence during optimization, potentially leading to suboptimal representation learning.

To address this limitation, our proposed AdaCL introduces an adaptive contrastive learning framework, where negatives with higher similarity to the anchor (e.g., clone negatives) are dynamically emphasized. By modulating the margin, AdaCL is able to force the model to pull them further apart from the anchor. AdaCL better encourages the model to learn finer-grained distinctions between semantically close representations.

In Fig. 4 of the manuscript, we included a case study to compare AdaCL with standard contrastive learning, demonstrating its effectiveness. We acknowledge that a theoretical analysis, as the reviewer rightly points out, could further strengthen our argument. We appreciate this valuable suggestion and plan to explore it in the future work.

(4) Q1: The method figure (Figure 2) is unclear, and neither the figure caption nor the text adequately explains what the "reference clone negatives" refer to.

In the revised manuscript, we have further included a more detailed explanation of Fig. 2 to address the your concerns. Specifically, reference clone negatives are those selected based on the Salient Score introduced in Sec. 3.3. Similarly, salient negatives are also identified through sorting by the Salient Score. We hope this clarification and the revised manuscript provides a clearer understanding of the concepts depicted in Fig. 2.

Thank you very much for the critical review of our work and your help in improving it. Here we provide a point-by-point response.

(1) W1: The experimental results are not convincing. The main contribution of the paper is the improvement of Contrastive Learning, with CLIP being the most representative of this field. To prove the effectiveness of the paper, a comparison within the OpenCLIP framework using larger datasets (e.g., LAION) should have been conducted, rather than experiments on the relatively small-scale COCO dataset.

We sincerely appreciate your constructive suggestion. In response, we have conducted additional experiments as recommended. For further details, please refer to our Meta Reply.

(2) W2: The proposed method appears to be somewhat complex, which goes against the inherently simple and scalable nature of contrastive learning. Additionally, the paper does not provide an analysis of the introduced hyperparameters or the potential increase in overall complexity.

We acknowledge that AdaCL introduces additional components, thus we have carefully analyzed its convergence efficiency in Appendix A.8 of the original manuscript. The experiments demonstrate that AdaCL maintains high convergence efficiency, ensuring its practicality in real-world applications. Furthermore, AdaCL can be seamlessly integrated into existing pipelines (image-text matching and pre-training) with minimal architectural modifications, making it a scalable and practical solution for a wide range of tasks.

Regarding the computational overhead in AdaCL, here we further compare AdaCL with vanilla CL. Assume we have a mini-batch of $N$ samples, and number of negatives for each sample is $M$, the computation comparison can be expressed as follows:

We can conclude that AdaCL introduces moderate computational overhead, primarily due to anchor selection. The additional cost scales linearly with $N$ and $M$. We also record the training time of AdaCL in CLIP pre-training, which is demonstrated as follows:

We record the average number of samples processed per second over the entire training process. We observe a slight decrease in efficiency for AdaCL compared to vanilla CL in the CLIP pre-training, which we recognize this as a limitation of AdaCL and have added this analysis to A.12 of the Appendix. Also, we believe this trade-off is acceptable given the context of pre-training, and AdaCL maintains a overall balance between effectiveness and efficiency. The above results and analysis have been updated to our revised manuscript to provide a thorough analysis of this work.

Dear reviewer 5tK1,

Thank you so much for your time and efforts in reviewing our paper. We have addressed your comments in detail and are happy to discuss more if there are any additional concerns. We are looking forward to your feedback and would greatly appreciate you consider raising the scores.

Best regards,

Authors