Debiased Orthogonal Boundary-driven Efficient Noise Mitigation

We thank the reviewer DHCR for the positive, patient and professional review, as well as the valuable suggestions for improvement. Our responses to the reviewer’s questions are below:

W1 : The claim of "table2 show that OSA outperforms all previous approaches on all metrics with a huge gap" is not fully adequete, due to NPC outperforms OSA on many occasions.

A1: Thank you for your careful review. In Table 2, OSA outperforms NPC in all metrics of R@1, which is the most important metric in image-text matching. For R@5 and R@10, OSA also outperforms NPC in most cases with a significant margin, especially in noisy scenarios. To accurately clarify this, we will revise the claim from 'on all metrics' to 'on most metrics' in the revised version.

---

W2 : The method is quite simple.

A2: The objective of our work is to develop a general and easily adaptable anti-noise method. Therefore, we hope the framework of our method is as concise as possible to ensure its practicality in complex real-world scenarios. To achieve this, we focus more on exploring anti-noise principles than sophisticated methods to mitigate the loss of generality arisen from complex methods.

---

W3 : It may be better to have a comprehensive introduction of related work in the main text to help the readers to understand the knowledge about existing methods for comparison.

A3: We are sorry to make this confusion. After your valuable reminders, we believe it would be beneficial to include more related work in the main text to enhance readers' understanding. In our next revision, we will transfer some content from the related work section to the main text.

---

W4 : How many pairs are used to calculate the space shift？

A4: During our training process, we randomly sample 256 images and 256 texts separately, forming $256 \times 256 = 65536$ pairs. We also evaluate different sample sizes ranging from $64 \times 64$ to $1024 \times 1024$, and found that the boundaries remain stable. Therefore, we ultimately set the sampling size to match the batch size (256) in most of our experiments.

---

W5 : There are some cases in which positive pairs get low similarity scores and negative pairs get high similarity scores, will it have a big impact on the identification of noise and clean data?

A5: Thank you for your constructive question. To explore this, we further conduct experiments on a very large real-world dataset, CC3M, which contains 3 million image-text pairs. These samples are collected from webpages and filtered by Google AI based on instance matching. This suggests that noise in this dataset is rare and semantically relevant, which can somewhat represent negative pairs with high similarity scores. Additionally, we found that zero-shot CLIP performs poorly on this dataset, achieving only about 29 R@1. This indicates that the dataset is somewhat out-of-domain for CLIP, and that there are likely some clean samples with low performance. We report the performance of zero-shot CLIP, the Baseline (CLIP fine-tuned on CC3M), and OSA applied to the Baseline in the table below:

We observe that OSA still provides noticeable performance improvement in this challenging scenario. This phenomenon further demonstrates the effectiveness and robustness of OSA. Therefore, cases where some positive pairs have low similarity scores and negative pairs have high similarity scores do not significantly impact OSA's performance

---

W6 : Is the clip you used as an estimator model off the shelf? For example, import the openai clip and calculate the similarity.

A6: Yes, we use the off-the-shelf CLIP model released by OpenAI.

---

W7 : Will it cost a lot of time to calculate the similarity? Once I have tried, it's a little bit slow.

A7: We evaluate the time cost on an NVIDIA RTX 3090, processing the MS-COCO dataset (566,435 pairs) with a batch size of 4096 takes about 153 seconds, using ~24 GB of GPU memory. At this rate, processing 1 billion samples would take approximately 75 hours on a single RTX 3090. In addition, this process can be further accelerated through parallel inference. We think this is an acceptable overhead for real-world industrial training.

---

W8 : How to indentify the noisy pair using CLIP in image classification?

A8: We follow the same image classification pipeline as shown in Figure 1(b), which is also the format used in the CLIP paper [1]. The specific format is: "This is an image of [CLS]."

Refs:

[1] Alec Radford et al, "Learning Transferable Visual Models From Natural Language Supervision", ICML, 2021.

---

We thank the reviewer Ho5e for the positive, patient and professional review, as well as the valuable suggestions for improvement. Our responses to the reviewer’s questions are below:

W1 : It is more meaningful when validated on large-scale datasets.

A1: Thank you for your insightful review. To further explore the effectiveness of our method in real-world scenarios, especially on large-scale and out-of-domain training, we conduct experiments on a very large real-world dataset, CC3M, which contains 3 million image-text pairs collected from webpage and filtered by Google AI. We find that zero-shot CLIP does not perform well on this dataset, achieving only 29.25 R@1 on i2t and 28.80 R@1 on t2i. Therefore, we believe this dataset can somewhat represent practical large-scale and out-of-domain scenarios. We report the performance of zero-shot CLIP, the Baseline (CLIP fine-tuned on CC3M), and OSA applied to the Baseline in the table below:

Although the samples in CC3M are filtered and have a lower noise ratio compared to natural ones, we observe that OSA still brings noticeable performance improvement. This phenomenon further demonstrates the effectiveness and robustness of OSA in real-world scenarios. Given the breadth and complexity of real-world domains, fully exploring all possible scenarios is extremely challenging. However, we believe this experiment could somewhat provide evidence and insights into OSA's effectiveness and robustness in real-world scenarios.

---

W2 : OSA’s effectiveness depends heavily on that estimator’s domain alignment, and may become less reliable in severely domain-mismatched scenarios.

A2: As mentioned in A1, in CC3M dataset, zero-shot CLIP only achieves a not good performance and this may somewhat represent domain-mismatched scenarios. In these scenarios, OSA still achieves improvements, indicating its reliablity across various domains. Furthermore, we also provide an optional domain adaptation (DA) solution in Section 3.2.1 to address edge-domain challenges in real-world scenarios, and it achieves significant improvements on noise detection accuracy in Table 5.

---

W3 & Q3: The repository authors provided in the abstract is expired. Hope the authors can fix the open-source repository they provided.

A3: We are sorry for this mistake. We have re-opened the repository, and it is now accessible!

---

W4 : If the target domain is drastically different, can zero-shot CLIP/ALIGN still provide a reliable boundary?

A4: Actually, we think that the boundary is an inherent property of the shared space. Therefore, we calculate the boundary using simulated images and texts to eliminate the influence of specific data domains. This suggests that the boundary itself is stable and independent of the target domain.

In cases where the target domain is drastically different—where CLIP is almost impossible to understand the target domain—samples are likely to be distributed around the orthogonal boundary, which may lead to substantial overlap. In such cases, domain adaptation may be necessary to enhance the estimator's recognition ability for the target domain. However, based on our experiments on the Stable Diffusion domain and CC3M, we find that it is uncommon for CLIP to entirely fail to understand the target domain in real large-scale training scenarios. As long as CLIP can recognize the target domain to some extent, the orthogonal boundary can effectively separate clean and noisy samples.

---

W5 : How to handle moderate overlap between clean/noisy distributions near β? Would a different shaping function or a more cautious threshold be better in such cases?

A5: This is an important question and highlights a common challenge in the field. In practice, it is hard to perfectly separate overlapping clean and noisy samples based on a threshold boundary. But in our work, we identify an inherent decision boundary in the model space with theoretical significance. Compared to the common strategy of using a strict threshold, this approach allows us to design more sophisticated and accurate methods for handling overlap. For instance, our high-degree scoring function, where the gradient trends follow the probability trends of random vectors near the orthogonal boundary, achieves nearly optimal weight ranking in Table 11, suggesting the effectiveness of the function design.

We therefore believe that designing a more suitable function based on the theoretical properties of the boundary is a better solution. Additionally, the overlap mainly arises from unfamiliarity with the target domain. As mentioned in A2, we also propose a domain adaptation technique to help the estimator better adapt to real-world scenarios.

---

We thank the reviewer PeYE for the positive, patient and professional review, as well as the valuable suggestions for improvement. Our responses to the reviewer’s questions are below:

W1 : Although the evaluation datasets cover diverse image and text content, they may not fully represent all possible real-world scenarios.

A1: Thank you for your valuable insights. To further explore the effectiveness of our method in real-world scenarios, especially on large-scale and out-of-domain training, we conduct experiments on a very large real-world dataset, CC3M, which contains 3 million image-text pairs collected from webpage and filtered by Google AI. We find that zero-shot CLIP does not perform well on this dataset, achieving only 29.25 R@1 on i2t and 28.80 R@1 on t2i. Therefore, we believe this dataset can somewhat represent practical large-scale and out-of-domain scenarios. We report the performance of zero-shot CLIP, the Baseline (CLIP fine-tuned on CC3M), and OSA applied to the Baseline in the table below:

Although the samples in CC3M are filtered and have a lower noise ratio compared to natural ones, we observe that OSA still brings noticeable performance improvement. This phenomenon further demonstrates the effectiveness and robustness of OSA in real-world scenarios. Given the breadth and complexity of real-world domains, fully exploring all possible scenarios is extremely challenging. However, we believe this experiment could somewhat provide evidence and insights into OSA's effectiveness and robustness in real-world scenarios.

---

W2 : Limited Exploration in Scoring Functions: The study on employing high-degree scoring functions is insufficiently comprehensive.

A2: The scoring function is a crucial component of our work, and we conduct an initial exploration in Appendix F.1. Specifically, we compare three types of functions: Linear, Cosine, and High-Degree functions. In Table 6, we observe that our carefully designed high-degree function (based on orthogonal boundary properties) outperforms other methods.

The rationale behind our design of the current high-degree function is as follows:

1. For cosine similarity values lower than the orthogonal boundary, there is a high probability that the sample is noise due to the huge gap caused by the orthogonal boundary in anisotropic space. To mitigate the impact of noise, we assign these samples a weight of zero to prevent them from influencing training.
2. On the positive side of the orthogonal boundary, the probability of a sample being noise decreases rapidly as the cosine similarity increases. Therefore, the gradient should initially increase rapidly as cosine similarity moves away from the orthogonal boundary. However, for samples with relatively high cosine similarity, we assume they have already been well-learned, so we assign them a lower weight to prevent overfitting.

Considering these factors, we designed a high-degree function in the form of Eq. 34 ($y=-x^2(x-1), x>0$). If plotted, our high-degree function exhibits a curve in the range of 0-1 that rises slowly at first, then steeply, before gradually decreasing after 0.7 on the positive side. We will provide a more detailed explanation and supplement these design considerations in the updated version.

---

W3 : The statement “Contrastive learning empowers the separation of clean and noisy samples” in Section 2.2 appears abrupt, lacking a clear lead-in and logical connection to the preceding discussion on the intersection boundary.

A3: We sincerely appreciate your reminder of the confusion maybe caused by this statement order. This claim aims to explain why the orthogonal boundary in the shared space of a pre-trained model can accurately and naturally separate clean and noisy samples. Inspired by your valuable suggestion, we believe this discussion would be better placed in Section 2.3, 'Qualitative Analysis of Robustness and Applicability,' to improve the logical flow. We will make this correction in our revised version.

---

W4 : Is the expression of “brings x_i and y_i closer when c_i=1” in section 3.1 (Line 187) a mistake? A noisy sample x_i should be far away from y_i.

A4: We sincerely appreciate your careful review. This is indeed a typo—it should be "brings $x_i$ and $y_i$ closer when $c_i=0$." We will correct this in our revised version.

---

W5 : It would be beneficial if the author could provide additional insights into the design of a score function, especially for the high-degree one.

A5: Thank you for your constructive suggestion. As mentioned in A2, there are two factors behind our high-degree function design. We would like to include all of these discussions in our revision.

---