Semantic Feature Learning for Universal Unsupervised Cross-Domain Retrieval

The necessity illustration of each loss and the theory justification

Firstly, we did not use six versions of contrastive loss. IPM combines INCE and PNCE with the intuitive goal of performing categorical semantic learning on unlabeled domain data. INCE forms the basis of unsupervised semantic learning. However, relying solely on INCE results in unclear boundaries between categories. Fortunately, introducing PNCE can enhance the distinction between categories, but determining the optimal timing to introduce PNCE requires careful consideration. We use a sigmoid function to weight the INCE training and then introduce varying degrees of PNCE accordingly. Additionally, we design a prototype conversion strategy to provide a cross-domain unified prototypical structure for PNCE. However, this IPM suffers from single prototype allocation mistakes and errors. To compensate for this issue, we propose SEL, which considers the relationship between a single instance and all prototypes, thereby alleviating erroneous optimization caused by single prototype allocation mistakes. IPM and SEL constitute the first stage of UEM optimization.

In the second stage of UEM, we recognize that the primary reason for the inaccuracy of existing cross-domain matching work is the neglect of the domain gap. However, regular domain alignment methods are not suitable for our UEM framework as they severely destroy the prototypical structure learned in the first stage. Therefore, we modified standard domain adversarial training by introducing semantic preservation constraints (SPR). We also design a more accurate cross-domain matching algorithm that aligns with the unified prototypical structure, which is SN^2M. SN^2M assesses the reliability of cross-domain pairs by the consistency of the nearest prototypes in their respective domains and then decides whether to treat the pair as a positive pair in contrastive learning. Accordingly, the second stage of UEM optimization is guided by SPR and SN^2M.

In our current work, we have only conducted theoretical analysis on IPM. Due to the high complexity of entanglement between feature geometry distribution and semantics in high-dimensional space, theoretical analysis of SPR and SN^2M is challenging in supervised learning. Considering unsupervised cross-domain scenarios makes it even more difficult. Thus, we will attempt to provide theoretical explanations for this part in future work.

Detailed description of clustering and Elbow approach

Thanks for your suggestion. We will incorporate the following description of clustering and the Elbow approach into our future revision. The Elbow approach requires a pre-set maximum cluster number, which is 100 in our implementation. Then we repeatedly apply K-Means to the memory bank of each domain with the cluster number increasing from 2 to 100. For each run, we record two metrics. One is the within-cluster sum of squares (WCSS) which measures the sum of squared distances between each data point and its assigned centroid, reflecting the compactness of the clusters. The other is the silhouette score (SS) which measures how similar a data point is to its own cluster compared to other clusters. Then we draw two curves with the cluster number as the x-axis and the metric value as the y-axis. To determine the elbow point, we select the farthest point below the line passing through the first point and the last point of the curve w.r.t Euclidean distance as the estimated cluster number. After obtaining the respective estimated cluster numbers from the WCSS and SS curves, we select the larger one as the final estimation.

More explanation of the IPM

Please refer to the explanation and description of cross-domain prototype conversion in the Global Response.

More explanation of the SN2M

Please refer to the explanation and description of cross-domain instance matching in the Global Response.

The central motivation of the two-stage training

As mentioned in the introduction, the successful achievement of cross-domain retrieval (CDR) relies on solving two problems: 1) effectively distinguishing data samples in each domain, and 2) achieving alignment across domains for samples of the same category. Most existing methods adopt a two-stage processing strategy, where self-supervised learning (SSL) is first used for categorical semantic learning on unlabeled data, followed by nearest neighbor searching for cross-domain categorical matching. Our UEM framework also employs this two-stage strategy and introduces several novel designs and algorithms during the training phases to address the U^2CDR problem.

We considered whether it might be possible to solve the U^2CDR problem with a single-stage, end-to-end framework. The answer is no because we cannot introduce additional constraints during the first stage of SSL, such as domain alignment or cross-domain categorical matching. These constraints would significantly affect the original optimization goals and directions of SSL, deviating the model from learning categorical semantics, which is the basis of everything. To validate our thinking, we carry out experiments with a single-stage design by incorporating domain alignment (from the second stage of UEM) and SN^2M into the first stage. Below are the experiment results for close-set, partial, and open-set UCDR (in each cell of open-set UCDR, the left number is Shared-set mAP@All, and the right is Open-set detection accuracy). As shown in the table, the performance of the single-stage process is indeed much poorer compared to the two-stage design.

Dear Reviewer tjhd,

As the rebuttal discussion phase ends soon, we want to express our gratitude for your engagement thus far. We really want to check with you whether our response addresses your concerns during the author-reviewer discussion phase. We have diligently addressed every concern and question you raised during the initial review, and our efforts are aimed at enhancing the clarity and quality of our work.

We genuinely hope our responses have resolved your concerns related to the design intuition behind our methodology, the detailed description of the Elbow approach, IPM and SN2M losses, and the central motivation of the two-stage design. Your thoughtful evaluation greatly aids in our paper's refinement and strength. Again, we sincerely appreciate your dedication and time.

Best regards,

Authors of Paper 5055

Description of the methodology novelty

Please refer to the novelty illustration in the Global Response. In addition to the ablation study, To further validate the novelty of semantic structure preservation and cross-domain matching, we carry out experiments with the replacement of another state-of-the-art binary graph-based semantic preservation approach (SPPA [1]) and nearest neighbor searching algorithm (CD^2NN [2]). SPPA preserves the instance-level cosine similarity within each cluster and the prototype-level Euclidean distance across clusters. CD^2NN determines cross-domain nearest instance pair by seeking neighboring instance consistency. The experiment results in open-set CDR (in each cell, the left number is the Shared-set mAP@All, while the right is the Open-set detection accuracy) are shown below, demonstrating that UEM's semantic structure preservation and cross-domain matching are more effective.

[1] Lin, et al. Continual semantic segmentation via structure preserving and projected feature alignment. ECCV 2022

[2] Liu, et al. Promoting Semantic Connectivity: Dual Nearest Neighbors Contrastive Learning for Unsupervised Domain Generalization. CVPR 2023

How does UEM maintain the relationship between positive pairs across domains

Thank you for your suggestion. In fact, cross-domain positive pairs are maintained through the construction of that unified prototypical structure. During the first stage of our UEM framework, all instances in each domain approach their nearest prototype. These prototypes undergo a prototype conversion process, making them cross-domain unified, meaning that prototypes of the same category have the same geometric relationship relative to other categories within each domain. In this scenario, cross-domain positive pairs will be close to the prototype of the same category in their respective domains.

The relationship of cross-domain positive pairs is preserved during the second stage of UEM. As the prototypical structures across domains are unified, the inner-domain relationship preservation achieved by our semantic-preserving domain alignment actually equals to preserving inter-domain relationships, which definitely includes the relationship of cross-domain positive pairs. Besides, the relationship of cross-domain positive pairs is further reinforced in SN^2M, which is a primary objective of cross-domain matching.

More analysis of the ablation study

Thanks for your suggestion. We provide more analysis and reasoning for the ablation study as follows, and will incorporate them into the future revision.

The performance degradation caused by not using Prototype Merging may be attributed to the geometry distinctness of instance discrimination learning within each domain (Theorem 3.1). The high-level intuition is that the same categories across domains are forced to distinct geometry locations due to distinct contrastive comparisons with different category spaces. In this case, there are plenty of mismatches and misalignments in cross-domain matching. Therefore, the UEM performs poorly without using Prototype Merging.

As for the performance loss due to SEL, we think it originated from the unavoidable errors of the arbitrary prototype allocation in the prototype contrastive loss. Different from this single prototype allocation, SEL considers the potential relationship between the instance and all prototypes. This design provides the possibility of error correction and can effectively compensate for the prototype contrastive loss.

According to Table 4, there is the largest performance gap between the 'Ours w/o SPDA' approach and the full 'Ours' approach. This indicates the semantic structure destruction during standard domain adversarial training. The semantic structure learned by the first stage is the basis of cross-domain matching, thus causing such a large performance drop.

To test the effectiveness of SN^2M, we replace it with the neighbor searching used in UCDIR, which searches for the nearest cross-domain instance and prototype in terms of cosine similarity distance to approach. Apparently, this search strategy has a lot of errors, which is worse if there is a domain gap. By contrast, our SN^2M can measure the reliability of the nearest cross-domain instance and then decide whether to approach it. Moreover, with the prototype translation and merging, the nearest cross-domain prototype is more accurate and reliable.

Any handling of instances belonging to uncommon categories

In the first stage of our UEM framework, we aim to build a unified prototypical structure across domains via the IPM loss. The IPM loss is a combination of instance and prototype contrastive losses. Given that the IPM loss is computed separately in each domain, we design a novel prototype merging strategy to merge highly potential common categories across domains. Then for the prototype contrastive loss, each instance searches for and tries to approach its closest prototype. There is no difference between instances corresponding to unmerged (uncommon category) or merged prototypes, as all prototypes should be treated equally to form the final prototypical structure -- otherwise, the uncommon category detection in partial and open-set cross-domain retrieval becomes much harder. As for the second stage, each instance of each domain searches for and decides whether to approach the nearest neighbor in the other domain. For instances belonging to merged prototypes (common category), there is a much higher possibility of being optimized to approach their nearest neighbors. While for instances that are not common categories, they usually shouldn't approach their cross-domain neighbors. However, we still allow these instances to approach their domain-translated prototypes, which means that we do have the handling for instances that are not common categories. We believe this is necessary to achieve more effective cross-domain matching.

Novelty of semantic structure preservation and adjacent feature matching

Please refer to the Global Response for a detailed description of the novelty. To further validate the novelty of semantic structure preservation and adjacent feature matching, we carry out experiments with the replacement of another state-of-the-art binary graph-based semantic preservation approach (SPPA [1]) and nearest neighbor searching algorithm (CD^2NN [2]). SPPA preserves the instance-level cosine similarity within each cluster and the prototype-level Euclidean distance across clusters. CD^2NN determines cross-domain nearest instance pair by seeking neighboring instance consistency. The experiment results in open-set CDR (in each cell, the left number is the Shared-set mAP@All, while the right is the Open-set detection accuracy) are shown below, which demonstrate that UEM's semantic structure preservation and adjacent feature matching are more effective.

[1] Lin, et al. Continual semantic segmentation via structure preserving and projected feature alignment. ECCV 2022

[2] Liu, et al. Promoting Semantic Connectivity: Dual Nearest Neighbors Contrastive Learning for Unsupervised Domain Generalization. CVPR 2023

Difference between universal and unsupervised domain adaptation

We suppose you are indicating the difference between universal and unsupervised cross-domain retrieval (CDR) rather than domain adaptation. First, the CDR problem considers two semantically distinct but similar domains and aims to retrieve data samples from one domain belonging to the same category as a query sample from the other domain. The unsupervised CDR specifies that these two domains only contain unlabeled data. Regular unsupervised CDR studies assume that the category spaces of these two domains are identical, while universal CDR focuses on scenarios where the category space across domains is distinct. In our work, we believe the assumption of identical cross-domain category space is unreasonable for many real-world applications, as the categorical composition of an unlabeled data domain is hard to acquire without detailed analysis and dedicated expertise.

Better statement of cross-domain conversion and close neighbor matching

Please refer to the Global Response for better descriptions of cross-domain conversion and close neighbor matching.

Dear Reviewer gZ1R,

The conclusion of the discussion period is closing, and we eagerly await your response. We greatly appreciate your time and effort in reviewing this paper and helping us improve it.

Thank you again for the detailed and constructive reviews. We hope our response is able to address your comments related to the handling of uncommon categories, the novelty description of semantic preservation and adjacent feature matching, and clarification of problem settings. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give us.

Best Regards,

Authors of Paper 5055