Bridging the Gap between Semantic Correspondence and Robust Visual Representation

• Futher comparision of ViTSC and LoFTR is required. What is the main difference of ViTSC and LoFTR in pipeline design.
• Several Multi-Scale method for semantic correspondence such as MMNet and VAT have to be introduced in the related work. The difference of ViTSC for using high-level and low-level features compared with other multi-scale pipelines should be analyzed.
• For auxiliary loss, what is the main difference of designed loss compared with WarpC
• I have recognized the performance of ViTSC. However, what is the performance of previous SOTAs when using DINO V2, can better backbone contribute to all the semantic correspondence pipelines?

[1] LoFTR: Detector-free local feature matching with transformers

[2] Cost aggregation with 4d convolutional swin transformer for few-shot segmentation

[3] Multi-scale matching networks for semantic correspondence

[4] Warp consistency for unsupervised learning of dense correspondences

1. The paper only conducts experiments on three standard benchmarks: SPair-71k, PF-PASCAL, and PF-WILLOW, which may not cover a broader range of datasets, especially those that include more diverse scenes and conditions. This limits the comprehensive assessment of the model's generalization capabilities.

2. The paper does not adequately test the model's performance under extreme conditions, such as in complex scenarios involving extreme lighting, weather conditions, occlusion, or rapid motion, indicating a lack of scalability.

3. The experiments may not evaluate the model's real-time performance and computational efficiency in practical applications, which is particularly important for resource-constrained environments such as mobile devices.

4. Although the ViTSC framework improves upon existing methods, the paper lacks a detailed description of the innovation in the model architecture, especially in comparison with existing technologies. While the proposed model includes a cross-attention module and a correlation-guided upsampler, its novelty is insufficient, and the analysis of these modules' impact on the overall architecture is not in-depth.

5. The experimental section of the paper lacks specific analyses, such as an in-depth exploration of the reasons behind the model's good performance in comparative experiments.

1. Lack of Novelty:
   • The interleaved attention module is a relatively simple modification of the cross-attention module input.
   • The correlation-guided upsampler has already been proposed in RAFT (Teed and Deng, ECCV 2020).
   • The auxiliary triplet loss is previously introduced in HardNet [A].

[A] Mishchuk et al., "Working hard to know your neighbor's margins," NeurIPS 2017.

Is there a difference in the way the modules proposed by RAFT or HardNet are applied or adapted in the semantic correspondence benchmarks covered in this paper?

2. Comparison to Stable Diffusion Backbones: Existing studies [B, C, D, E] suggest using stable diffusion backbones for solving semantic correspondence. The authors should include these baselines in Table 1 and provide a comparative discussion. This can be experimented with in the standard public semantic correspondence benchmarks PF WILLOW, PASCAL, and SPair.

[B] Tang et al., "Emergent Correspondence from Image Diffusion," NeurIPS 2023. [C] Hedlin et al., "Unsupervised Semantic Correspondence Using Stable Diffusion," NeurIPS 2023. [D] Zhang et al., "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence," NeurIPS 2023. [E] Luo et al., "Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence," NeurIPS 2023.

3. Limited Benchmark Testing: The ablation studies are primarily conducted on SPairs. Is there a reason why the ablation study was only performed on the SPair-71k? It is crucial to validate the results on other semantic correspondence benchmarks such as PF-WILLOW and PASCAL to ensure broader applicability.

4. Lack of Explanation for Figure 3: The manuscript does not explain Figure 3 adequately. It is essential to clarify why the interleaved feature enhancement is superior to symmetric and asymmetric feature enhancement from this perspective. In addition, the examples in Figure 3 visualization only covers left/right reflection symmetry, but in the real world, there exists rotational symmetry frequently. Does the proposed module handle the rotational symmetry cases?

5. Broken paper formatting: Table 4 citation is broken. Table 3 TransforMatcher row is wrongly cited.

1. In the abstract section, the paper repeatedly emphasizes the complexity of the matching module design in existing methods and underscores that the proposed method is both simple and effective. However, the experiments lack an analysis of the matching module's complexity, limiting the validation of the paper’s claims.

2. I strongly recommend that the authors make the original code publicly available upon submission to enhance the credibility of the proposed method.

3. The introduction of this paper outlines three issues that need to be addressed; however, the visualization experiments lack comparative results demonstrating performance improvements in scenarios related to these three issues.

4. The naming of this high-resolution low-level correlation-guided upsampling module is not concise enough and seems a bit long.