Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning

Q3: Clarify evaluation protocal.

Sorry for the confusion. None of the three downstream tasks in our main paper require training—they all directly use the same fine-tuned 3D-aware features. What we refer to as "training" on L160 is actually the onboarding phase, where we extract 2D dense features from the provided reference video and store them in our database. During inference, we match features between a single query image and our database.

For depth estimation, we follow DINOv2's protocol by adding a linear layer with classification loss to predict depths across 256 uniform bins. For instance recognition and semantic segmentation, we also adhere to DINOv2's evaluation protocol.

Thanks for the valuable suggestion, we will draw figures to illustrate the input and output for each task.

Q4: Move comparisons with FiT into the main paper.

Thanks for your insight. As our main goal is to show the fine-tuning of proximal rigid multi-view object centric tasks can improve downstream 3D understanding tasks, we did not put it into the main paper due to the space limit. But we added some text describing their performance in our revised manuscript (Section 3.2).

Q5: Why FiT is so bad? Clarifications on FiT.

FiT-Reg refers to FiT with DINOv2's registers[4]. Our experiments revealed that although FiT aims for 3D consistency, it significantly disrupts the semantics of certain parts, as shown in this figure and this figure. While this semantic disruption may be acceptable for FiT's original tasks like semantic segmentation and depth estimation—where an additional linear head can correct these issues—it becomes problematic for our tasks that require 3D-consistent, dense, pixel-level features. We hypothesize that FiT's poor performance stems from its naive approach to learning 3D consistency through an explicit 3D Gaussian field. When outliers or noise are present, the simple mean square error causes feature representations to shift toward these outliers.

Q6: Occlusion and symmetric objects' influence on the performance.

During fine-tuning, we handle self-occlusion by performing depth tests and discarding occluded samples.

For symmetric objects, we don't implement specific handling—instead, we rely on uniform point sampling, where symmetric features' gradients cancel each other out in our SmoothAP loss, leaving the loss dominated by features from distinct parts. From our Figure 1 in the main manuscript, we can see that symmetric parts share similar embeddings. While there is one minimum requirement: we cannot have all points on an object symmetric (like a perfect sphere), otherwise the model cannot learn any meaningful features.

[4] Vision Transformers Need Registers

Q1: Limited practical use case of the finetuned DINO features?

We would like to clarify that on OnePose-LowTex, our DINO fine-tuned method already outperforms OnePose++ on 3cm 3deg and 5cm 5deg metrics. However, on other datasets and tasks, the fine-tuned ViT still has some gap with state-of-the-art methods.

On one hand, we expect better ViT architectures to emerge, which will yield improved performance after our fine-tuning (as our method is agnostic to the particular ViT architecture). On the other hand, we focus on general-purpose ViT features that are more applicable than those domain specific features, and this paper’s main goal is not to beat the baselines.

The key advantage of these ViT features is their generality across different datasets and tasks. They can be applied to a wide range of scenarios. For example: SparseDFF[1] uses DINO to aggregate and fine-tune consistent feature representations across views for few-shot transfer manipulation policy learning; LERF[2] uses dense DINO features for regularization; Wild Gaussians[3] employs off-the-shelf DINO features as a strong prior to estimate occlusions and reconstruct 3D scenes in the wild. These tasks lack clear methods for training domain-specific structures, as they are open-set tasks with limited training data or demonstrations. Therefore, we believe studying these general-purpose ViT features remains promising.

To show our finetuned features can be useful in these general tasks, we conducted experiments on Wild-Gaussians and found that replacing the original features with our fine-tuned DINO features improved novel view synthesis quality in the wild, as shown in the following table. All results were produced using Wild-Gaussians' official GitHub repository.

Additionally, we visualized LERF 3D features after replacing its DINO regularizer with our fine-tuned version. When given the text query "plate", LERF with our fine-tuned DINO produced a more focused and accurate relevancy map compared to the original DINO features, with better localization of the plate region and reduced noise in irrelevant areas such as cookies, as shown in this image. We could only provide qualitative results for LERF since LERF has not released its quantitative evaluation code.

We also included these discussion in our supplementary (Section A.7).

Q2: Clarify why feature equivariance implies 3D awareness.

We agree that we should clarify SIFT as 2D affine invariant rather than 3D-aware. SIFT, by design, is robust only to 2D transformations (rotation, translation, scale). It can only match keypoints across views with small changes, as the 2D image patch distorts minimally in such cases. To illustrate this, we have demonstrated in this figure that under large viewpoint changes in 3D, our fine-tuned DINO (DINOv2-FT) features give much better correspondences on MVImgNet, while SIFT fails due to significant image patch distortion. This comparison indicates that ViTs possess better 3D awareness than SIFT descriptors and exhibit some 3D understanding ability. As our evaluation in based solely on 3D feature equivariance, it does imply 3D awareness.

[1] Sparsedff: Sparse-view feature distillation for one-shot dexterous manipulation.

[2] Lerf: Language embedded radiance fields.

[3] Wildgaussians: 3d gaussian splatting in the wild.

I appreciate the efforts the authors put into the rebuttal. The additional experiments and clarifications (e.g., the diagrams explaining how each task is evaluated) make the paper clearer and more convincing. The wild-Gaussian experiment is very interesting and should be presented in the main paper. It is also interesting to see that SIFT fails in extreme viewpoint while the proposed method is comparbly robust. A remaining concern (as agreed by other reviewers) is the inferior performance compared to SOTA on some tasks. Considering all these factors, I decide to increase my score from 5 to 6. A side note: the current version of the paper is 11 pages, which violates the authors guideline (https://iclr.cc/Conferences/2025/AuthorGuide). Please shrink it to 10 pages to avoid potential issues.

Q1: Comparison with state-of-the-art task-specific tasks.

Thank you for the suggestion. We indeed included baseline methods for pose estimation and tracking in our supplementary materials (Section A.3). For OnePose-LowTex, our DINO fine-tuned method already outperforms OnePose++ on 3cm 3deg and 5cm 5deg metrics. However, on other datasets and tasks, the fine-tuned ViT still has some gap with state-of-the-art methods.

On one hand, we expect better ViT architectures to emerge, which will yield improved performance after our fine-tuning (as our method is agnostic to the particular ViT architecture).

On the other hand, as you mentioned, these ViT features aren't designed to surpass domain-specific methods—a point also illustrated in DINOv2 paper’s table, which only compares general-purpose feature learning methods.

The key advantage of these ViT features is their generality across different datasets and tasks. They can be applied to a wide range of scenarios. For example: SparseDFF[1] uses DINO to aggregate and fine-tune consistent feature representations across views for few-shot transfer manipulation policy learning; LERF[2] uses dense DINO features for regularization; Wild Gaussians[3] employs off-the-shelf DINO features as a strong prior to estimate occlusions and reconstruct 3D scenes in the wild. These tasks lack clear methods for training domain-specific structures, as they are open-set tasks with limited training data or demonstrations. Therefore, we believe studying these general-purpose ViT features remains promising.

To show our finetuned features can be useful in these general tasks, we conducted experiments on Wild-Gaussians and found that replacing the original features with our fine-tuned DINO features improved novel view synthesis quality in the wild, as shown in the following table. All results were produced using Wild-Gaussians' official GitHub repository.

Additionally, we visualized LERF 3D features after replacing its DINO regularizer with our fine-tuned version. When given the text query "plate", LERF with our fine-tuned DINO produced a more focused and accurate relevancy map compared to the original DINO features, with better localization of the plate region and reduced noise in irrelevant areas such as cookies, as shown in this image. We could only provide qualitative results for LERF since LERF has not released its quantitative evaluation code.

We also included these discussion in our supplementary (Section A.7).

[1] Sparsedff: Sparse-view feature distillation for one-shot dexterous manipulation.

[2] Lerf: Language embedded radiance fields.

[3] Wildgaussians: 3d gaussian splatting in the wild.

Q3: Results on other models with different architectures like ConvNext.

Thank you for your insightful comment. We have applied our method to other architectures like ConvNeXt and found that we can consistently improve its performance on downstream tasks as well. However, we've also observed that ConvNeXt features are not as good as those of modern ViTs. This is one of the main reasons we chose to focus on ViT-based models—they are not only the most commonly used but potentially superior. Overall, we do expect and observe improvements in non-ViT based methods like ConvNeXt. This finding is particularly interesting as it teaches us a valuable lesson: with relatively simple 3D fine-tuning, we can achieve even better 3D features than those obtained through pretraining on a vast set of unstructured 2D images. We included this experiment in our supplementary (A.5).

Q4: How can the model learn from hemisphere shape?

Unlike a perfect sphere, the hemisphere we used is not completely symmetric and provides information about edges and viewpoint orientation. Our visualization of the learned embeddings shows that after fine-tuning on the hemisphere, the network achieves better edge correspondences and can differentiate between inward and outward views. Even though the object lacks texture, the shadows and edge features provide sufficient cues for the ViT features to develop 3D understanding.

Similarly, in cognitive science, scientists have discovered that the human brain also learns complex 3D structures from basic geometric primitives. Biederman's Recognition-by-Components (RBC) theory[4] suggests that humans recognize objects through simple 3D primitives called geons (geometrical ions)—basic shapes such as cubes, cylinders, and cones. We have included these discussions in our supplementary (A.9).

[4] ecognition-by-components: a theory of human image understanding.

Thanks the authors for providing the detailed rebuttal! I think most of my concerns are properly addressed (extracting features from other layers, other foundation models like ConvNeXt, the feature map for hemisphere shape). However, for my most major concern of comparing with recent methods, although the authors mention that in the pose estimation task on OnePose-LowTex, the proposed method surpasses the state-of-the-arts, in many other cases (like still the pose estimation task on YCB-Video, and what Reviewer unL9 has mentioned on the tracking task), the gap between the proposed method and the state-of-the-art is still very noticeable from my point of view.

I do think the presentation and motivation of this paper is very excellent, as I find myself enjoyable reading through the whole paper, but the sub-optimal performance compared to state-of-the-arts (the performance gap is a bit large I think) prevents me from further raising my scores, so I think 6 is a reasonable score for now.

Q1: General 3D understanding and correspondence tasks relationship.

Thank you for this insightful observation. Correspondence estimation is a fundamental component of 3D vision understanding, underlying key tasks such as epipolar geometry, stereo vision for 3D reconstruction, and optical flow or tracking to describe the motion of a perceived 3D world. Stereo cameras, and even human perception, rely on disparity maps—effectively, correspondences between projected 3D parts to understand depth and spatial relationships.

The three tasks we evaluated—pose estimation, video tracking, and semantic correspondence—were intentionally selected to cover diverse aspects of correspondence estimation, ranging from simpler to more complex scenarios:

1. Pose Estimation examines correspondences within the same instance under rigid transformations (SE(3));

2. Video Tracking extends this to correspondences for the same instance under potential non-rigid or articulated transformations, such as humans or animals in motion;

3. Semantic Correspondence requires correspondences across different instances with similar semantics, often under arbitrary transformations.

An qualitative illustration of these three different correspondences is shown in this link. We've also included more discussion in Section 2.1.2 to clarify these distinction.

A key contribution of our work is demonstrating that finetuning models using a simple SE(3) correspondence setup during training enables them to generalize across all three tasks, i.e., correspondence types at test time. This result highlights the non-trivial ability of vision models to extrapolate learned multiview equivariance to more complex and diverse scenarios.

To address the reviewer's concern, we are open to revising the title to more explicitly reflect our focus on 3D correspondences, ensuring it better aligns with the scope of our experiments and contributions. Additionally, we have expanded the discussion in Section 2.1 to provide a deeper analysis of the relationship between these tasks and 3D correspondence. Please let us know if an updated title is needed.

Q2: Inclusion of video tracking and semantic correspondence, and how this work relates to prior works A and B.

Thank you for your observation. We recognize the need to clarify how the selected tasks—pose estimation, video tracking, and semantic correspondence—fit into the broader scope of 3D understanding and how our work relates to previous studies. As discussed in our response to the previous question, these tasks were chosen to evaluate different aspects of correspondence estimation, a critical capability in 3D vision.

Regarding prior works:

• [A] explored multiview geometry correspondences, similar to our evaluation in Section 2. However, their experiments were conducted on relatively small datasets, such as NAVI (36 objects) and ScanNet paired views (1500 test pairs). In contrast, we used a large-scale dataset with approximately 1M image pairs, enabling more robust conclusions on multiview correspondence. More importantly, while [A] focused on evaluation, we went one step further and proposed a simple and effective fine-tuning approach that enhances 3D correspondence capabilities in vision models, demonstrating its generalization across multiple downstream applications.
• [B] studied a related but distinct problem—whether global tokens (up-down representations) vary across different views. Their work explores a complementary area to ours, as they focus on view-dependent global features, whereas we emphasize dense, pixel-level features that are invariant to viewpoint changes. Our results highlight the utility of these dense features for a variety of applications. See the next question for more discussion about B.

We have expanded the discussion in the introduction and related work sections to better articulate the positioning of our work within the 3D awareness landscape and how it relates with prior studies.

Thank the authors for the response. I think the revision has improved the clarity of the paper.

One of my concerns aligns with the W2 of unL9. I agree that having 3D understanding would benefit tasks such as video tracking, e.g., in the case demonstrated in Figure 4, but in general I think the major challenges of video tracking are still truncation, occlusion, and changes in appearances, etc.

I also agree with the W1 of unL9. I think the results achieved by finetuning on synthetic 3D models are interesting, but more analyses are needed, given how widely pretrained DINO has been adopted in various tasks and settings while it remains unclear how the proposed finetuning would affect downstream tasks.

In general I acknowledge the contributions of this work. The presentation is improved with the revision and discussions about related works. I think some minor changes in the title would also help clarify the main focus of the paper.

Q1: Why one conv layer performs better than multiple layers.

Thank you for the suggestion. Upon analyzing the effect of additional convolutional layers, we found that while one additional convolutional layer significantly improves the performance, adding two or three layers introduces noise into the features. This noise likely arises from the increased parameter freedom, which can overfit to local patterns and reduce the consistency of dense pixel-wise features. To illustrate this, we included feature visualizations in this link. These visualizations clearly show that the additional layers produce less coherent features, leading to a degradation in downstream task performance. This analysis has been added to the revised supplementary (A.8.3) for clarity.

Q2: Mathematical formulas and explanations for APE, PCDP, and SmoothAP metrics.

Thank you for pointing this out. We agree that including mathematical formulations for key metrics and the loss function would enhance the clarity and followability of our paper. To address this, we have added the following formulations for APE, PCDP, and SmoothAP to the revised supplementary (A.1).

• Average Pixel Error (APE): Suppose we have $N$ objects, each rendered from $k=42$ different views. For a pixel $x_1$ in the first image, the ground-truth corresponding pixel $x_2$ in the second image is determined via back-projection into 3D and re-rendering, excluding occluded points. The evaluated method predicts $\tilde{x}_2$. APE is computed as:

PCDP=\sum_N\sum_i^k\sum_j^k\sum_{x_1\rightarrow x_2}\mathcal{1}(\frac{|x_2 - \tilde{x}_2|_2}{\min(W,H)} < \delta)

        SmoothAP=\frac{1}{S_P}\sum_{i\in S_P}\frac{1+\sum_{j\in S_P}\sigma(D_{ij})}{1+\sum_{j\in S_P}\sigma(D_{ij})+\sum_{j\in S_N}\sigma(D_{ij})}