Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations

Thank you for your kind acknowledgment of our work, especially for recognizing our model’s technical universality. We hope the responses provided address your concerns effectively.

W1: Modality Dependency Limitations

It is crucial to consider the potential lack of data from certain modalities in multi-modal joint learning. To address scenarios where images or calibration data may be missing, we ablate the image usage ratio of the whole datasets(ranging from 0% to 100%) in the pretraining stage in Table 5(b) of the paper. The results demonstrate Concerto's ability to adapt to point cloud data even in the absence of image input. Additionally, we include VGGT-lifted point clouds into our datasets, which makes it possible to directly use images to reconstruct without camera calibration. We provide more quantitative results of this variant:

We will include the results for this variant in the Appendix of the final version.

Moreover, Concerto is robust to data with some degree of imperfection, such as noisy camera parameters, which is an advantage of our method. We begin with point fusion and then learning, which means we perform intra-modal self-distillation and cross-modal joint-embedding prediction at the abstract level, between fused points and image patches, instead of points and pixels. This approach ensures that slight misalignments between points and pixels do not cause significant differences in the loss calculations. Just as humans can understand noisy point clouds, Concerto shows a similar capability.

W2: High Computational Cost

Similar to DINOv2, Concerto is a pretrained self-supervised learning model that alleviates the heavy burden of downstream task learning. High computational cost at pretraining stage leads to strong and generalizable representations, resulting in data efficiency in downstream tasks. For example, Table 2(b) shows the strong linear probing results with less than 0.2M training parameters, Table 2(c) highlights outstanding performance in data-scarce situations, and Table 9 of the Appendix presents efficient LoRA-based adaptation.

W3: Shallow Language Alignment

Thank you for pointing that out. This language alignment is a future expectation for next-level evaluation criterion for self-supervised learning beyond single modality, which means although the model is trained without texts, we hope it can form concepts akin to human language.

Since it is an evaluation protocol, we intentionally use a shallow language alignment with a linear layer to prevent other components, such as complex decoders, from dominating the final results. The detailed explanation behind this is discussed in the Pilot Study section of the paper.

Q1: Cross-Modal Generalization

Compared with the RGBD camera commonly used in indoor scenes, LiDAR + camera needs standard spatial-temporal synchronization, including same-time triggering (temporal) and intrinsic and extrinsic calibration(spatial). Our methods need no additional calibration.

Q2: Long-Term Tracking Performance

In our work, we use VGGT to lift videos from the RE10K dataset. The videos in RE10K are divided into short sequences, and VGGT effectively handles these sequences for point cloud generation. Although sometimes the feed-forward reconstruction produces failure cases, Concerto shows robustness when training on these noisy data. We also look forward to further improvements in feed-forward reconstruction methods and will release our code for generating this data.

Q3: Lightweight Potential

Thank you for your suggestion! We recognize the demand for lightweight models suitable for mobile devices. Currently, we have models of varying scales, and we plan to release these versions in the future. Below are the quantitative results for these models using linear probing:

L1 and L2:

Please refer to our response to W1 and W3 for detailed clarification.

L3: Ecological Dependence

We conducted experiments on different 2D encoders, like SigLIP2 [1] and RADIOv2.5 [2], as Table 8 in the Appendix of our paper. The results show that the DINOv2 variant has the strongest performance, while the other encoders show comparable numerical performance.

The choice of DINOv2 as the 2D encoder in our pipeline is based on its strong performance as a self-supervised model. We observed that different modal representations from self-supervised learning benefit the 3D intra-modal self-distillation significantly. This is analogous to the human cognition process: the learning for one single modality is like an intra-modal self-supervised learning process, while humans learn concepts through multi-modal inputs that interact in the system and result in superior recognition of the world. By integrating self-supervised features from two different modalities in Concerto, it emerges more reliable representations than single modal representations and their naive concatenation.

[1] Tschannen, et al. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv:2502.14786 (2025).

[2] Heinrich, et al. RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models. CVPR 2025

Thanks for your acknowledgement of our work, especially your recognition of our data-efficient and ablation results. And we hope your concerns are well addressed below.

W1: Clarity of Writing

• Redundancy

  We will polish our paper to make it more concise and cohesive. We would also welcome more specific feedback.

• Lack of Formal Definitions

  Yes, some math formulas will help readers understand. Here we provide the mathematical presentation of our method:

  As shown in Fig. 3 of the paper, we obtain point representations $s_x$ and image representations $s_y$ from the point encoder and image encoder respectively, as

  $s_x = \Psi_{\text{pcd}}(x),$ $s_y = \Phi_{\text{img}}(y).$

  With camera parameters $z$ as conditions, the correspondence between the points and pixels can be captured, which is further explained in Appendix A.2. First, the intrinsic matrix $K$ and extrinsic matrix $[R|t]$, derived from $z$, are used to project 3D point coordinates $p_x$ to the 2D pixel coordinates $p_y$:

  $p_y = K[R|t] p_x.$

  Then, for each image patch $j$, we compute the mean of the features of points falling within it using

  $\hat{s}\_{y,j} = \frac{1}{|\mathcal{N}\_j|} \sum_{i \in \mathcal{N}\_j} s\_{x,i},$   where $\mathcal{N}\_j$ is the set of indices of points whose projected coordinates $p_y$ fall within the $j$-th image patch, and $|\mathcal{N}\_j|$ is the number of such points. $s\_{x,i}$ is the representation of the $i$-th point. Then, we compute the loss $D(s_y, \hat{s}_y)$ using cosine similarity

  $D(s_y, \hat{s}_y) = \left\langle \frac{s_y}{\||s_y\||_2}, \frac{\hat{s}_y}{\||\hat{s}_y\||_2} \right\rangle$

  between representations of point clouds and image patches. This process introduces 2D self-supervised features into 3D self-supervised learning and stimulates the point intra-modal self-distillation process.

• Confusing Statement

  Please refer to our responses to Q2 and Q3 for clarification.

W2: Figure 3 Improvement

We will improve Figure 3 to present our method more clearly. As mentioned in W1, we will include a more detailed version of the formulas to complement the figure.

W3: Further Explanation for Intra-Modal Self-Distillation Section

We agree that introducing the preliminary work is important for readers, especially those unfamiliar with it. Here’s a more detailed explanation:

In Intra-Modal Self-Distillation, we build upon the principles of multi-view consistency and masked reconstruction, illustrated in Part (a) of Figure 3. The main challenge in 3D point cloud self-supervised learning is the geometry shortcut, which means networks tend to focus on sparse point cloud geometry and neglect the rich semantic information, such as color and texture. To mitigate this, it is important to make the geometry clues harder for the network to capture. This approach is discussed in detail in the Sonata paper.  In addition to applying masking, cropping, and view rolling techniques, we also adopt the encoder-only structure and feature upcasting strategy from Sonata to reduce the over-reliance on local geometry and incorporate multi-scale features. This preliminary step ensures the point encoder refines its representations with richer semantic information.

We will revise this section, provide a detailed explanation in the Appendix, and refer readers to the Sonata [1] paper for more information.

W4: Zero-Shot Segmentation

Thank you for the suggestion to further improve our work. ConceptFusion [2] and FollowAnything [3] are both remarkable contributions to zero-shot semantic segmentation using existing foundation models(e.g., SAM, DINO, and CLIP) and providing various interfaces for users. However, we focus on building such a foundation model in the 3D domain, like DINOv2 in the 2D domain. The absence of such a 3D foundation model blocks the progress in improving downstream tasks. We acknowledge this as a key area for future work and will continue to push in this direction.

Additionally, we have noticed that evaluation datasets like unCoCo, used in ConceptFusion and FollowAnything, are currently not available. However, we provide a comparison between our method and ConceptFusion on ScanNet, as mentioned in W6.

W5: Long-Tail Concepts

We agree that long-tail concepts are crucial for evaluating the generalization ability of model. Although the unCoCo dataset is currently unavailable, we believe that the results on ScanNet200 in our paper can offer insights into the model's performance on long-tail concepts. ScanNet200 includes more rare classes than ScanNet, and our model achieves 37.4% mIoU on ScanNet200 using linear probing, compared to the previous SOTA of 29.3%. This improvement demonstrates Concerto's effectiveness in handling long-tail objects.

W6: Comparison to ConceptFusion

We followed the evaluation protocol from ConceptFusion, using the same subset and object classes without background concepts. Below is a comparison between Concerto and ConceptFusion on ScanNet Val:

However, the main focus of Concerto is point cloud representation learning. The language probing serves as an auxiliary evaluation method to demonstrate the potential of our representations. This shows that even without explicit text information, the representations learned by our method can inherently capture conceptual features related to language. With simple alignment, these representations can be transformed into text space. We intentionally use shallow language alignment to avoid complex components, such as decoders, dominating the evaluation results.

For W4-W6, we will include the related results and discussion in our final version.

Q1: Lines 92–94 (Sentence Starting with "Ideally")

This sentence refers back to Figure 2, which illustrates the "Apple" concept in cognition. Humans are exposed to multi-modal information from birth, and once they learn the concept of an apple, they can recall its taste when they see a picture of it. We aim to express a similar idea here. Specifically, while the model is trained using multi-modal joint representation learning, it is capable of outputting representations derived from 2D and 3D information, even when the input is a single modality, such as a 3D point cloud, during inference. This means a lot in real-world applications, because multi-modal input is not always available for a system.

We will rewrite this part in the final version to erase the confusion.

Q2: Lines 112–113 – Clarification

This paragraph describes our future expectation for a next-level evaluation criterion for self-supervised learning beyond single modality, which means that although the model is trained without texts, it can simultaneously form concepts akin to human language. Just as humans first learn concepts from the world before using language to express them, a strong self-supervised model should be able to form its own conceptual "language". This property can be validated during evaluation on text alignment, where the model’s representations can be aligned with the human semantic space through a simple transformation, such as a single linear layer. As mentioned in the W6, aligning representations with text is not the main contribution of this work but rather an auxiliary evaluation method to validate the quality of Concerto's representations, complementing the semantic segmentation tasks.

Then, come to the question. We here choose the CLIP text space to validate our representations. We align our point cloud representations with the image embeddings from LSeg, using the image as a bridge to connect our 3D point features with text embeddings. If any other text feature extractor is aligned with image pixels, Concerto’s representations can be transformed into that text space as well. The loss function used for alignment is cosine similarity. We freeze the pretrained backbone, add a linear layer on top, and align the output with the image embeddings from the LSeg image encoder.

Q3: DINOV2 as a Target – Clarification

Our model does not aim to simply copy 2D features from DINOv2, which ignores 3D information. Instead, we emphasize the importance of interaction between modalities. This leads to entirely new representations derived from different domains. The core of our method is the stimulation from the 2D modality to enhance intra-modal self-distillation in the 3D domain. As shown in Table 1 of the paper, our method outperforms each single-modality approach as well as their naive concatenation. This demonstrates that, rather than merely combining information from two modalities, Concerto captures richer representations through the interaction between 2D and 3D information.

Q4: Lines 145–146 – Ambiguity

Apologies for the ambiguity. What we meant to convey is that a scene with multiple images (>>4) is divided into several data pieces, where each piece consists of one point cloud and 4 images. We will revise this section in the final version to ensure better clarity.

[1] Wu, et al. Sonata: Self-Supervised Learning of Reliable Point Representations. CVPR 2025.

[2] Jatavallabhula, et al. ConceptFusion: Open-set multimodal 3d mapping. RSS 2023

[3] Maalouf, et al. Follow Anything: Open-set detection, tracking, and following in real-time RA-L 2024

Thanks for your suggestions. We are sincerely grateful for your efforts and feedback. We hope the concerns are well addressed.

W1: Comparisons with Other Methods on Spatial Understanding with Joint 2D-3D Data

Thanks for your suggestion. In response, we’ve included comparisons between Concerto and the methods you mentioned on spatial representation learning:

Large Spatial Model [1], PonderV2 [2], and FiT3D [3] represent different categories of joint 2D-3D learning, such as lifting projections, differentiable rendering, and direct distillation. For GGSD [4], since it focuses on open-vocabulary tasks using additional text and superpoint information, which is outside the scope of our current work, it is not included in this comparison. We will continue to extend our work to other downstream tasks in the future.

W2: Synergy from the 2D and 3D

Thank you for your feedback on our use of the term "synergy". To address your concern, we provide results on ScanNet where DINOv2 is fine-tuned with Concerto, showing an improvement in 2D tasks compared to the original DINOv2. This demonstrates that fine-tuning the 2D model alongside the 3D model enhances its performance on 2D tasks.

Here, when we refer to "synergy," we mean "1 + 1 > 2": with joint learning, our method allows for stronger and more reliable representations than what can be achieved with single modalities or the naive concatenation of 2D and 3D features. This comparison can be seen in Table 1 of our paper.

We acknowledge that our initial expression may have caused some misunderstanding about the nature of our method. We will revise the relevant section in the final version to make this clearer and will include the fine-tuning results for DINOv2 in the Appendix.

W3: 3D Intra-Model Self-Distillation as Preliminary

Thanks for your suggestion. The Intra-Modal Self-Distillation part is not a new contribution of this work, but rather a preliminary step. We introduced it here for clarity and to help readers better understand the foundational aspects of our method. A preliminary introduction is necessary for those who are not familiar with previous works. The other Reviewer, yoVm, also thinks this preliminary should be explained in more detail. In the final version, we will explicitly clarify that the intra-modal self-distillation is a preparatory step and not the primary contribution of our work. Our main contribution lies in the Cross-Modal Joint Embedding Prediction and its integration with self-distillation.

W4: Human Concept Learning in Cognition

Thank you for your advice. We will revise the explanation of human concept learning to improve clarity and the overall flow of the manuscript.

We introduce the concept of human cognition to help readers better understand the awareness in self-supervised learning with the other modality. Humans are exposed to multi-sensory information from birth, and once they learn a concept, such as an "apple," they can recall its taste when they see the fruit. In real-world applications, however, multi-sensory information is not often available to a system. This raises the question: How can we generate multi-sensory representations from a single modality, much like how humans recall information from different senses in their mind?

Our goal is to present a model that can produce representations derived from both 2D and 3D features with only point cloud input, mimicking the process by which humans recall information from other sensory modalities. This is different from the case where one model receives multi-modal input and outputs corresponding responses.

W5: Terms for DINOv2-Reg

Thank you for pointing this out. We will update the term DINOv2-Reg [5] in the final version to ensure consistency and clarity, and include the reference.

Q1-3:

Thank you for your questions. For detailed responses, please refer to the corresponding sections addressed in the previous  W1-3.

Q4: Concerto Variant Specifically for Videos

We appreciate your suggestions. To address the data limitations hindering progress in 3D self-supervised learning, we explore the use of abundant online video data. By employing feed-forward methods like VGGT, we can enlarge our point cloud datasets. To demonstrate the potential of this approach, we introduce a separate variant of Concerto that incorporates this additional video data.

For quantitative results of VGGT-lifted video data, we provide the results using the base model on ScanNet and ScanNet++:

Below, we provide mIoU evaluation results on ScanNet Val for different data scales using linear and decoder probing, showing scaling performance on video data. Here, 87k refers to 40k original point clouds and 47k VGGT-lifted point clouds.

The table above shows that, with additional video data, Concerto can further improve the performance. We believe that with the progress of feed-forward methods, the point cloud self-supervised learning has the potential to scale up to a larger level and finally result in a more reliable and robust spatial representation model. We will append these results to the Appendix in the final version.

[1] Fan et al. Large Spatial Model: End-to-end Unposed Images to Semantic 3D. NeurIPS 2024.

[2] Zhu et al. PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm. T-PAMI 2025

[3] Yue et al. Improving 2D Feature Representations by 3D-Aware Fine-Tuning. ECCV 2024.

[4] Wang et al. Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation. ECCV 2024.

[5] Darcet et al. Vision Transformers Need Registers. ICLR 2024.

Thanks for your acknowledgment of our work! Below, we provide detailed responses to your questions and comments. Hopefully, they will further clarify our approach.

W1: Clarity of Writing

• Clarity of Writing

  Thanks for your suggestion. We agree that adding math formulas will make the process clearer for readers, and we will revise this section in the final version.

  As shown in Fig. 3 of the paper, we respectively obtain point representations $s_x$ and image representations $s_y$ from the point encoder and image encoder, as

  $s_x = \Psi_{\text{pcd}}(x),$ $s_y = \Phi_{\text{img}}(y).$

  With camera parameters $z$ as conditions, the correspondence between the points and pixels can be captured, which is further explained in Appendix A.2. First, the intrinsic matrix $K$ and extrinsic matrix $[R|t]$, derived from $z$, are used to project 3D point coordinates $p_x$ to the 2D pixel coordinates $p_y$:

  $p_y = K[R|t] p_x.$

  Then, for each image patch $j$, we compute the mean of the features of points falling within it, as

  $\hat{s}\_{y,j} = \frac{1}{|\mathcal{N}\_j|} \sum_{i \in \mathcal{N}\_j} s_{x,i},$

  where $\mathcal{N}\_j$ is the set of indices of points whose projected coordinates $p_y$ fall within the $j$-th image patch, and $|\mathcal{N}\_j|$ is the number of such points. $s\_{x,i}$ is the representation of the $i$-th point. Then, we compute the loss $D(s_y, \hat{s}_y)$ using cosine similarity

  $D(s_y, \hat{s}_y) = \left\langle \frac{s_y}{\||s_y\||_2}, \frac{\hat{s}_y}{\||\hat{s}_y\||_2} \right\rangle$

  between representations of point clouds and image patches. This process introduces 2D self-supervised features into 3D self-supervised learning and stimulates the point intra-modal self-distillation process.

• Definition of "Image Usage"

  The term "Image Usage" refers to the ratio of the total number of images used during the pretraining process. If the Image Usage is 50%, it means we randomly selected 50% of the data with images, while the remaining is pure point cloud data. We will clarify this definition in the final version of our paper.

• Visible Points

  The control of visible points is crucial in the self-supervised learning process. The model needs to capture local cues to predict surrounding representations. Too many visible points can diminish this exploration: as training progresses, random views of the same data are similar, which reduces diversity in the model’s learning. Conversely, too few visible points make the task unmanageable. The value of 65536 is chosen experimentally by visualizing randomly selected views from the point clouds.

  Fully controlling the visible points for each data instance is impractical, as points in the point cloud do not have a one-to-one correspondence with pixels in the image. The chosen number of visible points (65536) is the maximum that ensures the model focuses on different regions of the data at an appropriate scale. Typically, the number of visible points falls within a user-defined proportional range of the total point count. Any count exceeding 65536 is capped at this maximum value.

  We will update the related section in the final version for better clarity.

• Linear Probing and Decoder Probing

  Linear Probing: In this approach, the pretrained encoder remains frozen. We upcast the features to their original resolution and adapt them to downstream tasks using a single linear layer, which accounts for less than 0.2% of the model’s total parameters.

  Decoder Probing: For this method, we use a lightweight, hierarchical decoder that comprises 13% of the total parameters. The encoder is frozen, and we train only the decoder, allowing it to learn task-specific representations.

  We will add this explanation to the final version. As these methods are described in detail in the Sonata paper, we will also refer readers to the original Sonata [1] work for further clarification.

W2&3: Data Scaling and Video-Lifted Data Potential

Thanks for your suggestion. Currently, 3D self-supervised learning faces the challenge of limited data, which restricts further development. To address this, we explore leveraging large-scale video data through feed-forward methods like VGGT to expand the dataset. Below, we provide evaluation results (mIoU) on ScanNet Val using linear and decoder probing for different data scales:

Here, 87k means 40k original point clouds and 47k VGGT-lifted point clouds. For quantitative results of VGGT-lifted video data, we provide the mIoU results with base model size on ScanNet and ScanNet++:

The table above shows that, with additional video data, Concerto can further improve the performance. As for model size, it should be increased with the data scale. Currently, the model is still in the stage of being data hungry. Therefore, we don't scale up our model size to a giant level but provide a tiny version for robotic downstream tasks and a large model trained with additional video data here. All the models are evaluated using linear probing with mIoU metric.

We believe that, as feed-forward methods continue to improve, point cloud training will scale more effectively, leading to more robust and reliable spatial representations. We will append these scaling results in the appendix of the final version and release a HuggingFace demo and website with more visualizations showcasing performance on VGGT-lifted point clouds.

Q1: Does batch size matter in the cross-modal training?

While batch size is an important parameter in self-supervised learning, Concerto shows strong adaptability to different batch sizes during pretraining. The following table presents results across various batch sizes on different benchmarks:

Q2: Can the method work with noisy camera parameters? Will it be sensitive if depth, camera poses, or intrinsics are inaccurate?

Yes, our method is robust to noisy camera parameters, which is one of its advantages. In our approach, we do fusion first and then learning. This allows minor misalignments between points and pixels without significantly impacting the loss calculations. Just as humans can understand noisy point clouds, Concerto shows a similar capability. Current feed-forward reconstruction methods may not always produce accurate point clouds and camera parameters, but Concerto shows strong adaptability to such noisy point clouds. Notably, we avoid cherry-picking video examples in our ongoing demos and paper, further proving the robustness.

[1] Wu et al. Sonata: Self-Supervised Learning of Reliable Point Representations. CVPR 2025.