MIXCON3D: SYNERGIZING MULTI-VIEW AND CROSS-MODAL CONTRASTIVE LEARNING FOR ENHANCING 3D REPRESENTATION

• Major: The proposed method is somewhat like some tricks that are incremental to me. The proposed joint alignment is very simple and makes sense, but some in-depth analysis is missing. For example, what is the synergy that is demonstrated? The paper claims that dense semantic information is within the 2D representations, while multi-view can further fortifying cross-modal alignment. The ablation studies is good, but it would be better if the authors could show some interesting and intuitive analysis that demonstrates what synergy is enhanced.
• Comparison: To the best of my knowledge, the proposed method is similar to GMC [1] and MCLEA [2], which also propose cross-modal contrastive learning between fused multimodal representations and single-modal representations, can authors explain the differences?
• Suggestions: i) It would be useful if the authors could provide the validation loss curves comparisons of different learning settings with baseline OpenShape to demonstrate the generalization improvements with the proposed techniques like joint representation alignment; ii) A visualization of learned association between 2D representation and 3D representation would be interesting.
• The improved training recipe is effective, but the insight behind this strategy is limited. It looks like a combination of useful tricks while a large amount of space is used to introduce these strategies. In this case, with all experiments showing benchmarking results, the current paper looks somewhat like an experimental report rather than a scientific paper.

1. Additional Comparisons: Your zero-shot classification experiments on the ModelNet40 and other datasets (Table 2) offer valuable insights. To further substantiate your findings, it could be beneficial to include comparisons with some additional methods that have shown promise in this domain, such as JM3D [1] and ULIP-2 [2]. Such comparisons could provide a more comprehensive view of how your approach stands in the context of the existing state-of-the-art methods and highlight the unique contributions of your work more effectively.

2. Clarification of Contribution: You have noted the introduction of multi-view images in the context of contrastive learning across multiple modalities as a key contribution to your paper. It is indeed an interesting aspect of your research. However, to strengthen your claim, it would be constructive to discuss how your approach differentiates from or builds upon prior works like JM3D [1] and ULIP-2 [2], which have also explored multi-view images within a similar framework. Clarifying this could help readers appreciate the novel aspects of your methodology and its significance in the field.

3. Expansion of Quantitative Experiments: The paper presents intriguing insights into the arithmetic of multi-modalities. To solidify these findings, it might be valuable to expand on the quantitative experiments. Additional data or more varied testing conditions could offer deeper validation of your hypothesis and might also uncover further interesting aspects of your model's behavior. This would not only bolster your current claims but could also provide a richer resource for readers interested in the intricacies of multi-modal contrastive learning.

[1] Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation, ACM MM 23

[2] ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding, arXiv: 2305.08275

1. Inequitable comparisons, MixCon3D employs ground truth rendered images as input data in zero-shot inference, which is certain to enhance its semantic comprehension performance.
2. Compared to ULIP&ULIP-2 [Liu et al., 2023], the improvements in the paper are incremental, with the sole modification being the introduction of $L^{(I, P)↔T}$.
3. The results for some methods in Table 2, such as ReCon, are inaccurate. The results of ScanObjectNN and ModelNet are reversed.

[Liu et al., 2023] ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding. arXiv preprint.

• Technical Novelty. Although this paper delivers impressive improvement in the area, the technical contribution is limited. The insights of this paper seem more focused on how to train a good model instead of solving a problem. For instance, in the introduction part, the authors wrote "synergy is underexplored, with each modality isolated in contrastive learning", after reading the whole paper, I still do not understand why this paper is able to perform multimodal synergy compared with ULIP, OpenShape, or other relevant methods. The difference is multi-view contrastive learning? How images and point clouds are synergized in the proposed method? Considering the method itself, multi-view projection between point cloud and images is directly adopted from existing methods, based on more visual clues, the proposed method offers improvement to baselines. Consequently, there are more constrastive pairs for learning objectives. However, I prefer this paper as a well-written engineering blog instead of an academic publication. And I sincerely appreciate the contributioins in this paper.

• Experimental Designs. An important ablation study. As shown in Table 4, with proposed training strategies, the method achieves 49.8% Acc. on the Objaverse-LVIS dataset. Introducing $\mathcal{L}^{T-I}$ holds back the performance by -0.9%. Why the loss function are not combined with multi-view projection? More specifically, why the performance of $\mathcal{L}^{T-I}$ and multi-view projection is not reported? In addition to this, the ablation seeting on the number of views projected to image is not ablated either. It's highly suggested to report the number of views on pretraining process. Is the optimal setting to project 4 views by 90 degress?

• Moreover, there are also shortcomings in this paper. The visualization results seem not supportive of the corresponding arguments. In Figure 3, the authors provide visualization results on text-point retrieval experiments. These results are top-3 ranking retrival results. For the left part, I do not think the proposed method significantly outperform OpenShape. Of cource, if the figure gets revised on top-10 or top-5 retrieval results, it would be more convincing.