UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting

Thank you for your detailed and careful feedback on our paper. We are pleased that you recognize the contributions of UniGS on the introduction of 3D Gaussian primitives for 3D-oriented topics. We are also glad to hear that you appreciate the novelty and effectiveness of UniGS. We will address your concerns point by point:

• W1, Q1(Comparisons to NeRF-based approaches) : "the paper does not provide comparisons with other relevant works ... These works utilize Neural Radiance Fields (NeRF)"

Table 1: Zero-shot classification on ShapeNetRender. Avg.: the mean average Top1 classification accuracy. Uni3D and UniGS are trained for 15 epochs on ShapeNetRender.

Comparisons to NeRF-based approaches: Thank you for the suggestion. We have now conducted additional comparisons with NeRF-based approaches. As shown in Table 1, UniGS outperforms nerf2clip[1] and nf2vec[2] with 9.93% and 6.64%, demonstrating significant improvement over NeRF-based approaches on cross-modalities learning.

Why not compared with NeRF-based approaches in the main paper: When utilizing point clouds for 3D representation, a significant challenge arises from the discrepancy between the 3D representation and other modalities. While NeRF can achieve pixel alignment with provided images, it has several drawbacks: its implicit representation is not as advantageous for 3D tasks as the explicit representation of 3DGS. Additionally, NeRF optimization is notably slow and demands a substantial number of viewpoints. In contrast, as illustrated in Table 5 of the main paper, 3DGS showcases potential compatibility with point clouds, offering promise for practical applications in real-world scenarios.

Summary: Thank you very much for your suggestions. As the additional experiments demonstrated, our framework achieves significant improvements with 3DGS representation over NeRF. We have incorporated the NeRF experiments and discussions into the paper to strengthen the persuasiveness and completeness of our work.

[1] Connecting NeRFs, Images, and Text

[2] Deep learning on 3D neural fields.

• Q2(Evaluation of Computational Time) : "3DGS ... requires an additional optimization process that point clouds do not, as point clouds represent raw data directly from 3D sensors"

Table 2: Comparisons of forward computational cost on Objaverse-Lvis.

Runtime analysis:Thank you for the comment. As shown in the Table 2, we further evaluate the FLOPs and runtime of UniGS and compare them with state-of-the-art approaches. With a slight increase in runtime, UniGS achieves significant improvement over ${CLIP}^{2}$, TAMM, and Uni3D on Objaverse-Lvis zero-shot classification.

Table 3: Ablation study on the FLOPs of UniGS modules. CNN encoder denotes the CNN layers to extract spatial information from 3D representation into features, and ViT blocks denotes the Transformer blocks understanding objects from extracted features. Cross-Attn denotes the Cross-attention layers between Fundamental and Advanced Encoder.

Moreover, as shown in the Table 3 and Figure 7 in Sec. J of the appendix, we further conduct an in-depth evaluation of the FLOPs for our UniGS framework, which provides a better understanding of the increased computational cost. Specifically, 76.5% of the total FLOPs (73.38G) is costed in the CNN layers of 3D Encoder to extract 3D spatial features for understanding. This is indeed a limitation of our method's general application. Fortunately, much progress is being made in compressing models [1] and 3D representations [2], and we expect these advances to facilitate the development of 3D understanding with 3DGS representation.

Time analysis of 3DGS optimization: Indeed, 3DGS involves additional time for optimization and the details of the required 3DGS optimization cost are discussed in Line at line 740 in the main paper. Specifically, preparing 800k 3DGS objects with 1024 3D Gaussian kernels takes a week to prepare, while 800k 3DGS objects with 10000 3D Gaussian kernels takes about 12 days. The preparation of 3DGS datasets demands significant efforts in terms of time and computational power. We will therefore make all prepared datasets publicly available to the community to support further advancements in 3DGS representation learning.

Moreover, leveraging image-to-3DGS approaches for dataset preparation is another promising step. For example, GS-LRM [3] and PF-LRM [4] takes only 0.23 and 1.3 seconds from 2-4 posed sparse images to generate 3DGS, respectively. Although recent advances have been made on image-to-3DGS [3,4], they are unfortunately still not open-sourced now. Given the popularity of 3DGS, we expect these representations to only become more and more efficient to construct.

[1] T3DNet: Compressing Point Cloud Models for Lightweight 3D Recognition

[2] LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS

[3] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting

[4] PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction

• W2,Q3(Initialization with Point Clouds) : "in some datasets (such as ABO in Table 2), the performance of 3DGS falls short compared to point cloud methods"

Why initialized with Point Clouds: As shown in Figure 4 and 5 in Sec. B of the appendix, we considered three common optimization settings of 3DGS for the ablation of the initialization of 3DGS: (1) flexible (ours): load surface points as initialization with flexibility of 3DGS location, (2) fixed: load surface points as initialization with fixed 3DGS location, (3) original: the vanilla optimization from 3DGS.

As shown in the results in Figure 4 and 5 in Sec. B of the appendix, pipelines that load surface points for initialization achieve better results over the vanilla optimization from 3DGS. Moreover, our "flexible" pipeline shows significant improvement with flexibility of 3DGS instead of fixing the location of 3DGS. Therefore, UniGS leverages the "flexible" pipeline for data preparation, which means 3DGS will learn the most representative locations of an object, rather than manually selected fixed points on the surface of the object.

Why Uni3D with 3DGS falls short compared to point cloud: Since the spatial information of 3DGS is not as regular as that of point clouds, which are typically located on the surface, it presents challenges to learning 3DGS features for 3D understanding. However, the 3D Encoder of Uni3D is designed for point clouds, which is not suitable for 3DGS and may deteriorate performance.

Therefore, we proposed Gaussian-Aware Guidance for UniGS to better understand the 3DGS features with the spatial query from the fundamental encoder. As shown in the Table 2 and 6 of the main paper, our UniGS without the spatial query from cross attention gets only 37.58% Top1 accuracy on ABO, which is close to the 37.79% Top 1 accuracy that we train Uni3D from scratch on 3DGS, demonstrating the effectiveness of our proposed Gaussian-Aware Guidance.

• W2, Q4(Clarification on Baseline Performance and Experimental Settings) : "how does the experimental setting of the results reported in your paper... differ from the Objaverse-LVIS zero-shot classification performance reported in Uni3D"

Table 4: Comparison results with 10000 points dataset on Objaverse-Lvis zero-shot classification.

Comparisons under the setting of Uni3D: Thank you for the suggestion. For comprehensive comparisons, we conduct further comparisons with the same setting of Uni3D and reformulate experiment results in Table 4. Under the same setting of Uni3D, UniGS still outperforms Uni3D on Objaverse-Lvis zero-shot classification.

Performance of the baseline models: UniGS outperform Uni3D under a fair setting of representing object with 1024 3D points and training with CLIP ViT-B-16. In the following, we briefly elaborate on the performance gap in Top 1 performance:

(1) model type: reported Top 1 performance of Uni3D uses the 1 billion parameter model (Uni3d-g) for better results while we only use the small version for consideration of over-fitting. The Top 1 accuracy of Uni3d-s on Objaverse-LVIS reduces to 50.34.

(2) training mode: we conduct experiments on Zero-shot classification, which means the test set will not be used in training, corresponding to the not ensemble mode of Uni3D. When not using the ensemble mode and leveraging the smaller model, the Top 1 accuracy of Uni3D-S on Objaverse-LVIS comes to 44.81.

(3) the number of points: we set the number of 3DGS for each object to 1024, while Uni3D is trained on 10000 3D point clouds, which leads to an additional gap in performance. Combined with points (1) and (2), if we directly evaluate Uni3D with 1024 point clouds on Objaverse-LVIS, the Top 1 accuracy comes down to only 33.61. For our reorganized Objaverse-Lvis, it increases to 38.17.

Thank you for your careful and insightful comments. We are glad to hear that you appreciate the novelty of our proposed UniGS for multi-modal pretraining. In the following, we will address your concerns in detail:

• W1(Benefit of 3DGS over point clouds) : "what numbers do you get in table 3 if you use the architecture used in UniGS but replacing the GS with point clouds?"

Table 1: Recognition on SUN RGBD. Avg.: the mean average Top1 accuracy across all categories. * denotes training from scratch.

Thank you for your suggestion. We have conducted additional experiments in Table 1. As shown in Table 1, UniGS shows superior performance with 3DGS representation and proposed Gaussian-Aware Guidance over common point clouds, while Uni3D deteriorates due to lack of 3DGS encoder. Moreover, UniGS with point clouds still outperforms Uni3D with point clouds, demonstrating the effectiveness of UniGS in modeling explicit features of color, opacity, scale, and rotation.

• W2,W3,Q5(Visual comparisons) : "It would be helpful to show some visual illustrations. You could show examples if 3D retrieval when the query is a 2D image"

Thank you for the suggestion. We visualize several examples of 3D retrieval with 2D image query.

As shown in Figure 8 in Sec. J of the appendix, Uni3D may mistakenly retrieve another similar object due to the similarity in point cloud structure. In contrast, UniGS demonstrates a superior 3D understanding of object color, shape, and texture with 3DGS representation and proposed Gaussian-Aware Guidance, resulting in better Image-to-3D retrieval.

• Q1(Figure 2 refinement) : "Caption in figure 2 could provide some details to help interpret the system sketch"

Thank you for the suggestion. We have refined Figure 2 in the main paper with the description of the information flow.

• Q2,Q3,Q4(Writing)

Thank you for your careful comment. We highlight the pertaining with Language-Image-3D at line 32 to demonstrate the pertaining strategy of UniGS. The global position and features at denotes the global understanding through the position and color features of point clouds, respectively. We will clarify these typos to make it more clear.

• Q4(Figure 3 refinement) : "It will be helpful if figure 3 had the same notation than the one used in section 3.4"

Thank you for your careful comment. As illustrated in equations 7 and 8, $f_{fun}$ and $f_{adv}$ denote the input of the fundamental encoder and advanced encoder, respectively. We add a single quote for $f_{fun}'$ and $f_{adv}'$ to represent the features of 3D understanding from the fundamental encoder and advanced encoder. We have added the same notation to Figure 3 of the main paper for the completeness of our work.

Thank you once again for taking the time to review our paper and for providing valuable comments to enhance its quality. We thank you for your constructive feedback, which has significantly contributed to the improvement of our work. Specifically, your comments guided us to enhance the visualizations throughout the paper, add more informative content to Figure 2, and include corresponding symbols in Figure 3 to better align with the equations. Additionally, your careful review helped us identify and correct typos in the manuscript, ensuring greater clarity and precision. Finally, your suggestion to conduct further experiments on SUN RGBD dataset has enabled us to provide a more comprehensive evaluation. Your thoughtful contributions have been invaluable in refining and strengthening our work.

We sincerely hope that the additional experiments and our response have addressed your concerns. If you have any further questions or suggestions, we would be glad to offer further clarifications. Thank you for your time and consideration!

Thank you for your thoughtful comments and positive recognition of our solid motivation and intuitive and nicely formulated pipeline. We are also glad that you appreciate the effectiveness of UniGS and the novelty of our Gaussian-Aware Guidance. We will address your concerns point by point:

• W1(Figure refinement) : "Figure 2 could provide an overall description of the information flow ... in the caption"

Thank you for the suggestion. We have now refined Figure 2 in the main paper with a description of the information flow.

• Q1(Comparisons to Depth- and NeRF-based approaches) : "the paper discusses the weakness of using point clouds as a 3D presentation due to the discrete representation. How about ... depth maps or NeRF?"

Table 3: Zero-shot classification on ShapeNetRender. Avg.: the mean average Top1 classification accuracy. Uni3D and UniGS are trained for 15 epochs on ShapeNetRender.

Table 4: Recognition on SUN RGBD. Avg.: The mean average Top1 accuracy across all categories. * denotes training from scratch.

Thank you for your suggestion. We have conducted additional comparisons to NeRF-based[1,2] and Depth-based[3,4] approaches. As shown in Table 3, UniGS outperforms nerf2clip[1] and nf2vec[2] with 9.93% and 6.64%, demonstrating significant improvement over NeRF-based approaches on cross-modalities learning.

Moreover, we supply comparisons to Depth-based approaches and reformulate the results in Table 4. As illustrated in Table 4, UniGS significantly outperforms PointCLIP[3] and Clip2Point[4] on the SUN RGBD datasets with over 31.64% and 12.74%, demonstrating the effectiveness of the 3DGS representation.

[1] Connecting NeRFs, Images, and Text

[2] Deep learning on 3D neural fields.

[3] Learning transferable visual models from natural language supervision

[4] Clip2point: Transfer clip to point cloud classification with image-depth pre-training

• W2(Computational cost) : "It seems that this paper does not provide the computation cost for inference"

Table 1: Comparisons of forward computational cost on Objaverse-Lvis.

Runtime analysis: Thank you for the comment. We have added an evaluation of the FLOPs and runtime of UniGS and compared them with state-of-the-art approaches. In the results in Table 1, observe that with a slight increase in runtime, UniGS achieves significant improvement over ${CLIP}^{2}$, TAMM, and Uni3D on Objaverse-Lvis zero-shot classification.

Table 2: Ablation study on the FLOPs of UniGS modules. CNN encoder denotes the CNN layers to extract spatial information from 3D representation into features, and ViT blocks denotes the Transformer blocks understanding objects from extracted features. Cross-Attn denotes the Cross-attention layers between Fundamental and Advanced Encoder.

Moreover, as shown in the Table 2 and Figure 7 in Sec. J of the appendix, we further conduct an in-depth evaluation of the FLOPs for our UniGS framework, which provides a better understanding of the increased computational cost. Specifically, 76.5% of the total FLOPs (73.38G) is spent in the CNN layers of the 3D Encoder to extract 3D spatial features for understanding. This is indeed a limitation of our method's general application. Fortunately, much progress is being made in compressing models [1] and 3D representations [2], and we expect these advances to facilitate the development of 3D understanding with 3DGS representation.

Time analysis of 3DGS optimization: Indeed, 3DGS involves additional time for optimization and the details of the required 3DGS optimization cost are discussed in Line at line 740 in the main paper. Specifically, preparing 800k 3DGS objects with 1024 3D Gaussian kernels takes a week to prepare, while 800k 3DGS objects with 10000 3D Gaussian kernels takes about 12 days. The preparation of 3DGS datasets demands significant efforts in terms of time and computational power. We will therefore make all prepared datasets publicly available to the community to support further advancements in 3DGS representation learning.

Moreover, leveraging image-to-3DGS approaches for dataset preparation is another promising step. For example, GS-LRM [3] and PF-LRM [4] takes only 0.23 and 1.3 seconds from 2-4 posed sparse images to generate 3DGS, respectively. Although recent advances have been made on image-to-3DGS [3,4], they are unfortunately still not open-sourced now. Given the popularity of 3DGS, we expect these representations to only become more and more efficient to construct.

Compatability with raw data representations: As illustrated in Table 5 of the main paper, 3DGS showcases potential compatibility with point clouds, such that the proposed approach can leverage both the processed 3DGS representations jointly with the raw data representations, offering promise for practical applications in real-world scenarios.

[1] T3DNet: Compressing Point Cloud Models for Lightweight 3D Recognition

[2] LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS

[3] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting

[4] PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction

Thank you once again for taking the time to review our paper and for providing valuable comments to enhance its quality. We appreciate your thoughtful comments, which have guided us in improving our work. Specifically, we have refined Figure 2 by incorporating information flow to improve clarity, added comparisons with NeRF- and Depth-based methods to enhance the completeness of the paper, and conducted additional experiments on computational cost and ablation studies to explore the potential for future applications in lightweight scenarios. Your insightful suggestions have been invaluable in helping us address these aspects comprehensively.

We sincerely hope that the additional experiments and our response have addressed your concerns. If you have any further questions or suggestions, we would be glad to offer further clarifications. Thank you for your time and consideration!

• W3,W4(Comparison with Uni3D under the same setting) : "it is better to use the similar settings as Uni3D for experiments."

Table 3: Comparison results with 10000 points dataset on Objaverse-Lvis zero-shot classification.

Thank you for the suggestion. UniGS outperforms Uni3D under a fair setting of representing objects with 1024 3D points and training with CLIP ViT-B-16. For comprehensive comparisons, we conduct further comparisons with the same setting of Uni3D and present the experiment results in Table 3. Under the same setting of Uni3D with a higher number of 3D gaussian kernels representing objects, UniGS still outperform Uni3D on Objaverse-Lvis zero-shot classification.

Thank you for your careful and insightful comments. We are glad to hear that you appreciate the effectiveness of our proposed UniGS for multi-modal pretraining. In the following, we will address your concerns in detail:

• W1(Advantage of 3DGS) : "What is the main advantage of proposed method using 3DGS"

Thank you for the comment. As shown in Figure 6 in Sec. J of the appendix, we highlight the difference between Uni3D and UniGS. Our UniGS is also capable of utilizing a single model to unify 3D representations from different models, with better performance with 3DGS representation and proposed Gaussian-Aware Guidance. Specifically, when using point clouds as a unified 3D representation, the main challenge is the divergence between the 3D representation and other modalities. In contrast, UniGS leverages 3DGS as the 3D representation, which effectively reconstructs the 3D target object as well as provides efficient correspondences between 3D and 2D images.

Through extensive experiments across the Objaverse, ABO, MVImgNet and SUN RGBD datasets with zero-shot classification, text-driven retrieval, and open-world understanding tasks, we demonstrate the effectiveness of UniGS in learning a more general and stronger aligned multi-modal representation.

• W2(Computational cost) : "Does it increase computational cost."

Table 1: Comparisons of forward computational cost on Objaverse-Lvis.

Thank you for the comment. We further evaluate the FLOPs and runtime of UniGS and compare them with state-of-the-art approaches in Table 1. With a slight increase in runtime, UniGS achieves significant improvement over ${CLIP}^{2}$, TAMM, and Uni3D on Objaverse-Lvis zero-shot classification.

Table 2: Ablation study on the FLOPs of UniGS modules. CNN encoder denotes the CNN layers to extract spatial information from 3D representation into features, and Trans. denotes the Transformer blocks understanding objects from extracted features. Cross-Attn denotes the Cross-attention layers between Fundamental and Advanced Encoder.

Moreover, to better understand the computational cost of 3D-based approaches, we present an additional ablation study of UniGS modules on FLOPs in Table 2 and Figure 7 in Sec. J of the appendix. Specifically, this helps us understand the difference as 76.5% of the total FLOPs (73.38G) is due to the CNN layers of the 3D Encoder to extract 3D spatial features. This is indeed a limitation of our method's general application. Fortunately, much progress is being made in compressing models [1] and 3D representations [2], and we expect these advances to facilitate the development of 3D understanding with 3DGS representation.

[1] T3DNet: Compressing Point Cloud Models for Lightweight 3D Recognition

[2] LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS

Thank you once again for taking the time to review our paper and for providing valuable comments to enhance its quality. As the deadline for reviewer-author discussions draws near, we look forward to your feedback on our response. If you have any additional comments, we would be glad to offer further clarifications. Thank you!

Thank you for your valuable feedback and suggestions. We appreciate your instrumental comments in refining our paper, and we have added the additional experiments in the appendix for the completeness of our work. We are glad that most of your concerns have been resolved. If you have any additional comments, we would be glad to offer further clarifications. Thank you!

Sorry for this unclarity. We have provided Table 1 and Table 3 in Section J of the appendix (please refer to Tables 13, 14, and 15), with the differences highlighted in red. We believe the additional experiments can be found in the latest updated PDF and we hope this can address your concern.

Thank you for your detailed and thoughtful feedback on our paper. We are pleased that you recognize the strengths of UniGS, including the state-of-the-art performance and the effectiveness of 3DGS cross-model representations in cross-model learning. We will address your concerns point by point:

• W1(Details of negative sample strategy) : "Do different tasks need to adjust the negative sample strategy?"

Table 1: Ablation study on the schedule of negative sampling. MoCo denotes Momentum Contrast for Unsupervised Visual Representation Learning.

Thank you for the comment. During training, batches of data are randomly sampled from the dataset, and will be gathered between GPUs when computing loss. For a single object, its positive image sample is its corresponding image, while its negative samples are derived from images of other objects. Similarly, for text data, the positive sample is the object's own corresponding text, whereas the negative samples are taken from other texts that differ from its corresponding text. This random negative sampling strategy works well across different downstream tasks.

Moreover, our framework is open to improvements in the negative sampling strategy. As shown in Table 1, we apply MoCo[1] as a better strategy of negative sampling for UniGS. Given more training iterations, UniGS can achieve better Top 1 accuracy with MoCo for different text-image models.

[1] Momentum Contrast for Unsupervised Visual Representation Learning

• W2(Impact of structural differences) : "There are significant differences in the structural and spatial characterization of 3DGS and traditional point cloud data. Does it affect the final performance?"

Table 2: Comparison results with 10000 points dataset on Objaverse-Lvis zero-shot classification. † denotes fine-tuning on 3DGS datasets.

Thank you for your suggestion. We conducted additional experiments to fine-tune UniGS using a pure 3DGS dataset. As shown in Table 2, UniGS outperforms Uni3D when augmented with point clouds under the same settings as Uni3D. It also achieves higher Top-1 and Top-3 accuracy after fine-tuning on the pure 3DGS dataset. Experimental results in Table 2 and the main paper further demonstrate that Uni3D is not fully compatible with both point clouds and 3DGS. However, with our proposed Gaussian-Aware Guidance, UniGS exhibits the ability to effectively understand objects in both point clouds and 3DGS, achieving superior results after fine-tuning.

Thank you once again for taking the time to review our paper and for providing valuable comments to enhance its quality. We appreciate your insightful comments on improving UniGS by employing an enhanced negative samples strategy and fine-tuning on a pure 3DGS dataset. We are grateful for your thoughtful feedback to explore the best performance of UniGS.

We sincerely hope that the additional experiments and our response have addressed your concerns. If you have any further questions or suggestions, we would be glad to offer further clarifications. Thank you for your time and consideration!