3D-SPATIAL MULTIMODAL MEMORY

Rebuttal Resources to Reviewer 2XXj: [rebuttal pdf: 2XXj], [video: pickup sponge], [video: pickup pineapple], [video: pickup screwdriver]

We sincerely appreciate reviewer 2XXj for the positive comments on our contributions, especially for the potential applications in diverse robotic tasks. To address reviewer 2XXj’s concerns, we provide both conceptual and experimental justifications.

Conceptually, we want to emphasize that the main contribution of this work is to introduce Gaussian Memory Attention (GMA) as an alternative way (other than simple feature distillation) to integrate spatial and semantic features. This contribution is acknowledged by reviewer a6FD and reviewer RUFb. The references provided by reviewer 2XXj are robotics applications of representations obtained via naive distillation, which is orthogonal to our method. Specifically, we provide extensive experiments to show that this memory attention mechanism is more effective than naive distillation on various downstream visual discriminative tasks even when the same set of reference features are used to optimize the representation.

Experimentally, due to the limited rebuttal timeframe and the methodological focus of our work to advocate an alternative paradigm to build representations, it is impractical to perform full-scale robotics experiments on all the provided references. Besides providing more results on 2D vision tasks, to motivate how our method can be applied to robotics tasks, we include a new set of real robot experiments to use GMA for the zero-shot grasping task setting proposed in LERF-TOGO [1], where the robot is tasked to grasp objects by parts specified via language to evaluate its part-level understanding ability.

[Q1] Real robot deployment with part-level understanding.

Geometry-based or semantic-part-based tasks using DINO or diffusion features, such as geometry-based object retrieval/grasping[1, 2], planning objects to the goal configuration[3].

We enclose a new [video: pickup screwdriver] on tabletop manipulation, the task is constituted of two steps:

1. Pick up the screwdriver on the handle.
2. Find the plate and put down the screwdriver on the plate.

The first step requires the feature field to be able to be language-grounded and part-level, and the second step requires language grounding as well as the effective integration of M3 with low-level control algorithms.

For the first and second steps, we also visualize the first-person view of the grounding masks ([grounding: step 1], [grounding: step 2]) to locate the screwdriver and plate.

[Q2] Real robot deployment with language-guided task.

Language-guided tasks using CLIP feature, such as like "pick up the red cube and place it on the blue book." (example references: [4, 5])

We show real robot deployment tasks with a language-guided query, below are the experiment results and language query.

1. [video: pickup sponge] Pick up the sponge and place it into the sink.
2. [video: pickup pineapple] Pick up the pineapple from the kitchen set and place it on to the table.

In addition, we kindly share the perspective, that language-guided task performance is equivalent to grounding task as shown in the table below:

References:

1. Rashid, Adam, et al. "Language embedded radiance fields for zero-shot task-oriented grasping." CoRL. 2023.

Dear Reviewer 2XXj, it is approaching the rebuttal deadline on Dec 2nd at 11:59 AoE, it would be appreciated if you could take a look at our rebuttal. We want to re-emphasis on our contents:

• Our main contribution is spatial memory via Gaussian splatting and foundation model embeddings.
• We deploy our model on the real robot to prove the effectiveness of M3. The real-robot experiments provide additional verification of our approach, instead of the main contribution.

Reviewer a6FD evaluates M3 with a focus on model architecture, while reviewer RUFb emphasizes feature representation learning. Additionally, reviewer UxH3 concentrates on Gaussian representation. All three reviewers have provided positive feedback on our paper. We would greatly appreciate it if you could review our rebuttal materials and share any further comments. We are happy to address any remaining concerns.

We sincerely appreciate reviewer UxH3for the positive comments on our contributions, especially for the proposed compression mechanisims in 2D-3D feature distillation. The suggestions in both the weakness and question sections are inspiring and encourage us to conduct more detailed analysis between our methods and previous works.

[Q1] Comparisons with previous methods (F-3DGS) on downstream tasks.

F-3DGS evaluated their approach through the task of novel view semantic segmentation. I see no reason why this could not be used in this paper as the experiment. This makes comparison between the two methods harder.

Thanks for pointing this out, we agree that only showing the direct feature comparison is not convincing enough for performance comparison. During the rebuttal, we included two new downstream tasks based on the reviewers’ feedback: grounding and retrieval, and we report their results here.

In the table below, we show the downstream task performance of both F-3DGS and M3 in comparison with GT features in three datasets including Train, Playroom, and Geisel. The evaluation metrics cover both grounding and retrieval tasks, with metrics spanning IoU, AP, I2T, and T2I. We use MaskCLIP embeddings for grounding and SigLIP embeddings for retrieval.

The table below shows the following insights:

1. With half the parameter size compared to F-3DGS, we not only achieve better results than F-3DGS by a reasonable margin (especially on AP50) but also outperform the original GT feature from MaskCLIP. This is caused by the Gaussian rendered feature usually having higher resolution as well as less noise.
2. For image retrieval, M3 performs significantly better than F-3DGS, while having relatively closer results with ground truth. Especially when we examine the top-1 results (I2T@1, T2I@1), M3 is much better than F-3DGS, this indicates that the feature map created via principal scene components does have a knowledge space more closely aligned with the original foundation model’s embedding space.

[Q2] Comparison on the dataset with F-3DGS, and F-Splat.

Similar to the previous point, I believe it would be beneficial to have some datasets between the proposed method and prior work in common.

Thank you for the constructive feedback. We agree that comparing M3 with previous works on common datasets is valuable. However, due to time constraints, we are still working on the implementation and will report the segmentation results on the dataset introduced in LERF shortly.

[Q3] Ablation study on memory efficiency in comparison with F-3DGS, F-Splat.

Further, the paper claims memory efficiency as a part of the method's benefits. Therefore an ablation and comparison to prior work from the memory perspective would be useful.

The memory efficiency of the proposed method is achieved by using lower-dimensional feature embedding for each Gaussian primitive. For example, the CLIP feature is stored with only 16 dimensions, compared to 64 dimensions in F-Splat and 256 dimensions in F-3DGS (or 64 dimensions with speedup). Below is a detailed comparison of the parameter counts between our method and F-Splat/F-3DGS for the Garden scene with all six features:

• Number of Gaussian Primitives: 138,766
• Dimension of rendering parameters (e.g. position, SHS, etc.): 59
• Feature dimension:
  • Ours: 160
  • F-Splat/F-3DGS w. speedup: 384
  • F-3DGS w/o. speedup: 1536

Parameter counts:

• F-Splat/F-3DGS w. speedup: (59 + 384) * 138766 = 61 M
• F-3DGS w/o. speedup: (59 + 1536) * 138766 = 221 M
• Ours (Gaussian primitives): (59 + 160) * 138766 = 30 M
• Ours (Principle scene component): 5 M
• Ours (Total): 30 M + 5 M = 35 M

I'd like to thank the authors for the rebuttal. I believe the extra experiment with a downstream task is very helpful. Personally, I'd suggest trying to fit it into the main paper. I also appreciate the additional ablation and the answers to my questions. Similarly, further experiments in the robot setup provided for another reviewer look convincing to me. Currently, the revised paper and a rebuttal text help to validate the claims of the paper. Given that, I increased my initial score.

[Q2] Feature reconstruction qualitative results in comparison with F-3DGS, F-Splat to show the effectiveness of PSC and GMA.

[…] I believe the paper would benefit from a comparison of reconstructed features between previous approaches and M3 […] Can the authors provide qualitative examples comparing the reconstructed features of M3 with those from previous methods, such as F-3DGS and F-Splat?

In the previous question, we have shown the feature reconstruction qualitative results in comparison with F-3DGS ([feature visualization]). This visualization clearly shows that the M3 feature is more compact, disentangled, and geometrically aligned. In this question, we show more qualitative results on the construction trace from the principal query to the principal scene components and the final rendered feature.

As shown in [visual: gaussian memory attention trace], we show the visualization of the principal query, psc, raw feature, and the final render feature. All the images, including the colored circles, are directly drawn by the algorithm. The trace clearly shows that the principal query will correctly attend to the principal scene components that correlate with the raw feature of the similar semantic/texture information for each spatial location. This shows that the claimed contribution in this work is evident.

[Q3] Terminology video memory is confusing for the static scene.

[…] Do the authors mean that the video contains the multi-view collection trajectory of an agent? If that is the case, I believe it should not claim to be a video memory and model the temporal aspect, as the temporal dimension in the video is just the dimension of new views of the same static scene.

Thanks for pointing out this potential confusion. Our method focuses on video clips captured along a camera trajectory in static scenes. We agree with the reviewer that the video memory involves only the multi-view aspect without the temporal aspect. To address this, we have clarified the terminology in the introduction section of the revised paper.

[Q4] Lack of input domain for many newly introduced concepts.

I believe the paper would benefit from better clarity when presenting the method. The authors introduce many concepts by naming them (VG, KS, V) but do not provide input domains for such concepts therefore lacking precision. […]

Thanks for pointing out this potential confusion. Below, we provide a detailed description of domain for the newly introduced concepts.

Consider a view $\mathbf{V} \in \mathbb{R}^{h\times w,3}$ (where $h,w$ denote the pixel dimensions) in the 3D scene $\mathbf{V}$. A signle view is represented as $\mathbf{V} =${$V^{1}, V^{2}, ..., V^{m}$}, where $V^{i} \in \mathbb{R}^{p, 3}$ is the $i^{\text{th}}$ granularity of the view $V$, $p$ is the number of pixels, and $m$ denotes the total number of granularities.

The foundation model features for each view are denoted as $\mathbf{F}(\mathbf{V}) =${$\mathbf{E}^{1}, \mathbf{E}^{2}, ... \mathbf{E}^{n}$} for each foundation model ( $\mathbf{F}$ ) and scene ($\mathbf{V}$), where $\mathbf{E}^{i} \in \mathbb{R}^{[h\times w,d]}$, and $d$ denotes the feature dimension. Note that we consistently use subscript to indicate each view (a total of $n$ views) and superscript to indicate each granularity or foundtaion model. (a total of $m$ foundatiaon models).

In addition, we have added a section in the supplementary material, using illustration figures and text descriptions for the input domain and newly introduced concepts.

[Q5] Typos.

Thank you for pointing this out. We have already addressed all the minor comments for writing in the revised paper.

References

1. Zhou, Shijie, et al. "Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields." CVPR. 2024.
2. Qiu, Ri-Zhao, et al. "Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing." ECCV. 2024.
3. Dong, Xiaoyi, et al. "Maskclip: Masked self-distillation advances contrastive language-image pretraining." CVPR. 2023.
4. Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." CVPR. 2023.
5. Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.
6. Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

Rebuttal Resources to Reviewer RUFb: [rebuttal pdf: RUFb]

We sincerely appreciate reviewer RUFb for the positive comments on our contributions, especially for the proposed GMA and PSC mechanisims. The suggestions in both the weakness and question sections are inspiring and encoure us to explore additional downstream tasks and conduct a more detailed analysis.

[Q1] Provide implementation details, qualitative, and quantitative evaluation for downstream tasks in comparison with F-3DGS, and F-Splat.

The paper misses implementation details and qualitative results for downstream task applications. […] Lacking quantitative comparisons with baselines for downstream tasks. […] […] Please share some explanation of which feature is used for what task, and how, and provide qualitative comparisons with previous works F-3DGS and F-splat.

Both F-3DGS and F-Splat are distillation-based methods with similar architecture. Thus, because of time constraints, we only compare with F-3DGS. Empirically and according to the results in Table 1, F-3DGS usually has better performance than F-Splat.

Implementation Details: We directly use the public codebase of F-3DGS. We use 64 dimensions in Gaussian primitives for F-3DGS, while we use 16 dimensions in Gaussian primitives for all foundation models except Llama3 and Llamav, which use 32 dimensions. For F-3DGS, they primarily use segmentation models and CLIP to build the raw feature, while in this work to have an apple-to-apple comparison, we only use the foundation model embeddings without semantic clustering. The input and testing data are the same for both M3 and F-3DGS.

Qualitative Results: We compare the feature PCA visualization of M3 and F-3DGS on the Train and Geisel dataset with the foundation model CLIP and SigLIP. As clearly shown in the [feature visualization], the F-3DGS features appear smoother and more dispersed, with less defined edges and structures, resulting in a hazier and less spatially coherent representation where features overlap more. In contrast, the M3 features exhibit sharper boundaries, better-defined object structures, and localized color separations, indicating more compact, disentangled, and geometrically aligned feature representations.

Quantitative Results: In the table below, we show the downstream task performance of both F-3DGS and M3 in comparison with GT in three datasets including Train, Playroom, and Geisel. The evaluation metrics cover both grounding and retrieval tasks, with metrics spanning IoU, AP, I2T, and T2I. We use MaskCLIP embeddings for grounding and SigLIP embeddings for retrieval. The table below shows the following insights:

1. With half parameter size in comparison with F-3DGS, we not only achieve better results than F-3DGS with a reasonable margin (especially on AP50) but also have better results in comparison to the original GT feature from MaskCLIP. This is caused by the Gaussian rendered feature usually having higher resolution as well as less noise.
2. For image retrieval, M3 performs significantly better than F-3DGS, while having relatively closer results with ground truth. Especially when we look at the top-1 results (I2T@1, T2I@1), M3 is much better than F-3DGS, this indicates that the feature map created via principal scene components does have closer knowledge space with the original foundation models embedding space.

We report the result on grounding and retrieval here.

Rebuttal Resources to Reviewer a6FD: [rebuttal pdf: a6FD], [video: pickup screwdriver]

We sincerely appreciate reviewer a6FD for the positive comments on our contributions, especially for the combination of spatial and semantic features. The suggestions in both the weakness and question sections are very inspiring and encourage us to explore further in the direction of the GMA architecture and beyond.

[Q1] Discuss the motivation and implementation details of Gaussian Memory Attention.

The Gaussian Memory Attention component is a central contribution of the paper but is rather briefly discussed in the paper. It would be good to provide a clearer rationale for why this method was chosen and why it achieves the claimed result?

Thanks for pointing out the key components of M3 with the correct emphasis that we need more motivation for the Gaussian Memory Attention (GMA) module. The GMA module is designed with the following rational steps:

• The principal query is a compressed index of the principal scene component ($PSC$).
• The up-sampled principal query ($PQ$) should have the same dimensionality as the principal scene components ($PSC$) → [L298: $Q_p \times W_m$].
• The up-sampled principal query ($PQ$) can directly compute the cosine similarity with principal scene components ($PSC$) → [L298:  $\text{softmax}\left(Q_p W_m \times PSC^T\right)$].
• The weighted sum of the principal scene components ($PSC$) based on the similarity could create the final rendered feature ($R$) → [L298: $\text{softmax}\left(Q_p W_m \times PSC^T\right) \times PSC$].
• The final rendered feature is a direct sum of principal scene components so it should be in the knowledge space of the original foundation models’ embeddings.

We clearly show in the visualization [visual: gaussian memory attention trace] that the Gaussian Memory Attention operates as described above.

[Q2] Compare Gaussian Memory Attention with classic transformer architecture.

How does it compare to standard memory mechanisms (such as in transformers)

The scaled dot-product attention operator is computed as follows:

While the Gaussian memory attention operator is computed as:

While the attention mechanism is quite similar, the major difference is the transformer architecture has projection layers for $Q, K, V$, as well as the output (could be in a multi-head way). This shifts the distribution of both the principal query and principal scene components leading to the output space out-of-distribution of the foundation model knowledge space. Thus, the implementation of GMA directly fits the 3D feature field reconstruction scenario.

[Q3] Potential Variants of Gaussian Memory Attention (GMA).

and what could be the alternative implementations.

• The simplest variant of Gaussian Memory Attention is removing the memory bank (Principal Scene Components) and the weighted sum procedure, preserving only the projection. This variant is known as distillation. Our baselines F-3DGS, F-Splat are both distillation methods.
• We currently implement a version of GMA with softmax temperature. In our experiments, softmax without temperature significantly hurt performance.
• We appreciate the reviewer mentioning transformer architecture, which is quite similar to ours.
• The PSC could be initialized with raw features with learnable parameters, while it is fixed currently.

[Long Horizon Real Robot Tasks]: We introduce three new real robot deployment videos in the supplementary material, they contain (1) Multiple steps, and (2) Multiple scenes.

1. [video: pickup sponge] Pick up the sponge and place it into the sink.
2. [video: pickup pineapple] Pick up the pineapple from the kitchen set and place it on to the table.
3. [video: pickup screwdriver] Pick up the pineapple from the kitchen set and place it on to the table.

The third video requires our representation to be able to recognize part-level semantics, identifying the graspable part of the screwdriver. All the video is achieved by the language-guided queries, at the same time for those videos with two scenes, we use different foundation models on each scene.

References

1. Zhou, Shijie, et al. "Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields." CVPR. 2024.
2. Dong, Xiaoyi, et al. "Maskclip: Masked self-distillation advances contrastive language-image pretraining." CVPR. 2023.
3. Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." CVPR. 2023.

[Paper Revision Updates]: We have submitted a revised version of our paper, with changes highlighted in red. The updates include:

• Added the downstream evaluation results of grounding and retrieval to address the feedback from Reviewer RUFband Reviewer 2XXj.
• Added the ablation study on memory footprint, performance, and training time to address the feedback from Reviewer UxH3.
• Added the clarification on terminology and notations and the visualization of GMA to address the feedback from Reviewer a6FD.
• Added the new long-horizon real robot deployments results to address the feedback from Reviewer a6FD and Reviewer 2XXj.
• Added additional feature PCA visualization between different methods to address the feedback from Reviewer RUFb.
• Revised the M3 Preliminaries section to enhance the clarity of notations based on feedback from reivewer RUFb.
• Revised the abstract and introduction sections to address the feedback from Reviewer RUFband Reviewer UxH3.

We sincerely appreciate the reviewers' constructive feedback. We believe this revision addresses most comments, but we are willing to address any additional feedback.