Single-View 3D Representations for Reinforcement Learning by Cross-View Neural Radiance Fields

Comment 1:

The proposed representation requires synchronized multiview video data to be trained, so it can only be trained on limited data. It would be good to compare against embeddings such as DinoV2 which do not have explicit geometry-aware nature but can be trained on a lot more data and probably have a notion of “view-invariance” to some degree due to their training strategy.

Response 1:

Thank you for the suggestion. We conducted experiments using DINOv2 embeddings as the representation for our RL setup. However, the results were not favorable.

Anonymous link to RL with DINO-v2:

https://drive.google.com/drive/folders/13WU3Rq2vZvt68vlgjdVWngLL52skPuM_?usp=sharing

We believe there are two primary reasons for this outcome. First, DINOv2 is trained on real-world data, which introduces a significant sim-to-real gap in our experimental setup. This mismatch in data domains likely hampers performance in the simulated environment we use for RL.

Second, while DINO has been applied successfully in contexts such as temporal segmentation of robot trajectory for imitation-based approach [1], the requirements for RL differ significantly from methods like imitation learning. In RL, particularly with Bellman operator-based optimization, simply incorporating an off-the-shelf pre-trained backbone does not guarantee effective learning. Instead, RL demands a careful integration of embeddings with the underlying optimization process. Related works, such as [2] highlight that the effectiveness of a pre-trained vision model highly depends on the downstream policy learning method, and the pre-trained representations often require specific adaptations to align with RL's optimization dynamics.

This observation reinforces the need for representation learning approaches specifically designed for RL, as demonstrated in our method.

[1] Wan, Weikang, et al. "Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery." 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.

[2] Hu, Yingdong, et al. "For pre-trained vision models in motor control, not all policy learning methods are created equal." International Conference on Machine Learning. PMLR, 2023.

Comment 2:

Table 1 is a bit misleading. While during deployment the proposed algorithm can indeed be run on single view input, if I’m not mistaken during training of the actual embedding the requirement is still for posed multiview data. Perhaps it would be better to disentangle the deployment and training stages in this Table for the proposed method and for baselines as applicable.

Response 2:

Thank you for your suggestion. We have modified the table to separate the pre-training and deployment phases for a clearer comparison.

Anonymous link for modified table

https://drive.google.com/file/d/1fhHpJtBrUcDoLxycD13AnSD-aVv6RKX6/view?usp=sharing

The revised manuscript has been uploaded.

Comment 3:

I think the paper focuses slightly too much on the few- or single-view reconstruction results visually and wr.t. view synthesis metrics, which I don’t think is particularly informative. Single- and few-view reconstruction is a huge field by itself and there are much stronger baselines to compare against if this is the goal such as PixelNeRF, NeRDi, GS-LRM, ZeroNVS, Cat3D, Reconfusion, etc. etc. The goal of the paper is not to solve single or few view 3D reconstruction but to learn 3D-aware representations for downstream RL.

Response 3:

Thank you for your insightful feedback. We acknowledge that the current manuscript seems slightly focused on the single-view reconstruction results, which we think are indirect validations for our learned latent representation $z_t$’s 3D understanding. We have uploaded a revised manuscript to make clear that our work’s focus is learning effective representation for RL.

Comment 1:

The evaluation is limited to MetaWorld environments, which are relatively simple by 2024 standards. Testing on more complex manipulation scenarios would strengthen the paper. There are a lot of other simulated environments like RLBench. Can you explain why MetaWorld is used?

Response 1:

Thank you for your comments. Our primary focus in this work is on the algorithmic development of 3D representation learning for RL, as this is the first attempt at a single-view inference framework for 3D-aware representation, to the best of our knowledge. So, we followed the environment setup from prior work [1], while modifying the environment to a more realistic and challenging setup, incorporating additional textures from elements like a table and robot body. However, it is not an inherent limitation of our work, and we can consider performing experiments in other simulated environments like RLBench.

[1] Shim, Dongseok, Seungjae Lee, and H. Jin Kim. "Snerl: Semantic-aware neural radiance fields for reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

Comment 2:

The quantitative results in Figure 3 show an apparent contradiction - NeRF-RL achieves higher PSNR despite producing visibly blurrier reconstructions. This needs a better explanation. Can you explain why NeRF-RL images are blurry but the PSNR is higher than SinCro?

Response 2:

As you rightly pointed out, NeRF-RL achieves a higher PSNR in the multi-view input setting. This is because slight blurring at edges or differences in high-frequency details can reduce PSNR without significantly impacting perceived image quality. For example, NeRF-RL renders sharp boundaries in non-salient areas, such as table textures (which occupy most of the pixels) and the edges of the robot arm, whereas our method shows minor degradation in these areas. Also, blurred regions, such as the green peg in Figure 3, constitute only a small portion of a scene and thus have a limited impact on the overall PSNR while significantly contributing to the downstream RL performance.

Comment 3:

The figures could be improved - Figure 2 is a PNG instead of vector graphics which reduces quality.

Response 3:

Thank you for your feedback. We replaced Figure 2 with a vector graphic format to ensure higher quality and better readability.

If you have any questions or need more discussion, please let us know. We would be happy to improve our work based on your valuable feedback.

Comment 1:

The writing quality requires improvement...

Response 1:

We acknowledge that there are some unclear sentences as you rightly pointed out. For example, ‘randomly select a viewpoint’ is more clear expression compared to ‘randomizing viewpoints’. We have uploaded a new manuscript with the revised parts highlighted in magenta color.

Comment 2:

In line 93, the authors do not clarify the concept of a calibrated camera, ...., Consequently, the image encoder and neural radiance fields (NeRF) are effectively learning a fixed RGB->pose mapping.

Response 2:

First, we would like to kindly ask the reviewer to refer to the Common Response that addresses some misunderstandings regarding our work. We would like to note that our work is NOT a high-quality novel view synthesis framework without calibration, and the 3D reconstruction results in the manuscript are included as indirect validations for our learned latent representation $z_t$’s 3D understanding. In this work, we have focused on learning effective latent representation for downstream image-based RL, which itself is a huge field in RL.

Our work consists of 2 steps. 1) pre-train the 3D scene encoder to extract effective representation by leveraging NeRF (require calibrated camera at this phase for volume rendering), 2) perform inference on the pre-trained encoder and utilize the output representation of the encoder as an input for downstream RL. At this phase, we do NOT perform rendering (In Figure 1, the deployment phase does not include NeRF). For this reason, we said that camera calibration is not required during the downstream RL phase, while other prior representation learning works in RL require synchronized multi-view input or camera pose even during the downstream RL phase (Table 1).

We acknowledge that the manuscript may have been slightly unclear on the above points, and we have uploaded a revised manuscript to clarify them. Regarding overfitting, please refer to the Response 4.

Comment 3:

Furthermore, recent multi-view stereo (MVS) reconstruction models demonstrate that a calibration matrix is not essential for creating MVS 3D models. The authors should explore relevant literature in the domains of lightweight regression models (LRM), single-view LRMs, LRM with Gaussian distributions, and indoor LRM-like methodologies.

Response 3:

We assume your comment refers to a Large Reconstruction Model [1], not a Lightweight Regression Model, since most keywords you mentioned are addressed in [1]-related works. If we misunderstood your comments, please kindly let us know.

Similar to Response 2, we would like to emphasize that our work is not a high-quality novel view synthesis framework without calibration. Our focus is on learning effective latent representations for downstream image-based RL. Since the referenced works mostly focus on 3D reconstruction generalization capability or novel-view synthesis using single-view frameworks, we believe they fall outside the scope of our work and are not direct baselines for comparison.

Nevertheless, we investigated the rendering results of LRM. The novel-view synthesis results of LRM were significantly distorted when perturbations from the input viewpoint exceeded approximately 10 degrees, as illustrated in the following anonymous link:

https://drive.google.com/file/d/1AXY0UjwR2ctwuqNSxSPskNuaL2hWNsnr/view?usp=sharing

[1] Hong, Yicong, et al. "Lrm: Large reconstruction model for single image to 3d." arXiv preprint arXiv:2311.04400 (2023).

Question 1:

The author should explain more details about the camera calibration.

Answer 1:

We addressed this question in Response 2. If it remains unclear, please let us know.

Question 2:

The author should add some baselines with explicit 3d representation in RL.

Answer 2:

We addressed this question in Response 5. If it remains unclear, please let us know.

Question 3:

The visualization result of snerl looks much worse than it original paper, it will be great to see the visualization on the same env and setting.

Answer 3:

As mentioned in the revised manuscript (lines 309-311), the experimental environment used in this work is a modified version of the SNeRL environment, designed to provide a more realistic and challenging setup by incorporating additional textures from elements such as a table and robot body.

The reproduced results of SNeRL confirm that it can capture objects in the environment of the SNeRL paper, as illustrated in the anonymous link provided:

https://drive.google.com/file/d/1ePRBSd16gDrBkSF737O0L8HSZ1LYn8t2/view?usp=sharing

We believe that SNeRL performed well in the original paper since it was tested under simpler environments. However, it struggles as the complexity of the scene and the number of viewpoints increase. This limitation stems from its reliance on a simple CNN and the absence of masked reconstruction and cross-view completion strategies, making it unsuitable for scaling to more complex scenes and a larger number of viewpoints.

If you have any questions or need more discussion, please let us know. We would be happy to improve our work based on your valuable feedback.

We have conducted additional experiments to compare our method with another explicit 3D representation baseline, GNFactor [3], which utilizes posed RGB-D to construct 3D volumes and reconstruct the images from multiple viewpoints. Due to the limited response window, we could not conduct downstream RL experiments with this explicit representation. Thus, we only perform the pre-training phase, which could be utilized as indirect validations for the learned representation’s 3D understanding.

Anonymous link for the single-view volume rendering results: https://drive.google.com/file/d/1G3zERn-KFOwYT09ASIHK6Xvfp0AJRlQU/view?usp=sharing

The results demonstrate that the explicit representation baseline also struggles to capture fine-grained object details, such as the green peg, despite using depth information. As shown in the RL experiments in the manuscript, capturing these details is correlated to the downstream RL performance. Therefore, we expect that our method has the potential to outperform the explicit 3D representation baseline in downstream RL tasks. Furthermore, since this explicit representation baseline was originally designed to work in a single fixed viewpoint, the RL performance would further degrade as the input viewpoint varies.

[3] Yanjie Ze, et al., “Multi-Task Real Robot Learning with Generalizable Neural Feature Fields.” CoRL, 2023.

Comment 1:

The time contrastive loss (Eqn. 3) repulses state features at different timesteps. However, this does not hold for static scenes where the actor remains stationary between timesteps $t$ and timestep $t’$.

Response 1:

Thank you for your insightful comments. As you mentioned correctly, repulsing state features at different timesteps may not hold for totally static episodes. However, since the RL algorithm itself is inherently designed to explore the environment to maximize the reward, the agent continues to move during the episode rollout, preventing static episodes. Furthermore, this time contrastive loss has been widely adopted and validated in other robotics-related research [1,2,3], demonstrating its effectiveness in various settings.

[1] Sermanet, Pierre, et al. "Time-contrastive networks: Self-supervised learning from video." 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018.

[2] Li, Yunzhu, et al. "3d neural scene representations for visuomotor control." Conference on Robot Learning. PMLR, 2022.

[3] Nair, Suraj, et al. "R3m: A universal visual representation for robot manipulation." arXiv preprint arXiv:2203.12601 (2022).

Comment 2:

The 3D encoder-decoder model is trained on multi-view images, with scene representation ($z_t$ … ). How can it generalize when the inputs are from the same viewpoint, as in  ($z_t$ …) (line 291)?

Response 2:

The ability of the proposed model to perform single-view inference is enabled by the following factors (also described in the revised manuscript at lines 285-288).

1. Time Contrastive Loss: The time contrastive loss ensures that the state feature $v_t$ remains consistent regardless of which viewpoint image is provided for a given timestep $t$. This objective encourages the model to produce similar state features for all images corresponding to the same underlying scene state, even if the viewpoints differ.
2. 3D Geometry Awareness: A fully 3D-aware encoder will output the same representation for any image of a scene at a specific timestep $t$, regardless of the viewpoint. To achieve this level of 3D geometry awareness, we employ training objectives such as cross-view completion and multi-view reconstruction. These objectives help the encoder learn the underlying 3D structure of the scene, ensuring consistent outputs for any combination of primary or reference inputs.

While these individual factors do not achieve the desired property perfectly, they work complementarily, as demonstrated in the ablation study (Section 5.4).

Comment 3:

Table 1 claims that the proposed method does not require camera calibration. However, camera poses are needed to render multi-review reconstruction from $z_t$, making this claim inaccurate.

Response 3:

Thank you for your comment. First, we would like to kindly ask the reviewer to refer to the Common Response that addresses some misunderstandings regarding our work.

Our work consists of 2 steps. 1) pre-train the 3D scene encoder to extract effective representation by leveraging NeRF (require calibrated camera at this phase for volume rendering), 2) perform inference on the pre-trained encoder and utilize the output representation of the encoder as an input for downstream RL. At this phase, we do NOT perform rendering (In Figure 1, the deployment phase does not include NeRF). Therefore, camera calibration is no longer required during the downstream RL phase.

Since there is a similar comment (Comment 2 from reviewer JDLs), we have modified Table 1 to separate the pre-training and deployment phases for a clearer comparison.

Anonymous link for the modified table:

https://drive.google.com/file/d/1fhHpJtBrUcDoLxycD13AnSD-aVv6RKX6/view?usp=sharing

Comment 4:

In the volume rendering experiments, the authors should also include comparisons with NeRF baselines for sparse views, such as RegNeRF, pixelNeRF, etc.

Response 4:

Thank you for your comments. As mentioned in the Common Response, we would like to clarify that the main goal of this work is to enhance downstream RL performance through NeRF-based representation learning, rather than achieving high-quality rendering or novel-view synthesis. As such, we believe that visual comparisons with NeRF variants like RegNeRF or pixelNeRF, which are optimized for sparse-view novel-view synthesis, are not directly aligned with the main objectives of this study.

Even though we could consider integrating other NeRF models, we just utilized vanilla NeRF to focus on algorithmic-level development for representation learning. However, leveraging NeRF variants capable of better novel-view synthesis might improve the downstream RL performance under larger levels of viewpoint perturbation, which could be an interesting direction for future exploration.

Thank you to the authors for their response. They have adequately addressed my concerns, and I have no further questions.

Comment 1:

The paper misses various key recent results both for 3D representation learning using NeRFs [1] and for baking in viewpoint awareness for policy learning [2,3].

Response 1:

As you rightly commented, NeRF-MAE [1] learns 3D representation. However, this method is not suitable for the RL setup because the encoder’s input consists of voxels, which must be obtained through a pre-trained NeRF model. Specifically, 1) we must have an individual pre-trained NeRF model for each scene (corresponding to each timestep in our work), even during the downstream task, to compute the voxel’s 4-channel values. Additionally, 2) these 4-channel values are computed by averaging the outputs of the pre-trained NeRF model across all viewing directions, which requires access to multi-view images during inference. This reliance makes NeRF-MAE incompatible with the single-view inference requirements of the RL setup. Therefore, we believe that NeRF-MAE is not directly comparable to our method.

Both RoVi-Aug [2] and VISTA [3] use ZeroNVS [4] for data augmentation by synthesizing novel-view images, focusing on generating novel-view data rather than learning effective representations. In contrast, our approach centers on pre-training an image encoder to extract effective 3D-aware representations for downstream RL tasks by leveraging NeRF, based on the assumption of a given multi-view dataset. Consequently, prior works [2, 3, 4] are not used as direct baselines. Rather, these methods can serve as complementary components, as we could replace the multi-view dataset assumption with a dataset augmented by ZeroNVS, as in [2, 3], to further enhance our approach.

To provide preliminary insights, we conducted data augmentation experiments to evaluate the policy’s robustness to viewpoint changes, similar to [2,3]. As mentioned in Response 3, ZeroNVS does not produce reasonable zero-shot synthesis results. Therefore, we used ground truth images obtained from simulation as a proxy for the data augmentation effect of ZeroNVS (i.e., assuming ZeroNVS synthesizes 100% accurate images). Specifically, we performed an RL experiment similar to Section 5.3, while following CNN+view randomization described in the manuscript, but with 30 viewpoints. The results are available in the following anonymous link:

https://drive.google.com/drive/folders/1oUNOOcUNQGkIcCjlpFKT_B5tyhr-HJ3J?usp=sharing

As shown in these figures, our method outperforms this baseline despite utilizing images from significantly fewer viewpoints (Ours: 6 views, Baseline: 30 views). We attribute this result to the proposed 3D-aware representation learning scheme, which enhances the encoder’s implicit understanding of the 3D world. This finding underscores that simply increasing the number of viewpoints is not always the optimal approach; instead, carefully designed representation learning schemes play a far more critical role.

Comment 2:

The paper doesn't show any real-world evaluation results while both [2,3] show real-world results. Is it an inherent limitation of the method that it only works in simulation?

Response 2:

This is not an inherent limitation, and real-world experiments are certainly feasible. However, as a 3D representation learning framework for RL has emerged only recently, there remain many challenges and open questions. Consequently, most of the prior 3D representation-based RL works (and our method) are more focused on developing algorithms for the effective 3D representation learning framework rather than deploying them directly in real-world settings. Therefore, we leverage simulations where the proposed algorithm can be extensively evaluated and analyzed in multiple environments.

Our method, which supports single-view inference for downstream RL, is well-suited for future extensions into real-world applications. Specifically, we plan to address the requirement for synchronized multiple cameras during pre-training by exploring the use of videos captured from moving cameras with varying viewpoints (as mentioned in Section 6). This direction will enable seamless transitions to experiments with real robots, spanning pre-training to downstream online RL, while building on the strengths of the current proposed single-view inference framework.

We have updated some of the attached anonymous links as there were access issues with the previous ones. Please let us know if you still encounter any problems accessing the links.