Latent Radiance Fields with 3D-aware 2D Representations

Thank you very much for your time and effort in reviewing our work and providing the constructive comments.

W1. Decoder finetuning: Thank you for the valuable comment. The view inconsistency cannot be fully addressed by the encoder fine-tuning. However, with our fine-tuned encoder, the latent radiance field can be built more effectively, which serves as a foundation for further fine-tuning the decoder for better view inconsistency in the decoding process.

Current view-consist decoding methods often leverages temporally consistent decoders (e.g., 3D VAEs) [r1, r2, r3] to achieve view-consistent decoding. However, pre-trained 3D VAEs typically do not outperform standard VAEs in terms of performance; their primary advantage lies in computational efficiency. While fine-tuning these 3D VAEs can improve view consistency, they remain ineffective for handling large view gaps (often in our setting) due to the limitations of their Nx1x1 temporal kernel size.

In contrast, our approach focuses on encoder fine-tuning, which inherently enables our decoder to achieve view consistency without requiring substantial adjustments. Consequently, this work took a more general approach for decoder fine-tuning to demonstrate the effectiveness of our encoder fine-tuning.

[r1] CV-VAE: A Compatible Video VAE for Latent Generative Video Models

[r2] Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution

[r3] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

W2. Trying sparse views.:

Thank you for your very insightful suggestion! The table below demonstrates that our method indeed outperforms the other image-space approaches in the sparse-view setting (3 views) on the LLFF dataset. Our method is even better than the state-of-the-art 3DGS methods with standard input image resolutions.

W3. Train/test split: We split the training and test data by selecting camera poses whose indices are divisible by 8 as the test dataset, following the same train/test split setting of 3D Gaussian splatting.

W4. Color shift in teaser:

Thank you for pointing this out. There are still noticeable color shifts (i.e., the distribution shifts discussed in Section 3.3) in the novel view latent rendering. In general, our method has addressed most of the color shift issues. As shown in Figure 6, comparing 3DGS-VAE and our method, there is a significant improvement in color consistency. However, some inevitable color shifts remain due to the imperfection of applying RGB-based NVS pipeline to latent features. The SHS faces challenges in effectively handling view-dependent latent representations.

Q1. More view-consistent visual results: We evaluate the 3D consistency of different latent representations by constructing 3DGS models in the latent space. A higher PSNR in NVS tasks indicates better 3D consistency, and our method achieves a higher PSNR compared to the vanilla VAE, as shown in Figure 6 and Table 2 of the paper. Please visit our project website https://latent-radiance-field.github.io/LRF for videos showing better consistency between video frames (i.e., different views) rendered by our method.

Q2. Train/test split: Please refer to our response to W3.

Thank you again for your time and effort in reviewing our work and providing the constructive comments.

W5. More recent text-to-3D generation: We have conducted comparison with more recent text-to-3D generation method, as shown in Sec. 5.3 of the revised paper. As shown in Fig. 5, our method can boost the performance under extremely complicated text prompts, and achieve complex geometry while preserving the multi-view consistency. For instance, in the result of GSGEN, the train on the cake appears distorted and inconsistent shapes across views. Our method achieves consistent generation across views.

Q1. Correspondence point mapping:

The mapping is performed by grid sampling, where pixel-level correspondences are mapped to the latent space through bilinear interpolation. This ensures that each latent position receive correspondence information from its associated pixels accurately.

Q2. COLMAP settings:

Please refer to our response to W2.2.

Q3.1. How to balance the weight: We set both $\lambda_{\text{train}}$ and $\lambda_{\text{novel}}$ to 0.5. Our dataset consists of 80% training view images and 20% novel view images. By assigning equal weights to these terms, we ensure that the decoder learns not only to decode effectively from the training views but also to generalize and perform well on the novel views.

Q3.2. Over-fitting risk: Thank you for the valuable comment. To avoid over-fitting of VAE fine-tuning, we held a validation set from the 1K scenes and selected the checkpoint with the best performance on the validation set. Additionally, we have performed extensive out-of-domain evaluations on unseen datasets such as NeRF-LLFF, MVImageNet, and MipNeRF 360 to demonstrate the strong generalization ability of our approach.

Q4. Comparsion Protocol: Please refer to our response to W4.

Thank you very much for your time and effort in reviewing our work and providing the constructive comments.

W1. What are the computational advantages over image-space methods?:

Please refer to the following figure with the details about the training time, GPU usage, storage, and rendering speed. Our method reduces input resolutions, model storage space, and GPU usage for photorealistic NVS, which is particularly useful in cases with limited communication bandwidth and storage.

W2.1. Scale mismatch:

Thank you for the insightful comment. We have studied the proposed correspondence loss at multiple spatial levels by applying it to different downsampling layers of the VAE encoder. The performance gains were found to be trivial. To balance computational efficiency and performance, we decided to apply correspondence loss only to the output feature maps of VAE encoder.

W2.2. COLMAP details:

Regarding the setting of COLMAP, the correspondence points for each scene will be pre-computed before the model fine-tuning process. We use sequential matcher with the number of overlapping images of 10 and number of quadratic overlap of 1. Such overlapping searching strategy ensures our model not only learns from easy and dense correspondence, but also from challenging cases. Moreover, we set the minimum number of inliers and minimum ratio of inliers to 15 and 0.25 with the loop detection to make sure the extracted correspondence is accurate enough. Even though our correspondence calculation is robust, more ablation studies will be added in the camera ready revision to investigate the performance of different noise levels of correspondence points to reveal the impact of outlier correspondence.

W3.1. Why introduce SH basis functions? A direct learnable decoder or simpler view-dependent representation might suffice:

Thank you for the great insight. We agree that per-scene view-dependent layer may also work, but it requires much more model optimization time and rendering time compared to SHS. In this work, we need to build a large dataset of latent radiance fields for 1K scenes. Therefore, we chose SHS as the 3D representations for an efficient optimization.

W3.2. Distribution shift in latent representations:

We observed that there are distribution shifts between latent features rendered from 3D representations and encoded by VAEs. To verify this, we calculate KL divergence to quantify the distribution shift before and after novel view synthesis by LRF.

Please also refer to Fig. 6 of the paper, where no-decoder-finetuning results in worse visual quality. These two evidences indicate the necessity of fine-tuning the decoder.

W4. Comparison protocol: The purpose of comparing our method to 3DGS with 8x downsampled input resolution is to evaluate the performance of 3DGS models under the same input resolutions. When the latents share the same resolution as the RGB images, the 3D latent representation outperforms all existing methods of 3D RGB representation. This finding provides valuable insights for downstream tasks requiring latent representations while addressing limited computational constraint.

Thank you for the further comments. We would like to reemphasize the motivation of this work, as to "bridge the gap between 2D and 3D representations, providing an efficient and generalizable solution for tasks requiring multi-view consistency in the feature space". Targeting at this, our focus on 3D reconstruction is feature-space reconstruction rather than image-space reconstruction. Compared with the feature-space reconstruction baselines such as Feature 3DGS and Latent-NeRF, our method achieves better reconstruction performance, demonstrating the effectiveness of introducing 3D awareness into the 2D feature space.

The gap between feature-space reconstruction and full-resolution image-space 3DGS is mainly due to the image quality degradation incurred by image VAE reconstruction. As evidenced in the table below, reconstructing images by VAE in the 2D space (without 3D reconstruction) only maintain 24.59 PSNR, degrading from the infinite PSNR (i.e., comparing two identical images). This serves as the upper-bound of feature-space reconstruction methods. Our method conducts 3D reconstruction in the feature space while approaches this upper-bound with a 22.45 PSNR. The image-space methods shown in the paper, like 3DGS and Mip-Splatting, serve as a reference to feature-space methods while reveal an insight:

• Under the same input resolution to 3D models, feature-space reconstruction is much more effective than RGB reconstruction.

Computation efficiency: To demonstrate the efficiency advantage of feature-space 3D reconstruction, we estimated the GPU usage of 3DGS for a large outdoor scene comprising 1,100 images (1408 × 1024, captured by a drone) over 500m × 500m area. The standard 3DGS requires 40 GB GPU memory, while our method needs only 13.1 GB. Training standard 3DGS on such scenes is impractical with consumer-grade GPUs like the RTX 4090 of 24GB memory.

Additional pre-training overhead: The 3D-aware autoencoder is generalizable, such that it can be directly loaded and used in real-world applications without any per-scene fine-tuning. Extensive out-of-domain evaluations (training on DL3DV, evaluating on NeRF-LLFF, MVImageNet, and MipNeRF-360) demonstrate the strong generalization ability of our 3D-aware autoencoder. This is different from existing latent rendering methods, such as Latent-NeRF, which require per-scene fine-tuning.

Thank you for the beneficial discussion. Feature-space 3D reconstruction (i.e., the Latent Radiance Field in Stage-II) only serves as one part of our entire framework. Feature-space reconstruction evaluates the multi-view consistency of 2D encoder proposed in Stage-I, and serves as a foundation for fine-tuning the decoder in Stage-III. Again, our goal is to bridge the gap between 2D and 3D representations by constructing a 3D-aware autoencoder, rather than advancing image-space 3D reconstruction algorithms.

Why do we need 3D-consistent 2D representations? As demonstrated by Stable Diffusion models, optimizing in the latent space instead of the image space can significantly boost generation efficiency. With a 3D-consistent latent space and photorealistic decoding capability, many tasks such as text-to-3D generation, latent NVS, sparse-view NVS, efficient NVS, and 3D latent diffusion model can be improved.

Thank you very much for your time and effort in reviewing our work and providing the constructive comments.

Motivation. This work aims to bridge the gap between 2D and 3D representations, providing an efficient and generalizable solution for tasks requiring multi-view consistency in the feature space. Targeting at this, there are two key challenges:

• 1. A key challenge is that 2D latent features often lack 3D view consistency, which impairs the performance of novel view synthesis (NVS). To address this. We introduce an efficient correspondence-aware autoencoding fine-tuning strategy that enforces multi-view consistency of 2D latent features, minimizing changes to the 2D latent space while achieving significantly better geometric consistency.
• 2. Another key challenge is that, applying RGB-based NVS methods to latent features would cause data distribution shift (see below table). We mitigate this through an alignment fine-tuning process that uses latent radiance fields (LRFs) as guidance to fine-tune decoder, resulting in improved latent representation fidelity with significant PSNR gains, as shown in Table 2 and Figure 6 of the paper.

What can our method benefit?

• Input data compression of 3D reconstruction: Our method reduces input resolutions, model storage space, and GPU usage for photorealistic NVS, which is particularly useful in cases with limited communication bandwidth and storage. For instance, some individuals may not have GPUs with large memories, where our method is an efficient solution for them to run photorealistic NVS algorithms.

• Text-to-3D Generation: The 3D-aware 2D representations enhance both latent and image-space text-to-3D generation frameworks. Please refer to Sec. 5.3 of the paper for this improvement on three different generation frameworks.

• Sparse View Reconstruction: We find that the improved 3D feature consistency enables better reconstruction of missing information for few-shot NVS task. The results are even better than the state-of-the-art 3DGS methods with standard input image resolutions. The experiment is conducted on the LLFF dataset with three input views.

What can our method potentially benefit in the future?

• 3D latent diffusion model: The efficient latent radiance field would serve as a robust foundation for optimizing 3D latent diffusion models.
• Integration with other NVS compression methods: Our framework could seamlessly integrate with existing NVS compression methods to enhance their efficiency and scalability.

Reconstruction accuracy:

The purpose of demonstrating these comparisons is to evaluate the performance of 3DGS models under the same input resolutions. When the latents share the same resolution as the RGB images, the 3D latent representation outperforms all existing methods of 3D RGB representation. This finding provides valuable insights for downstream tasks requiring latent representations while addressing limited computational constraint.

Rendering speed:

Please refer to the following figure for the inference speed.

Storage space reduction:

Please refer to the below table for storage space comparisons. Our method demonstrates a significant advantage in storage efficiency. While 3DGS with the same input resolution achieves slightly smaller storage, its PSNR (14.03) is much worse than our method (22.45).

Other applications:

Please refer to Sec. 5.3 of the revised paper, where we additionally compare with the state-of-art text-to-3D generation method, GSGEN [r1]. Our method still works well on more complicated text prompts and generated geometries. For instance, in the results of vanilla GSGEN, the train on the cake appears distorted and lacks multi-view consistency. However, with the help of our model, we achieve consistent generation across views.

[r1] Z. Chen, F. Wang, and H. Liu, “Text-to-3d using gaussian splatting”, arXiv preprint arXiv:2309.16585, 2023.

Thank you very much for your time and effort in reviewing our work and providing the constructive comments.

Q1. Math definition of average pose error:

To compute $\lambda_{ij}$, we first calculate the Absolute Pose Error (APE) for each pose pair using the formula:$E_{ij} = P_i^{-1} P_j,$where $P_i$ and $P_j$ are the different camera poses respectively. After obtaining $E_{ij}$, the APE is calculated as:

$APE_{ij} = \|E_{ij} - I_{4 \times 4}\|_F,$

where $I_{4 \times 4}$ is the identity matrix and $F$ represents the Frobenius norm. In each iteration, the APE values are normalized across all image pairs to derive the weights $\lambda_{ij}$, as:

$\lambda_{ij} = \frac{APE_{ij}}{\sum_{k} {{APE}_{k}}},$

where $k$ represents each image pair within one iterations. This normalization ensures they reflect the relative contributions of each pose error in a consistent manner. This method is implemented based on the APE computation approach in the evo library [r1].

Thank you for your suggestion, we will add the above explanations in the camera ready verison.

[r1] Michael Grupp. evo. https://github.com/MichaelGrupp/evo/blob/d71da47342082626b7c90404e96363e89a05cc22/notebooks/metrics.py_API_Documentation.ipynb#L191

Q2. Since equation (6) becomes multi-objective optimization, does this change largely increase the training convergence time? Did you experience any convergence issues?

The convergence time does not increase significantly. Please refer to the following table for the rendering performance corresponding to different VAE training steps. The training converges in around 50k steps. Compared with the official training steps of 250k in Stable Diffusion VAE [r2], our method only needs 1/5 training time, which is comparably efficient.

[r2] https://huggingface.co/stabilityai/sd-vae-ft-mse

Q.3 Inference speed:

Please refer to the below table for the inference speed. Our method outperforms other latent-space approaches with the fastest inference speed and better rendering quality. Compared to image-space methods which avoid using a decoder, their image quality are significantly worse.

Q.4 Discuss CaesarNeRF:

Thank you for the helpful reference. Both CaesarNeRF and this work observe that the inconsistent feature representation would degrade NVS performance. However, our approaches are totally different. In CeasarNeRF, the inconsistent features from the reference views are calibrated by the relative camera rotations to achieve more consistent rendering results at the novel views. They employ a global semantic calibration for the entire scene, whereas we use a 3D-aware autoencoder for each image patch (corresponding to the receptive field of one pixel). The global semantic calibration fails to ensure local feature consistency, which is essential for capturing fine-grained geometric details. We did not experimentally compare with CaesarNeRF because it focuses on few-shot, feed-forward generalizable methods, while our approach is optimization-based, making the comparison unfair. We will discuss CaesarNeRF in the Related Work.