Generalizable Dynamic Radiance Field in Egocentric View

Thank you for your careful review and helpful suggestions!

Q1&Q2: How the model could generate novel views of dynamic scenes at different timestamps. The time-variant motion features may be mixed into one triplane representation.

A1&A2: We are sorry to make you confused. We clarify it in revised paper. Concretely, our method is as follows: we use a learned triplane for 3D representation. When rendering novel views at target time $t_1$, we use the learned triplane as the initialization and update the learned triplane by aggregate features from source views at other times using view-attention module. To compute attention scores, we also take time $t_1$ and other times {t}$_{t=1}^S$ as query and key, respectively. In this way, the view-attention module can leverage temporal correlation to aggregate features. Thus, at different times, the triplane feature is updated based on source views and the associated temporal difference.

Q3: The paper claimed that it could learn semantic information with the proposed pipeline, but it only tests its image encoder performance on image classification tasks, which is somehow weak. Semantic learning of dynamic scenes should be demonstrated by challenging semantic tasks like segmentation, tracking, completion, and generalization in Semantic Flow [1] paper.

A3: For our method, it is a self-supervised approach. Unlike Semantic Flow[1], we do not leverage any semantic labels. As shown in semantic learning section, the high classification performance on related categories demonstrates that our method can emerge the capability of semantic learning without any semantic supervision. From this point, our method is a promising approach for representation learning. For challenging tasks in Semantic Flow [1] paper, we think it can be solved by our method when the training data increases. Due to training data issues, we leave it for future work. We must note that we provide a promising path to learn semantic in self-supervised manner in this paper.

Q4: The paper uses many non-egocentric view datasets for evaluation. It is weird to conduct experiments on these dataset as the goal of the paper is to build generalizable dynamic radiance fields in egocentric view. I want to know why.

A4:

egocentric view: In general, egocentric view is only a modeling approach. It takes observer as world origin to model dynamic scenes. For each video frame, we use camera center as world origin. That means under ego-view modeling, all videos can be taken as egocentric datasets.

many non-egocentric view datasets for evaluation: For the evaluation, we need use some specific datasets that have multi-view image and pose annotations. NVIDIA Dynamic Scenes satisfies the dataset requirement, thus we use it for evaluation.

Q5: Some citation formats in this paper seem wrong. Please double-check citation formats.

A5: Thanks for your suggestions. We have solved it and updated the paper.

Thank you for your detailed response.

After carefully reading the rebuttal. I believe that the current problem of this paper is the unclear details of method. Although the revised version of the paper provides more technical details, there is no clear overview of how the model is trained (same point with PbCf). In this way, it is not clear why the propose methods achieve better generalization ability. Besides, I am still confused the propose of the "egocentric" in the title, as there is no clear connection between the "egocentric" word and the model details in the proposed method. "use camera center as world origin" is just a definition of camera coordinate and can be applied to any other dynamic scene reconstruction models.

I think the contribution of paper the authors try to propose is interesting and valuable. However, I believe that it needs further revision due to these unclear expression in the paper. Hence I keep my rating.

Thank you for your careful review and helpful suggestions!

Q1: I think a major weakness is in the evaluation of the approach, particularly regarding generalization.

A1: We thank the reviewer for this very helpful and insightful comment! 

To make generalizability of our method clear, we conduct more experiments and update the experiment section. Specifically, we use PGDVS+ (generalizable version of PGDVS) as baseline. As shown in Table 1, our method outperforms PGDVS+ on all metrics by a large margin. We also provide qualitative comparison in Figure 3. Compared with PGDVS+, our method has superior performance on both dynamic and static scenes. In addition, we evaluate our method on DAVIS datasets and show the results in Figure 4. To demonstrate the effectiveness of our method, we also provide the video samples in supplementary material.

Q2: Comparison with other multi-view approaches (even if they're static or optimization-based)

A2: In the paper, our method focus on monocular video setting, which is more challenging than multi-view settings. Thus, it is unfair and unreasonable to compare with other multi-view approaches.

Q3: more distinct views from input views to better understand the limitations

A3: We have presented the limitations for our method. For more distinct views from input views, the render views have blurred or black areas, which are unseen content that does not exist in source views. For our method, we only take 6 sequent images as source views. It is easy to meet unseen content when rendering more distinct views from input views. To solve it, we will introduce diffusion models to generate unseen content in the future.

Q4: results and comparisons regarding first-person vs other types of samples.

A4: Thanks for your suggestion. As shown in Introduction, our work focuses on modeling dynamic world in egocentric view to simulate human. Other types of samples is beyond our focus.

Q5: The paper does not have any analysis nor comparisons regarding latency and speed. This would make clear the efficiency advantage over optimization-based approaches.

A5: We evaluate the inference speed of our method on an L40 machine using NVIDIA scenes. At a resolution of 288x512, our approach renders each target frame in just 15 seconds. In comparison, the PGDVS method requires 91.4 seconds per frame (without depth overfitting)—approximately six times longer. This substantial difference in processing speed underscores the efficiency of our method.

Q6: Some closely related literature on Gaussian Splatting (GS) is not mentioned

A6: Thanks for your suggestion. We will discuss these works in the final version.

Q7: How is the initial triplane learned/initialized?

A7: The goal of the learnable triplane is to learn a good initialization, which is useful to aggregate image features. Although the learned triplane is shared by all scenes, it is updated by aggregating the image features from source views at different times. Thus, the updated triplane is different for any target scene.

Q8: The concept of feature "similarity" is used in a few places but is not properly defined.

A8: Thanks for your suggestion. “similarity" refers to the attention scores computed between 'query' and 'key' elements within each attention module.

Q9: "epipolar feature" appears to be simply projected pixel features. If so, the use of term "epipolar" is uncommon and confusing.

A9: Thanks for your suggestion. "epipolar feature" indeed refers to projected pixel features. We will revise it in our paper.

Q10: "time bias" on L200 is not defined.

A10: “time bias” means the target time and source view times. To compute attention scores in (temporal-aware) view attention, we also take time $t_1$ and source times {t}$_{t=1}^S$ as query and key, respectively. In this way, the view-attention module can leverage temporal correlation to aggregate features. We will clarify this term in the final version.

Thank you for your careful review and helpful suggestions!

Q1: The generalizability is tested on only one dataset. While the main focus is on generalizability, the primary results table (Table 1) does not reflect this aspect.

A1: We also evaluate the generalizability of our method on RealEstate 10K, as shown in Table 2. To make generalizability of our method clear, we conduct more experiments and update the experiment section. Specifically, we use PGDVS+ (generalizable version of PGDVS) as baseline. As shown in Table 1, our method outperforms PGDVS+ on all metrics by a large margin. We also provide qualitative comparison in Figure 3. Compared with PGDVS+, our method has superior performance on both dynamic and static scenes.

Q2: Could the algorithm be evaluated on additional datasets to assess its generalizability?

A2: Thanks for your suggestion. We evaluate our method on DAVIS datasets and show the results in Figure 4. To demonstrate the effectiveness of our method, we also provide the video samples in supplementary material.

Q3: Additionally, in Table 1, it states, "Our model is trained on scenes from NVIDIA, EPIC, Plenoptic, and nuScenes (train set)" (Line 370). Why does the performance remain limited compared to other methods?

A3:

Limited performance: Other methods are scene-specific approaches. They overfit each scene by optimization. It is well known overfitting method is better than generalizable method. We must point out that our method is a data-driven method and has a potential performance improvement. If we train it with large-scale and diverse scenes, our method can have a better performance and generalize to any scene.

Q4: Missing reference: DynPoint: Dynamic Neural Point For View Synthesis NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos

A4: Thanks for your suggestion. We will discuss these works in related works.

Thank you for your careful review and helpful suggestions!

Q1&Q2: The methodology part is not straightforward to understand. The methodology is not aligned to the title or motivation of this paper, focusing on 'egocentric views'

A1&A2: We update the paper and clarify the methodology part. In our methodology, 'egocentric views' is a modeling approach. It takes observer as world origin to model dynamic scenes. For each video frame, we use camera center as world origin. To implement this modeling, we set triplane center as camera center.

Q3: Experimental results are far from 'comparable' to the previous approaches.

A3: Thanks for your suggestion. We have revised it in our paper.

Q4: This paper includes shallow ablation study.

A4: I disagree with this statement. Only having a strong 4D representation, can our model render high novel views at different time. Thus, the novel view results in ablation study can reflect the ability of our model on 4D scene modeling.

Q5: Overall qualitative results are not curated well to present the effectiveness of the proposed method.

A5: Thanks for your suggestion. We have provided new qualitative results in Figure 3. The comparison is between PGDVS+ (generalizable version of PGDVS, baseline) and our method. The results demonstrate that our method beats PGDVS+ on both dynamic and static scenes by a large margin.

Q6: Can you explain what exactly the suggested attention modules in 4D transformer are trying to 'learn'? For example, what is 'axis-attention', implicitly? What does it mean, and how does it help to understand 4D scene?

A6:

view-attention module(temporal-aware): As shown in paper, this module is used to aggregate feature from source views to update triplane. To compute attention scores, we also take time $t_1$ and other times ${t}_{t=1}^S$ as query and key, respectively. In this way, the view-attention module can leverage temporal correlation to aggregate features. Besides, we utilize epipolar priors(3D geometry) to extract image features from source views. Thus, temporal-aware view-attention module has a 4D understanding (temporal+3D geometry) on scene.

axis-attention: As shown in paper, axis-attention is proposed to reduce the cost of directly applying self-attention on virtual point features. Due to huge virtual point, the cost of applying self-attention on these point directly is expensive.

plane-attention module: As shown in paper, epipolar priors used in temporal-aware view-attention is invalid for dynamic scenes. The virtual point features for dynamic scenes need to refine. Thus, in plane-attention module, we leverage 3D priors to refine. 3D prior is derived from two sources. First, triplane itself is a 3D-structured representation. Second, the rendering is based on triplane and high quality rendering means that triplane has 3D prior knowledge. In the training, triplane should learn these 3D prior knowledge from training dataset.

Q7: Can you explain how the suggested modules are specifically designed for 'egocentric' views?

A7: In general, egocentric view is only a modeling approach. It takes observer as world origin to model dynamic scenes. For any video frame, we use the camera center as world origin. That means under ego-view modeling, any video can be taken as egocentric datasets. To implement this modeling, we set triplane center as camera center.

Q5: However, their strength in terms of generalization and dynamics representation is unclear. It would be great to prove the proposed 4D-aware transformer's effectiveness in both generalization and dynamics representation by comparing it to previous methods. For instance, compared to previous dynamic content embedding methods (Li et al. (2023); Tian et al. (2023)), is the View-Attention Module superior to representing dynamic content?

A5: First, the design philosophy of our 4D-aware transformer is shown in A2. Second, Only having a strong 4D representation and generalization, can our model render high novel views at different time. Thus, the novel view results in ablation study can reflect the strength of each module.

Comparison with previous dynamic content embedding methods (Li et al. (2023); Tian et al. (2023))

We must point out that it is unreasonable for our (temporal-aware) view-attention module to compare with these dynamic content embedding. The motivation and methodolgy are different. Our method is a self-supervised approach, thus can not take priors as input, such as 2D mask and optical flow. But, for Li et al. (2023) and Tian et al.(2023), they rely on 2D mask and optical flow. Moreover, to our knowledge, these priors may have non-negligible errors, which are harmful to dynamic scene modeling, see Figure 3 (new version of paper).

Thank you for your careful review and helpful suggestions!

Q1: However, if the target training set gets diverse, the target of the learnable triplane is unclear. In the current training scenario, what is the learning goal of the learnable triplane, and what do they learn?

A1: Although the learned triplane is shared by all scenes, it is updated by aggregating the image features from source views at different times. Thus, the updated triplane is different for any target scene. In this way, we can use the updated triplane to render high quality novel views. The goal of the learnable triplane is to learn a good initialization, which is useful to aggregate image features.

Q2: Also, it would be great to discuss whether the current framework is suitable for dynamic radiance field generalizatoin. The current frameworks seem unsuitable for radiance field generalization to handle totally unseen and out-of-distribution domain data, such as mountains, caves, or endoscopes.

A2: The design philosophy of our method is to learn a good 3D representation for a scene at target time $t_1$.

In our framework, we take a learnable triplane as the 3D representation. For generalization, the learnable triplane should be updated according to different scenes and target times.

For a scene at target time $t_1$, we should update the triplane feature by aggregating image features from source views at times {t}$_{t=1}^S$. During the aggregation, we also utilize temporal correlation between target time and source times to compute attention scores. This is the motivation of (temporal-aware) view-attention module.

Due to epipolar priors used in (temporal-aware) view-attention is invalid for dynamic scenes, the virtual point features have much noise. To solve this issue, we propose plane-attention module and axis-attention module to refine these features.  Axis-attention module can reduce the cost of directly applying self-attention on virtual point features. Plane-attention module can leverages 3D priors to remove the noise from virtual point features.

As shown in experiments, the effectiveness of the proposed method is validated by the generalization performance.

generalization to handle totally unseen and out-of-distribution domain data

For our method, it is a generalizable approach and can be applied to any scenes. But it is a data-driven method. With limited training scenes, our method can not be applied to scenes that you mentioned, due to lack of related semantic knowledge. We must note that generalization ability means the ability of assembling learned semantic knowledge when encountering totally unseen objects/scenes. It is well known that semantic knowledge is learned from large-scale datasets. Thus, our method have a potential generalization when trained with large-scale datasets. We hope the reviewer to think: if you are required to describe out-of-distribution domain data, like God, what the image of God in your mind is?

Q3: In novel view generation in both the seen and unseen domains, the proposed method doesn't outperform the pseudo-generalized methods MonoNeRF Tian et al. (2023) and PGDVS Zhao et al. (2024).

A3: We must point out that our method outperforms the pseudo-generalized methods MonoNeRF Tian et al. (2023), as shown in Table 1.

PGDVS is a scene-specific approach. It overfits each scene by optimization. It is well known overfitting method is better than generalizable method.

We must point out that our method is a data-driven method and has a potential performance improvement. If we train it with large-scale and diverse scenes, our method can have a better performance and generalize to any scene.

Q4: If the method is truly generalizable, it should be tested with totally out-of-distribution data.

A4: We must note that generalization ability means the ability of assembling learned semantic knowledge when encountering totally unseen objects/scenes. It is well known that semantic knowledge is learned from large-scale datasets. Unlike existing methods, our method is a data-driven approach and can benefit from large-scale dataset. Our method have a potential generalization when trained with large-scale datasets.

To make generalizability of our method clear, we conduct more experiments and update the experiment section. Specifically, we use PGDVS+ (generalizable version of PGDVS) as baseline. As shown in Table 1, our method outperforms PGDVS+ on all metrics by a large margin. We also provide qualitative comparison in Figure 3. Compared with PGDVS+, our method has superior performance on both dynamic and static scenes.

I appreciate the author's detailed response. I carefully read all other reviews and their corresponding responses.

Most reviewer has similar concerns in the definition and experiments regarding generalization (D5Cb,PbCf,Bx6N), presentation issue of the proposed method (aFqB, fo5V), and the learning objective of triplance representation (aFqB, fo5V,PbCf,Bx6N). Even after reading all the reviews and responses, the learning objective and the definition of generalization in this paper are unclear to me. Also, as mentioned by Reviewer Bx6N and PbCf, the current experimental setup can be regarded as in-domain distribution (i.e., similar environments were already seen during training), which is far from generalization. So, I think the paper needs clarification on writing, problem definition, and problem formulations.
Therefore, I maintain my initial rating.