DEGS: Deformable Event-based 3D Gaussian Splatting from RGB and Event Stream

(1)The article frequently mentions low-framerate RGB as a premise. When RGB is sparse, low-quality 3DGS is a common phenomenon, which becomes more pronounced in dynamic scenes. However, DEGS does not clearly indicate whether the sparsity of RGB is a result of comparison with event signals or if it is simply a result comparison conducted in an environment with limited RGB data. A more detailed explanation of the results is expected. (2)Although DEGS achieves good scene representation results, much of the framework's design and usage relies on combining existing methods. The core of its approach is not an original design that breaks the framework of previous methods. As DEGS mentions in the conclusion, the current design has limitations and is not a tightly integrated fusion.

• The evaluations on real datasets could be more thorough. Why only three scenes are chosen from HS-ERGB dataset? How about the performance on other sequences?

• The results on real datasets are not always the best, as seen in Table 4, even the proposed method exploit additional event measurements. The proposed method performs only slightly better, or sometimes worse, than other methods. It would be helpful for the authors to analyze the reasons. Providing more results on real data could offer further insights.

• The presentation in Figure 1 could be further improved, in the part on how event measurements interact with the other parts.

1. The contribution of the proposed method is limited. The main goal of the proposed method is to accelerate the training speed of the previous event-based methods and reduce rendering costs. The most significant contribution to achieving this goal is to change the baseline (Deformable NeRF [R1]) of DE-NeRF to Deformable 3DGS, which appears to lack novelty.
2. The proposed unsupervised finetuning of the event flow predictor is unreliable. In section 3.2, the authors use equation 8 to finetune EV-flownet. However, the text lacks further analysis or related work to support that maximizing the variance of predicted optical flow (which differs from CM loss [R2]) could facilitate achieving more accurate flow prediction. A comprehensive study with convincing validation is necessary.
3. The details of the experiments lack clarity. The authors should explain how to create the train-test sets in the nine scenarios to ensure that the test set is captured in a novel view. It is worth noting that the three realistic scenarios from the HS-ERGB dataset were recorded using only one static camera, which implies that they cannot be utilized to evaluate the performance of novel view synthesis.
4. The writing quality of this paper is poor. Several crucial parts of the exposition are unclear. For example, why is it necessary to build event-Gaussian association in section 3.3 (since you can directly use optical flow to optimize the warp field without the association)? The utilization of symbols in section 3.2 (e.g., t_{r}ef), section 3.4 (e.g., the Proj function whose timestamp corresponding to the camera transformation matrix is not specific), and section 3.5 (e.g., the timestamp t1 and t2) is also confusing.

[R1] Park K, Sinha U, Barron J T, et al. Nerfies: Deformable neural radiance fields[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 5865-5874.

[R2] Shiba S, Aoki Y, Gallego G. Secrets of event-based optical flow[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 628-645.

1. The base assumptions in this paper somewhat diverge from real-world conditions. Although event data, due to its high temporal resolution, can provide detailed inter-frame motion clues for large or rapid movements, the large motions (fast movements) would, in practice, result in motion blur in RGB frames captured by regular cameras. However, the experiments in the paper assume that sparse, clear RGB frames are available—a point that warrants reconsideration. Using event data to deblur RGB frames could be a potential solution.

2. Using 2D optical flow estimates obtained from events as supervision signals introduces substantial noise into the optimization process for 3DGS and deformation fields, as the optical flow estimates are not entirely accurate (and the authors did not analyze the intermediate results of the optical flow estimation in their experiments).

3. I have concerns about the plausibility of the authors' proposed method for fine-tuning a pre-trained optical flow estimator using the event flow from a single scene. First, the paper does not provide details on the dataset used for fine-tuning, nor does it offer any quantitative evaluation of the fine-tuned estimator. The qualitative comparison in the two presented frames lacks Ground Truth, rendering it unconvincing. Moreover, the Contrast Maximization (CM) framework [3] used by the authors is only a model-based event flow estimation method, and its performance is often unstable and not accurate enough in various situations. Thus, the claim that fine-tuning EvFlowNet [1] using CM can improve performance is questionable.

4. Further discussion reveals that LoRA [2] is typically used to fine-tune large models trained on massive datasets to adapt to specific domains or tasks. However, the rationale for applying LoRA [2] to fine-tune a small model focused on a specific task like optical flow estimation using event flow from a single scene is questionable. Additionally, the pre-trained EvFlowNet [1] model mentioned in the paper has not been trained on a broad dataset; it is generally trained in very limited scenarios, such as those encountered in autonomous driving or drone perspectives. Therefore, the performance of the fine-tuned optical flow estimator and its ability to generalize to different motion patterns are concerning.

5. The authors' method of directly applying Structure from Motion (SfM) to multi-view inconsistent image sequences for obtaining point cloud initialization may not be well-defined and has certain limitations. While it is true that 3DGS does not require a completely accurate initialization, it is important to note that a high proportion of dynamic elements in the scene or low texture in static areas can hinder SfM from providing a reasonable initialization point cloud. Consequently, this method may fail in some scenarios.

6. The selection of comparison pipelines in the authors' experimental section is not reasonable, thus lacking persuasiveness. Among the four baselines compared, only one is an event-based radiance field reconstruction method, while the others are based on RGB data. The authors should compare their method with more radiance field approaches that utilize both event and RGB modalities, such as EvDNeRF [4].

7. The experimental setup for comparisons with RGB-based methods may be unfair, as the paper does not detail the number of RGB frames used by different pipelines during evaluation. Since RGB-based methods do not benefit from additional event data supervision, relying solely on sparsely sampled RGB frames makes it difficult to achieve good performance. This may also explain why the performance of the RGB-based methods in this paper is generally lower than that reported in the original papers. Therefore, using the same number of sparse RGB sequences for all pipelines while allowing the authors' pipeline to utilize additional event information constitutes an unfair comparison. The authors should establish a more equitable experimental setup that enables the RGB-based pipelines to perform at their normal levels. This situation reflects the authors' underlying assumption that ordinary cameras can still capture clear RGB images during large or rapid movements, which limits the practical value of their pipeline. Conversely, if ordinary cameras could obtain dense and clear RGB sequences, the necessity of incorporating event data would warrant reconsideration. Event cameras typically demonstrate their advantages over traditional cameras under high-speed motion and low-light conditions.

[1] Zhu, A. Z., Yuan, L., Chaney, K., & Daniilidis, K. (2018). EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. arXiv preprint arXiv:1802.06898.

[2] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... & Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

[3] Gallego, G., Rebecq, H., & Scaramuzza, D. (2018). A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3867-3876).

[4] Bhattacharya, A., Madaan, R., Cladera, F., Vemprala, S., Bonatti, R., Daniilidis, K., ... & Gupta, J. K. (2024). Evdnerf: Reconstructing event data with dynamic neural radiance fields. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 5846-5855).