Event-3DGS: Event-based 3D Reconstruction Using 3D Gaussian Splatting

Thank you for your suggestions and affirmation. The figures mentioned below can be found in the attached PDF.

W1: Combining 3DGS with Event cameras is somewhat meaningless.

R1: The combination of these two approaches is meaningful:

(1) Pure event data excels in SLAM, 3D reconstruction, and autonomous driving, especially in high-speed or low-light scenes where it outperforms RGB frames.

(2) For high visual quality, our method can be extended to a multimodal framework combining RGB and event data. This joint approach is useful for tasks like high-speed motion de-blurring (see Fig. 6) and HDR imaging.

W2: A more comprehensive comparison, including EventNeRF, E-NeRF, and Ev-NeRF.

R2: Per your suggestion, Ev-NeRF and EventNeRF can be used for comparison, but E-NeRF has a bug that prevents it from being tested.

(1) For Ev-NeRF, we report experimental results on both synthetic datasets (see Table R1 and Fig. R1) and real-world datasets (see Table R2 and Fig. R2). Note that, our Event-3DGS significantly outperforms Ev-NeRF in terms of SSIM, PSNR, and LPIPS.

Table R1: Comparison of Ev-NeRF and our Event-3DGS in the synthetic dataset

Table R2: Comparison of Ev-NeRF and our Event-3DGS in the real-world dataset.

(2) For EventNeRF, it processes color event data unlike our format. Thus, we converted its synthetic data to our classic event data format. Table R3 and and Fig. R3 shows that EventNeRF and our Event-3DGS perform comparably. Note that, EventNeRF’s data differs from the original paper due to additional color adjustments made before metric calculation. This adjustment may be not entirely reasonable, so we compare only the raw reconstructed images in the two methods. Besides, while EventNeRF requires a long training time for each scene, our Event-3DGS trains each sequence in under 10 minutes. In short, Event-3DGS offers better efficiency while maintaining comparable reconstruction quality.

Table R3: Comparison of EventNerf and our Event-3DGS in the synthetic dataset.

(3) For E-NeRF, we attempt deployment but faced insurmountable issues. A review of related GitHub discussions reveal that others have encountered similar problems without resolution from the authors, leading us to abandon this comparison.

Overall, we have tested many sequences with high-noise real event data, proving its practicality. The real object data used in EventNeRF comes from a different color event camera, which makes it unsuitable for our algorithm.

W3: Making the threshold a learnable variable and optimizing it with the 3DGS scene representation may be unreasonable.

R3: Emphasizing adaptive thresholds is valuable. We explore a basic approach using an adaptive threshold for a single sequence. While varying scene or pixel-level thresholds could improve reconstruction quality, this would likely increase computational complexity, which we acknowledge as a limitation. These points are important and could be explored further in future work.

Q1: Can the author provide rendering speed and training times on various datasets?

A1: Our Event-3DGS optimization is fast. For instance, using the Ficus sequence, we achieved the reported results in approximately 9 minutes and 47 seconds on an RTX 3080 Ti. However, the original 3DGS has high GPU memory requirements, and running out of memory can significantly slow down subsequent reconstructions.

Q2: How is the threshold initialized? How can it be optimized with the scene?

A2: We can start by setting the threshold to the manufacturer’s standard value or an estimated range (typically between 0 and 1). During training, this threshold can then be optimized as a learnable parameter alongside the scene parameters. This approach differs from a learning rate schedule, which is a manually designed strategy not directly tied to backpropagation or training but helps facilitate the training process. We will add these details in the revised version.

Q3: How does the experiment generate blurred images through integration in Fig. 6?

A3: To accurately simulate the blur effect, we start with a high-quality 3D model and render the blurred images. During rendering, we calculate the spectral integration of the camera's motion based on its speed to produce the final blurred image. Given the ground truth data of the 3D scene distribution, the resulting blurred images closely approximate real-world conditions.

Thank you for your recognition of our work and for raising your score from a 5 to a 6. We appreciate your constructive feedback and support, which have been instrumental in improving our research.

The figures (i.e., Fig. R1 and Fig. R2) mentioned below can be found in the attached PDF.

Q1: The paper is not well written and contains many errors, it is very difficult to follow.

A1: Thanks. We will clarify the writing to improve reader understanding in the camera-ready version.

Q1-1: In Eq. 9, why it equals to the negative of $E$?

A1-1: Eq. 9 is a parameter set used to simplify assumptions, which is convenient for the theoretical derivation and representation of subsequent filtering. $w$ should not be treated as a Gaussian noise term. The negative term is reasonable, as it accommodates step-like changes in light intensity, which are typically within the feasible range of $w$ . We will clarify this explanation in the revised version.

Q1-2: $V$ is used to denote signal after Laplace transformation.

A1-2: The notation $V$ should indeed be different after the Laplace transform. We will correct this in the camera-ready version.

Q1-3: Eq. 12 conflicts with Eq. 10.

A1-3: Eq. 10 represents the pure integration method, which estimates light intensity differences under ideal conditions but performs poorly in noisy environments. In contrast, Eq. 12 employs high-pass filtering, making it more robust to noise. Thus, Eq. 10 and Eq. 12 are two different approaches for estimating light intensity differences and are not contradictory. We will add the details in the revised version.

Q1-4: It is confusing on "when the time is ..." in Line 127.

A1-4: The camera pose $T$ can be regarded as a function of time $t$ , which is generally not one-to-one. For example, a camera might return to the same position with the same orientation at different times. However, within a sufficiently small time interval, this function can be treated as one-to-one, meaning each pose corresponds to a unique point in time. We will provide a more detailed description of this in Line 127 in the revised version.

Q1-5: Line 147, the typo minor.

A1-5: The omission of the subscript was a mistake, and we will correct it in the revised version.

Q2: Experimental evaluations against prior methods are not sufficient. It misses the comparisons on the synthetic datasets. The evaluations on real data against Ev-NeRF also misses some sequences in Table 2.

A2: Thanks for your comments. We have conducted additional experiments comparing our Event-3DGS with Ev-NeRF on both synthetic (see Table R1) and real-world datasets (see Table R2). Our Event-3DGS significantly outperforms Ev-NeRF in terms of SSIM, PSNR, and LPIPS. Additionally, we provide representative visualization examples for both the synthetic data (see Fig. R1) and the real-world dataset (see Fig. R2). Note that, we have added 3 additional sequences in the real dataset that are not shown in Table 6 of the appendix. These results demonstrate that Event-3DGS achieves superior reconstruction quality compared to Ev-NeRF. We will present these experimental results for Ev-NeRF in the camera-ready version.

Table R1: Comparison of Ev-NeRF and our Event-3DGS in the synthetic dataset.

Table R2: Comparison of Ev-NeRF and our Event-3DGS in the real-world dataset.

Q3: In terms of the ablation studies on the contribution of each component, why the authors chose the baseline that integrates E2VID and 3DGS for event based 3D reconstruction. It should be experiments for the proposed method instead of E2VID.

A3: Sorry for the writing mistake that may have misled you. We confirm that the data in Table 3 is correct, but the description of the baseline in Lines 250-251 is incorrect. The baseline should be described as a pure integration image without the adaptive threshold and event loss, not E2VID+3DGS. Additionally, comparing Table 2 and Table 3 shows that the ablation baseline performance in Table 3 does not match the E2VID performance in Table 2, confirming that the baseline is not E2VID. We will correct the baseline description in Lines 250-251 in the camera-ready version.

Q4: Dependency on known poses. It is usually difficult to obtain GT poses for event camera alone.

A4: In general, obtaining accurate camera poses in normal scenes using only event cameras is challenging compared to RGB frames, due to the sparse sampling of event cameras. However, recent works [1, 2] show that event cameras can be effective for pose estimation, particularly in challenging scenarios such as high speeds or low light. For example, Muglikar et al. [1] developed the E2Calib tool, which uses E2VID to reconstruct gray frames, calibrate camera extrinsics, and estimate poses. In our work, we use E2VID to reconstruct frames and then apply COLMAP to process frames to estimate camera poses. Thus, the camera poses estimated using event data enable our Event-3DGS to produce high-quality reconstructions in real-world applications.

[1] How to calibrate you event camera, CVPRW 2021.

[2] EVO: A geometric approach to evnet-based 6-DOF parallel tracking and mapping in real time, IEEE RAL 2017.

Thanks for the effort in detailed rebuttal. My questions are addressed and I would like to upgrade my rating. It is suggested to further improve the writing and incorporate all the new experimental results into the final version.

Thanks for your positive evaluation.

W1: In the writing, it is better to avoid abbreviations like “it’s” and write “it is”.

R1: we will add the missing words and change “it’s” to “it is” in the camera-ready version.

W2: A word is missing in Line 147.

R2: We missed "event sensor" in Line 147, and will correct it in the camera-ready version.

W3: Questions about Equation (7).

R3: Our intention is to use $t$ as the subscript and $p$ as the superscript to represent the time when the $i$-th event is triggered at pixel $p$, without implying an exponent. We will correct this in the camera-ready version.

W4: It seems that numbers in the text Line 230 do not match the numbers in Table 2.

R4: Actually, Line 230 corresponds to the numbers in Table 1, not Table 2. The statement "our Event-3DGS has improved by 0.017 and 2.227, respectively" refers to the performance metrics for the synthetic dataset shown in Table 1, not the real-world dataset in Table 2.

Thanks for you insightful suggestions and affirmation. The tables and figures mentioned below can be found in the attached PDF.

W1: Rewriting should be clear for the NeurIPS audience.

R1: We will add a high-level introduction to Gaussian Splatting in the main manuscript and provide clearer explanations in the appendix of the revised version.

W2: Some terminology may be hard to understand. Not focus on reconstruction but on NVS.

R2: Thanks. We will clarify the terminology in the revised version.

Besides, existing 3DGS methods mainly evaluate NVS to align with common image reconstruction quality metrics. Therefore, we also use NVS to assess our Event-3DGS. However, our method is not restricted to NVS and can be extended to obtain depth and reconstruct 3D meshes.

W3: The work seems not to compare against any NeRF techniques. Please explain the performance contributions.

R3: First, we compare our Event-3DGS with a typical method (i.e., Ev-NeRF) on both synthetic datasets (see Table R1 and Fig. R1) and real datasets (see Table R2 and Fig. R2). Note that, our Event-3DGS significantly outperforms Ev-NeRF in terms of SSIM, PSNR, and LPIPS.

Second, our Event-3DGS benefits from both 3DGS itself and our proposed modules (see the ablation test in Table 3). Unlike NeRF’s ray tracing, 3DGS uses raster-based rendering to generate the entire image at once, which can introduce errors if many pixels are not sampled when events occur. Our loss design and filtering techniques address these errors, improving performance.

W4: Some writing minor points.

R4: Thanks. We will address these writing minors one by one.

Q1: In Line 112, what is meant by "point mapped to this pixel"?

A1: The transparency of the 3D Gaussian function is first mapped to screen space. Next, pixel transparency is determined based on each pixel's position within the 3D Gaussian function. The color is determined from the spherical harmonics lighting coefficients. We will clarify this description in the revised version.

Q2: In Line 200, elaborate on the theoretical analysis.

A2: An event camera cannot capture intensity information between events, and the intensity can vary within the threshold range (see Eq. 8). The parameter selection in Line 200 considers this, ensuring that the rendered intensity within the positive and negative threshold range minimizes the loss, aligning with the theoretical analysis. We will add this analysis in the camera-ready version.

Q3: The setting and optimization of $\alpha$.

A3: $\alpha$ is a hyperparameter chosen to provide a good initial value for 3DGS. It doesn't need to be exactly 0, and any small value can work. While setting $\alpha$ to the maximum value is possible, it may lead to local optima issues.

Q4: The datasets setting and the length of the event sequences sampled.

A4: We split each synthetic or real dataset, reserving a portion of the images for testing while using the rest for reconstruction training. The event sequences vary in length, but generally contain more than a million events. We will add these details in the experimental settings in the revised version.

Q5: What is the impact of different values for $\tau$? Is there a reason why it is set to 0.05?

A5: This hyperparameter $\tau$ controls the frequency range of the high-pass filter. A narrower range reduces noise but may discard more original event information. Besides, our tuning experiments show that setting this parameter to 0.05 yields better results.

Q6: What is meant by "$E(p, t)$ is ideal" in Line 164?

A6: "The ideal" means that an event perfectly represents intensity changes without any noise. In practice, events are affected by background noise, refractory periods, leak noise, and bandwidth limitations. We will provide more details in the revised version.

Q7: What is the intuition of high pass filtering. Is the noise assumed to be low frequency?

A7: Yes, due to the higher noise levels in event cameras compared to traditional RGB cameras, especially in low-light scenes, the noise primarily consists of low-frequency signals compared to the effective events.

Q8: What is the purpose of the subscript $i$ in Eq. 14.

A8: The symbol used here seems inappropriate, as it was intended only to differentiate it from the subsequent event loss. We will correct this in the camera-ready version.

Q9: Line 253, in what sense is the threshold adaptive?

A9: The threshold for detecting light changes in event cameras may not be directly obtained from event data, and it can be influenced by the scene environment (see Lines 148-150). Accurate threshold estimation is crucial for obtaining light intensity from a mathematical optimization perspective. To improve reconstruction quality, we design an adaptive threshold for each scene (see Table 3).

Q10: Table 7, goes down as training progresses.

A10: The training has essentially converged at a certain point, with subsequent steps involving only minor perturbations. In practice, although there are fluctuations after convergence, the overall performance remains stable.

Q11: Explain how the combined method.

A11: We have provided the combination of events and frames in the appendix (see Lines 469-488). Briefly, we compare the rendered image with the RGB frame to obtain a loss, which is then added to the total loss (see Eq. 16 and Eq. 19).

Q12: Visualizations created.

A12: In pure event-based imaging, restoring absolute light intensity is generally not possible. We specify the initial light intensity (e.g., setting the background intensity) as done in previous methods. Typically, this value can be set to 0, representing pure black.

Q13: Colors is not realistic.

A13: We assume each pixel has three channels of events, differing from the Bayer pattern. Therefore, our algorithm first interpolates the raw data into a three-channel format. In the future, we could extend our Event-3DGS to handle raw color event data in the Bayer pattern.

Thank you for your responses. Many of my points have been addressed and I still see the paper still on the accept side. The reason I tend to not increase my score further is that some of the promised changes require a significant change in writing and it is hard to judge those without having another full round of reviews.