V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation

Response to Reviewer bJ8Q

Thank you for your detailed feedback. We will address your concerns below.

S1. Clarification of contributions.

Thank you for your appreciation of our contributions. However, we would like to clarify that V2V is not a specific dataset, but a general framework for efficient event data simulation, especially for large scale datasets. It can be used to generate synthetic data for a large range of event-based vision tasks, including but not limited to video reconstruction and optical flow estimation.

W1. Similarities to existing simulators.

It is indeed common for event simulators to first calculate the amount of events triggered on each pixel -- in other words, the voxel -- before assigning timestamps to the individual events. This paradigm can be found in V2CE, V2E, EventGAN and many other simulators. Our key novelty lies in removing the final, expensive voxel-to-event step. By arguing that the intermediate event stream is unnecessary for many deep learning pipelines, V2V directly generates discrete voxels, fundamentally improving efficiency and avoiding the ill-posed problem of interpolating continuous temporal data. We will add this discussion to our related works section.

W2. Insuffucient exploration of model design.

The focus of our paper is to propose a general data framework that could boost model performance accross different tasks and different models. It would indeed be beneficial to explore specific model designs that can better utilize the V2V framework, but this is beyond the scope of our current work.

W3. Limitations of synthetic datasets.

Our V2V framework indeed produces synthetic data, and does not provide large-scale real data. However, compared to existing data synthesization methods, the V2V framework scales up the amount of realistic video-based synthetic data, and also allows for more flexible data augmentation methods. This help boosts the performance and robustness of the models trained, which is the source motivation of pursuing large-scale real datasets.

W4 & Q1. Disentanglement of V2V framework & data scaling.

The key advantages of our V2V framework are efficiency, scalability and flexibility. The efficiency and scalability contribute to the model performance by allowing the use of large-scale video datasets, so their contributions cannot be disentangled from data scaling. The flexibility, however, can be disentangled and verified.

The flexibility refers to the variety of data augmentation methods applicable when using the V2V framework. For example, in each epoch, a video can be combined with different event thresholds and noise levels, serving as different training samples. Existing event simulators do not support changing thresholds after simulation, so we can evaluate the benefits of the V2V framework by disabling the threshold changing functionality.

Specifically, to disable the threshold changing functionality, we randomly sampled threshold values for each video, and fixed them throughout the training process. We annotate these ablation experiments as "V2V-Fixed" yellow dots in Figure 5. The full methods, with thresholds randomly sampled in each epoch, are annotated as "V2V-Random" green dots.

As visualized in Figure 5 (qualitative results are in Table 2), disabling the threshold changing functionality leads to a performance drop, no matter which scale the training dataset is (100 -> 1K -> 10K). This verifies that the flexibility of the V2V framework contributes to the model performance, disentangled from data scaling.

Q2. Different performance gains over EVAID and HQF.

The more significant gains on EVAID compared to HQF stem from EVAID being a more challenging and higher-quality dataset, offering more room for improvement.

1. EVAID has a higher frame rate (18 ~ 150 FPS) than HQF (17 ~ 26 FPS). This requires models to generalize to higher temporal resolutions and sparser event streams.

2. EVAID is captured with a Prophesee EKV4 event camera, with resolution 720x1280. HQF is captured with DAVIS346, with resolution 346x260. To gain good performance on both EVAID and HQF, the model would need to generalize to higher spatial resolutions, and also generalize to the different noise characteristics.

3. There are long periods in EVAID where the camera is static, and no events are generated in the background (an example begins 01:40 of the supplementary video). This requires the model to "remember" the background information reconstructed in the beginning, and distinguish the texture it needs to remember from the noise it should gradually smooth out.

As a result, existing models meet more challenges when tested on EVAID, while the metrics over HQF are near saturation. Hence, our V2V-retrained versions, with better generalization capabilities, can achieve more significant performance gains over EVAID than HQF.

L1. Missing limitations.

Due to the space limit, we moved our limitations section to the appendix (Section G). We apologize for the resulting inconvenience.

Response to Reviewer rSE5

Thank you for your positive feedback and insightful suggestions. We will address your concerns below.

W1 & Q1. Comparison against existing video-based simulators.

Thank you for your suggestion. In order to compare our V2V framework to a "frame interpolation + event simulation" approach, we conducted two new experiments based on the WebVid-100 dataset.

1. Frame interpolation: We interpolated the WebVid-100 videos by 8x using the PerVFI [1] model. We selected this model since it was proposed recently (CVPR 2024) and code is available. If required, we can change it to other models.

   [1] Perception-Oriented Video Frame Interpolation via Asymmetric Blending. Guangyang Wu, Xin Tao, Changlin Li, Wenyi Wang, Xiaohong Liu, and Qingqing Zheng. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

2. Event simulation: We used the popular V2E simulator to generate events. We used the "clean" noise mode. For the thresholds, we used the same random thresholds as in the "fixed thresholds" ablation experiment. Since the logarithm mapping of V2E is different from the ESIM simulator (our V2V framework follows ESIM), we scaled the thresholds linearly. In particular, the logarithm mappings of V2E and ESIM are:

   def ESIM_log(x):
   	# The output has range [-6.908, 0], with range length 6.908.
   	return np.log(0.001 + x / 255.0)
   
   def V2E_log(x):
   	# The output has range [0, 5.545], with range length 5.545.
   	if x <= 20:
   		return (1.0 / 20) * math.log(20) * x
   	else:
   		return np.log(x)
   

   So, for a video corresponding to thresholds $c_+$ and $c_-$, we set the V2E thresholds to $c_+ * 5.545 / 6.908$ and $c_- * 5.545 / 6.908$, respectively.

   In the simulation, a video (1066695484.mp4, with original size 1.2MB) produced 14GB of events. This caused our post-processing step to fail due to out-of-memory issues. Since the rebuttal stage is time-limited, we decided to discard this video, leaving us with a WebVid-99 dataset. The WebVid-99 dataset (in raw events) sums to 12 GB space, while the original WebVid-100 videos sum to 210 MB.

3. Voxel stacking. We stacked the events into interpolated voxel bins. In this stage, we need to select the durations of the voxel bins. In the V2V experiments, each voxel corresponds to the events between 6 real frames, and the E2VID models reconstruct 24/5 FPS videos. We tried two binning strategies in our V2E experiments:

   • In strategy 1, we encode all the events between 1+1 original frames (crossing 7 interpolated frames) into a single voxel. This makes the E2VID models reconstruct 24 FPS videos, utilizing the high temporal resolution from frame interpolation. We annotate this as "V2E-1".

   • In strategy 2, we encode all the events between 5+1 original frames (crossing 35 interpolated frames) into a single voxel. This makes the E2VID models reconstruct 24/5 FPS videos, aligning to the V2V experiments. We annotate this as "V2E-5".

4. Training and results. During training, we used the same augmentation noise (gaussian noise + hot pixels) as the ESIM protocol. For the V2E-5 experiment, we train for 8000 epochs, aligning to the V2V-WebVid-100 experiment. For the V2E-1 experiment, it produces 5x more training samples, so we train for 1600 epochs.

   The test results are shown in the table below. The V2V-1 experiment produced better results than the V2V-5 experiment, showing that training on shorter bins may indeed benefit model performance. However, both V2E experiments produced metrics worse than the "WebVid-100-Fixed" experiment, which also uses fixed thresholds for each video.

   Idx	Model	Train Dataset	Loss	Epochs	HQF MSE	HQF SSIM	HQF LPIPS	EVAID MSE	EVAID SSIM	EVAID LPIPS
   (b)	E2VID	WebVid100-Fixed	V+L1+TH	8000	0.049	0.611	0.330	0.073	0.604	0.451
   (e)	E2VID	WebVid100	V+L1+TH	8000	0.042	0.626	0.327	0.078	0.594	0.441
   (u)	E2VID	WebVid99-V2E-1	V+L1+TH	1600	0.057	0.524	0.459	0.116	0.484	0.561
   (v)	E2VID	WebVid99-V2E-5	V+L1+TH	8000	0.077	0.470	0.569	0.117	0.483	0.588

Q2. Simulation parameters.

Details of the selection of simulation parameters are as follows.

1. We first sample a threshold $c$ from the range [0.05, 2]. Note that we calculate logarithm images with np.log(0.001 + video/255.0) (following the ESIM simulator), which means that the amount of change between two frames is in the range of [0, 6.908].

2. When using real event cameras, it is improbable that the positive and negative thresholds are too different, say 0.1 vs 1. So we uniformly sample the "threshold ratio" $r$ from the range [1, 1.5]. Then, with 50% probability, we set $c_+ = c, c_- = c * r$; with the other 50% probability, we set $c_+ = c * r, c_- = c$. The parameters $c_+$ and $c_- are the positive and negative thresholds, respectively.

3. The standard deviation of the background gaussian distribution is uniformly sampled from the range [0, 0.1].

4. The standard deviation of the hot pixel gaussian distribution is uniformly sampled from the range [0, 10].

5. The fraction of hot pixels is uniformly sampled from the range [0, 0.001] (i.e., 0 to 0.1%).

These configuration details will be included in our code release to insure reproducibility.

Q3. Other grid representations.

The idea of using the V2V framework on other representations by neglecting intra-bin differences is interesting. Intuitively, we believe that this could work. However, further experiments would be needed for validation.

Response to Reviewer 92jc

Thank you for your positive evaluation and constructive feedback. We will address your concerns below.

S4. Experiment mistake.

Thank you for your understanding. We have reconducted the experiment with the correct configuration, and we will update the results in the final version of the paper. The test results before and after correction are as follows.

W1 & Q. Justification for noise simulation modeling.

Our noise model was designed by referencing two widely used simulators, ESIM and DVS-Voltmeter. The Gaussian noise added to the output voltage is adapted from DVS-Voltmeter's model, which simulates the voltage change as a Brownian motion process caused by photon reception randomness. This is more physically plausible than adding noise post-voxelization (as in the ESIM training protocol), as it allows noise to interact with the signal during event generation. For simplicity, we did not include other noise types like leak noise, but they could be easily incorporated into our framework.

W2 & L1. Adaption to higher temporal resolution.

When applying V2V-trained models to real event streams, the temporal resolution of the outputs can be adjusted by changing the bin durations of the voxels. For example, the EVAID dataset has ground truth frame rates up to 150 FPS (see Table 5), so we set the duration of each bin to 1/150/5 = 1.33 ms. The resulting high frame rate video outputs achieved good results qualitatively (Figure 6) and quantitatively (Table 2).

This raises a question: Why can the model, trained on 5 FPS synthetic voxels, adapt to 150 FPS real event inference? We believe the answer lies in the vast diversity of our training data. The WebVid dataset contains scenes with a wide range of motion speeds, from rapid action to nearly static shots. Statistically, voxels from long bins in slow scenes can appear similar to voxels from short bins in fast scenes. This diversity allows the model to learn representations that are robust to different bin durations and, therefore, different temporal resolutions.

L1. Missing limitations.

Due to the space limit, we moved our limitations section to the appendix (Section G). We apologize for the resulting inconvenience.

Dear 92jc, I see your review is positive, but we need you to show up and say something that engages with the rebuttal.

Response to Reviewer WNjQ

Thank you for your thorough review and insightful comments. We will address each concern below.

W3 & W4. Clarity of writing.

We will improve the clarity of our writing in the final version.

• We will add citations (including [a]) to our preliminary discussion on the inherent ambiguity of event-based vision, as suggested.
• We will highlight the "Ours" methods in Table 2 and Figure 6 to improve readability.

The "ESIM-280-Intp" in Table 2 refers to a training dataset created from ESIM, composed of 280 scenes, and converted to the Interpolated voxel representation (L259-L261). The corresponding experiments (l) and (m) are also video reconstruction experiments. They are designed to compare the interpolated voxel representation against the discrete voxel representation. We apologize for any confusions.

W5 & Q. Limitations of the voxel representation.

We acknowledge the reviewer's concern on discarding intra-bin timing, which is a limitation we discuss in Section G. However, we believe that this limitation does not "defeat the entire purpose of an event camera".

1. Voxel representations preserve key advantages of event cameras. As detailed in Section 3.2, while discrete voxels discard fine-grained intra-bin timing, they preserve essential inter-bin temporal dynamics. This allows for:

   • High temporal resolution: By adjusting bin durations, voxel grids can effectively encode high-speed motion. In our work, we successfully reconstruct videos at 150 FPS on the EVAID dataset.
   • High dynamic range (HDR): The HDR property of event cameras, which relies on per-pixel logarithmic changes, is fully preserved in the voxel representation.
   • Sparsity and efficiency: The principle of sparsity can be maintained by adapting bin durations based on event activity or by processing only local patches with sufficient event density, preserving the low-power potential of event cameras.

2. Voxel-based methods are widely and successfully used across diverse event-based tasks, including latency-sensitive ones like SLAM. The simplicity and effectiveness of voxel grids have made them a cornerstone of modern event-based vision research. Examples include:

   • SLAM: [CVPR 2019] Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion.
   • Action recognition: [CVPR 2022] E^2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition.
   • Head pose estimation: [ECCV 2024] Event-based Head Pose Estimation: Benchmark and Method.
   • Object detection: [CVPR 2024] Scene Adaptive Sparse Transformer for Event-based Object Detection.
   • Eye tracking: [CVPRW 2024] Event-Based Eye Tracking. AIS 2024 Challenge Survey.
   • HDR imaging: [CVPR 2023] Learning event guided high dynamic range video reconstruction.
   • Image deblurring: [CVPRW 2025] NTIRE 2025 Challenge on Event-Based Image Deblurring: Methods and Results.

   Our V2V framework can boost models that accepts a voxel-based input, making it broadly applicable. We chose video reconstruction and optical flow as representative examples, and we hope to inspire the community to explore more tasks given task-specific large-scale datasets.

3. While the ultimate goal may be end-to-end spiking neural networks on neuromorphic hardware, the current ecosystem faces challenges with hardware cost and computational limitations. High-performing, GPU-trained voxel-based models can demonstrate the value of event cameras, driving the widespread adoption needed to scale the industry. As the hardware matures, these successes can pave the way for a transition to more neuromorphic approaches.

4. Regarding whether training on synthetic data can improve performance on real data, our intuition is that it can. Existing literature, such as [CVPR 2020] Video to Events: Recycling Video Datasets for Event Cameras, suggest that training on synthetic data and fine-tuning on real data often yields the best results. However, this is beyond the scope of our current study and awaits future validation.

Thank you for the response!! Let me first describe my rationale for the rating despite strong results and good presentation -- The contribution of the paper is a system-level insight, i.e. "instead of generating event streams, one can generate voxel accumulations, and this allows us to leverage videos while not blowing up I/O and other costs during training". Closing the loop, i.e. an end-to-end evaluation actually tells us if the model trained on all of these videos is actually useful in real situations or not.

On limitations of voxel representations - (a) Agreed on Point 1 -- real event data does preserve inter-bin temporal dynamics. Your training simulates event data from videos with lower temporal resolution as far as I understood the paper. Are you claiming when you "successfully reconstruct videos at 150 FPS on the EVAID dataset", it's done by training on only 30 FPS videos? This does partially resolve my concern if it's the case.

(b) Yes, voxel based methods are used in prior work for downstream tasks -- agreed. I think my core issue is that it would be much more impactful to show that your method enables leveraging thousands and millions of videos on one of these tasks (or improved generalization to a different dataset for the same task).

(c) Agreed.

(d) I disagree this is beyond the scope of the problem. The whole point of the work is to leverage all of these thousands (and potentially millions of) videos -- which result in synthetic data is to help event camera related tasks. As pointed in [CVPR 2020, Gehrig et al] which shows corroborating experiments, I'm asking the same question "can we reuse weights trained on such video scale datasets, and finetune on limited real events data for other event tasks". Experiments along the line of [Gehrig et al] would significantly improve the work.

Feel free to correct me if I have missed something about this work and in my assessment.

(a) Thanks for this critical point, maybe I missed it from the paper but it does help my understanding a lot. This is good to know to that the model generalizes to temporal resolutions beyond what it's trained for.

(b) Yes, this makes sense. I think my question is better answered in response to (d), where we discuss the evaluation.

(d) Discussion on evaluation.

(i) Okay on re-reading the paper, I see that your evaluation on A_1 (HQF) and A_2 (EVAID) (Table 2) is in some sense zero-shot, not trained on the real dataset at all -- very cool. ESIM is the event data simulator, whose performance is worse. Can the authors clarify why ESIM is tagged with "Pretrain" -- is the model then trained on HQF or EVAID? Please correct me if I'm wrong in my understanding. In general, Table 2 is very hard to interpret because there is experimental combinations/ablations that are all shown together, it's difficult to understand the effect of one specific axis. Breaking the table into a few tables (some for the data size ablations or loss ablations) would help bring clarity and surface up these points better.

(ii) Worry of overfitting on a specific dataset is reasonable -- I'm not hoping that the finetuned model would generalize to both the datasets. It would be nice to see if fine-tuning on a specific dataset additionally helps, and it's an insight that strengthens the argument for (b) - Training on WebVid + Small Event Dataset >> Training on Small Event Dataset.

(iii) This is the kind of evaluation/validation I'm hoping for -- to know for a fact that a model trained on diverse even if synthetic data helps in a better event-based object detection (or some other method, like SLAM) pipeline. Ideally the pipeline ought to be event "native" -- uses event streams by accumulating them and also in other ways, but I suppose most pipelines resort to accumulation (point 2 in author's initial response).

(iv) Noted. I do think a general transformer like model is a ripe direction in that case to simplify parameter sharing across tasks.