LaSe-E2V: Towards Language-guided Semantic-aware Event-to-Video Reconstruction

Thank the reviewer for the constructive comments and valuable concerns.

Can the model handle the situation when the description of the scene is unknown?

• Yes, our model can handle such scenarios. If the text description is unavailable, our framework defaults to a conventional event-to-video approach. As demonstrated in Tab. 2 (row 1), even without text input, our method still achieves reasonable performance.
• Additionally, event-based multi-modality models such as EventBind[77] and EventClip[78] can also be employed to generate text descriptions directly from the event stream.

About running times and GPU memory costs for inference.

• Thanks for the question. We have included a complexity comparison in Global Response to All Reviewers. As illustrated in Tab. A-1, our method requires significant inference time due to the multiple denoising steps, showing a known limitation of diffusion models. However, while diffusion models have been employed in tasks like super-resolution and depth estimation, “they are still new to the event-based imaging community”(uWG8). As such, our approach could hopefully “provide new ideas for subsequent works”(uWG8) in this field.

Ambiguity of SSIM metric.

• The SSIM metric raises ambiguity because it involves several hyper-parameters that may differ across various codebases. For example, in the structural_similarity function of the skimage package, parameters like gaussian_weights and sigma are used for spatial weighting of each patch with a Gaussian kernel, use_sample_covariance indicates whether to normalize covariances, and K1 and K2 are algorithm-specific parameters that need to be set.

About explanations for the artifact of text reconstruction in Fig.4.

• Thanks for the insightful question. Although our method achieves superior overall performance, some artifacts persist in the text part reconstruction. This issue arises because we depend on the prior of the pre-trained diffusion model (SD2 [42]), which also faces challenges in text generation. However, the recently released SD3 [79] claims to show improved text rendering capabilities, which could potentially address this problem.

Typo: "evet".

• Thanks for pointing out the typo. We will correct it in the final version.

Additional Reference:

[77] Zhou J, Zheng X, Lyu Y, et al. E-clip: Towards label-efficient event-based open-world understanding by clip[J]. arXiv, 2023.

[78] Wu Z, Liu X, Gilitschenski I. Eventclip: Adapting clip for event-based object recognition[J]. arXiv, 2023.

[79] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis[C]//ICML. 2024.

Thank the reviewer for appreciating our work with valuable suggestions. We address the comments below.

About the "hallucination" issue of diffusion-based models.

• Yes, this is indeed a problem for the diffusion models. To address this, we have proposed novel techniques (i.e., ESA module, Event-aware Mask Loss, and Event-aware Noise Initialization) to mitigate the diversity objectives of the diffusion model. The proposed techniques thereby enhance the consistency between events and the video and reduce the hallucination issue.
• Please be noted that, in static areas without event data, the model tends to reconstruct these regions with hallucination. By incorporating text descriptions, the hallucination issue can be mitigated to some extent, as shown in Fig. 7 (left).

Text could introduce ambiguity and lead to a rial-and-error process.

• Unlike existing image synthesis pipelines, our method primarily relies on event data, with text descriptions serving as the supplementary guidance only when events are too sparse or not trigged. This approach minimizes the potential for ambiguity.
• Additionally, our method requires only coarse text descriptions to effectively leverage the semantic prior in the T2I diffusion model. Most of these descriptions simply identify objects, such as "car" or "road," generated by the tagging model RAM, which introduces little ambiguity. As shown in Tab. 1 and Fig. 4, our experiments confirm that this coarse-grained text is effective and does not necessitate a trial-and-error process.

Computational Resources and Inference Times.

• Thanks for the suggestion. We have included a complexity comparison in Global Response to All Reviewers. As shown in Tab. A-1, our method requires considerable inference time due to multiple denoising steps, showing a limitation of diffusion models. However, our approach “could inspire future work in the field”(ueNs) by incorporating language guidance and proves “it is possible to use diffusion-based models for the event-based video reconstruction task”(H8fU).

The artifacts in Fig. 3 are not captured by traditional metrics in Tab. 1.

• The reviewer might misunderstand Fig.3. Fig. 4 presents the qualitative results corresponding to the quantitative data in Tab. 1, derived from the widely-used ECD, MVSEC, and HQF datasets. These results demonstrate a significant improvement in reducing artifacts. Conversely, Fig. 3 illustrates qualitative results from a different dataset (HS-ERGB) that involves fast motion.
• Additionally, the traditional metrics (MSE, SSIM, and LPIPS) used in Tab. 1 are widely recognized in recent E2V research. These metrics have been proven to effectively assess results at the pixel level (MSE), structure level (SSIM), and in terms of human perception (LPIPS).

How is temporal consistency between reconstructed sections resolved?

• As mentioned in Line 184, we condition the frame from the previous section to enable an auto-regressive pipeline for long video reconstruction. The ablation study in Tab. 2 (2nd row vs. 3rd row) has demonstrated the effectiveness of the previous frame condition to ensure the temporal consistency between sections. Theoretically, the auto-regressive pipeline imposes no limitations on the number of frames.

Metrics for temporal consistency.

• Upon the suggestion, we evaluate the results based on the temporal quality metrics from VBench [76]. As shown in Tab. A-4, our method significantly outperforms others on subject consistency and background consistency, while achieving comparable performance on motion smoothness. These results demonstrate the effectiveness of our approach in maintaining temporal consistency. We will include these results in the camera-ready version.

Table A-4: Quantitative Comparison on temporal consistency on ECD based on VBench metrics.

Clarification on the Variable "V".

• Thanks for the question. The V in bold represents N segments. We will clarify this in the final version.

Clarification on the Variable $\epsilon$.

• Yes, the noisy input latent representation is typically denoted as $z_t$ during the training process, while $\epsilon$ refers to the sampling noise input during inference. Sorry for the confusion. We will clarify it in the final version.

Adding reference to ESA.

• Thanks for the suggestion. We will add the reference for this module in the revision. While SD [42] serves as a baseline architecture to incorporate text embedding, we introduce ESA to enhance spatial-temporal consistency between the event data and video by considering the unique characteristics of event data.

Adding Cross-Attention to section title of ESA.

• Thanks for the insightful comment. We will revise the title in the final version to better reflect the inclusion of cross-attention.

About the value of $\lambda$.

• Thanks for the question. The value of $\lambda$ is set to 0.01 for all the experiments. We will clarify it in the implementation details.

Additional Reference:

[76] Huang Z, He Y, Yu J, et al. Vbench: Comprehensive benchmark suite for video generative models[C]/CVPR. 2024.

Thank the reviewer for elaborating on the points. We address the reviewer's concerns below:

Hallucinations

• Hallucination is a potential side effect introduced by incorporating semantic priors from the diffusion model into our framework. In this regard, our proposed techniques are demonstrated to be effective in reducing hallucinations, but not entirely eliminating them, which is still a challenge in the diffusion model research community. Extensive experimental results (Tab. 1 and Fig. 4) show that our method significantly improves fidelity and reduces hallucinations in the scenes with sufficient event data. Our method also exhibits better reconstruction performance for the scenes even with insufficient event data (Tab. A-6). Please be reminded that it is impossible to totally remove the hallucination effect for regions with insufficient events. However, our method indeed effectively leverages the semantic prior in the diffusion model to reconstruct the image more closer to the distribution to the real scene, whereas the previous E2V methods (e.g., HyperE2VID) fail in these regions, as shown Fig. 1.
• Our framework mainly focuses on providing a potential pipeline to reconstruct the video from events with the guidance of language. This is based on our finding that language naturally conveys abundant semantic information, which is beneficial in enhancing the semantic consistency for the reconstructed video (see Lines 39-41). The text guidance can serve as a form of human intervention through text prompts, which is similar to the way of the text-guided denoising [80,81] and super-resolution[82] methods.
• As shown in Fig. 1, it compares the difference between our method and the previous E2V method (i.e., HyperE2VID). For region with insufficient events, although our results still differs from reality in some details, they indeed reconstructs a scene according to human preference. In contrast, the previous E2V method always reconstructs haze in these regions. For Fig. 7, we will update the results from previous E2V methods for clearer comparison. Quantitatively, Tab. A-6 also shows the superiority of our method in the scene with insufficient events.

Ambiguity in Text Prompts

• Our framework incorporates text descriptions as complementary information, marking the first instance of allowing language guidance in the E2V pipeline. This approach is believed to "inspire future work" (ueNs) and "provide new ideas" (uWG8). While this may introduce some ambiguity, it also offers the flexibility to manually adapt the reconstructed video according to user preferences, as demonstrated in Fig. 8. The text guidance manner is also similar to the way of the text-guided denoising [80,81] and super-resolution[82] methods.
• In regions with sufficient event data, our method improves reconstruction performance by leveraging the semantic priors from the text prompts, ensuring high fidelity. For regions with insufficient event data, the method relies solely on the text prompts to reconstruct a scene that aligns with human preferences. Although this approach may produce textures or details that differ from the real ones, it still generates images that are closer to reality. In contrast, the previous E2V models always reconstruct these regions as indistinct haze, far from the true distribution of real images. Fig. 1 shows the qualitative comparison. Tab. A-6 also shows the quantitative comparison and demonstrates the superiority of our method. We will add more comparisons on these scenes with insufficient events in the revision.

Computational Resources and Inference Times

• Table A-5 provides the GPU memory for different E2V models. It shows similar memory costs among different methods.

Table A-5: GPU memory cost of different E2V methods. All tests are conducted on one NVIDIA Tesla 32G-V100 GPU.

Artifacts Not Captured by Traditional Metrics in Figure 3

• Tab. A-6 provides a quantitative comparison based on three sequences (horse_11, horse_12, horse_13) from the HS-ERGB dataset. Existing E2V methods typically fail to reconstruct regions without events, leading to significantly worse quantitative results. Although our method may not perfectly reconstruct every detail for reality, it does generate a reasonable output that aligns with human preference and is generally close to the distribution of the real scene. Therefore, while our results on the HS-ERGB dataset may be less significant than those on "constantly moving" datasets (ECD, MVSEC, HQF), our method is still substantially better than baseline methods.
• Regarding the artifacts in the letters, as also noted by reviewer uWG8, these issues arise because our method relies on the pre-trained diffusion model (SD2 [42]), which also struggles with text rendering. However, the recently released SD3 [79] claims to have improved text rendering capabilities, which could potentially address this problem and improve the performance. We believe that the MSE can capture these artifacts at pixel level. However, although minor artifact in small regions exists, our method excels in overall image quality, since previous methods tend to reconstruct misty areas in front of the poster in Fig. 4 (3rd column)

Table A-6: Quantitative comparison of HS-ERGB. Results are conducted on 3 sequences with a total 497 frames.

Reference to ESA

• Our ESA module is specifically designed to enhance spatio-temporal consistency between events and videos. In contrast, the original SD [42] serves as a baseline attention mechanism for integrating conditional input. Our approach differs in attention design. While the original SD relies solely on simple cross-attention to incorporate various feature conditions, our ESA module takes into account the unique spatial and temporal characteristics of event data. It introduces two distinct attention mechanisms respective to the spatial domain and temporal domain, which fully leverage the constraints provided by the event data and ensure spatio-temporal consistency.

Additional Reference:

[80] Duan H, Min X, Wu S, et al. UniProcessor: A Text-induced Unified Low-level Image Processor[C]//ECCV. 2024.

[81] Qi C, Tu Z, Ye K, et al. Tip: Text-driven image processing with semantic and restoration instructions[C]//ECCV. 2024.

[82] Gandikota K V, Chandramouli P. Text-guided Explorable Image Super-resolution[C]//CVPR 2024.

Thank the reviewer for the valuable suggestions. We address the questions below.

Detailed comparison of inference speed and model size.

• Please refer to the Global Response to All Reviewers. As shown in Tab. A-1, our method requires considerable inference time due to multiple denoising steps, showing a limitation of diffusion models. However, our approach “could inspire future work in the field”(ueNs) by incorporating language guidance and proves “it is possible to use diffusion-based models for the event-based video reconstruction task”(H8fU).

Textual information is difficult to obtain and compromises fairness.

• We respectfully disagree. The textual description is not difficult to obtain for our framework. Please note that our method only requires coarse text descriptions to leverage the capabilities of the pre-trained T2I diffusion model. These descriptions, generated by the tagging model RAM, simply identify objects in a tagging style, such as "car," "road," and "tree". These can be easily provided by humans in the absence of APS frames.
• Also, please note that it is not uncommon to introduce additional priors for event-to-video (E2V) reconstruction task. For instance, Zhang et al. [72] incorporated optical flow to address event-to-image reconstruction as a linear inverse problem. Our work is the first to introduce text as a guiding prior for event-to-video reconstruction, which, as noted by reviewer ueNs, "could inspire future work in the field".

Why is it necessary for the event activation regions of adjacent frames to be similar?

• Sorry for the misunderstanding. It should be the regions without events to be similar. The value zero is assigned to the event-activated area, and vice versa. We will rectify Line 196 in the version.

Thanks for bringing up this thoughtful discussion.

About the hallucinations issue

• We respectfully disagree with the reviewer. Please note that all reconstructed results are not unreliable. In regions with sufficient event data, our method enhances performance with text guidance while ensuring fidelity. As demonstrated in Tab. 1 and Fig. 4, our method significantly outperforms previous methods. We also encourage the reviewer to recheck the demo video provided in our supplementary materials. For regions with insufficient event data, our method relies primarily on text prompts to reconstruct a scene that aligns with human preferences. As demonstrated in Fig. 1, HyperE2VID struggles with severe artifacts (haze-like artifacts), typically in the background regions, far from the true distribution of real images. Whereas, our method exhibits higher-quality reconstruction results that are closer to the ground truth. The results indicate that our method can subtly leverage the semantic information from language and thus ensure semantic consistency of the reconstructed video. To further demonstrate the effectiveness of our method, we provide a quantitative comparison of the HS-ERGB dataset, which includes larger regions lacking event data, as shown in Tab. A-6. Our method substantially outperforms baseline methods with an MSE of 0.083. Apparently, the quantitative results verify the superiority and reliability of our method.
• In this work, we mainly focus on a new research direction of exploring E2V reconstruction from a language-guided perspective, as affirmed by other reviewers: "This approach is expected to "inspire future work" (ueNs) and "provide new ideas" (uWG8). We did not focus much on the computation efficiency of the diffusion model, which will be left as a future work. Please be noted that using language guidance does not actually hamper the application value. Such a text-guided manner is also similar to recent methods used in text-guided denoising [80,81] and super-resolution [82], which have provided broad applications for low-level vision in a user-friendly manner. Our research is application-significant as the language provides abundant semantic information, beneficial for ensuring the semantic consistency of the reconstructed video. The text prompts also provide a way of human intervention to control the reconstruction process.

What is the significance of the text in this model?

• The text information serves as a crucial component to activate the semantic prior in the diffusion model. As clarified above, our method effectively enhances performance in regions both with sufficient and insufficient event data by utilizing text prompts. To simplify text prompts and enhance the model's robustness, we utilize only coarse, tagging-style descriptions for training. Without text prompts, it is challenging to exploit the semantic prior in the diffusion model to ensure semantic consistency. As shown in Tab. 3 (1st row), without the text prompt, our method yields modest results. However, when text information is incorporated, it significantly improves the model's ability to leverage the semantic prior in the diffusion model, thereby enhancing performance.

Text-related parts of the experiments

• For the qualitative comparison in Fig. 4, we respectfully suggest the reviewer recheck the supplementary video material. It shows not only an improvement in contrast. The previous methods tend to reconstruct misty images with distinct artifacts, especially in the region with insufficient event data. Our method effectively exploits the semantic prior of the diffusion model with the guidance of text prompts, which demonstrates semantic consistency. Additionally, Fig. 1 shows another extreme case. While the previous method (i.e., HyperE2VID) fails in the regions without event data, our method still reconstructs a reasonable scene according to the text guidance, which is closer to the ground truth. Tab. A-6 also shows the quantitative comparison in this dataset (i.e., HS-ERGB) and demonstrates the superiority of our method.
• Regarding the artifacts in the letter “g”, as also noted by reviewer uWG8, these issues arise because our method relies on the prior from the pre-trained diffusion model (SD2 [42]), which also struggles with text rendering. However, the recently released SD3 [79] claims to have improved text rendering capabilities, which could potentially address this problem and improve the performance.

Thank the reviewer for valuable comments and suggestions. We address the concerns below:

About ablation experiments for ESA on raw events and text.

• The reviewer might misunderstand the ESA module. Indeed, the ESA and the text are separate parts in our framework. As outlined in Sec. 4.2 (Line 157-183), the ESA module consists of event spatial attention and event temporal attention layers, which compute attention based on the hidden state $z$ and event feature $z_e$, respectively. This way, ESA module facilitates spatio-temporal consistency between event data and video. The text input is integrated separately via a cross-attention layer that follows the original LDM [42], and it is not involved in the ESA module. Since event input is essential for the E2V reconstruction task, we have already provided ablation studies on both ESA and text with event input to demonstrate their individual effectiveness. Hereby, we reiterate the results as follows:
  • To demonstrate the impact of text description, the ablation study in Tab. 2 (1st row vs. 3rd row) and Fig. 7 (left) compare the E2V results with (w) and without (w/o) text. Please check them.
  • Tab. 3 (1st row) compares the performance of the baseline trained with simple channel-wise concatenation of events, without the ESA module, showing a significant performance drop. This ablation result confirms the effectiveness of ESA module.

About FLOPs and parameters comparison.

• Thanks for the suggestion. Please refer to Global Response to All Reviewers.

About the tagging models and the corresponding ablation experiments.

• In Line 216, we employ the off-the-shelf tagging model RAM [61], which serves as a prompting model to provide text descriptions for the datasets. In fact, a recent work (SeeSR [70]) has also demonstrated the superiority of RAM compared to other models because of rich objects and concise description.
• As suggested, we also tested BLIP[71] on a sampled sequence (i.e., boxes in HQF) to further evaluate the influence of the prompting model, as shown in Tab. A-2. BLIP can generate reasonable text prompts in caption-style and show nearly reconstructed performance on MSE (0.025 vs 0.027). A detailed discussion will be provided in the camera-ready revision.

Table A-2: Comparisons between different prompting models on boxes of HQF.

Additional comparison methods.

• We mainly compared with the latest state-of-the-art method, e.g., HyperE2VID (TIP 2024). By default, our method is superior to the methods published in 2022 and 2023. However, upon the suggestion, we have provided more comparison methods in Tab. A-3, which further demonstrates the superior performance of our method. We will update Tab.1 of main paper in the camera-ready revision.

Table A-3: Comparison with more event-to-video methods.

About SSIM and SSIM* metric.

• The SSIM metric raises ambiguity because it involves several hyper-parameters that may differ across various codebases. For example, in the structural_similarity function of the skimage package, parameters like gaussian_weights and sigma are used for spatial weighting of each patch with a Gaussian kernel, use_sample_covariance indicates whether to normalize covariances, and K1 and K2 are algorithm-specific parameters that need to be set. For this reason, we reevaluated all comparison methods by using a unified metric, denoted as the SSIM* scores. We will further clarify this point in the revision.

How to define spatial consistency and how does noise initialization enhance the consistency?

• Spatial consistency denotes the consistency between event data and the reconstructed video. Specifically, events typically occur at the edges/texture part of the scene context, and the reconstructed video needs to align the event data in the spatial structure.
• Noise Initialization mainly focuses on mitigating the train-test gap during inference, which is a common challenge in diffusion models. Intuitively, accumulated event data provides structural information (e.g. edges) in the scene, acting as an additional constraint during the denoising process.

Additional Reference:

[71] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//ICML, 2023.

[72] Zhang Z, Yezzi A J, Gallego G. Formulating event-based image reconstruction as a linear inverse problem with deep regularization using optical flow[J]. TPAMI, 2022, 45(7): 8372-8389.

[73] Zhu L, Wang X, Chang Y, et al. Event-based video reconstruction via potential-assisted spiking neural network[C]//CVPR. 2022.

[74] Liu S, Dragotti P L. Sensing diversity and sparsity models for event generation and video reconstruction from events[J]. TPAMI, 2023, 45(10): 12444-12458.

[75] Liu S, Dragotti P L. Enhanced Event-Based Video Reconstruction with Motion Compensation[J]. arXiv, 2024.