Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation

We appreciate the reviewer’s careful evaluation, including both the comprehensive review and the thoughtful critiques. The following responses aim to address the raised concerns in detail.

---

Clarification on Learnable Noise Embeddings (W1, Q1)

Thank you for pointing this out. Here we clearly explain, from both technical and experimental aspects, how learnable noise embeddings are beneficial for improving fitting ability and enhancing learning stability.

Technical explanation:

From an optimization perspective, the learnable noise embeddings serve to enhance the model's fitting capacity and learning stability by introducing additional degrees of freedom into the cross-modality interaction process. This added flexibility allows the model to better adapt to heterogeneous modalities during learning.

From the viewpoint of the cross-attention mechanism, in the absence of noise, the query (Q) tends to attend excessively to its own features in the key (K), thereby suppressing signals from the other modality and impeding effective fusion. Introducing noise embeddings mitigates this issue by perturbing the attention space in a controlled, learnable manner, encouraging richer and more balanced cross-modal interactions.

Conceptually, this mechanism is inspired by prompt-based learning, where learnable prompts are used to guide model behavior. In our case, the noise embeddings act as fixed prompts that help regulate and stabilize the cross-modal attention patterns during training.

Experimental explanation:

We conduct an ablation study by removing the learnable noise embeddings on three datasets: DERS-XS, DSEC-Semantic, and DSEC-Xtrm. As shown in Table R8, the removal of noise embeddings leads to a 1.05% and 0.81% mIoU drop on DERS-XS and DSEC-Xtrm, respectively, confirming their contribution to improved fitting ability and stability. The performance drop on DSEC-Semantic is minimal (0.18%), which we attribute to its reliance on RGB-based pseudo-labels. As the supervision signal is already biased towards RGB, the model naturally relies less on event modality. In such cases, the role of learnable noise embeddings in facilitating cross-modal interaction becomes less significant.

Table R8: Ablation study on learnable noise embeddings.

---

Justification of Challenging RGB and DSEC-Night Evaluation (W2, Q2)

We would like to clarify that the use of dark or degraded RGB inputs is intentional and consistent with our primary motivation, that is, to facilitate resilient semantic segmentation by addressing the core challenge of inferior optimization issues of heterogeneous event and RGB in modality imbalance and failure situations. In this context, using challenging RGB conditions and applying spatial occlusions via local masking is not a source of artificial bias, but a deliberate design choice to evaluate the fusion capabilities under representative adverse scenarios.

That said, we fully understand the reviewer’s concern about broader real-world applicability. Notably, our method also achieves consistent gains on DSEC-Semantic, a dataset that represents more typical, non-extreme conditions. This demonstrates that the effectiveness of our approach is not limited to extremely degraded RGB scenarios.

In addition, to directly address the reviewer’s suggestion, we include a new evaluation on DSEC-Night, which was originally proposed for unsupervised cross-modality domain adaptation by CMDA. As this dataset only contains 150 annotated frames for testing, we adapt it to a supervised setting by splitting the labeled sequences into training and testing sets: 54 frames from zurich_city_09_a and zurich_city_09_b for training, and 96 frames from zurich_city_09_c/d/e for testing. All models are trained for 100 epochs with the same settings for fair comparison. As shown in Table R9, our method outperforms CMX by 6.55%, CMNeXt by 6.28% mIoU, and EISNet by 5.95%, demonstrating the strong efficacy of our method in real-world extreme conditions.

Table R9: Quantitative comparisons on DSEC-Night.

Lastly, we agree that a more comprehensive evaluation on diverse extreme conditions would be valuable. As part of our ongoing work, we are actively collecting a new, large-scale real-world dataset covering a broader range of challenging scenarios, including underexposure, overexposure, motion blur, and lens flare. Due to the high cost of manual annotation, this dataset could not be completed during the rebuttal period, but we plan to release it in future work.

---

Efficacy in Non-Extreme Real-World Conditions (W3)

We respectfully disagree with the characterization of the 2.01% mIoU improvement as marginal. The significance of a performance gain should be evaluated in the context of task difficulty, dataset characteristics, and the strength of the baseline. Achieving a 2.01% improvement over a strong baseline like CMNeXt is non-trivial, particularly given that DSEC-Semantic represents non-extreme real-world conditions where existing methods already perform well and leave limited room for further improvement.

For reference, CMNeXt itself reports only 0.2% and 0.6% mIoU improvements on the MFNet and NYU Depth V2, respectively. In this context, our 2.01% improvement is both meaningful and practically relevant. Moreover, in real-world deployment scenarios, even modest increases in mIoU can lead to noticeably improved segmentation quality.

---

Justification of DERS-XR Dataset (W4)

We would like to clarify that the DERS-XR dataset is constructed from real-world captures under naturally occurring extreme conditions. Our intention is not to deliberately diminish the RGB signal, but rather to faithfully reflect real-world scenarios. Although the RGB images appear severely degraded, they still retain essential semantic cues for segmentation.

Regarding the dataset scale, we acknowledge that DERS-XR contains a limited number of frames. This limitation is primarily due to the high cost of accurate pixel-level annotation under extreme conditions. The original motivation behind DERS-XR is to provide a benchmark for evaluating model performance under real-world extreme conditions with reliable ground truth. Given the scarcity of such labeled data, DERS-XR fulfills this purpose effectively despite its modest size.

---

Efficiency Performance Trade-off with Lighter Backbones (W5, Q3)

To address the reviewer's concern, we conduct additional comparative experiments on DERS-Xtrm using smaller backbones (2×MiT-B0), reducing the FLOPs of ESC to be even lower than CMNeXt. In addition, we measure the end-to-end inference latency of CMNeXt and our ESC (both standard ESC and reduced variant) on a single NVIDIA GeForce RTX 3090 GPU with a batch size of 1. All latency measurements are performed with a fixed input size of 512 × 512, with each measurement calculating the average execution time over 100 inferences, and we repeat 3 times for stability.

As shown in Table R10-A, the reduced ESC variant still outperforms CMNeXt, achieving 49.06% mIoU vs. 45.16%, despite lower FLOPs (60.658G vs. 62.805G) and significantly fewer parameters (14.184M vs. 58.687M). This suggests that the performance gains stem from architectural design rather than merely an increased computational cost. Furthermore, as shown in Table R10-B, the reduced ESC variant has an average inference latency of 29.37 ms, which is shorter than that of CMNeXt (29.79 ms), demonstrating its potential for more efficient deployment.

Above results indicate that even with lightweight backbones, our model maintains strong performance with higher inference speed, highlighting the effectiveness of our design beyond raw FLOPs. This reflects a favorable trade-off between efficiency and performance, which is essential for practical deployment in real-world systems.

We hope these additional results and analyses adequately address the reviewer’s concern.

Table R10-A: Ablation study on lighter backbones on DSEC-Xtrm.

Table R10-B: Inference Latency comparison on lighter backbones.

We thank the reviewer for the constructive feedback. The comments have been very helpful in guiding us to improve the manuscript. Please find our detailed responses below.

---

Framework Generality and Alternative Dictionary Strategies (W1)

We appreciate the reviewer for the valuable suggestion. In our framework, VQ-VAE-based edge dictionary not only provides a feasible edge encoding method, but also provides a feasible optimization mechanism for modality edge features, deriving a series of reliable indicators based on confidences and uncertainties to guide the fusion of the heterogeneous event and RGB. The utilization of alternative encoding strategies, such as a prototype-based encoding method, will have a significant change in the current training paradigm, for example, the confidence indicator based on classification softmax logit cannot be obtained. This results in different edge encoding strategies not being able to be replaced losslessly in the framework.

Despite this, we still do our best to re-implement our framework with a simple prototype-based edge encoding method. We pre-define 24 categories of edge prototypes, which represent 11 categories of semantic boundaries and one non-boundary category for each modality. The prototypes are updated every epoch based on the average feature vectors of each category. In the training and testing stages, the nearest neighbour look-up algorithm is used to match the prototypes with edge features at each spatial position, and the matched prototypes reconstitute the re-coded edge embeddings $**Γ**^\mathcal{I}$ and $**Γ** ^\mathcal{E}$.

As shown in Table R6, for the prototype-based edge encoding method, the mIoU on DERS-XS drops by 1.20%, on DSEC-Semantic by 0.12%, and on DSEC-Xtrm by 0.36%, compared to the original VQ-VAE-based edge encoding method. These results suggest that our framework has generality in different edge encoding designs, while the VQ-VAE-based edge dictionary is uniquely designed in our proposed framework after careful consideration.

Table R6: Ablation study on prototype-based edge encoding method and VQ-VAE-based edge encoding method.

---

Differences Between ESC and ESEG (W2)

We appreciate the reviewer for suggesting this paper, and we would like to discuss and clarify any discrepancies between ESEG and our ESC.

Both ESEG and our ESC utilize edge information to guide the learning process; however, there are several main discrepancies as follows.

(a) Different tasks. ESEG is a uni-modality method that relies solely on event input, aiming to approximate the segmentation performance typically obtained from RGB data. Our ESC is a multi-modality semantic segmentation method with event and RGB as inputs, which mainly focuses on the inferior optimization issues in modality imbalance and failure situations. For this reason, it is acceptable to use RGB-based pseudo-labels for ESEG, while our ESC needs more accurate labels for assessment.

(b) Different motivations and learning objectives of the utilization of the edge. ESEG utilizes edge-semantic messages to explicitly inform the interested region to the model as guidance, thus ESEG structure introduces direct edge supervision with different semantics of the objects. Our ESC utilizes edge information as an intermediate commonality for heterogeneous event and RGB, which realigns event and RGB into a unified semantic space and jointly optimizes them based on the confidence and uncertainty indicators derived from latent edge distributions.

(c) Different pipelines even if ESEG's detection branch replaced by RGB input. For ESEG, if the dense-semantic branch is replaced by RGB input, the entire pipeline will become that events only represent edge semantics, and RGB only represent dense semantics. However, our assumption is that event and RGB both contain edge semantic information. Therefore, our pipeline realigns event and RGB into the unified semantic space to jointly optimize their edge semantics, with implicit edge information as a crucial clue.

Although this work differs from ours in many ways, it is still a key message for us. We are grateful to the reviewer for his keen observation, and we will discuss and cite this work in our related work section with the above analysis.

---

Rationale Behind Event Sampling Strategy (W3)

The DSEC-Semantic event input is built by sampling a fixed number of events per voxel grid rather than a fixed time window, which follows the setting of ESS [34]. In ESS, the event input is built with 100,000 events per voxel grid. We found that a 50-ms fixed time window or 100,000 fixed number of events is relatively large, which reduces the performance of data preprocessing, and may lead to insufficient edge characteristic representation. After the above trade-offs, we decided to use 50,000 events per voxel grid for DSEC-Semantic as our event sampling strategy in our work.

To further demonstrate the impact of different event sampling strategies, we conduct experiments on DSEC-Semantic with a fixed time window of 50 ms, a fixed number of events of 100,000 per voxel grid, compared with 50,000 events per voxel grid in the main paper. As shown in Table R6, different event sampling strategies present comparable results, with mIoU slightly lower (0.20% and 0.19% respectively) than the fixed number of events of 50,000 in the main paper.

Table R7: Ablation study on different event sampling strategies.

---

Data of Broader Range Extreme Conditions (W4)

Given the specificity of this field and the high cost of precise annotation, there is currently a lack of publicly available event-RGB semantic segmentation benchmarks that feature a broader range of extreme real-world conditions with true labels. Existing datasets such as DDD17 and DSEC-Semantic primarily rely on RGB-based pseudo labels and do not explicitly differentiate between different conditions.

To address this gap, we are currently dedicated to collecting a set of larger-scale, more diverse, and truly challenging real-world scenes data as our future work, which will include challenging scenarios such as underexposure, overexposure, motion blur, and lens flare. Nevertheless, we stress that the datasets employed in our current study are carefully designed and sufficient to demonstrate the core effectiveness and robustness of our method.

---

Deployment Efficiency and FLOPs Ablation (W5)

We appreciate the reviewer's concerns regarding the deployment efficiency. We address this from both a hardware evaluation and an algorithmic ablation perspective.

Hardware-wise, we report the end-to-end inference latency, throughput, and peak memory usage of NVIDIA GeForce RTX 3090 GPUs and an AMD EPYC 7642 48-Core CPU, with a fixed input resolution of 512 × 512. As shown in Table R4, our framework achieves favorable deployment efficiency on standard hardware.

Table R4: Inference latency, throughput, and peak memory usage of GPU and CPU.

Algorithm-wise, we further evaluate our framework with lighter backbones (2×MiT-B0). As shown in Table R10-A, even with FLOPs lower than CMNeXt, the reduced ESC variant achieves better performance on DSEC-Xtrm compared to CMNeXt (49.06% vs. 45.16% mIoU) and a significantly lower parameter count. Moreover, as shown in Table R10-B, the average inference latency of the reduced variant is also slightly lower than CMNeXt (29.37 ms vs. 29.79 ms). These results highlight the effectiveness of backbone scaling in achieving a favorable balance between deployment efficiency and performance.

Table R10-A: Ablation study on lighter backbones on DSEC-Xtrm.

Table R10-B: Inference Latency comparison on lighter backbones.

Dear Reviewer 53vx,

Thank you for your final acknowledgement and for taking the time to review our rebuttal. We appreciate your thoughtful engagement throughout the review process and are glad that our response helped to address your concerns. We will reflect the necessary improvements in the final version.

Sincerely,

The Authors

We sincerely appreciate the reviewer’s time and effort in thoroughly reviewing our manuscript. We also appreciate the insightful suggestions and provide detailed responses below.

---

Other Techniques Investigation and Comparison (W1)

We appreciate the reviewer for highlighting this important research direction. Our ESC framework lays emphasis on the inferior optimization issues of heterogeneous event and RGB in the case of modality imbalance and failure situations. Motivated by this, our work emphasizes developing more robust and resilient fusion strategies under the existing extreme input conditions. Therefore, we center our investigation around recent token-level and attention-level fusion baselines that address similar concerns.

Other techniques suggested by the reviewer, such as cross-modal contrastive pretraining (e.g., Event-Camera-Data-Pre-training [A], SAM-Event-Adapter [B], CM3AE [C]) and adaptive modality dropping (e.g., Missing-Modality-Prediction [D], RAGPT [E]), indeed provide valuable and complementary perspectives in the broader multimodal learning field. However, cross-modal contrastive pretraining approaches primarily aim to acquire informative and effective pretrained backbones, while adaptive modality dropping methods mainly tackle issues of incomplete modality inputs. In contrast, our focus lies in inferior optimization issues of modality imbalance and failure, which are prevalent yet underexplored challenges in real-world scenarios. This distinction reflects different motivations and problem settings, rather than limitations of any particular method.

For completeness of comparison, we include in Table R5 the reported performance of two representative contrastive pretraining methods ([A], [B]) on the DSEC-Semantic dataset. The results are cited directly from the original papers, as [B] does not release its code, and we rely on their published numbers.

In the final version of the paper, we will expand the above discussion and theoretical comparison in the related work section to include these approaches and cite the relevant literature, including [A-E]. We believe that this will help situate our work more clearly within the multimodal fusion landscape and highlight its distinct contributions.

Table R5: Comparison on DSEC-Semantic with two cross-modal contrastive pretraining methods.

---

Formatting and Layout Issues (W2)

Thanks for your careful review and valuable suggestions. We will carefully adjust the texts embedded in figures for more convenient reading in our final manuscript version. As for tables, we will also trim them neatly in the final version.

It must also be seriously pointed out that we DO NOT violate any NeurIPS style guidelines. No tables spill over margins or have inconsistent column widths.

References

[A] Yan Yang, Liyuan Pan, and Liu Liu. "Event camera data pre-training." ICCV, 2023.

[B] Bowen Yao, Yongjian Deng, Yuhan Liu, Hao Chen, Youfu Li, and Zhen Yang. "Sam-event-adapter: Adapting segment anything model for event-rgb semantic segmentation." IEEE ICRA, 2024.

[C] Wentao Wu, Xiao Wang, Chenglong Li, Bo Jiang, Jin Tang, Bin Luo, Qi Liu. "CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework." ACM MM, 2025.

[D] Donggeun Kim, and Taesup Kim. "Missing modality prediction for unpaired multimodal learning via joint embedding of unimodal models." ECCV, 2024.

[E] Jian Lang, Zhangtao Cheng, Ting Zhong, and Fan Zhou. "Retrieval-augmented dynamic prompt tuning for incomplete multimodal learning." AAAI, 2025.

Dear Reviewer BVgg,

Thank you again for your time and effort in reviewing our submission. We hope that our rebuttal has addressed your concerns and clarified the key points. If there are any remaining questions or if further clarification would be helpful, we would be glad to provide additional information.

We truly appreciate your contributions to the review process.

Sincerely,

The Authors

We are grateful to the reviewer for the detailed and professional review, which reflects a deep understanding of the topic. We sincerely appreciate the positive comments, valuable concerns, and suggestions on our work. Here is our response to the mentioned weaknesses and questions.

---

Edge Dictionary Domain Transferability (W1, Q1)

Thank you for the insightful question. Theoretically, the discrete edge dictionary learned by VQ-VAE is expected to have good transferability across datasets. As an intermediate representation, semantic edge exhibits relatively simple and consistent structures, and the latent distributions of semantic edge derived from segmentation labels tend to vary only slightly across different datasets. Therefore, we expect the performance degradation under dictionary transfer settings to be minimal.

Following the reviewer’s suggestion, we conduct cross-domain evaluations by (i) using an edge dictionary pretrained on DSEC to evaluate on DERS-XS, and (ii) using a dictionary pretrained on DERS-XS to evaluate on DSEC-Semantic and DSEC-Xtrm. As shown in Table R1, the performance drops are small in all cases. Specifically, the mIoU on DERS-XS drops by 0.66%, on DSEC-Semantic by 0.10%, and on DSEC-Xtrm by 0.21%, compared to the original non-exchanged dictionary settings. These results suggest that the learned edge dictionary generalizes well across domains, and our method remains robust under moderate domain shifts.

Table R1: Ablation study on edge dictionary exchange settings.

---

Hyperparameters Sensitivities (Q2)

Thank you for raising this important point. We observe that the model's performance shows noticeable variation under different dictionary size $K$, but remains robust across a reasonable range. As shown in Table 4 of the main paper (reproduced here as Table R2), using too small a dictionary leads to insufficient latent edge representations, while an overly large dictionary introduces ambiguity in modality-specific learning, potentially due to increased difficulties in selecting appropriate dictionary entries for modality-specific edge patterns. Both extremes negatively impact performance. The best results are achieved with a moderate dictionary size (e.g., $K=128$), which offers a good balance between representation capacity and stability in codebook learning.

Table R2: Ablation study on key usage (dictionary size) on DERS-XS.

As for the uncertainty threshold, our framework does not rely on any manually set threshold as a hyperparameter. Instead, the uncertainty indicators are dynamically inferred and represented as continuous values in the range of 0 to 1. These maps serve as soft attention weights, allowing the model to adaptively determine the fusion strength between heterogeneous event and RGB.

To further examine the role of these learned confidence and uncertainty maps, as well as the contribution of the event modality, we conduct additional experiments on DERS-XS by injecting controlled modifications during the inference phase, specifically by zeroing out the confidences of event modality. As shown in Table R3, this results in a 1.49% drop in mIoU, indicating a noticeable degradation in performance. This ablation underscores the effectiveness of our dynamically learned uncertainty indicators, demonstrating that the model can adaptively control the fusion strength of each modality based on the input content, without relying on a manually tuned threshold.

Table R3: Ablation study on injecting modifications to confidence and uncertainty indicators.

---

End-to-end Inference Latency, Throughput and Peak Memory Usage (W2, W3, Q3)

Certainly, and thank you for the suggestion. Here are the end-to-end inference latency, inference throughput, and peak memory usage of GPU and CPU in Table R4, averaged over three runs for stability. All experiments are performed on a machine equipped with NVIDIA GeForce RTX 3090 GPUs and an AMD EPYC 7642 48-Core CPU. Inference latency is measured on a single GPU with a batch size of 1, while throughput and memory usage are measured on two GPUs with a batch size of 16 per GPU, and only the peak memory usage of the process on GPU 0 is reported. All measurements are conducted with a fixed input resolution of 512 × 512. For inference latency and throughput, each measurement calculates the average execution time over 100 inferences.

These results indicate that ESC achieves competitive inference efficiency and moderate memory usage, suggesting good potential for deployment in practical scenarios. Combined with the reported FLOPs and parameter count in Table 5, this further supports the practical feasibility of our proposed ESC framework.

Table R4: Inference latency, throughput, and peak memory usage of GPU and CPU.