Zero-Shot Event-Intensity Asymmetric Stereo via Visual Prompting from Image Domain

Thank you for your detailed feedback. We address your concerns below.

W1. Novelty and contributions of the visual prompting module:

While we draw inspiration from the EDI model for event-based image deblurring, our visual prompting module is not a straightforward rearrangement. We re-purpose its core idea to bridge the modality gap between events and frames specifically for stereo matching, which is not the original intention of the EDI model. Our method efficiently tackles challenges of non-negligible duty time between exposures and unknown triggering thresholds, which are crucial aspects of event-intensity asymmetric stereo and not addressed by prior work (L147-L156). This tailored reformulation, enabling the effective use of off-the-shelf stereo models without retraining on events in a zero-shot manner, forms a core novelty of our work and demonstrates "ingenuity in adapting well-established methods from the image domain to the event domain" (Reviewer dhj9 Strength 2). It is potentially "widely applicable for cross-model tasks" (Reviewer Ej3V Strength 3) and "other asymmetric vision tasks" (Reviewer ggyM Question 3).

W1&W2. Novelty and contributions of the disparity refinement module:

The disparity refinement module is a novel component of our approach that addresses key limitations inherent to event-intensity asymmetric stereo:

1. Handling static regions: Event cameras excel at capturing dynamic scenes but suffer from information sparsity in static regions. The refinement module leverages monocular cues to effectively address this, particularly for challenging regions with sparse events or textureless areas (L162-L164).

2. Addressing unknown scale: Monocular depth models often predict relative depth with an unknown scale. Our refinement module overcomes this by using a linear transformation guided by event density confidence to align the monocular prediction with the stereo disparity map (L170-L172).

This linear transformation-based fusion strategy, which utilizes numerical optimization techniques guided by event density confidence, is specifically tailored to the unique challenges of event-intensity asymmetric stereo, which is one of the novel aspects introduced by our approach.

The inclusion of the disparity refinement module brings improvements, particularly in preserving sharp depth boundaries for objects like cars and buildings in challenging scenes with sparse events or textureless areas, as evident in the visual comparisons between Ours-DS and Ours-DS-DA in Figures 6, 7, 12, and 13. The marginal gains observed in the evaluation metrics (Table 2, Rows 7-8) are likely due to the sparsity of the ground truth disparity provided in the DSEC dataset (Figure 5).

We profiled the runtime and found that the disparity refinement module takes about 306.82 ms, accounting for 48.6% of the total inference time (630.36 ms) of the Ours-CR-DA variant. We can reduce this overhead while maintaining comparable performance by decreasing the number of iterations, as shown in Table C in the response to Global - Q2.

W3&Q1. Linear transformation in disparity refinement:

The choice of a linear transformation to connect monocular and binocular predictions is not a standard practice but rather a novel aspect of our approach. It is motivated by the observation that monocular depth models often predict relative depth up to an unknown scale and shift (L170-L172). A linear transformation offers a simple yet effective way to align these predictions with the stereo disparity map, particularly in preserving sharp depth boundaries for objects like cars and buildings in challenging scenes with sparse events or textureless areas. While exploring more complex transformations is an interesting avenue for future work, our experiments demonstrate the effectiveness of this simple approach in achieving good performance.

Thank you for your insightful comments and positive feedback on our work. We address your concerns below.

S1. Comparison to more methods:

We appreciate the suggestion to compare with Mostafavi et al. [17]. However, for the slightly different setting where both frames and events are used in both views, no pre-trained models are publicly available. We have, however, included a comparison with the event-based stereo setting from their follow-up CFF [18] using their released checkpoint. Due to the character limit, please refer to the response to EJ3V - W1 for more details about the availability of publicly released code and checkpoints for related works in Table E.

W1. Disparity refinement module:

The inclusion of the disparity refinement module brings improvements, particularly in preserving sharp depth boundaries for objects like cars and buildings in challenging scenes with sparse events or textureless areas, as evident in the visual comparisons between Ours-DS and Ours-DS-DA in Figures 6, 7, 12, and 13. The marginal gains observed in the evaluation metrics (Table 2, Rows 7-8) are likely due to the sparsity of the ground truth disparity provided in the DSEC dataset (Figure 5).

We profiled the runtime and found that the disparity refinement module takes about 306.82 ms, accounting for 48.6% of the total inference time (630.36 ms) of the Ours-CR-DA variant. We can reduce this overhead while maintaining comparable performance by decreasing the number of iterations, as shown in Table C in the response to Global - Q2.

We appreciate the suggestion regarding fine-tuning a stereo network on synthetically transformed frame pairs. While promising, this approach presents challenges in accurately simulating event data from frames and potential domain gaps between synthetic and real data. Our zero-shot method offers a practical and robust solution that can easily integrate future advances in frame-based stereo matching and monocular depth estimation.

W2. Related works:

We will expand the related works section in the final version as suggested to include paragraphs on "Visual Prompting" and "Event-Intensity Stereo Fusion," citing and discussing relevant works, including Mostafavi et al. [17].

W3. Visual prompting terminology:

We agree that our usage differs slightly from [1] and is more aligned with [NeurIPS'23], which also uses a non-learnable transformation as a visual prompt. We will clarify this in the final version, emphasizing that while we don't use learned perturbations, our approach shares the core concept of modifying inputs to adapt pre-trained models without fine-tuning.

[NeurIPS'23] Yang et al., Fine-Grained Visual Prompting.

Q. Minor comments:

Thank you for your careful reading. We will correct all the issues you identified: 1) The reference will be changed to Eq. (4). 2) We will use the correct notation $W^{(0)}_\mathbf{p}$ in Eq. (14). 3) It will be adjusted to align the input order. 4) We will clarify that it refers to $C$ in Eq. (12). We will ensure all references and definitions are accurate in the final version.

Thank you for your kind and constructive feedback. We appreciate your insights and the opportunity to address your questions and suggestions.

Q1. Analysis of improvement of the refinement module in edge and textureless areas

An analysis of the EPE improvements on the interlaken_00_c sequence is shown in the table below. We differentiate between "textureless areas" and "edge areas" according to the values in the left-view temporal gradient images, which are derived from the differences in pixel values between consecutive frames. The results show that the refinement module yields more improvements in static areas. While depth estimation is more challenging for stereo matching between events and frames in these regions, it is generally easier for monocular depth estimation due to the large smooth areas.

Table G. Analysis of improvement of the refinement module in edge and textureless areas

Q2. Finetuning versus refinement

Thank you for the suggestion and the clarification. We have attempted a similar approach prior to exploring the zero-shot approach, motivated by the desire to leverage prior knowledge in the frame domain. However, we found that fine-tuning a large model without encountering catastrophic forgetting posed significant challenges given our computational resources. We will explore this approach more in-depth in future work.

Q3. Global scale versus spatially-varying scales

Thank you for your suggestion. In the examples showcased, a global scale approach could indeed perform better. However, for most cases, spatially-varying scales outperform global scales, as demonstrated in our comparison with the ablation study "DA w/ GT scale", which serves as an upper bound for all global-scale-based methods. Our proposed spatially-varying scale scheme can encompass global scales, and it might be beneficial to design a hybrid method that automatically switches between the two models based on specific criteria. We will explore this in future work.

Thank you for your thoughtful feedback and positive assessment of our work. We address each point below.

W1. Comparison with more methods:

In the main text, we have compared our approach with several methods (Table 1): the deep learning-based method DAEI [33], after obtaining their code upon request; the traditional methods HSM [13] and SHEF [24], both of which have released code; and the event-based stereo method CFF [18], using their released checkpoint. Notably, DAEI [33] is recognized in the literature for achieving state-of-the-art performance.

In response to the reviewer's suggestion, we conducted a thorough survey of the availability of publicly released code and checkpoints for related works, as summarized in Table E. Unfortunately, to the best of our knowledge, there are currently no publicly available implementations for deep learning-based event-intensity asymmetric stereo matching methods [39, 33, 3]. Even for the slightly different scenario where both frames and events are used in both views, pre-trained models are not publicly available.

We will continue to track the progress of these methods and actively engage with the authors to request access to their code or checkpoints. We are also in the process of reproducing the results from [39] and plan to include them in the final version of our paper. Our commitment is to provide a comprehensive evaluation and comparison as more resources become available.

Table E. Availability of publicly released code and checkpoints for related works.

W2. Written mistakes:

We apologize for the error. This will be corrected in the final version, along with a thorough review to address any other minor mistakes.

W3&Limit. Disparity refinement:

We agree that the optimization details for cue-guided disparity refinement should be in the main text and will move this information from the appendix in the final version.

Please refer to the response to Global - Q2 for the computational overhead introduced by the refinement module in Table C.

Q1. Evaluation on additional datasets:

The DSEC dataset covers a wide range of scenarios, including various lighting conditions, motion patterns, and scene complexities, as illustrated in Figures 4 and 14. However, we agree with the reviewers that evaluation on more datasets would provide a more comprehensive assessment.

To demonstrate the generalization ability of our approach, we evaluated Ours-CR-DA on sequences from two additional datasets: MVSEC [38] (DAVIS sensor, 2018) and M3ED [CVPRW'23] (Prophesee sensor). The quantitative results are shown in Table F, with the qualitative results in Figure B of the attached PDF. These results demonstrate ZEST's robust generalization across different environments, motion patterns, and sensor characteristics, supporting its applicability in diverse scenarios.

We acknowledge the current limitations in our comparison with other methods on these datasets and the small number of sequences used. The distinct data formats of the new datasets and slow download speeds posed challenges to the timely completion of our experiments, preventing us from providing more extensive evaluations within the given timeframe. In the final version, we will include more sequences and conduct comparisons with additional methods to offer a more thorough evaluation.

[CVPRW'23] Chaney et al., M3ED: Multi-Robot, Multi-Sensor, Multi-Environment Event Dataset.

Table F. Quantitative results of the proposed zero-shot disparity estimation method on additional datasets.

Q2. Alternatives for constructing scale and shift maps.

We considered guided filtering-based alternatives without explicit optimization, which apply a linear transformation to respect its structure while maintaining the absolute amplitudes of the edges of the binocular disparity prediction. However, we found that the guided filtering-based method produced inferior results with blurry boundaries, as shown in Figure C of the attached PDF. We agree this is an interesting direction for future work to potentially improve efficiency.

Q3. Resolution differences:

Our method assumes input data from both views have the same spatial resolution, aligning with the DSEC dataset. If necessary, different resolutions between image and event sensors can be handled through appropriate resampling.

Thank you for your valuable comments and insightful suggestions. We address each concern below.

W1. Evaluation on more datasets:

The DSEC dataset covers a wide range of scenarios, including various lighting conditions, motion patterns, and scene complexities, as illustrated in Figures 4 and 14. However, we agree with the reviewers that evaluation on more datasets would provide a more comprehensive assessment.

To demonstrate the generalization ability of our approach, we evaluated Ours-CR-DA on sequences from two additional datasets: MVSEC [38] (DAVIS sensor, 2018) and M3ED [CVPRW'23] (Prophesee sensor). The quantitative results are shown in Table F in the response to EJ3V - Q1, with the qualitative results in Figure B of the attached PDF. These results demonstrate ZEST's robust generalization across different environments, motion patterns, and sensor characteristics, supporting its applicability in diverse scenarios.

[CVPRW'23] M3ED: Multi-Robot, Multi-Sensor, Multi-Environment Event Dataset

W2&Q1&Limit2. Computational efficiency and scalability:

Due to the character limit, please refer to the response to Global - Q1 for an analysis to computational efficiency in Tables A and B.

We have also examined the scalability of our method with varying input resolutions. As presented in Table D, GPU memory usage and runtime only increase marginally as the resolution scales up. Notably, the Depth Anything model internally uses a fixed resolution for inference, which keeps its memory usage constant.

Various strategies can be applied to speed up our method. For example, we can run the stereo matching module and the monocular depth estimation module in parallel. Due to the flexibility of our framework, our method can also be accelerated by using more lightweight models. We will further explore lightweight alternatives for the stereo and monocular modules (e.g., Depth-Anything-Small) that could achieve speedups for real-time applications, and we will provide the results and discussions in the final version. Our modular design also allows for further optimizations like model pruning and quantization.

Table D. Performance and computational cost comparison for varying input data sizes.

W3. Comparison with more methods:

In the main text, we have compared our approach with several methods (Table 1): the deep learning-based method DAEI [33], after obtaining their code upon request; the traditional methods HSM [13] and SHEF [24], both of which have released code; and the event-based stereo method CFF [18], using their released checkpoint. Notably, DAEI [33] is recognized in the literature for achieving state-of-the-art performance.

In response to the reviewer's suggestion, we conducted a thorough survey of the availability of publicly released code and checkpoints for related works, as summarized in Table E in the response to EJ3V - W1. Unfortunately, to the best of our knowledge, there are currently no publicly available implementations for deep learning-based event-intensity asymmetric stereo matching methods [39, 33, 3]. Even for the slightly different scenario where both frames and events are used in both views, pre-trained models are not publicly available.

We will continue to track the progress of these methods and actively engage with the authors to request access to their code or checkpoints. We are also in the process of reproducing the results from [39] and plan to include them in the final version of our paper. Our commitment is to provide a comprehensive evaluation and comparison as more resources become available.

Q2. Robustness to input data quality:

Our method exhibits robustness to variations in data quality. Figure 4 demonstrates consistent performance under challenging conditions: sparse event inputs in Column 2, rapid scene changes in Column 3, and extreme low-light conditions in Column 4. Figure 14 further showcases robustness in diverse scenarios. These results indicate stable performance even with moderate input quality degradations.

Limit1. Zero-shot generalization.

We recognize the limitations of a purely zero-shot setting and plan to explore techniques for efficient adaptation to new domains with limited data in future work.

Limit2. Computational overhead.

We profiled the runtime and found that the disparity refinement module takes about 306.82 ms, accounting for 48.6% of the total inference time (630.36 ms) of the Ours-CR-DA variant. We can reduce this overhead while maintaining comparable performance by decreasing the number of iterations, as shown in Table C in the response to Global - Q2.

The inclusion of the disparity refinement module brings improvements, particularly in preserving sharp depth boundaries for objects like cars and buildings in challenging scenes with sparse events or textureless areas, as evident in the visual comparisons between Ours-DS and Ours-DS-DA in Figures 6, 7, 12, and 13. The marginal gains observed in the evaluation metrics (Table 2, Rows 7-8) are likely due to the sparsity of the ground truth disparity provided in the DSEC dataset (Figure 5).

We appreciate your thorough review and insightful questions. We are pleased that you found our work technically strong, novel, and impactful. We address each point below.

W1&Q1. Computational efficiency analysis:

We profiled the computational complexity of our framework's modules on a machine with an Intel i7-13700K CPU and an NVIDIA RTX 4090 GPU, using an input resolution of 480x640. The performance values were evaluated on the interlaken_00_c sequence of the DSEC dataset unless otherwise specified. A breakdown of the computational cost in each algorithm stage is shown in Table A in the response to Global - Q1, and the total computational costs of each method variant, along with the compared methods, are shown in Table B in the response to Global - Q1.

The Ours-CR-DA variant achieves a runtime of about 630.36 ms per frame with 7454 MB GPU memory usage. The disparity refinement module is the most computationally intensive component, consuming 48.6% of the total runtime. In the Ours-DS-DA variant, the DS model dominates the computational cost, while the DA and refinement modules add minimal overhead. We will explore optimizations to improve efficiency in future work and include this discussion in the final version.

W2. Technical clarity:

We will add explanations for technical terms and concepts to enhance readability and accessibility for a broader audience.

Q2. Unknown event triggering thresholds: Our method manages unknown event-triggering thresholds with a normalization operation (L404-406). The normalization eliminates the threshold $c$ in Eq. (17), bridging temporal event integrations and temporal image gradients. In the final version, we will move this explanation to the main text for clarity.

Q3. Extending visual prompting: We see potential in extending the visual prompting technique to other tasks, such as optical flow estimation or object detection. Challenges include adapting the prompting mechanism to accommodate different data characteristics and ensuring prompt effectiveness across diverse modalities. For instance, in optical flow estimation, the prompts could encode temporal information about object motion, while for object detection, they could highlight salient features relevant to object recognition.

Q4. Sensitivity to stereo model choice:

As shown in Table 1, the proposed method, when combined with different stereo matching models such as CREStereo (CR) [14] and DynamicStereo (DS) [12], exhibits variations in performance. Overall, variants using DS perform better in terms of EPE and RMSE, while those using CR excel in 3PE and 2PE. Importantly, the proposed method consistently improves accuracy across various stereo matching models, highlighting the robustness of our intermediate representation.

Q5. Evaluation with sparse ground truth: Consistent with previous works on the DSEC dataset, we evaluate metrics using only pixels with valid ground truth disparity values. This approach focuses on meaningful disparities and excludes void or uncertain areas. However, we acknowledge that this evaluation strategy has limitations due to the sparsity of the ground truth. Future work could explore alternative evaluation metrics specifically designed for sparse data.

Q6. "Spatial Integ." and stereo matching: "Spatial Integ." refers to the spatial integral of events, capturing accumulated changes over time in different positions of the sensor corresponding to the same physical position.

The slight differences in textures between views are due to the asynchronous nature of events and manufacturing imperfections. Despite the differences, our method performs well due to the surprising robustness of state-of-the-art stereo models, which are able to exploit high-level feature similarities despite low-level texture inconsistencies.

Q7. "GT scale" in ablation study:

In our method, we model the relationship between the monocular-predicted disparity $D^\text{mono}$ and the stereo-predicted disparity $D^{\text{bino}}$ as

$$D_{i,j}^{\text{bino}} = k_{i,j} D_{i,j}^{\text{mono}} + b_{i,j},$$

where $k$ and $b$ are spatially-varying scale and shift maps obtained by optimization. However, there also exists a simpler model where $k$ and $b$ are globally consistent

$$D_{i,j}^{\text{bino}} = k_{\text{global}} D_{i,j}^{\text{mono}} + b_{\text{global}}.$$

In the "GT scale" ablation experiment, we prove that the later model is insufficient. We calculate the optimal $k_{\text{global}}$ and $b_{\text{global}}$ by fitting the ground truth disparities to the relative disparities linearly for each frame. The corresponding result, which is the upper-bound of all algorithms using global scales, is worse than the currently proposed method, proving that spatially-varying scales and shifts are essential.

Limit. Failure cases:

Representative failure cases are shown in Figure A of the attached PDF.

• Row 1 demonstrates the impact of noisy events on the visual prompts. When the events are noisy, the visual difference between the two views increase. The stereo model can robustly manage this inconsistency in most cases, such as in row 1 & 2 of Figure B, but it may occasionaly fail. In row 1 of Figure A, the CR stereo model produced erroneous results. The monocular DA predictions improved the disparity estimation via the refinement module, but the final result was still suboptimal.

• Row 2 shows the impact of sparse events. The sparse events did not provide sufficient information for stereo matching, which the refinement module could not fully compensate for.

We will expand our discussion of potential failure cases in the final version.