Rethinking Scale-Aware Temporal Encoding for Event-based Object Detection

[W1, Q1]: Based on Weakness #1: While your integration of recurrent modules at earlier stages is effective, the key building blocks (ConvLSTM, deformable convolution, and FPN) are well-established. Can you clearly elaborate on which aspects of these modules were specifically adapted or uniquely designed for event-based detection in your work?

[A1]: Thank you for pointing out this important issue. We fully agree that ConvLSTM, deformable convolution, and FPN, as fundamental modules, have been widely used in other tasks. The innovation of our work indeed does not lie in inventing entirely new components, but in the targeted adaptation and reconstruction of these modules to meet the specific needs of event-based vision. The following are the core design adjustments we have made for asynchronous event data:

(1) Decoupled deformable-enhanced recurrent module: Considering the motion information in events, we designed a decoupled structure where deformable convolution first performs spatial sampling and alignment based on local motion, followed by accumulation in the temporal dimension via ConvLSTM. This sequence is specifically tailored to the sparsity of event data and its characteristic of containing motion information. The effectiveness of this design has been verified in ablation experiments.

(2) Shallow recursion to preserve fine-grained temporal information: Given the sparsity of events, shallow features may contain more critical information. Thus, unlike traditional methods that insert recurrent layers into deep semantics, we place the ConvLSTM modules earlier to better capture typical short-term motion patterns and local dynamics in event streams. Additionally, as mentioned in RED, a key reason for placing recurrent layers at high scales is to avoid recurrent layers from dynamically modeling low-level features unnecessary for the given task. However, the decoupled deformable-enhanced module we added before the recurrent layers can significantly reduce these unnecessary features through the smoothing effect of deformable convolution (as shown in Supplementary Materials Figures S3 and S4).

(3) Time-aware multi-branch encoding and FPN fusion: Our multi-branch backbone network performs temporal downsampling independently at each scale to avoid cross-scale interference. Features are fused only after scale-specific temporal encoding. Ablation experiments have also confirmed the effectiveness of the multi-branch design.

---

[W2, Q2]: You evaluated your method on two standard datasets (Gen1 and 1Mpx). How well do you expect your architecture’s advantages to generalize beyond these specific benchmarks? Have you considered or tested scenarios with significantly different event distributions or sensor characteristics? The recent trend has shifted to incorporating more RGB data in event-based detection frameworks. Can you discuss and compare your method (perhaps in event-frame fusion mode) with existing methods, such as DAGR [R1], RENet [R2], FlexEvent [R3], and/or CAFR [R4]?

[A2]: Thank you for your suggestion. We are attempting to train on other datasets (such as EvDET200K) to demonstrate the generalization ability of our model, but this may take some time. Once we have results, we will report them immediately during the discussion. Additionally, thank you for pointing out fusion methods like DAGR, RENet, FlexEvent, and/or CAFR. These works demonstrate the great potential of fusing RGB with events, and we believe that combining work with multimodal frameworks is a direction well worth exploring. Although this paper currently focuses on pure event data input, the Decoupled Deformable-enhanced Recurrent Layer (DDRL) has good modular characteristics and can be embedded into the event branch to provide dynamic auxiliary information for RGB. Especially in scenarios where RGB is degraded, such as low light and blurriness, our method can provide temporal supplementary signals. We plan to focus on exploring this fusion direction in future work.

---

[W3, Q3]: Given the inherent complexity of recurrent and deformable convolution modules, have you performed a detailed runtime or complexity analysis under varying conditions (e.g., higher resolution or event density)? How might these computational considerations affect practical deployments on resource-constrained platforms?

[A3]:

Thank you for this insightful question. We fully agree that for practical deployment, it is crucial to gain a deep understanding of the computational complexity and resource consumption impacts brought by recurrent and deformable modules, especially on resource-constrained platforms.

We tested the inference time and memory usage under different input resolutions on a single RTX 3090, all based on the same model weights (trained on Gen1 with a resolution of 256×320). It can be seen that when the resolution is less than 512×640, our method still maintains relatively advanced inference time. Although the computational cost increases significantly at high resolutions (e.g., 1024×1280), considering the typical application scenarios of event cameras (such as real-time perception in autonomous driving, which usually adopts medium resolutions), our method can balance accuracy and efficiency under mainstream input sizes.

---

[W4, Q4]: Can you provide a more thorough discussion of scenarios or cases where your proposed DDRL module may perform poorly or have limited benefits? Understanding these boundary conditions would significantly strengthen the clarity and practical applicability of your contributions.

[A4]: Thank you for your constructive question. We have found that the benefits of DDRL are limited in the following scenarios:

• Highly crowded scenes with fast-moving and overlapping objects: In scenes where multiple fast-moving objects are severely overlapping in both space and time (such as a congested intersection), deformable offsets may cause motion ambiguity when extracting motion information, leading to reduced alignment effectiveness and even feature blurring.

• Static or near-static scenes: Although the introduction of recurrent layers can to some extent improve the detection performance of slowly moving objects, due to the limited sequence length, the contribution of temporal modeling remains limited in certain scenarios.

[W1]: The proposed methods do not present new insights or theoretical advancements in the context of event-based representation learning, particularly in areas like asynchronous data processing or novel feature extraction techniques. This approach primarily improves on existing RNN-based models without introducing a groundbreaking shift in understanding the underlying event-based object detection problem.

[A1]: Thank you for your perspective. Our method mainly focuses on spatiotemporal feature extraction approaches from a new direction. Firstly, starting from the design of recurrent layers, we developed a decoupled deformable-enhanced module based on the characteristic that events contain motion information, aiming to reuse the state features in the recurrent module. This module integrates adaptive spatial sampling and temporal modeling capabilities, significantly improving the alignment effect of temporal features. Secondly, considering that low-scale features may contain richer motion details, we introduced recurrent layers at low scales and re-examined the temporal modeling method under scale-specific conditions. As mentioned in the previous work RED, one of the important reasons for placing recurrent layers at high scales is to prevent recurrent layers from dynamically modeling low-level features that are unnecessary for the given task. However, the decoupled deformable-enhanced module we added before the recurrent layers can greatly reduce these unnecessary features through the smoothing effect of deformable convolution (as shown in Supplementary Materials Figures S3 and S4).

---

[W2]: (1) Lack of recent state-of-the-art event-based object detection methods, such as: SpikeYOLO [1], EGSST [2], EventPillars [3], HsvT [4], SNN-ANN [5], etc; (2) Compared to recent state-of-the-art methods, the performance improvement is limited; (3) Has the proposed method been tested for other alternative event representations (such as time surfaces, histogram)? (4) Can the proposed method be tested on the EvDET200K [6] dataset?

[A2]:

Thank you for mentioning many recent and meaningful SOTA methods. Compared with these SOTA methods you referred to, our method still remains competitive, especially in comparison with SpikeYOLO, HsvT-B, and SNN-ANN. EGSST-E has shown good performance on both the Gen1 and 1Mpx datasets. We believe this is partly attributed to its proposed dynamic label enhancement strategy, which improves the accuracy and adaptability of annotations by dynamically adjusting the time window for label matching. Additionally, it is worth noting that when the YOLO detection head is used to replace the RT-DETR detection head (i.e., EGSST-E-Y), the performance of EGSST decreases. Regarding (3) and (4), we are currently training on the EvDET200K dataset using histogram representation, and will share the results immediately during the discussion once available.

[W1, Q1]: The fixed 5-bin, 50 ms voxelization may not capture the fastest or smallest event dynamics, yet no ablation evaluates finer temporal discretization.

[A1]: Thank you for your valuable comments. We have added an ablation study on the number of voxel bins and the temporal window size. To reduce data generation and training time, we randomly selected one-third of the Gen1 training set for training and evaluated on the full Gen1 test set. All other settings were kept consistent with those in the main paper. Our experiments are divided into two groups:

• Fixed number of bins (5), varying temporal window sizes (20ms, 50ms, 100ms):
  The results show that a 50ms window yields relatively better performance. This may explain why several prior works such as RED, RVT, and SSM have also adopted 50ms as the default window size.

• Fixed temporal window size (50ms), varying number of bins (5, 8, 10):
  The results indicate that increasing the number of bins does not lead to significant performance gains. This may be due to the sparse nature of event data—when using voxel representation, 5 channels are already sufficient to capture most of the meaningful information available in the event stream.

---

[W2, Q3]: Please compare or discuss your approach relative to SODFormer (Li et al., 2023) and the Time-Surface Pyramid (Manderscheid et al., 2019). Benchmarking against these fine-grained temporal-encoding methods would help clarify the paper’s novelty and positioning.

[A2]: Thank you for your suggestion. Based on your feedback, we have improved the Related Work section. Compared to SODFormer, both methods aim to model temporal dynamics in event data, but differ in design: SODFormer performs event - frame fusion using asynchronous attention to compensate for frame degradation under fast motion or low light. Our method focuses on event - only modeling, emphasizing early - stage temporal encoding to capture fine - grained motion without relying on frame input. SODFormer models long - term dependencies in deep layers via global attention, which is computationally heavy. In contrast, we apply recurrent modeling at lower feature stages, better preserving temporal cues like fast edge motions that tend to degrade after downsampling. Regarding Time - Surface Pyramid, it targets corner detection by building velocity - invariant patterns through local decay, aiming to normalize static temporal changes. Our DDRL module, by combining deformable convolution and recurrence, directly models motion dynamics and better serves event - based object detection tasks

---

[W3, Q2]: There is no visualization or quantitative analysis of the learned offsets ($\Delta$x ), nor any comparison to optical - flow ground truth to validate that motion alignment is actually occurring.

[A3]: Thank you very much for your suggestion. We fully agree that visualizing the learned offsets of the deformable convolution is important for better understanding its behavior. We have visualized the offsets from the first deformable - enhanced block using heatmaps. We chose this layer because it retains a higher spatial resolution and the features are less abstract, making the offsets more interpretable. The visualization reveals that the offsets primarily capture motion information within the current frame. And we will include these offset visualizations.

---

[W4]: Combining deformable convolution with ConvLSTM could over-smooth spatial detail, but no effective-receptive-field or smoothing analysis is provided.

[A4]: You make a valid point — combining deformable convolution with ConvLSTM may indeed risk over-smoothing spatial details. To investigate this, we provided a visual comparison in the supplementary material (Figures S3 and S4) between features output with and without deformable convolution in the recurrent layers. The results show that while adding deformable convolution introduces some smoothing, it primarily suppresses irrelevant or noisy information that does not contribute to object detection. We will further enhance this analysis by adding more visualizations.

---

[W5, W6, Q4]: Can you report end-to-end inference latency, peak GPU memory usage (batch 1), and a small/medium/large object AP breakdown on both Gen1 and 1 Mpx inputs, measured on specified hardware?

[A5]: Thank you for your suggestion. We re-conducted the tests on an Nvidia RTX 3090, comparing with advanced methods such as RVT and SSM. The batch size was set to 1, and the results on the Gen1 dataset are as follows:

Here, AP_S, AP_M, and AP_L represent the detection accuracies for small, medium, and large objects, respectively. It can be seen that our method achieves the best performance while ensuring high inference speed and low GPU peak memory, especially in small object detection.

---

[Q5]: Would moving one or two key supplementary results (e.g., the Table S1/S2 ablations or a representative offset heat map) into the main manuscript improve transparency around your architectural choices?

[A6]: Thank you very much for your valuable suggestion. We fully agree with your idea and will make some adjustments.

[W1, Q1 generalization experiment]: The use of deformable convolution in the recurrent layer design raises questions about generalization. It is important that the authors evaluate the performance of the model when trained on one dataset and tested on a different one.

[A1]:

Thank you for raising this important question regarding the generalization of deformable convolution in the recurrent layer design. We agree that cross-dataset evaluation is essential to verify the model's generalization ability. We conducted generalization experiments using our model, carrying out cross-dataset generalization between the Gen1 and 1Mpx datasets, and compared them with two advanced Transformer and Mamba models (RVT, SSM). We directly used the weights trained on one dataset for testing on the other dataset without fine-tuning the weights.

It can be observed that despite using deformable convolution to dynamically learn offsets in event data, the generalization ability of our model is still comparable to, or even better than, the other two types of methods, especially in the generalization experiment from Gen1 to 1Mpx.

Generalization: 1Mpx → Gen1

Generalization: Gen1 → 1Mpx

---

[W2, Q2]: The paper is repetitive about the high-level points that temporal patterns in event data appear in low-level features and are best captured before spatial abstraction. This point and several related ones are made multiple times in the Intro, Related Works, Method sections. On the other hand, the review of related work is quite brief and very few qualitative results are presented. I recommend that the authors reduce the repetitiveness of the high-level motivation for the approach and instead add more of: technical details, review of additional related works, qualitative results.

[A2]: Thank you very much for your constructive suggestion. Based on your feedback, we will make the following improvements:

• Reducing redundancy: We will revise and streamline the Introduction and Related Work sections to eliminate repetitive statements about our method. The discussion on the decoupled deformable-enhanced recurrent layer and low-level spatiotemporal modeling will be condensed and clarified within the Method section.

• Expanding related work: The original Related Work section was kept concise due to space limitations. We will expand this section to include recent works such as ERGO-12 and S5-ViT, providing a more comprehensive overview of event data modeling approaches.

• Qualitative visualization analysis: Figures 5 and 6 present qualitative results from our ablation study to demonstrate the effectiveness of our proposed module. We acknowledge that the detection boxes may have been too thin, making it difficult to observe differences. In several scenes, other ablated variants exhibit missed detections or false positives.

---

[W3, Q3 Table1]: The authors should include the time and accuracy results for all methods in Table 1. Also, the best and second-best method should be highlighted for the time results as well.

[A3]: The missing entries in Table 1 correspond to earlier works published before the release of the 1Mpx dataset, and these methods generally exhibit lower performance. Additionally, their original papers did not report inference time, which makes it difficult for us to provide fair or reproducible runtime comparisons.

---

[Q4]: Minor comments
[A4]: Thank you for your careful reading. We have addressed your suggestions one by one and thoroughly checked the manuscript for any other potential issues.