Event Camera Object Detection at Arbitrary Frequencies

Q3: "Provide the latency, parameter count, and computational cost, to enable a fairer performance comparison."

A: Thanks a lot for your suggestion.

• We have included comparisons of parameter count and inference time (in milliseconds, ms) in the table below, as well as in Tab. 7 of the revised manuscript. | Method | Param (M) |||20 Hz|||27.5 Hz|||30 Hz|||36 Hz|||45 Hz|||60 Hz|||90 Hz|||180 Hz|||200 Hz |:-:|:-:|:-:|:-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:| | RVT | 18.5 ||| 9.20 ms ||| 8.67 ms ||| 8.35 ms ||| 7.93 ms ||| 7.61 ms ||| 7.51 ms ||| 7.19 ms ||| 6.77 ms ||| 6.34 ms | | SAST | 18.9 ||| 14.06 ms ||| 13.11 ms ||| 12.68 ms ||| 12.37 ms ||| 11.95 ms ||| 11.63 ms ||| 11.52 ms ||| 11.10 ms ||| 10.36 ms | | SSM | 18.2 ||| 8.79 ms ||| 8.26 ms ||| 8.08 ms ||| 7.71 ms ||| 7.55 ms ||| 7.30 ms ||| 6.90 ms ||| 6.54 ms ||| 6.12 ms | | DAGr-50 | 34.6 ||| 73.35 ms ||| 65.73 ms ||| 60.02 ms ||| 55.11 ms ||| 51.00 ms ||| 48.00 ms ||| 45.29 ms ||| 43.89 ms ||| 37.58 ms | | FlexEvent | 45.4 ||| 14.27 ms ||| 13.53 ms ||| 13.32 ms ||| 13.00 ms ||| 12.79 ms ||| 12.58 ms ||| 12.47 ms ||| 12.37 ms ||| 12.12 ms |

• As can be seen from the results, while FlexEvent has around 9 M parameters more (compared to the SOTA method DAGr), our inference time (under different frequency setups) is comparable to the event-only method SAST and significantly faster than the event-frame fusion method DAGr.

• We also observe that all methods will become slower as the frequency decreases due to the need to process more event data. These findings highlight FlexEvent's efficiency and rapid performance.

Q4: "Based on this premise, is changing the frequency and adjusting the event sparsity through data augmentation essentially the same?"

A: Thanks a lot for your question.

• Changing the event frequency and adjusting event sparsity are fundamentally different processes:
  • In our method, we represent events using voxel grids, a representation that is known to lose temporal information. For low-frequency training, the event stream is divided into $T$ bins, where all events within a bin are treated as occurring at the same time. This temporal information is lost due to the floor operation, as described in Eq. (2) of our manuscript.
  • For high-frequency training, the same event stream is divided into $T$ bins over shorter intervals. If the frequency increases by $10$x, the temporal resolution also increases by $10$x, retaining more temporal information. Thus, the temporal information loss caused by voxel grids is significantly reduced at higher frequencies.
  • In contrast, adjusting the sparsity of low-frequency events necessarily results in a greater loss of temporal information than high-frequency training. Unless the sparsity is extreme (e.g., no events in a bin), this loss cannot be avoided.

Q5: "When addressing the issue of missed detections at low frequencies, is it more the RGB information that compensates for this, rather than the method you proposed?"

A: Thanks a lot for asking.

• We assume the reviewer’s concern relates to FAL’s impact at low frequencies compared to the frame branch. This has been reflected in Table 4. Specifically:

  • At 20 Hz, FAL improves performance by 1.4%, as seen in rows 1-2 of the table.
  • At 180 Hz, FAL improves performance by 7.5%, a much larger margin for the performance gain.

• This phenomenon is expected because FAL is designed to improve generalization at higher frequencies. A model trained solely on 20 Hz data (e.g., Variant 1 in Table 4) is naturally better suited for low-frequency scenarios, while FAL enhances performance at higher frequencies by leveraging additional high-frequency data. These results align with our design goals and validate the effectiveness of FAL.

Last but not least, we sincerely thank Reviewer wFUC again for the valuable comments provided during this review.

We sincerely thank Reviewer wFUC for the valuable comments provided during this review.

We have revised the manuscript based on your suggestions, please refer to the purple-colored texts. The point-to-point responses are as follows:

Q1: "The authors should clearly state in the main text whether the comparison methods are retrained on DSEC-Det, DSEC-Detection, and DSEC-MOD. The authors should ensure consistency in the training modalities of all models, comparing them either exclusively in the Event-RGB mode or using purely event-based training."

A: Thanks for your question. We provide explanations on the use of datasets, methods, and experimental settings as follows:

• The datasets:
  • DSEC is a high-resolution, large-scale multimodal dataset specifically designed for event-based perception. For our experiments, we use three comprehensive versions of DSEC, i.e., DSEC-Det, DSEC-Detection, and DSEC-MOD, which are tailored for the object detection task. These versions provide a robust platform for benchmarking and evaluating event-based object detection methods.
  • Among them, DSEC-Det is the most recent, largest, and most comprehensive one, annotated and released by the original DSEC authors. Thus, we prioritized it as the primary benchmark for reporting results, ensuring relevance and reliability. DSEC-Detection and DSEC-MOD are datasets used by two recent event-frame fusion methods CAFR and RENet, so we also report results on these two datastes to validate our method’s effectiveness.
• The comparison methods:
  • To evaluate the effectiveness of our method, we compared both event-only and event-frame fusion methods.
  • For event-only methods:
    • We included state-of-the-art detectors (RVT, SAST, LEOD, and SSM), which were originally trained on event-only datasets like Gen1 and 1Mpx.
    • To ensure fairness, we retrained these methods on DSEC-Det. Table 1 demonstrates that our method outperforms these detectors.
  • For event-frame fusion methods:
    • For DSEC-Det, we compared with DAGr, as it has been evaluated on DSEC-Det. We report the scores of DAGr from the original paper to ensure fairness.
    • For DSEC-Detection and DSEC-MOD, we trained our model by following the standard training and evaluation settings, and compared again the state-of-the-art methods, CAFR and RENet, in Table 2 and Table 3. For other methods on DSEC-Detection and DSEC-MOD, we reference the results from the CAFR paper and the RENet paper, respectively.
    • Since DAGr's training code is not publicly available, we are not able to reproduce DAGr on DSEC-Detection and DSEC-MOD. We have been contacting the authors about this but haven't gotten an update.
• These comparisons ensure a fair and comprehensive evaluation while adhering to resource and code availability constraints.

Q2: "Provide more ablation and visualization results to compare whether FAL shows significant differences from other data augmentation methods."

A: Thanks a lot for your valuable suggestions. We have carefully studied the suggested data augmentation methods, i.e., EventDrop and Shadow Mosaic. We have also included detailed discussions and comparisons with them in the revised manuscript.

Here, we would like to clarify that our Frequency-Adaptive Learning (FAL) mechanism differs in nature from the above two methods from the following perspectives:

• Methodology:

  • Our FAL mechanism is fundamentally different from EventDrop and Shadow Mosaic, as it is a self-training framework rather than a simple data augmentation. FAL leverages unlabeled high-frequency data to enhance the generalization ability of our model at high frequencies by:
    1. First, training the model on labeled high-frequency data (pretraining).
    2. Then, using this pretrained model to infer pseudo labels on all high-frequency data.
    3. Next, refining pseudo labels using strategies like bidirectional data augmentation, NMS, and tracking-by-detection to filter noisy labels.
    4. Last, mixing pseudo labels with ground-truth labels and train again.
  • This iterative process (pretrain-> generating labels → refining labels → retraining) gradually improves pseudo label quality and enables better generalization at high frequencies by utilizing all available data. This approach contrasts significantly with EventDrop and Shadow Mosaic, which enhance diversity through simple data augmentation techniques, such as discarding events over a time period or increasing temporal density to improve robustness across scenarios.

• Experimental Setting:

  • Our work focuses on inference across arbitrary frequencies, conducting comprehensive comparisons with SOTA methods under varying frequencies. In contrast, EventDrop and Shadow Mosaic emphasize generalization through data augmentation but do not evaluate performance across other frequencies, making their experimental setting fundamentally different from ours.

• Comparative Study:

  • We conduct experiments using data augmentation methods from EventDrop and Shadow Mosaic. As Shadow Mosaic is not open-sourced, we reproduce the results as best as we can. The results are shown in the table below, as well as in Tab. 8 of the revised manuscript. | Drop | Mosaic | FAL | 20.0 Hz | 27.5 Hz | 30.0 Hz | 36.0 Hz | 45.0 Hz | 60.0 Hz | 90.0 Hz | 180 Hz | Average | |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| |✗|✗|✗| 53.2% | 54.0% | 53.5% | 52.0% | 49.4% | 45.9% | 38.8% | 22.9% | 46.2% | |✓|✗|✗| 53.6% | 54.4% | 53.8% | 52.7% | 50.2% | 47.2% | 40.2% | 24.5% | 47.1% | |✗|✓|✗| 53.7% | 54.4% | 54.0% | 53.9% | 51.4% | 48.6% | 41.8% | 27.8% | 48.2% | |✗|✗|✓| 54.6% | 54.9% | 54.9% | 54.3% | 53.3% | 50.7% | 44.6% | 30.4%| 49.7% |
  • Compared to the plain baseline (first row), EventDrop and Shadow Mosaic demonstrate good performance enhancement, credited to the strong generalization ability brought by the spatial and temporal manipulations of the event data.
  • Meanwhile, the results demonstrate that FAL significantly outperforms other methods by leveraging high-frequency event data, especially in high-frequency scenarios.
  • The iterative refinement through self-training in our method ensures that the model remains robust across different motion dynamics and frequency settings.

Response to Your Second Concern:

"I still maintain my point of view: I believe the model essentially benefits from the significant increase in the amount of labeled data."

The reviewer suspect that the model improvement essentially benefits from the increase of data amount. We would like to provide evidence from our original ablation studies to demonstrate that the improvements are primarily due to our proposed modules rather than just an increase in labeled data.

1. Ablation Study Results:

   • We conducted comprehensive ablation experiments to isolate the effects of each component, as shown in the Tab.4 in the main paper:

   Modality	FAL	EFF	20.0 Hz	27.5 Hz	30.0 Hz	36.0 Hz	45.0 Hz	60.0 Hz	90.0 Hz	180 Hz	Average
   E	✗	✗	53.2%	54.0%	53.5%	52.0%	49.4%	45.9%	38.8%	22.9%	46.2%
   E	✓	✗	54.6%	54.9%	54.9%	54.3%	53.3%	50.7%	44.6%	30.4%	49.7%
   E+F	✗	✓	58.0%	59.6%	60.0%	59.6%	59.0%	57.6%	54.8%	49.2%	57.2%
   E+F	✓	✓	57.4%	60.0%	60.0%	60.1%	59.5%	58.8%	56.5%	50.9%	57.9%

   • Notations:
     • E: Event modality.
     • F: Frame (RGB image) modality.
     • EFF: Adaptive Event-Frame Fusion module.
     • FAL: Frequency-Adaptive Learning module.

2. Analysis of Improvements:

   • Comparing rows 1 and 2 (Event modality without and with FAL), we observe an average improvement of 3.5%, indicating that FAL contributes to performance gains without increasing the amount of event data.
   • Comparing rows 1 and 3 (without and with EFF, without FAL), we see a significant average improvement of 11.0%, demonstrating that the EFF module is the primary contributor to performance enhancement, which is one of our contributions. Without FAL, these two variants don’t introduce additional pseudo-label, use the base ground data and complete fair setting according to the reviewer.

3. Key Observations:

   • The significant improvement from EFF underscores that our adaptive event-frame fusion strategy is the main factor enhancing performance, not merely an increase in labeled data.
   • FAL provides additional benefits, particularly in high-frequency scenarios, by improving the model's ability to generalize across frequencies.
   • All models, with or without FAL, use the same event dataset; FAL enhances label information without adding new event samples.

4. Relation to Data Augmentation Methods:

   • Data augmentation methods like EventDrop and Shadow Mosaic focus on increasing data diversity through spatial and temporal manipulations on the original data. Our FAL approach is a self-training framework that improves temporal resolution without altering the original data.
   • These are two parallel methods to improve generalization in different scenarios. We have already added a comparison with these two data augmentation methods in our previous reply, proving our method's effectiveness. The extra pseudo labels come from our own method and are one of our contributions.
   • EventDrop and Shadow Mosaic demonstrate good performance enhancement, credited to the strong generalization ability brought by the spatial and temporal manipulations. These methods are not mutually exclusive and can be integrated. Combining data augmentation with our FAL self-training framework would be a interesting direction in the future. And we have included comparisons with these data augmentation techniques in the related work in our revised manuscript in Line 126-129.

We hope that this detailed explanation addresses your concerns. Our results indicate that the performance improvements are primarily due to our proposed modules, especially the EFF, rather than solely from an increased number of labels. We are committed to maintaining fairness in our comparisons and ensuring that our contributions are clearly demonstrated.

Dear Reviewer wFUC,

We sincerely appreciate your time and effort in reviewing our manuscript and for providing insightful questions for discussion.

We hope that our previous replies have adequately addressed your concerns.

We look forward to participating in the Author-Reviewer Discussion session and welcome any additional feedback you may have on the manuscript or our reply.

Once again, thank you for your valuable contributions to improving the quality of our work.

Best regards,

The Authors of Submission 589

We sincerely thank Reviewer FU5g for the valuable comments provided during this review.

We have revised the manuscript based on your suggestions, please refer to the cyan-colored texts. The point-to-point responses are as follows:

Q1: "(a) Fig. 2 does not clearly illustrate the process of feature fusion at different frequencies; (b) For Eq (2), the inputs to the network for all scales except the initial one which corresponds to $E^a$, $E^b$, $F$, should correspond to $^{(i-1)}h_E^a$ , $^{(i-1)}h_E^b$ , and $^{(i-1)}h_F$."

A: Thanks for your comment.

• For question (a):

  • We have modified Fig. 2 to provide a better explanation of the fusion of features at different frequencies. Please refer to Fig. 2.

• For question (b):

  • The event branch and frame branch output features from four different scales. At each scale, the inputs to the FlexFuser are:
    • Event features: $^{(i)}h_{E}^a​$ and $^{(i)}h_{E}^b$​;
    • Frame feature: $^{(i)}h_{F}$​;
    • Fused feature from the previous scale: $^{(i-1)}h_{\text{fuse}}$.​
  • The fusion process is as follows:
    • First, the event features under different frequencies are fused with frame features to obtain $^{(i)}h_{\text{fuse}}^a$ and $^{(i)}h_{\text{fuse}}^b$​;
    • Then, the fused features from different frequencies are combined, i.e., $^{(i)}h_{\text{fuse}} = ^{(i)}h_{\text{fuse}}^a + ^{(i)}h_{\text{fuse}}^b$;
    • After processing all scales, the multi-scale fused features $^{(i)}h_{\text{fuse}}$​ are concatenated and fed into the detection head.

For better clarity, we have further enhanced the elaboration. Please refer to Sec. 3.2 and Fig. 3.

Q2: "Sec. 3.3: the formula related to pre-training with low-frequency labels lacks numbering, which may confuse readers."

A: Thanks for your suggestion. The equation in Sec. 3.3 is intended as an inline equation that does not require numbering. However, for better clarity, we have modified this equation to a standalone equation with the number. Please refer to Eq. 8.

Q3: "Sec. 4.2: the evaluation methods across the three datasets (DSEC-Det, DSEC-Detection, DSEC-MOD) differ."

A: Thanks for your question. We provide explanations on the use of datasets, methods, and experimental settings as follows:

• How we chose the three datasets:
  • DSEC is a high-resolution, large-scale multimodal dataset specifically designed for event-based perception. For our experiments, we use three comprehensive versions of DSEC, i.e., DSEC-Det, DSEC-Detection, and DSEC-MOD, which are tailored for the object detection task. These versions provide a robust platform for benchmarking and evaluating event-based object detection methods, covering a wide range of scenarios, object categories, and environmental conditions.
  • Among them, DSEC-Det is the most recent, largest, and most comprehensive one, annotated and released by the original DSEC authors. Thus, we prioritized it as the primary benchmark for reporting results, ensuring relevance and reliability. DSEC-Detection and DSEC-MOD are datasets used by two recent event-frame fusion methods CAFR and RENet, so we also report results on these two datastes to validate our method’s effectiveness.
• How we chose the methods:
  • To evaluate the effectiveness of our method, we compared both event-only and event-frame fusion methods.
  • For event-only methods:
    • We included state-of-the-art detectors (RVT, SAST, LEOD, and SSM), which were originally trained on event-only datasets like Gen1 and 1Mpx.
    • To ensure fairness, we retrained these methods on DSEC-Det. Table 1 demonstrates that our method outperforms these detectors.
  • For event-frame fusion methods:
    • For DSEC-Det, we compared with DAGr, as it has been evaluated on DSEC-Det. We report the scores of DAGr from the original paper to ensure fairness.
    • For DSEC-Detection and DSEC-MOD, we trained our model by following the standard training and evaluation settings, and compared again the state-of-the-art methods, CAFR and RENet, in Table 2 and Table 3. For other methods on DSEC-Detection and DSEC-MOD, we reference the results from the CAFR paper and the RENet paper, respectively.
    • Since DAGr's training code is not publicly available, we are not able to reproduce DAGr on DSEC-Detection and DSEC-MOD. We have been contacting the authors about this but haven't gotten an update.
• These comparisons ensure a fair and comprehensive evaluation while adhering to resource and code availability constraints.

Q4: "Inconsistency in training iterations from Sec. 4.1 and Appendix Sec. A.3."

A: Thanks for your comment. The training iteration count is 100,000, as stated in Sec. 4.1. We have revised this inconsistency in Appendix Sec. A.3 to ensure correctness. Please take a look.

Last but not least, we sincerely thank Reviewer FU5g again for the valuable comments provided during this review.

Dear Reviewer FU5g,

We sincerely appreciate your time and effort in reviewing our manuscript and for providing insightful questions for discussion.

We hope that our previous replies have adequately addressed your concerns.

We look forward to participating in the Author-Reviewer Discussion session and welcome any additional feedback you may have on the manuscript or our reply.

Once again, thank you for your valuable contributions to improving the quality of our work.

Best regards,

The Authors of Submission 589

Dear Reviewer FU5g,

We sincerely appreciate your time and effort in reviewing our manuscript and for providing insightful questions for discussion.

We hope that our previous replies have adequately addressed your concerns.

We look forward to participating in the Author-Reviewer Discussion session and welcome any additional feedback you may have on the manuscript or our reply.

Once again, thank you for your valuable contributions to improving the quality of our work.

Best regards,

The Authors of Submission 589

Dear Reviewer FU5g,

We thank you for your time and effort in reviewing our manuscript. We have made corresponding revisions based on your insightful comments.

The Author-Reviewer Discussion session is near the end. We really cherish the opportunity to discuss with you and welcome any additional feedback you might have on the manuscript or the changes we have made.

Please let us know if there are additional aspects that we can look at to improve this work further.

Best regards,

The Authors of Submission 589

We sincerely thank Reviewer uLco for the valuable comments provided during this review.

We have revised the manuscript based on your suggestions, please refer to the pink-colored texts. The point-to-point responses are as follows:

Q1: "The authors claim to be the first to address arbitrary-frequency object detection using event cameras. This phrasing could be more precise. Some prior works have explored high-frequency inference by combining events and frames, such as CovGRU and SODFormer."

A: Thanks for your comment. We have refined the contribution claim in the Introduction section. Please refer to the pink-colored texts. Specifically:

• The FlexEvent framework is designed to tackle the challenging problem of event camera object detection at arbitrary frequencies.
• There is only a limited number of relevant works in the literature, making us one of the early attempts to tackle this interesting yet challenging problem.
• To solve this, we propose an adaptive event-frame fusion (FlexFurser) module and frequency-adaptive learning (FAL) mechanism.
• We demonstrate that our approach achieves state-of-the-art performance in event-based object detection across three large-scale datasets, particularly in high-frequency scenarios.

Here, we further clarify the differences between CovGRU and SODFormer:

• For CovGRU:
  • This work proposes Recurrent Asynchronous Multimodal (RAM) networks to generalize traditional RNNs for asynchronous and irregular data from multiple sensors.
  • However, this work primarily focuses on monocular depth estimation. In the experiments, the event data frequency is fixed at a standard 25 Hz, while the image frequency is varied between 5 Hz and 1 Hz. No experiments are presented to evaluate the impact of varying event data frequencies.
• For SODFormer:
  • This work employs an asynchronous attention-based fusion module that supports varying frequencies.
  • However, their primary experiments are conducted under normal frequencies, with only low-frequency quantitative results and high-frequency qualitative results presented in their ablation studies.
• In contrast to these two works, our work focuses explicitly on "arbitrary-frequency" scenarios, leveraging the temporally rich nature of event cameras. Our key contribution lies in benchmarking and validating our method across varying frequencies, showcasing its effectiveness compared to other state-of-the-art approaches.
• To ensure accurate claims, we have revised the manuscript to better reflect the differences between CovGRU, SODFormer, and FlexEvent. Please kindly refer to Lines 135-138.

Q2: "Table 1: could you clarify the number of classes used for this evaluation? Is it two classes or eight?"

A: Thanks for your question. We clarify this setup as follows:

• In our experiments on DSEC-Det, we trained and evaluated the model under the two-class setup, which is consistent with the DAGr paper and their code implementation.
• For other detectors that we reimplemented on DSEC-Det, we trained and evaluated them under the two-class setup to maintain consistency and fairness.

Q3: "Sec. 3.1: the background and motivation would be more appropriate in the Introduction or Related Work sections."

A: Thanks a lot for your feedback. Based on your suggestion, we have revised the manuscript. The background and motivation parts have been moved to the Introduction section, while Sec. 3.1 is primarily focusing on problem formulation and data processing. Please kindly refer to the pink-colored texts in Introduction and Sec. 3.1.

Last but not least, we sincerely thank Reviewer uLco again for the valuable comments provided during this review.