Rebuttal to Reviewer 7nZC

We sincerely value the reviewer's recognition of our work's contributions, particularly regarding:

• Innovative Method: Introduces a PEAS module for adaptive event frame selection and an MSG loss to guide this process.
• Technical Impact: Outperforms SOTA by +8.31% (HARDVS) and handles highly variable event lengths with generalization to different sampling frequencies.
• New Benchmarks: Introduces 3 datasets for broader evaluation of long and realistic event sequences.
• Clear Presentation: well-written, easy-to-read, well-motivated, and technically sound.

Additionally, we greatly appreciate the reviewer's thoughtful assessment and valuable suggestions for improving our work. Below we address each concern with concrete revisions:

Q1. PEAS Module Interpretability

• Added in revision: Currently, the qualitative analysis of the PEAS module was presented in the supplementary Figure 6 due to page limitations. Considering that this part is very important for your valuable suggestion, we will move this part to the main text in the final version.
• Explanation for PEAS frame selection: At epoch 0, the PEAS module randomly selects event frames, resulting in unnecessarily padded frames and redundant event frames with repetitive information. After 100 epochs, the eight chosen frames exclude redundant frames and non-informative padding, resulting in selected frames that are semantically meaningful.
• Key findings: This visualization result demonstrates the effectiveness of the PEAS module and the MSG loss. We can conclude that the MSG loss can effectively guide the PEAS module to select the most representative event frames, thus minimizing the randomness and redundancy of the encoded features during the PEAS selection process.

Q2. Temporal coherence of the PEAS selection

• Selection randomness: We admit the PEAS module can introduce some randomness for the frame selection at the onset of training due to the random weight initialization of the PEAS module. Consequently, during the start of training, the model can only optimize performance based on the distribution of these randomly selected frames, rather than improving the PEAS module for adaptive selection of input events.
• Solution: MSG loss to facilitate effective optimization.
  • WEIE Loss: To reduce the randomness of the selection process to ensure the selected sequence features can encapsulate the entirety of the sequence.
  • IEMI Loss: To guarantee that each selected event feature stands out from the others.
  • Both of the two losses are motivated by the information entropy and mutual information from the information theory so as to guide the PEAS module to select the most representative event frames without the loss of temporal coherence.
• Evidence:
  • The ablation study in Table 4 (main paper), 'PEAS + $L_{MSG}$,' proves the effectiveness of the MSG loss, which achieves 94.83% and 93.85% accuracy gain on the PAF and ArDVS100 datasets, respectively.
  • The qualitative analysis of the PEAS module in Figure 6 (Supplementary) also proves the temporal coherence of the selection event frames.

Q3. Ablation Study with Random Sampling + MSG Loss

We thank the reviewer for this valuable suggestion. We have conducted the additional ablation study comparing random sampling with learned selection (PEAS), both with and without MSG loss. The updated results in the following Table 4 (main paper) reveal several key insights:

Table 4: Ablation study for the PEAS module with additional 'Random Sampling + L_MSG' experiment result. We will update Table 4 (main paper) in the final version.

Key Findings:

• 1. MSG Loss Requires Learnable Selection:
  • Random Sampling + MSG performs worse (92.64%/91.89%) than baseline random sampling (92.98%/92.23%)
  • This confirms MSG loss cannot meaningfully guide completely random selection
• 2. PEAS Enables Effective Optimization:
  • PEAS alone outperforms random sampling (93.33%/92.84% vs 92.98%/92.23%)
  • PEAS + MSG achieves best results (94.83%/93.85%), demonstrating the importance of learnable frame selection and also proves MSG loss's ability to optimize PEAS's selection process

Conclusion: The MSG loss provides meaningful guidance only when combined with a learnable selection mechanism (PEAS), as it requires gradient pathways for optimization.

This analysis has been added to Table 4 in the main paper and will be discussed in Section 4.3 of the revised manuscript.

Rebuttal to Reviewer Tnzz

We sincerely value the reviewer's recognition of our work's contributions, particularly regarding:

• Innovative Method: Introduces the PEAS module for adaptive event encoding, and the novel MSG loss minimizes feature redundancy.
• Technical Impact: First SSM adaptation for ultra-long event streams (up to 10⁹ events).
• SOTA Performance: Outperforms prior methods on multiple datasets.
• Dataset Contribution: Establishes new benchmarks for long-duration event recognition

Additionally, we greatly appreciate the reviewer's thoughtful assessment and valuable suggestions for improving our work. Below we address each concern with concrete revisions:

Q1. Baseline Scaling & Fair Comparison (Primary Concern)

We sincerely appreciate the reviewer's thoughtful feedback regarding baseline comparisons. Below, we provide a detailed justification for our scaling methodology and experimental design:

1. Scaling Methodology of PASS

Our PASS framework builds upon the VideoMamba SSM backbone [1], with the PEAS module (a lightweight two-layer 3D convolutional scorer) serving as the primary parameterized component. Since PEAS’s non-parametric operations (e.g., frame selection) introduce negligible parameters, the total model size is dominated by the SSM backbone. For fair comparisons, we adopt the standard ViT scaling protocol, producing Tiny (7M), Small (25M), and Middle (74M) variants by adjusting depth and embedding dimensions proportionally.

2. Challenges with Suggested Baseline Modifications

• Challenges with Suggested S5-ViT-M-K(16): The S5-ViT baseline [2] combines an S5 [3] SSM with a ViT for event-based object detection. While we faithfully reproduced their official implementation (detailed in supplementary line 655 - line 670), substituting their Base ViT with a Middle variant S5-ViT-M-K(16) would introduce an unfair comparison for two reasons:

  • Architectural Mismatch: S5-ViT is a hybrid SSM+ViT model, while PASS is purely SSM-based. Scaling the ViT component independently disrupts parameter parity (e.g., S5-ViT-M’s parameters diverge from PASS-M’s 74M).
  • No corresponding ViT-M model: There is no corresponding ViT-M model publicly available since the ViT paper only provides the Small (22M), Base (86M), Large (307M), and Huge (632M) versions; none of them match our PASS-M (17.5M) parameter count.

• Challenges with Suggested EventDance+SSM: EventDance [4] utilizes the convolution network (ResNet-18, 34, 50) combined with an EvFlowNet [5] as the flow estimation model and E2VID [6] as the reconstruction model. The whole architecture is designed for cross-modal adaptation from image to event modalities. Appending an SSM to this already complex pipeline would:

  • Architectural Conflict: EventDance uses ResNet+EvFlowNet [5]+E2VID [6] for cross-modal adaptation. SSM insertion would require complete pipeline redesign.
  • Skew Parameter Comparisons: ResNet-50 alone introduces ~25M parameters. The combined model size would dwarf our SSM-centric approach

3. Additional Comparative Analysis

Due to the above reasons, we tried our best to make a fair comparison with S5-ViT by adding the additional evaluation results for S5-ViT-B-K(2) on the N-Caltech101 datasets and S5-ViT-B-K(2) on the PAF, SeAct, and HARDVS datasets. The results are shown in the following table 1 and table 2:

Table 1: Comparison with previous methods for event-based object recognition with additional evaluation results for S5-ViT-B-K(2) on the N-Caltech101 datasets. We will update Table 1 (main paper) in the final version.

Key Insight:

PASS-T-K(2) achieves superior accuracy (+1.28%) with 2.5× fewer parameters (7M vs 17.5M) than S5-ViT-B-K(2) on the N-Caltech101 datasets.

Table 2: Comparison with previous methods for event-based action recognition with additional evaluation results for S5-ViT-B-K(16) on the PAF, SeAct, and HARDVS datasets. We will update Table 2 (main paper) in the final version.

Key Insight:

PASS-T-K(16) achieves 94.83% performance on the PAF datasets, which is better than the S5-ViT-B-K(16) performance (93.12%) with fewer parameters (7M vs. 17.5M). Besides, our PASS-M-K(16) achieves 98.37% performance on the HARDVS datasets, which is better than the S5-ViT-B-K(16) performance (93.12%) with fewer parameters (74M vs 17.5M). The above results show PASS's consistent performance improvement and better parameter efficiency when we enlarge the selected event frame number.

4. Broader Advantages of PASS

Beyond parameter efficiency, our method demonstrates:

• Fine-grained Temporal Recognition: 89.00% on the TemArDVS100 dataset (vs. 79.62% for S5-ViT).
• Real-World Robustness: 100% accuracy on the Real-DVS10 dataset.
• Inference Frequency Generalization: Max accuracy drop of -8.62% (vs. -20.69% baseline).
• Strong Long Spatiotemporal Modeling: Consistent performance across a broad distribution of event length (1-10^9).

These advantages stem from our novel PEAS module and MSG Loss, which enable effective spatiotemporal event modeling (see ablation studies in table 4 of the main paper). We will incorporate these additional results in the final version and welcome further discussion on evaluation protocols.

---

[1] Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., & Qiao, Y. (2024, September). Videomamba: State space model for efficient video understanding. In European conference on computer vision (pp. 237-255). Cham: Springer Nature Switzerland.

[2] Nikola Zubíc, Mathias Gehrig, and Davide Scaramuzza. State space models for event cameras. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5819–5828, 2024.

[3] Smith, J. T., Warrington, A., & Linderman, S. W. (2022). Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933.

[4] Zheng, X., & Wang, L. (2024). Eventdance: Unsupervised source-free cross-modal adaptation for event-based object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17448-17458).

[5] Zhu, A. Z., Yuan, L., Chaney, K., & Daniilidis, K. (2018). EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. arXiv preprint arXiv:1802.06898.

[6] Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D. (2019). Events-to-video: Bringing modern computer vision to event cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3857-3866).

Q2. Experiment Result Reresentation

We sincerely appreciate the reviewer’s valuable feedback regarding the presentation of our experimental results. We acknowledge that the organization and clarity of our results could be improved, and we propose the following revisions for the final version:

1. Restructured Results Section:

- We will reorganize Section 5.2 into clear subsections:
    - 5.2.1: Event-based recognition results
    - 5.2.2: Inference frequencies generalization results
    - 5.2.3: Event frame selection visualization results

2. Improved Table Presentations:

- All tables will be reformatted to include:
    - Clearer headers with unified terminology
    - Better grouping of comparable methods
    - Consistent precision in reported metrics (all accuracies to 2 decimal places)

3. Supplemental Analysis:

- We will include in the supplement:
    - Complete per-dataset breakdowns
    - Training curves
    - Additional ablation studies

We believe these changes will significantly improve the clarity and impact of our results presentation. We sincerely thank the reviewer for this constructive suggestion, which will undoubtedly strengthen our paper’s quality.

Rebuttal to Reviewer ot6Q

We sincerely value the reviewer's recognition of our work's contributions, particularly regarding:

• Novel Technical Contribution:
  • Introduces PASS framework with two key innovations:
    • PEAS module: Adaptive event aggregation with path selection
    • MSG loss: Reduces feature redundancy through end-to-end optimization
• Comprehensive Empirical Validation:
  • Outperforms SOTA across 8 datasets
  • Demonstrates superior generalization:
    • Across event lengths (1-10^9 events)
    • Under varying inference sampling frequencies
  • Introduces valuable new benchmarks:
    • ArDVS100/TemArDVS100 for long-sequence analysis
    • Real-ArDVS10 for real-world applicability
• Presentation Quality
  • Clear and well-organized manuscript
  • Thorough ablation studies and analysis

We sincerely appreciate the reviewer’s thoughtful critique, which has helped us improve the manuscript. Below, we address each weakness and question with concrete revisions:

Q1. Component Ablation & Prior Work Discussion

Concern:

• Needs clearer ablation of PEAS vs. MSG contributions
• Additional discussion of frequency-adaptive prior work ([R1], [R2], [R4])

Response:

Table 1: Ablation study for the PEAS module with additional 'Random Sampling + $L_{MSG}$ ' experiment result. We will update the corresponding table 4 of the main paper in the final version.

• Added new ablation studies in Table 1, showing:
  • MSG Loss Requires Learnable Selection:
    • Random Sampling + $L_{MSG}$ performs worse (92.64%/91.89%) than baseline random sampling (92.98%/92.23%)
    • This confirms MSG loss cannot meaningfully guide completely random selection
  • PEAS Enables Effective Optimization:
    • PEAS alone outperforms random sampling (93.33%/92.84% vs 92.98%/92.23%)
    • PEAS + MSG achieves best results (94.83%/93.85%), demonstrating the importance of learnable frame selection and also proves MSG loss's ability to optimize PEAS's selection process
  • Conclusion: The MSG loss provides meaningful guidance only when combined with a learnable selection mechanism (PEAS), as it requires gradient pathways for optimization.

Table 2: The differences and similarities for evaluation settings of the inference frequencies generalization among the existing works [R1], [R2], and [R4]. We will add this table in the supplementary in the final version.

• Added discussion in Table 2, showing:
  • The existing works [R1], [R2], and [R4] only evaluate the inference frequencies generalization on the 20 Hz training frequency with inference frequencies ranging from 20 Hz to 200 Hz with different intervals.
  • Our PASS model evaluates the inference frequencies generalization on the 20 Hz, 60 Hz, and 100 Hz training frequencies and the inference on the 40 Hz, 60 Hz, 80 Hz, 100 Hz, and 200 Hz inference frequencies.
  • Our evaluation settings are more comprehensive; they not only investigate the inference frequency generalization on a model trained with 20 Hz and applied to 40 to 200 Hz, but also investigate models trained with 60 Hz and 100 Hz and applied to 20 to 200 Hz and 20 Hz to 80 Hz, respectively.

Q2. Interpretability analyses for the PEAS module

Concern:

• Needs clearer visualization of PEAS’s frame selection

Response:

• Added in revision: Currently, the qualitative analysis of the PEAS module was presented in the supplementary Figure 6 due to page limitations. Considering that this part is very important for your valuable suggestion, we will move this part to the main text in the final version.
• Explanation for PEAS frame selection: At epoch 0, the PEAS module randomly selects event frames, resulting in unnecessarily padded frames and redundant event frames with repetitive information. After 100 epochs, the eight chosen frames exclude redundant frames and non-informative padding, resulting in selected frames that are semantically meaningful.
• Key findings: This visualization result demonstrates the effectiveness of the PEAS module and the MSG loss. We can conclude that the MSG loss can effectively guide the PEAS module to select the most representative event frames, thus minimizing the randomness and redundancy of the encoded features during the PEAS selection process.

Q3. Computational complexity analyses

Reviewer Concern:

• Needs runtime/memory analysis

Response:

• Added detailed benchmarks (Table 3):
  • As suggested, we provide additional detailed computational resource requirements on one A100 GPU with 80GB memory.
  • As shown in Table 3, when our PASS model is applied in the real-world scenario, we can select different model versions based on the application requirements. For example:
    • If the application requires high accuracy, we can select the Middle model with 74M parameters and 12.7G FLOPS and 24.7 FPS.
    • If the application requires low latency, we can select the Tiny model with 7M parameters, 1.1G FLOPS, and 243.9 FPS.

Table 3: The computational complexity and memory consumption of the PASS model. We will update the corresponding table 7 in the supplementary in the final version.

Additional questions (Minor)

Q4. Datasets coverage

• We have tried our best to evaluate our PASS model across a wide range of event lengths and event frequencies, covering five publicly available event datasets: PAF, SeAct, HARDVS, N-ImageNet, and N-Caltech101.
• Given the existing datasets only provide events within a limited distribution of event length (10^6 for objects and 10^7 for actions). We introduce the ArDVS100 and TemArDVS across a broad distribution of event length (10^6 to 10^9), synthesized by concatenating event streams with varying meta actions, thus capturing action transitions over time.
• Additionally, to assess the model’s real-world applicability, we created a real-world dataset, named Real-ArDVS10, comprising event-based actions lasting from 2s to 75s, encompassing 10 distinct classes selected from the ArDVS100 datasets.

Q5. Discussion on potential scalability issues

PASS demonstrates strong scalability through PEAS adaptively encoding (handling 10^9 events) and MSG loss to guide the PEAS module to select the most representative event frames. While larger models (74M params) show diminishing returns, we propose three potential improvements:

• Model compression via quantization like INT8, INT4, etc.
• Dynamic K-selection adapting to event density.
• Hardware optimizations like CUDA kernel fusion.

Q6. Modification for readability

We appreciate this helpful suggestion for improving readability. In the revised manuscript, we will clearly define all acronyms upon first use:

• PEAS: Path-selective Event Aggregation and Scan
• MSG: Multi-faceted Selection Guiding loss
• SSM: State Space Model
• WEIE: Within-Frame Event Information Entropy
• IEMI: Inter-Frame Event Mutual Information

Q7. Discussions on tasks beyond recognition

We appreciate this valuable suggestion. In future work, we will expand discussions on broader applications:

• Object Detection: PASS's adaptive frame selection (PEAS) could enhance [R3]'s event transformers by improving long-term temporal modeling and complement [R4]'s SSM detection framework through more efficient event representation.
• Semantic Segmentation: The MSG loss's redundancy reduction may benefit [R5]'s feature learning, while PEAS could optimize [R6]'s open-vocabulary pipeline by filtering noisy events.

Q8. Hyperparameter K tuning

We appreciate this valuable suggestion. In future work, we will expand discussions on dynamic hyperparameter K tuning adapting to:

• event length
• event density

Q9. Strategies for mitigating the overfitting issue of large variants of SMM model

We propose three potential strategies to address overfitting in our large variants of the SMM model:

• Stochastic Path Dropout: Randomly masking 20% of PEAS paths during training to reduces overfitting
• Temporal Augmentation: Event frame shuffling improves generalization
• Regularized MSG Loss: Added L2 penalty on frame selection scores (λ=0.1)

Rebuttal to Reviewer oNQK

We sincerely appreciate the reviewer's recognition of our work's contributions, particularly regarding:

• Novel Method: Our proposed PEAS effectively encode variable-length event streams into fixed representations, achieving superior superior performance against the existing baselines.
• SSM Optimization: Addresses SSMs' forgetting problem by shortening sequences while preserving critical information.
• Innovative Loss: Introduces MSG Loss to optimize frame selection, validated through ablations.
• Strong Experiments: Demonstrates superior performance across multiple datasets with thorough analysis.
• Generalization across Inference Frequencies: Demonstrates PASS's superior performance in handling varying inference frequencies.

We also sincerely appreciate the reviewer’s insightful feedback and constructive critiques. Below, we address each concern in detail, with additional clarifications and experimental support.

Question 1: SSM Utilization of PEAS-Generated Frames.

Concern:

The reviewer asks whether the SSM backbone effectively utilizes the fixed-length event frames or disproportionately favors later information.

Response:

We acknowledge the reviewer’s concern about potential biases in SSM’s utilization of event frames. Due to SSM’s recurrent nature, its hidden state updates can be influenced by input sequence length and feature order, particularly for long-range dependencies. To mitigate this, our PEAS module encodes events into fixed-length representations, ensuring balanced utilization of temporal information.

Key evidence supporting this:

• Ablation Study (Table 4 main paper): The additional baseline 'No Sampling' (using all event frames) achieves lower accuracy (92.90% on PAF, 92.31% on ArDVS100) compared to PEAS (93.33%, 92.84%), indicating that raw long sequences harm performance due to SSM’s limitations.

• Comparison to Random Sampling: 'PEAS' outperforms 'Random Sampling' (+0.35% on PAF, +0.61% on ArDVS100), demonstrating its ability to preserve critical information.

Table 4: Ablation study for the PEAS module with additional 'No Sampling' experiment result. We will update the Table 4 (main paper) in the final version.

Question 2: Evaluating Information Loss in PEAS.

Concern:

The reviewer raises concerns about quantifying information loss and its acceptability in different scenarios.

Response:

We admit the PEAS module can introduce some randomness for the frame selection at the onset of training due to the random weight initialization of the PEAS module. Consequently, during training, the model can only optimize performance based on these randomly selected frames, rather than improving the PEAS module for adaptive selection of input events. Howerver, evaluating information loss is challenging since there is no universally accepted metric to define and measure it. In this paper, we addressed it through indirect and qualitative analyses.

• Quantitative Evaluation (Table 4 in main paper):

  • 'PEAS' improves accuracy over 'No Sampling' (+0.43% on PAF, +0.53% on ArDVS100), suggesting that the selected frames retain task-relevant information despite compression.
  • 'PEAS' boosts accuracy compared with 'Random Sampling' (+0.35% on PAF, +0.61% on ArDVS100), demonstrating its ability to preserve critical information.
  • With MSG Loss ('PEAS + $L_{MSG}$'), performance further improves to 94.83% (PAF) and 93.85% (ArDVS100), validating that $L_{MSG}$ mitigates randomness in frame selection. The MSG loss consists of two parts: (1) WEIE Loss is designed to reduce the randomness of the selection process to ensure the selected sequence features can encapsulate the entirety of the sequence, and (2) IEMI Loss is designed to guarantee that each selected event feature stands out from the others. They are motivated by the information entropy and mutual information from the information theory so as to guide the PEAS module to select the most representative event frames without introducing information loss.

• Qualitative Analysis (Figure 6 in supplementary): It demonstrates that the PEAS module can select the most representative event frames that are more informative and discriminative for the event recognition task after 100 training epochs. Currently, this part is presented in the supplementary material due to page limitations. Considering that this part is very important and has been mentioned by multiple reviewers, we will move this part to the main text in the final version.

• Task Suitability:

  • We can adjust the hyperparameter K, which is the selected number of event frames, to adapt to the different applications.
  • Fast-moving scenarios: Larger K may be needed to capture rapid changes as the event camera triggers more events and the resulting event frames differ a lot.
  • Slow-moving scenarios: Smaller K may be enough to capture the main information of the event stream as the resulting event stream is dense.

Question 3: Discussion on Improving the Spatiotemporal Event Modeling.

Concern:

The reviewer asks for thoughts on enhancing the event modeling module.

Response:

The improvement of the spatiotemporal modeling of event data can be discussed from two aspects: event representation and network structure.

• Event Representation (Sec. 3.1):
  • Currently, there is no commonly used event representation for event data. Event representation varies from task to task and adapts to different network architecture.
  • In this paper, we use the event frame as the event representation. Ablations in Table 6 (main paper) show that the event frames are optimal for the spatiotemporal modeling of event data using SSM.
• Network Structure (Sec. 3.2):
  • Different network architectures are tailored to specific event representations for spatiotemporal modeling. For instance, event point clouds and graph representations are compatible only with specialized networks, such as point transformers or graph neural networks.
  • In this work, we address the challenge of processing long event streams, where the number of aggregated event frames varies significantly due to the high temporal resolution of events. This variability complicates spatiotemporal modeling.
  • Therefore, we propose the PEAS module to select the most representative event frames. followed by SSM to model the spatiotemporal information based on the selected event frames. The PEAS can be viewed as a dynamic path selection module, which can adapt to the different event stream lengths. As shown in Table 3 (main paper), our approach achieves robust performance across a wide event distribution (10^6 to 10^9 events), attaining 97.35% on ArDVS100. Moreover, it demonstrates fine-grained temporal recognition, scoring 89.00% on the challenging TemArDVS100 dataset.

Question 4: Applicability to Other Tasks.

Concern:

The reviewer questions whether PASS generalizes to complex tasks.

Response:

We appreciate the suggestion to extend our method to other tasks. However, due to time constraints, we are unable to conduct comprehensive experiments to evaluate its performance on additional tasks at this stage. We plan to extend our method to other tasks like event-based object detection and event-based action recognition in future work.

We would also like to clarify that event recognition remains a challenging task, as evidenced by current benchmark results. For instance:

• On the N-Imagenet object recognition task, state-of-the-art methods achieve only 40-60% accuracy (our method reaches 61.32%, as shown in Table 1 in the main paper), significantly lower than the ~90% accuracy typical for standard ImageNet classification.

• On the SeAct action recognition dataset, existing methods attain 50-60% accuracy (our method achieves 66.38%, Table 2 in the main paper).

• For long-length event streams (~10^9 events), baseline methods achieve 79.62% accuracy on the TemArDVS100 dataset with fine temporal labels, while our method reaches 89.00% (Table 3 in the main paper).

These results demonstrate that while event recognition performance has saturated on small-scale datasets, substantial improvements are still needed for large-scale or temporally complex scenarios.