PPLNs: Parametric Piecewise Linear Networks for Event-Based Temporal Modeling and Beyond

Dear Reviewer,

We appreciate the discussion with you previously in our submission to another venue. We have answered the questions you listed earlier but did not receive your response. May we ask whether you think our rebuttal has clarified your concerns? If so, we kindly encourage you to consider raising the rating. Thanks!

Comment: Could you give the time complexity of PPLNs, or the inference time on the three downstream tasks, such as steering prediction, human pose estimation, and motion deblurring?

Response: Thank you for your comment! We appreciate the opportunity to provide additional information. Below is a table comparing various performance indicators between our PPLN approach and baseline methods for different tasks. All timing measurements were conducted on the same Titan V server in our institution. For motion deblurring, the baseline approach is U-Net, which has an identical network architecture to PPLNs constructed from regular convolutional layers. For steering prediction and human pose estimation, the baseline approaches are "Calabrese" and "Hu", respectively.

Importantly, we emphasize that in steering prediction and human pose estimation, PPLNs output ten times as much information as the baseline, which contributes to their slower performance in these two tasks.

Comment: Which hyperparameters have the greatest influence on LLPNs? Please explain the analysis from an experimental perspective.

Response: Thank you for your comment! We appreciate the opportunity to address this question from an experimental perspective. Our analysis indicates that the integral normalization constant has the greatest influence on PPLNs, as demonstrated in Table 5. Particularly in motion deblurring tasks, where the ground-truth constant value is available, the MSE improvement attributable to integral normalization can be as high as 42.4%. In tasks such as steering prediction and human pose estimation, where the constant value must be predicted, integral normalization still enhances performance but to a lesser extent. This result demonstrates the importance of using a good value as the integral normalization constant.

Furthermore, our experiments reveal that the number of line segments per layer also impacts prediction quality, as evidenced by Table 7 in the paper. We recommend utilizing three segments per layer as it strikes a balance between quality and parameter count, resulting in optimal performance.

Sincerely,

Authors

Dear Reviewer,

Here are the responses to the concerns in your review. We look forward to your additional input during the discussion period. Meanwhile, we encourage you to consider raising the rating if you deem it appropriate.

Comment: The paper lacks a thorough analysis of the computational costs associated with PPLNs. Without this information, it's difficult to assess the practical viability of the approach, especially in comparison to existing methods.

Question: What is the exact computational overhead of PPLNs compared to traditional neural network layers?

Response: Thank you for your comment and question! We analyze the computational cost of PPLNs and the baseline approaches in Section A.8 of the supplementary material. For your convenience, below is a table comparing various performance indicators between our PPLN approach and baseline methods for different tasks. All timing measurements were conducted on the same Titan V server in our institution. For motion deblurring, the baseline approach is U-Net, which has an identical network architecture to PPLNs constructed from regular convolutional layers. For steering prediction and human pose estimation, the baseline approaches are "Calabrese" and "Hu", respectively.

Importantly, we emphasize that in steering prediction and human pose estimation, PPLNs output ten times as much information as the baseline, which contributes to their slower performance in these two tasks.

Comment: While the authors claim biological inspiration, the paper does not adequately explore how closely PPLNs actually mimic neuronal behavior. The connection to biological systems seems superficial and not well-substantiated.

Question: Can you provide more evidence to support the biological plausibility of PPLNs beyond the initial inspiration?

Response: Thank you for your comment and question! In Section A.4 of the supplementary, we discuss how PPLNs are connected with the biological neuromorphic mechanism in detail. In particular, we highlight that PPLNs focus on an explicit parameterization of the membrane potential, which is underexplored in existing work. While they do not follow all neural principles in biology, we hope you agree that PPLNs are closer to real neural systems than conventional artificial neural networks.

Comment: The paper's claims about PPLNs' applicability to conventional frame-based tasks are not adequately supported. The limited experiments in this domain do not provide convincing evidence of the method's broader applicability beyond event-based vision.

Response: Thank you for your comment! We agree with the reviewer that the experiments in the paper do not justify the use of PPLNs in conventional frame-based tasks. In fact, for most conventional frame-based tasks, we would advocate against the use of PPLNs. As discussed in the paragraph starting at line 195, most conventional frame-based tasks, such as object detection and segmentation, are associated with sufficient high-quality training data. PPLNs are most applicable to scenarios when the training data is limited, causing modeling to play a more important role. Such scenarios include most tasks in event-based vision, as well as very niche situations in conventional frame-based vision.

Comment: The paper fails to provide a detailed analysis of how sensitive PPLNs are to hyperparameter choices, such as the number of line segments in the parameterization. This omission raises questions about the robustness and reproducibility of the results.

Question: Can you provide a detailed sensitivity analysis for the key hyperparameters of PPLNs?

Response: Thank you for your comment and question! We appreciate the opportunity to clarify. Our analysis indicates that the integral normalization constant has the greatest influence on PPLNs, as demonstrated in Table 3. Particularly in motion deblurring tasks, where the ground-truth constant value is available, the MSE improvement attributable to integral normalization can be as high as 42.4%. In tasks such as steering prediction and human pose estimation, where the constant value must be predicted, integral normalization still enhances performance but to a lesser extent. This result demonstrates the importance of using a good value as the integral normalization constant.

Furthermore, our experiments reveal that the number of line segments per layer also impacts prediction quality, as evidenced by Table 7 in the supplementary material. We recommend utilizing three segments per layer as it strikes a balance between quality and parameter count, resulting in optimal performance.

Comment: However, a more comprehensive discussion of potential negative societal impacts and failure cases would be beneficial.

Response: Thanks for your comment! We fail to observe a noticeable improvement when having sufficient high-quality training data, which is rarely the case in event-based applications. However, this impedes PPLNs from being applied widely to conventional vision tasks. A promising direction is to integrate PPLNs into Parameter-Efficient Fine-Tuning (PEFT) techniques to fine-tune model parameters on small datasets.

Sincerely,

Authors

Dear Reviewer,

Here are the responses to the concerns in your review. We look forward to your additional input during the reviewer-author discussion period. While we are grateful for the weak accept vote, we encourage you to consider raising the rating if you deem it appropriate.

Comment: The authors did not compare with some of the latest event-based deblurring algorithms, such as REFID[1].

Response: Thank you for your comment! We recognize that REFID is an important work in event vision. However, REFID takes two blurry frames as input and performs frame interpolation. The design of REFID relies on the fact that there are two conventional frames available as input, which deviates from our setting that assumes only one input frame. We will make sure to discuss REFID and why it is not included as a baseline approach in our revision.

Comment: In addition, the authors should provide a direct comparative experiment using a KAN-based backbone to further demonstrate the superiority of the proposed solution.

Response: Thank you for your comment! Unfortunately, there are two reasons why we are unable to provide such a direct comparative experiment. First, KANs do not have temporal modeling, and there is not yet consensus on how KANs can be modified to support temporal modeling. Additionally, KANs are very slow (c.f., page 33 of 2404.19756). This is why KANs are experimented on very simple toy datasets. Sadly, we do not have access to the computational resources to evaluate KANs on larger-scale applications discussed in our submission.

Comment: The piecewise linear nature of this paper is akin to an input-dependent activation function. An SNN using multi-layer LIF neurons with learnable parameters also seems to achieve a similar level of biological plausibility. Could the authors provide further explanation from this perspective?

Response: Thank you for your comment! We agree that the proposed PPLN is conceptually very similar to the SNN. However, we highlight four unique characteristics of PPLNs. First, PPLN focuses on representing the membrane potentials instead of the interconnection between artificial neurons. Second, PPLN models the membrane potential using a real value instead of explicit binary spikes. Third, PPLN does not restrict the sign of the slope. Lastly, PPLN does not explicitly enforce any line segment to be flat. For more details, we encourage you to check out the discussion in Section A.4 of the supplementary material.

Comment: Based on Figure 2d, I am still unclear whether the input t for PPLN is an external query (you can choose your desire timestamp t) or an inherent timestamp of the event. If it is the former, does this mean that all neurons query with the same input t? If it is the latter, how is this implemented? Given that there are many input events with numerous timestamps, how does the method handle this?

Response: Thank you for your comment! We appreciate the opportunity to clarify. The model input consists of two parts, the non-temporal component $\mathbf{x}$, and the temporal component $t$. For each $\mathbf{x}$, we typically hope to make inferences at multiple timestamp $t$'s. For example, we are interested in the sharp frames at different timestamps $t \in [0, 1]$ that correspond to the input blurry image and events ($\mathbf{x}$). From this perspective, the input $t$ is an external query, and all the neurons receive the same $t$ value.

Comment: A good response to the Weaknesses and Questions will improve my initial rating.

Response: We sincerely appreciate your willingness to improve the rating! Please let us know if our response has addressed your concerns. We look forward to discussing the submission more with you.

Sincerely,

Authors