TDFormer: Top-Down Attention-Controlled Spiking Transformer

We sincerely thank the reviewer for the detailed and constructive feedback. We address each of the raised concerns below:

---

Question 1: Evaluation on more advanced models like Spikformer V2

Answer: We appreciate the reviewer’s suggestion regarding the use of more advanced models for evaluation. However, we would like to clarify a misunderstanding:

1. Qkformer was the most advanced model available at the time of our submission (excluding Spike-driven V3). QKFormer significantly outperforms Spikformer V2 in top-1 accuracy on ImageNet at 224 resolution (e.g., 84.22% vs. 80.38%), as reported in the original paper. The reason we did not evaluate on Spike-driven V3 is that its integer-based training removes the temporal dimension from the network dynamics, making our method inapplicable.

2. Qkformer is also a more recent and peer-reviewed work, published in October 2024, whereas Spikformer V2 remains a preprint uploaded in January 2024.

Therefore, we believe that the significant improvements demonstrated on Qkformer, one of the most advanced models at the time, across multiple datasets, especially achieving a state-of-the-art accuracy of 86.83% on ImageNet, provide strong evidence of the effectiveness of our method.

---

Question 2: Concern about the method’s applicability under resource constraints due to the power-accuracy trade-off

Answer: While our method slightly increases the number of parameters and power consumption, it achieves a significantly better performance-to-power ratio, which demonstrates its effectiveness for energy-constrained scenarios.

1. In Table 1, our lightweight variant increases power consumption from 38.91 to 38.93 (less than a 1% increase), and parameters from 64.96M to 65.55M (a 0.91% increase). However, accuracy improves from 84.22% to 85.37%, even surpassing Qkformer at 288 resolution (85.37% vs. 85.20%). In addition, our power usage is still significantly lower than Qkformer (39.93mJ vs. 64.39mJ), representing a 38.0% reduction.

2. At 224 resolution, our model achieves 85.57% accuracy, which is comparable to Qkformer at 384 resolution (85.57% vs. 85.65%), while consuming much less power (39.10mJ vs. 113.64mJ), resulting in a 65.6% reduction in energy cost.

In addition, we conducted ablation experiments based on Spikformer to further validate that the performance improvement comes from our proposed method rather than from a trade-off between accuracy and increased power or parameters.

When applying our method to Spikformer-2-384, we achieved better performance than the baseline Spikformer-4-384 (95.02% vs. 94.73%) while reducing parameters by 33% (6.21M vs. 9.33M). In addition, Spikformer-3-384 (TDAC) further improved performance to 95.14%, with a 14.36% reduction in parameters (7.99M vs. 9.33M).

These results clearly demonstrate that our method achieves a better performance-to-energy ratio, making it highly suitable for energy-constrained deployment scenarios. All results are averaged over five random seeds for fairness.

Table 4: TDAC achieves higher accuracy with fewer parameters on Spikformer

---

Question 3: Limited improvement over competitive baselines.

Answer: When applying our method to Qkformer, as shown in Table 1, it achieves a new state-of-the-art accuracy of 86.83% using only 69.09M parameters and 113.79 mJ of energy. Furthermore, we consistently observe improvements of over 1% on ImageNet across multiple input resolutions. On CIFAR10-DVS, our method achieves a significant gain of 1.83% at 16 simulation steps (85.83% vs. 84.00%). These results clearly demonstrate that our gains are not "only limited improvements," but instead represent meaningful and practical advances over the competitive baselines.

---

During the discussion period, we conducted additional experiments to address your concerns, as summarized below:

---

1. Marginal Gains on CIFAR10-DVS

To address your concern regarding the marginal improvement on the CIFAR10-DVS dataset, we conducted further experiments. The results are as follows:

Table 5: Additional Experiments on CIFAR10-DVS

Numbers with ∗ denote our implementation, representing results that differ from those reported in the original papers. These results demonstrate that the performance improvement of our method on CIFAR10-DVS is not merely marginal.

---

2. Inference time Evaluation on ImageNet

To better evaluate the training and inference efficiency of our method, we conducted inference-time measurements on the largest QKFormer model (HST-10-768). The results are shown in the table below:

The number of parameters is expressed in millions (M), energy consumption in millijoules (mJ), and training time is measured on a single batch with a batch size of 60. Despite the negligible overhead in energy consumption (+0.05%), parameter count (+0.91%), and inference time (+3.17%), our model achieves a significant accuracy improvement, highlighting its superior performance–efficiency trade-off.

---

3. Can the proposed method help accelerate the training process of SNNs?

Our proposed method does not accelerate the training process of SNNs. Due to the introduction of feedback connections across timesteps, it inevitably incurs a slight increase in energy consumption and parameter count, which leads to longer training times—as shown in Table 6. This overhead is mainly observed under the standard PyTorch framework, where sequential execution limits temporal parallelism.

---

4. Are there any ways to accelerate the proposed method?

While our method introduces feedback connections across timesteps—where outputs from previous steps are used to influence subsequent computations—this does not fundamentally imply a proportional increase in training time. The current implementation, based on PyTorch and SpikingJelly, any form of temporal dependency inherently incurs latency, since later timesteps must wait for the completion of earlier ones. However, this is a limitation of the high-level framework's execution model, not of our method itself.

In theory, this sequential bottleneck can be overcome through a custom CUDA implementation. For example, intermediate results from earlier timesteps can be buffered and accessed via shared memory or device-level parallelism, allowing multiple timesteps to be computed in an overlapped manner. Moreover, since our feedback mechanism is relatively lightweight—only requiring simple operations like accumulation or gating—it is well-suited to efficient hardware acceleration. Therefore, while the current PyTorch-based implementation does not realize this potential due to framework constraints, the proposed method could theoretically achieve comparable training speed to non-feedback SNNs when implemented with custom low-level CUDA kernels optimized for asynchronous execution.

While I acknowledge the authors’ additional experiments and clarifications, my main concerns remain unaddressed.

1. Marginal Gains & Reproducibility Concerns The reported accuracy improvements on CIFAR10-DVS (e.g., +1.83% for QKFormer) are small and could fall within the margin of reproduction error, particularly given the variability seen in event-based datasets. The added experiments do not convincingly demonstrate that the gains are statistically significant or robust across settings.

2. Increased Overhead Without Speed Benefit The authors confirm that the proposed TDFormer does not accelerate SNN training; in fact, it increases training time (+3.17%) and parameter count (+0.91%), and introduces additional energy consumption, albeit small (+0.05%). In my view, the lack of training speed improvement combined with added overhead undermines the practical benefit.

3. Lack of Novelty The core mechanism is a direct adaptation of an existing ANN approach ([1] Shi et al., CVPR 2023) with minimal modification. This method transfer is not tailored to the specific properties of SNNs—temporal sparsity, event-driven computation—and, as such, does not fully leverage the unique advantages of the SNN domain.

4. Domain Suitability More critically, the borrowed mechanism appears ill-suited for SNNs, as evidenced by the negligible accuracy gains (<2%) alongside increased power and training cost. This suggests a mismatch between the method’s design assumptions and the constraints/opportunities in SNN architectures.

Given the limited empirical improvement, absence of speed-up, added overhead, and lack of substantive methodological innovation, I believe the overall contribution is not sufficiently strong for NeurIPS. I would recommend the authors consider a venue such as Neural Networks, where a more incremental adaptation of ANN ideas to SNNs might be a better fit.

Reference

[1] Shi, Baifeng, Trevor Darrell, and Xin Wang. "Top-down visual attention from analysis by synthesis." CVPR, 2023.

.

Weaknesses 1: The two feedforward stages introduced additional FLOPS and parameters costs.

Answer: Our model, as described in Section 4.1, does not require two feedforward stages. Our proposed method introduces only one feedforward pass and one lightweight feedback pass. The feedback mechanism incurs minimal additional computational cost, as detailed in Appendix C.1.

For example, as shown in Table 1, our lightweight model only increases power consumption from 38.91 to 38.93 (less than a 1% increase), and parameters from 64.96M to 65.55M (a 0.91% increase). These overheads are negligible compared to the significant performance gains achieved by our method, demonstrating that the design is both efficient and effective.

---

Weaknesses 2: The paper do not adequately reflect additional costs.

Answer: Our paper explicitly reports the additional FLOPs, power consumption, and parameter costs introduced by our method. For example, in Table 1, we present the comparison of power and parameter overhead introduced by our method. Furthermore, Appendix Table 6 provides parameter counts and additional FLOPs on small-scale datasets, offering a complementary perspective on the cost across different scenarios. In addition, we provide a detailed discussion of power consumption and efficiency in Appendix C.1, showing that the introduced feedback mechanism contributes minimal overhead while significantly improving performance.

---

Weaknesses 3: Accuracy gains come at the cost of over 40% more power consumption, undermining SNN Efficiency

Answer: As shown in our response to Weakness 1 and detailed in Appendix C.1, our model on ImageNet only incurs less than 1% additional power consumption (from 38.91 to 38.93 mJ). This is significantly lower than 40%. Furthermore, our model is fully event-driven, and the proposed feedback mechanism adds only minimal overhead in both computation and power. Therefore, it preserves the core advantages of SNNs, including low power, low latency, and low computational cost.

---

Question 1: Is this method outperforms simply because of doubling the same time steps of the baseline?

Answer: We did not double the time steps of the baseline, and there is no evidence in the paper suggesting a doubling of energy consumption. For detailed clarification, please refer to Weakness 1, Weaknesses 3 and Appendix C.1.

---

Question 2: The biological relevance of the paper is questionable.

Answer: Our goal is not to replicate biological mechanisms precisely, but to draw inspiration from top-down feedback circuits in the brain to address inherent limitations in SNNs, ultimately aiming to achieve high-performance spiking neural networks.

In traditional SNNs, feature representations across time steps are often independent, as the membrane potential is the only temporal link and is limited in capturing rich temporal dynamics. Inspired by the progressive nature of the human visual system — where earlier stimuli modulate later perception through feedback — we design feedback paths that flow from deeper to shallower layers and from earlier to later time steps, aiming to enhance temporal coherence in representation and thereby improve the performance of SNNs.

---

Question 3: During training, will spiking attention use BPTT to propose a decline? Why does using both L1 and L2 in the function promote model convergence?

Answer:

1. Yes, BPTT is the standard method for training SNNs, and we adopt it in our model.

2. Regarding the use of L1 and L2: as described in Section 4.2 of our paper, our method does not design or rely on L1 or L2 loss functions for convergence.

---

Question 4: The paper lacks visual case studies.

Answer: In the appendix of our paper, we provide visual evidence of our model's attention patterns through detailed visualization studies. Figure 4 presents representative results on CIFAR-10-C, while Figure 5 extends this analysis to include ImageNet-C. These visualizations clearly illustrate that our top-down enhanced architecture focuses on more semantically meaningful, sparse, and task-critical regions compared to conventional methods.

---

Question 5: Concerns about the novelty and contribution of this paper.

Answer:

1. While [1] introduces top-down mechanisms in conventional ANNs, our work is, to the best of our knowledge, the first to explore top-down attention in SNNs, with significant performance and robustness gains. Moreover, [1] primarily focuses on visual question answering (VQA), whereas our work emphasizes architectural advances in SNNs and systematically evaluates performance across both static and dynamic datasets.

2. TDAC addresses a core limitation in SNNs—namely, the temporal independence of features across time steps. It significantly increases temporal feature correlation, leading to more coherent and consistent representations. We further investigate temporal connections in SNNs through forward and backward propagation experiments, along with a theoretical analysis provided in Theorem 4.3.

3. The feedback connection in [1] mainly computes similarity with a prompt vector, whereas our PM implements a full attention mechanism. We further analyze its statistical properties in Proposition 4.1 and Proposition B.4, showing that it provides improved numerical stability. The CM, by contrast, is a commonly used component that serves as a lightweight channel-wise feature mixing module.

Thank you for your insightful feedback. We have carefully studied your comments and argue that your concerns can be addressed.

---

Weaknesses 1: Our method’s performance on datasets from other domains

Answer: We agree on the importance of evaluating generalization across different domains. While extending our method to domains such as object detection and natural language processing is valuable, such tasks typically require substantial computational resources and extensive training efforts. Given these constraints, we instead conducted experiments on two real-world datasets from distinct application domains, medical imaging and industrial detection, to assess the adaptability of our method. Specifically, we performed preliminary evaluations on: Alzheimer's MRI and PV cell defect detection (ELPV).

TDAC was applied to multiple SNN backbones (Qkformer, Spikformer, Spike-driven Transformer), and we observed consistent performance improvements:

Table 2: Performance of TDAC on medical imaging and industrial detection

TDAC consistently improved performance on both tasks, demonstrating promising cross-domain compatibility.

---

Weaknesses 2: Is the method hardware-friendly?

Answer: While full Spiking Transformer models are currently challenging to deploy on neuromorphic chips due to architectural complexity, our method’s core component — the top-down feedback —can be deployed on hardware. We demonstrate this by providing code that shows how the feedback mechanism can be deployed on Intel’s Loihi chip using the Lava framework.

#Import 

class Add(AbstractProcess):
    ...
 
#ADD         
@implements(proc=Add, protocol=LoihiProtocol) 
@requires(CPU)  
class PyAddModel(PyLoihiProcessModel): ...
    def run_spk(self):
        self.a_out.send(a_in1.recv() + a_in2.recv())
           
class SpikeInput(AbstractProcess): ...
    
class OutputProcess(AbstractProcess):...
    
@implements(proc=SpikeInput, protocol=LoihiProtocol)
@requires(CPU)
class PySpikeInputModel(PyLoihiProcessModel): ...
            
class ImageClassifier(AbstractProcess):
    def __init__(...):    
        # Using pre-trained weights and biases
        ...
        
        #ADD
        self.w_feedback = Var(shape=(64, 64), init=fb_weights) 
        
        ## Up-level currents and voltages of LIF Processes for resetting
        ...
      
@implements(ImageClassifier)
@requires(CPU)
class PyImageClassifierModel(AbstractSubProcessModel):
    def __init__(...):
        #Connect core layers
        
        #ADD
        self.feedback_dense = Dense(weights=proc.w_feedback.init)
        self.add = Add(shape=(64,))
        self.output_lif.s_out.connect(self.feedback_dense.s_in)
        self.feedback_dense.a_out.connect(self.add.a_in2)
        self.dense0.a_out.connect(self.add.a_in1)
        self.add.a_out.connect(self.lif1.a_in)
        
        # Create aliases of SubProcess variables
        ...

# Create Process instances

# Connect Processes

# Connect Input directly to Output for ground truth labels
  
# Loop over all images     
for img_id in range(num_images):
    mnist_clf.run(
        condition=RunSteps(num_steps=64),
        run_cfg=Loih i1SimCfg(select_sub_proc_model=True, select_tag='fixed_pt'))
        
    out_voltage = mnist_clf.oplif_v.get().astype(np.int32)
    mnist_clf.feedback_dense.a_in.set(out_voltage)
    
    mnist_clf.run(
        condition=RunSteps(num_steps=64),
        run_cfg=Loihi1SimCfg(select_sub_proc_model=True, select_tag='fixed_pt'))
  
    # Reset internal neural state of LIF neurons

# Gather ground truth and predictions before stopping exec

# Stop the execution    

---

Question 1: Is there a more suitable timestep (T)?

Answer: Yes. We conducted experiments with different values of T on the CIFAR10 and CIFAR100 datasets, using Spikformer V1. Specifically, we tested T = 2, 4, and 6. All results are averaged over five random seeds.

As shown below, under the Spikformer architecture, T = 2 yields the largest performance improvement, T = 4 provides moderate gain, while T = 6 brings little additional effect.

Table 3: Performance under different timesteps (T)

We hypothesize the following reasons:

1. A smaller number of timesteps (T) may already be sufficient to capture the essential temporal dynamics required for the task. Once the key temporal dependencies are modeled, increasing T further may lead to performance saturation.

2. In models equipped with top-down modulation (such as ours), the network’s effective capacity is enhanced, allowing it to focus on critical features early in the temporal sequence. As a result, adding more timesteps may not provide additional useful information and could instead introduce redundancy or even lead to overfitting by amplifying irrelevant or noisy signals.

---

Question 2: Validity of theoretical assumptions

Answer: We have the following reasons to alleviate this concern:

1. In practical large-scale settings such as ImageNet, modern models often have a large number of channels — for example, Qkformer uses 768 channels. This statistically narrows the gap between theory and real-world usage,
   making the large-C assumption more valid for analyzing modern architectures.

2. Although in real-world scenarios C cannot truly be infinite, assuming a large channel dimension (e.g., C → ∞) is a widely adopted and accepted theoretical simplification in the deep learning community. Theoretically, the large-C assumption underpins numerous foundational studies in deep learning. For example:

   • "Neural Tangent Kernel: Convergence and Generalization in Neural Networks":
     Analyzes convergence using the NTK framework, leveraging the C → ∞ assumption for kernel stability.

   • "Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent":
     Shows that under infinite-width conditions, networks behave like Gaussian Processes.

   • "A Mean Field Theory of Batch Normalization":
     Uses the large-C perspective to derive tractable analytical expressions for BN behavior.

These examples indicate that, although idealized, the large-C assumption provides a theoretically meaningful perspective that can offer valuable insights into the behavior of modern large-scale models.

Dear Reviewer 7Yb8,

We hope this message finds you well. As the discussion period approaches its end, we sincerely appreciate the time and effort you've dedicated to reviewing our manuscript and rebuttal. We are writing to check whether there are any additional questions or clarifications you would like from us at this stage — we are more than happy to provide further details or support if needed.

In particular, we have tried to directly address the key concerns you raised in your reviews, including:

• Cross-domain generalization: We added experiments on two real-world datasets — Alzheimer's MRI (medical imaging) and ELPV (industrial defect detection) — and consistently observed performance gains across multiple SNN backbones, demonstrating the adaptability of our method.
• Hardware friendliness: We show that the top-down feedback mechanism can be implemented on Intel’s neuromorphic Loihi chip using the Lava framework, with code examples included to illustrate feasibility.
• Choice of timestep (T): We conducted systematic experiments on CIFAR10 and CIFAR100 showing how smaller T values yield better gains.
• Theoretical assumptions: We clarified the role of the large-channel (large-C) assumption, supporting its validity through both empirical architecture sizes and references to foundational theoretical work in the deep learning literature.

We welcome any additional feedback regarding our method or the responses in the rebuttal.

Thank you for your insightful comments. We first list your advice and questions, then give our detailed answers.

Question 1: Lack of theoretical support

Answer: We appreciate the concern regarding the lack of a rigorous theoretical derivation for the core problem. In this paper, we focus on empirical evidence: comprehensive performance gains and targeted visualizations support that the proposed architecture effectively addresses the problem. Although we have been seriously considering theoretical foundations, we acknowledge that a complete and rigorous proof remains an open challenge at this point. We fully agree on its importance and will prioritize a thorough theoretical analysis in future work.

---

Question 2: Insufficient comparison with prior work

Answer: Compared to prior methods, our approach introduces several notable differences, as outlined below.

1. Our method differs from prior approaches in where features are selected for temporal fusion along the time axis. Unlike prior works such as TIM and TE-Spikformer, which perform temporal feature fusion within the same semantic space — typically by aggregating or averaging queries across different time steps at the same feature level. Our proposed TDFormer adopts a fundamentally different design inspired by the feedback circuits in the human brain. In particular, TDFormer introduces a top-down feedback mechanism, where high-level features from earlier time steps guide lower-level feature extraction at later steps. This approach enhances temporal coherence and enriches representations, offering a more biologically inspired alternative to standard temporal attention mechanisms.

2. The three approaches differ in how they fuse temporal information. TIM fuses time causally with a recurrent update: a single 1D convolution along the spatial axis refines the carry from the previous step, then an EMA-style gated blend merges it with the current input. TE-Spikformer, applies transformations to form new QKV tokens and reuses attention for temporal fusion which significantly increases compute and power due to the added attention operations. By contrast, our method uses a feedback path that combines a spatial mixer and a channel mixer to refine the previous timestep’s representation before fusing it with the current input. The spatial mixer captures token-level structure, while the channel mixer models cross-feature dependencies, enabling selective amplification or suppression of historical cues. This feedback refinement gives greater capacity to process.

3. Unlike TIM and TE-Spikformer, which are evaluated solely on dynamic datasets and primarily with a single baseline, our work significantly expands the evaluation scope. Specifically, we conduct comprehensive experiments on both dynamic and static datasets, including ImageNet, and validate our method across multiple baseline architectures.

---

Question 3: Inferior performance on CIFAR10-DVS compared to TIM

Answer: While TIM demonstrates strong performance on dynamic datasets, several concerns suggest that it may not be fair when compared to our model and Spikformer:

1. Unaligned training settings: Our model strictly follows all training hyperparameters used in Spikformer in the original paper to ensure a fair comparison. However, TIM significantly alters these settings — for instance, training for 500 epochs compared to Spikformer’s original setup of 96 epochs. This results in over 5× more training iterations, which may lead to significantly higher training costs and energy consumption, ultimately making the comparison unfair.
2. Different data augmentation: Data augmentation can have a significant impact on model performance, especially for dynamic datasets like CIFAR10-DVS. Our method strictly follows the data augmentation strategies described in the original Spikformer paper to ensure a fair comparison. But TIM adopts different data augmentation strategies compared to Spikformer, which may affect the fairness of comparison. For example:
   • Resizing input frames to 64×64 (default is 128×128).
   • Omitting mixup, which is enabled in Spikformer.
   • Applying Gaussian blur after resizing.
3. Not fully spike-Driven: TIM does not fully adhere to the spike-driven paradigm. For example, it uses fixed alpha values. Such non-spike-driven components may contribute to performance improvements. In contrast, our method strictly follows the spike-driven paradigm, ensuring all computations are spike-based and align with neuromorphic principles.

Due to the above reasons, these modifications can influence performance and make direct comparisons less reliable.

---

Question 4: Baseline reproduction concerns

Answer: We strictly followed the hyperparameter settings and released code from the original papers to reproduce the results. We also noticed on GitHub that some users reported significant reproduction gaps on the Spikedriven CIFAR10-DVS benchmark in the issues section , suggesting that the problem is not specific to our reproduction. (For more details, please kindly refer to the GitHub repository linked in the original paper.)

To mitigate such variance, we conducted multiple-seed averaging experiments to eliminate reproduction bias. Each result shown is averaged over five random seeds; please refer to Table 1 below for detailed results.

Table 1: Random seed averaged results on CIFAR10-DVS

---

Question 5: Lack of ablation to disentangle gains from parameter count and temporal modeling

1. We conducted ablation experiments based on Spikformer to further validate that the observed performance improvements stem from our proposed method, rather than being a trade-off between accuracy and increased energy or parameter count.
   Given the similarity of concerns, please refer to our response to Reviewer Sx25 - Question 2 for detailed results and analysis.

2. While it is true that TDFormer introduces a slight increase in parameter count, we emphasize that the performance gains observed cannot be solely attributed to this minor change. For example, as shown in Table 1, TDFormer at 224² resolution outperforms QKFormer at 288² in terms of top-1 accuracy (85.57% vs. 85.20%) while consuming significantly less energy (39.10 mJ vs. 64.27 mJ). This substantial improvement in both accuracy and efficiency, despite a modest increase in parameters, strongly suggests that the gains cannot be attributed solely to increased parameter count.

Thank you for the detailed responses. While I appreciate the authors' efforts to address my concerns, several fundamental issues remain unresolved:

The lack of rigorous theoretical foundation is acknowledged but not addressed - this is a core weakness for a paper claiming to solve temporal information extraction problems in SNNs. The substantial baseline reproduction gaps (0.8-3%) continue to raise questions about experimental reliability, and the marginal improvements on some datasets may simply reflect these reproduction issues rather than genuine methodological advances.

While the explanations regarding fair comparison with prior work are reasonable, the fundamental question remains whether the proposed method truly outperforms existing approaches under equivalent conditions.