SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network

Thank you very much for your valuable comments!

Q1: The work of training SNNs using KD principles has already been done, and SpikeCLIP lacks algorithms and neuron designs specific to SNNs, so is the contribution of SpikeCLIP relatively limited?

R1: The work you mentioned [1] employed KD principles to train single-modal CNN-based SNNs in a step-wise manner, whereas our work effectively trained multi-modal Transformer-based SNNs using the paradigm of aligned pre-training followed by dual-loss fine-tuning. The difficulties and emphasis between the two works are different. Due to the scarcity of SNNs in the field of natural language processing (mainly due to the absence of authoritative architectures), we attempted to represent both modalities' information effectively through direct training with spike signals, aligning and fusing them, which is more challenging than distilling single-modal information. To achieve this, we not only modified the original Spikingformer on the image encoder side (adding TSW, as shown in A.1) but also compared the effectiveness of different text encoder architectures. While it is indeed feasible to easily transition from CLIP to the SNN domain using methods such as weight transfer, our focus is on validating whether modal information represented by spike signals can effectively undergo modal fusion, similar to what ANNs achieve. Our work's contribution lies in emphasizing this point.

Q2: The performance of SpikeCLIP on downstream datasets is not superior, and the author did not fully list recent research achievements in single-modal SNNs in Table 1?

R2: We have included several representative works of single-modal SNNs in Table 1, but not all, as there is indeed a considerable amount of relevant research. We have also listed the latest works on single-modal SNNs, such as [2] (CIFAR10: 95.95%; CIFAR100: 80.37%, 4-time steps), which outperforms the one you mentioned [3] and is currently the best-performing.

Additionally, you noted that the performance of SpikeCLIP is inferior to a single-modal SNN trained directly through ResNet-19. This is inevitable because comparing the accuracy of a single-modal model with a multi-modal model is inherently unfair and meaningless. As SpikeCLIP is the first multi-modal SNN, we intentionally included in Table 1 a fair comparison between the best single-modal and multi-modal models in both the SNN and ANN domains to better assess its performance. Table 1 compares the accuracy of models on CIFAR10/100. In the ANN domain, the best single-modal model has an accuracy advantage of 0.68/4.50% over the best multi-modal model, while in the SNN domain, the best single-modal model outperforms SpikeCLIP by 1.47/2.68%. Compared to the gap in ANNs, the gap in SNNs is within an acceptable range, and on CIFAR100, the gap in SNNs is even smaller. We believe this sufficiently reflects the performance level of SpikeCLIP. Furthermore, SpikeCLIP possesses zero-shot learning capabilities, a feature absent in traditional single-modal SNNs. Faced with new downstream tasks, SpikeCLIP can follow the fine-tuning paradigm without starting training from scratch, a characteristic worth considering. In conclusion, demanding that multi-modal SNNs surpass single-modal SNNs in performance while leveraging advantages unique to single-modal models is the next important challenge.

Dear commenter EDPh:

We greatly appreciate your time and effort in reviewing our work. We have considered your questions carefully and made the necessary changes. Specifically, we have revised section A.1 to elaborate in more detail on our introduction of TSW, an architectural setup that significantly improves SNNs performance. We have revised 3.3 to express in more detail the special insights and significance of our "alignment pre-training + dual-loss fine-tuning" training framework; We have added Table 1 to include the baseline you mentioned in the comparison range; Finally, we give our considerations for choosing to use the LN layer in A.3. You can also refer to our official review for a more comprehensive response on the motivation and contribution to this study, the training framework, the performance of SpikeCLIP, and the implementation of LN in SNN.

Your expertise and insights are invaluable and we are keen to ensure that our research meets the highest standards. We look forward to hearing more from you and will be happy to answer further questions.

Dear Reviewer EDPh:

We sincerely thank you for taking the time out of your busy schedule to conduct a thorough review of our paper and provide valuable comments.

As the Author-Review Discussion period is drawing to a close with only two days remaining, we would like to ensure that all your concerns have been adequately addressed. If there are any questions or unresolved issues, we are eager to provide further clarification or make necessary revisions.

Best regards,

The Authors

Thanks for your response. Considering that all the concerns of reviewers are the same, Maybe I have been a bit strict with the authors, so I have changed my score. However, I agree with other reviewers that the more important thing is actually to find the specifics of spikes in those architectures or settings, not just apply the SNN to other fields. I advise that the authors could give more special insights and meanings for the alignment pre-training + dual-loss fine-tuning framework. About the performance, the performance of the SpikeCLIP on zero-shot tasks is really much worse. I don't think that just SNN can save energy thus its performance can be ignored. So the response can not convince me. In addition, BN can be folded into the weights in the inference, but LN can not be. So the authors' responses to the questions of other reviewers are not good enough.

Thank you very much for your valuable comments!

Q1: The novelty seems limited; is the alignment pre-training + dual-loss fine-tuning just a knowledge distillation (KD) method?

R1: Currently, most research on SNNs is primarily concentrated in the field of computer vision, with an extremely limited presence in natural language processing and a complete absence in the application of SNN architectures in the multimodal domain. There is currently no existing work exploring the feasibility of SNN architectures in the multimodal domain and whether the modality information represented by pulse signals can fuse modalities as effectively as the floating-point information in ANNs. However, with the growing prevalence of multimodal applications and the enormous energy consumption pressure brought by ANNs, the application of SNN architectures in the multimodal domain is becoming inevitable. Our work, inspired by CLIP, is the first attempt to explore the feasibility and effectiveness of using the spike-based computing paradigm for modality fusion, urging researchers to pay more attention to the application of SNNs in the multimodal context. We propose the alignment pre-training + dual-loss fine-tuning framework, which effectively trains the first competitive and zero-shot learning-capable multimodal SNN.

While our framework draws inspiration from CLIP and incorporates the idea of knowledge distillation (KD), it successfully accomplishes the task on its own merit. Knowledge distillation is a universally effective concept, and the focus of our alignment pre-training + dual-loss fine-tuning framework is not on whether KD principles are used, but on how they are applied. In our paper, we argue that the initial alignment pre-training step enables SpikeCLIP to acquire general image/text representation capabilities. In the dual-loss fine-tuning stage, as described in Section 4.3, we determine the use of two different losses by discerning the impact at different data granularities. Moreover, SpikeCLIP is not merely a transformation of CLIP into SpikeCLIP. SpikeCLIP and CLIP have distinct architectures (beyond the SNNs vs. ANNs difference) and different training processes, among other differences. Since CLIP conducts representation comparison at the end of a dual-stream architecture when performing multimodal tasks, a condition also achievable by current SNNs, we chose CLIP as a guide for SpikeCLIP. SpikeCLIP is not presented as a direct transformation from CLIP, and perhaps a better "conversion" could be achieved through weight transfer. However, this is not the focus of our paper.

Q2: SpikeCLIP's performance is not good; Its accuracy is lower than other state-of-the-art (SOTA) methods, and it is much worse than CLIP. Does SpikeCLIP have limited value?

R2: Without a doubt, SpikeCLIP's accuracy is expected to be lower than the current best single-modal SNNs and significantly worse than CLIP. However, this comparison is unfair and meaningless because, at the same level, single-modal models perform worse than multimodal models, and under the same conditions, SNNs perform worse than ANNs.

At the same level, multimodal models have zero-shot learning capabilities, a feature we validated in Section 4.2.2 for SpikeCLIP, but currently, all single-modal SNNs lack this capability. At the same level, SNNs have lower accuracy than ANNs because, unlike ANNs, SNNs use integer operations to process data streams. While this results in an acceptable decrease in accuracy, it also comes with higher energy efficiency. In Section 4.5, we demonstrate that SpikeCLIP reduces energy consumption by 77.06% to 78.66% compared to its ANN counterpart.

To more fairly evaluate SpikeCLIP's performance, we compare the performance gaps between the best single-modal and multimodal models in SNNs and ANNs on CIFAR10/100 in Table 1. SpikeCLIP is currently the only multimodal SNN. In Table 1, the performance gap between SpikeCLIP and Spikingformer is 1.47%/2.68%, while the gap between CLIP and ViT is 0.68%/4.50%. These comparisons indicate that, as a multimodal SNN, SpikeCLIP's performance is competitive. Of course, it is not our intention to simply discuss the performance of SpikeCLIP. Due to the gap in the topic of multimodal SNN, our work is only a preliminary attempt to determine whether modal information represented by SpikeCLIP can be fused, which is also claimed in our paper. We believe that our work will lead to more outstanding research focusing on the performance weaknesses and application challenges of multimodal SNNs.

Dear commenter njnp:

We greatly appreciate your time and effort in reviewing our work. We have considered your question carefully and made the necessary changes. Specifically, we have revised section 3.3 to elaborate in more detail on the specific insights and implications of our training framework. We have also revised 4.2.1 so that everyone can more clearly compare SpikeCLIP's performance from a fair point of view. Finally, we detail our considerations for choosing the LN layer in section A.3.

Your expertise and insights are invaluable and we are keen to ensure that our research meets the highest standards. We look forward to hearing more from you and will be happy to answer further questions.

Dear Reviewer njnp:

We sincerely thank you for taking the time out of your busy schedule to conduct a thorough review of our paper and provide valuable comments.

As the Author-Review Discussion period is drawing to a close with only two days remaining, we would like to ensure that all your concerns have been adequately addressed. If there are any questions or unresolved issues, we are eager to provide further clarification or make necessary revisions.

Best regards,

The Authors

Thank you very much for your valuable comments!

Q1: Does the MLP text encoder sacrifice the generalization ability of language understanding? Regarding the use of text templates in zero-shot classification.

R1: In order to imbue the text decoder of SpikeCLIP with text representation capabilities and for the purpose of comparison with CLIP, our constructed dataset $D_{txt}$ is derived from 27 datasets used to evaluate CLIP's zero-shot learning ability. As mentioned in A.4, for each dataset, we combine its labels with the corresponding template set, thus constructing $D_{txt}$ with a total of 115,708 entries. We sincerely apologize for not providing more detailed construction details of $D_{txt}$ in the paper, causing confusion. However, these template sets encompass various templates, and you can refer to the supplementary material for a detailed examination. For the image classification datasets used in this paper, training a text decoder using $D_{txt}$ can yield a sufficiently effective result. However, as you rightly pointed out, a simple MLP text encoder inevitably sacrifices the generalization ability of language understanding, despite our comparison of the effectiveness of two different architectures used as text encoders. Therefore, constructing a better text encoder structure remains an important problem we aim to address in our future work.

Q2: Why is the test accuracy after further fine-tuning on ImageNet itself not reported? Does SpikeCLIP have sufficient representational capacity and scalability?

R2: Following the paradigm of alignment pre-training + dual-loss fine-tuning, we deliberately set the downstream datasets to be different from the pre-training dataset. This setup aims to demonstrate the effectiveness of pre-training and the zero-shot learning capability of SpikeCLIP. In Section A.6, we report SpikeCLIP's performance when the distributions of the pre-training dataset and the downstream dataset are dissimilar, indirectly validating the good performance when using ImageNet-1k for both alignment pre-training and dual-loss fine-tuning.

Furthermore, this is the first attempt to use the spike-based computing paradigm for multimodal classification tasks. Unlike CLIP, we do not have access to larger datasets for alignment pre-training, which indeed imposes significant limitations on SpikeCLIP's representational capacity and scalability, as you rightly mentioned. However, in Section 4.3, we experimentally verify the potential for improved SpikeCLIP performance with the existence of more data. Certainly, this is a challenging task, but it doesn't hinder us from confirming the feasibility of SNNs in multimodal scenarios.

Dear reviewer edZQ:

We greatly appreciate your time and effort in reviewing our work. We have considered your questions carefully and made the necessary changes. Specifically, we further analyze our considerations for choosing the MLP architecture as the text encoder in section A.3. Although the architecture is simple, in our work, there is also not much text information consisting of labels of data sets, which makes the simple MLP architecture suitable for the text processing capabilities of SpikeCLIP. In addition, we provide more details of $D_{txt}$ in A.4 (including different templates), and finally, you can refer to our official review for A more comprehensive summary of SpikeCLIP's performance.

Your expertise and insights are invaluable and we are keen to ensure that our research meets the highest standards. We look forward to hearing more from you and will be happy to answer further questions.

Dear Reviewer edZQ:

We sincerely thank you for taking the time out of your busy schedule to conduct a thorough review of our paper and provide valuable comments.

As the Author-Review Discussion period is drawing to a close with only two days remaining, we would like to ensure that all your concerns have been adequately addressed. If there are any questions or unresolved issues, we are eager to provide further clarification or make necessary revisions.

Best regards,

The Authors

Thank you very much for your valuable comments!

Q1：The paper lacks novelty, SpikeCLIP architecture implementation claims "the first......" However, the bright spot could not be found ?

R1: Our work attempts to align the two modalities with the spiking computing paradigm, which is not done before. As mentioned in the paper, the current implementation methods of SNNs mainly include weight transfer and direct training using surrogate gradients. However, due to the sparsity of data and integer operation in SNN, direct training of SNNs is not easy. Our work has used alignment pre-training and dual-loss fine-tuning to effectively realize the alignment of the two modalities and achieve competitive performance. In previous research on SNNs, the vast majority of work in the field of computer vision has been done, so many questions about architecture design have been conducted around computer vision tasks. However, the lack of work on SNNs in the field of natural language processing makes it difficult to use SNNs for multimodal tasks. Considering that the advanced Spikingformer is based on Transformer, which is derived from natural language processing tasks, we have experimented with different text-side architectures in Transformer and MLP for multimodal classification tasks.

We sincerely apologize for emphasizing "the first..." in the paper. However, due to the lack of exploration in the field of SNNs in the multimodal context, our work represents only a preliminary exploration of the feasibility of this topic. We also hope that in the future, there will be more SNNs-related research that pays close attention to the implementation of multimodal tasks, similar to the attention given to computer vision tasks.

Q2: The comparison of energy efficiency between SpikeCLIP and its counterpart, ScratchCLIP, is unfair, and there are omissions in energy calculations?

R2: To ensure a fair comparison of the performance and energy consumption between SpikeCLIP and its counterpart ScratchCLIP, we configured both with identical settings (e.g., layer number, dimensions, training process) except for the spiking neurons. In this setting, energy consumption calculations were based on [1], where SNN, as a neural morphic computing algorithm, can smoothly execute on sparse neural morphic chips, requiring only spike-based accumulate (AC) operations, while ANNs involve numerous multiply-and-accumulate (MAC) operations. Therefore, the additional energy required for membrane potential update processes and maintaining the membrane potential of each neuron does not need to be considered. Specific chip implementation details can be found in [2] and [3]. Regarding BN/LN operations, as SpikeCLIP and ScratchCLIP architectures have identical BN/LN operations, their energy consumption cancels out when calculating the energy consumption ratio.

While it is acknowledged that the data input for SNN is greater than that for the corresponding ANN, SNN can leverage this additional input to abandon more precise floating-point calculations, sacrificing precision for efficient energy consumption. SpikeCLIP, in its preliminary attempt at multimodal alignment tasks in SNNs, exhibits a performance gap with the single-modal Spikingformer within a reasonable range. At the same time, Spikeclip also shows zero-shot learning ability (which is not available in previous single-modal SNNs). Superior energy efficiency is only an inherent but indispensable advantage of SpikeCLIP.

Dear reviewer edTp:

We greatly appreciate your time and effort in reviewing our work. We have carefully considered your questions and made the necessary changes. Specifically, we went deeper into the special insights and implications of our "alignment pre-training + dual-loss fine-tuning" training framework in 3.3, and we modified section A.6 to make it clearer how energy consumption compares with ScratchCLIP. Finally, you can refer to our official review for a more comprehensive summary of our motivations and contributions to this study and SpikeCLIP's performance.

Your expertise and insights are invaluable and we are keen to ensure that our research meets the highest standards. We look forward to hearing more from you and will be happy to answer further questions.

Dear Reviewer edTp:

We sincerely thank you for taking the time out of your busy schedule to conduct a thorough review of our paper and provide valuable comments.

As the Author-Review Discussion period is drawing to a close with only two days remaining, we would like to ensure that all your concerns have been adequately addressed. If there are any questions or unresolved issues, we are eager to provide further clarification or make necessary revisions.

Best regards,

The Authors

3. The performance of SpikeCLIP.

R3: SNNs still lag behind ANNs in terms of accuracy. Through intensive research on SNNs in recent years, the performance gap between deep neural networks (DNNs) and SNNs is constantly narrowing. SNNs cannot currently outperform DNNs on the datasets that were created to train and evaluate conventional DNNs (they use continuous values). Such data should be converted into spike trains by spiking neurons before it can be fed into SNNs, and this conversion might cause a loss of information and result in a reduction in performance. Therefore, the comparison is indirect and unfair. In our study, we have conscientiously chosen existing SNNs as baselines for evaluation to provide a fair and relevant benchmark. As illustrated in Table 1, SpikeCLIP exhibits accuracy rates of 94.48% and 77.69% on CIFAR10 and CIFAR100, respectively. These results surpass most of the previous single-modal SNN, and the performance of SpikeCLIP only decreases by 1.47% and 2.68% compared with the current optimal single-modal SNN, Spikingformer[3]. It is worth noting that single-modal and task-specific models typically outperform multi-modal models. Similarly, The traditional CLIP, designed to excel in zero-shot situations, was not optimized for achieving state-of-the-art performance on a specific dataset. In addition, the performance gap of 2.68% between SpikeCLIP and the state-of-the-art single-modal Spikingformer is substantially smaller than the 4.50% gap between CLIP and its single-modal counterpart (i.e., Vit) on the CIFAR 100 dataset. This comparison underscores the reasonably good performance of our model.