Dear Reviewer,

We would like to sincerely thank you for taking the time to review our paper and provide valuable feedback. We are glad that you think our paper is well presented. We are thankful to have the chance to address your questions and concerns.

Q1. In lines 93-94, it should be 'nearly lossless generation'.

A1. We sincerely appreciate you for this thoughtful suggestion. In the updated version, we have revised this term.

Q2. How would the method scale to larger models such as LLaMA-70B in multi-GPU settings?

A2. Thank you for raising this meaningful question. We are very happy to discuss it with you. In fact, we think that CoreInfer can easily adapt to multi-GPU settings. This is because the computation of core neurons is done at the layer level, and different layers do not affect each other. In multi-GPU settings, a large model is typically split by assigning certain layers to individual GPUs, meaning that computations within the same layer still occur on a single device. Therefore, the calculation of core neurons is unaffected by multi-GPU settings, making it easy to scale and deploy.

Q3. Would this method be compatible with quantization methods?

A3. We sincerely appreciate you for this constructive suggestion. We deployed CoreInfer using the Bitsandbytes library under 4-bit and tested its performance on four commonsense reasoning tasks. We found that CoreInfer achieved lossless sparse inference even with applying 4-bit quantization, as the activation of neurons was not affected by quantization. After combining CoreInfer with the FP4 and NF4 quantization formats, the average accuracy improved from 69.98 and 68.83 to 69.99 and 68.89, respectively. Therefore, CoreInfer can be combined with state-of-the-art quantization methods for further acceleration. We have added these experimental results in Appendix 4.3.

Q4. The authors should also discuss related works on sparse KV cache [1-4], another important direction to accelerate LLM inference.

A4. Thank you for providing this valuable suggestion and these references. Activation sparsity-based inference can indeed be combined with sparse KV cache techniques to further accelerate the model. In the updated version of the paper, we have added a discussion on KV cache methods in the Background and explored the references provided.

We would like to once again express our sincere gratitude for your valuable suggestions. Your feedback has made our work more complete and thorough. If you have any further comments, we would be more than happy to discuss them and address any questions you may have.

Best regards,

Authors

Q6. Task performance comparison with predictor-based methods

A6. We sincerely appreciate you for this valuable suggestion.

We compared the performance of CoreInfer and PowerInfer on four commonsense reasoning tasks, with the experimental results provided in Appendix A.4.2. The experimental results for PowerInfer are taken from the original paper. We observed that both CoreInfer and PowerInfer achieved near-lossless performance on these tasks. However, PowerInfer requires additional predictors for training and inference, leading to increased overall cost. For example, for OPT-30b, PowerInfer requires approximately 10.45GB for the predictors, which accounts for about one-sixth of the total model size.

Regarding the sparsity of other methods, previous predictor-based approaches train an MLP predictor for each layer of the model to predict which neurons will be activated. The level of sparsity in these methods depends on the model, the number of layers, and the input content. In general, assuming a prediction accuracy close to 100%, the sparsity can be estimated as the ratio of activated neurons to the total number of neurons. In the OPT model series, the proportion of activated neurons is around 10% to 15%. Additionally, the memory size of the MLP predictors introduced by these methods is approximately 10% - 20% of the overall model size.

We once again thank you for taking the time and effort to thoughtfully consider our work! We truly appreciate your recognition of our efforts. If you have any further questions, do not hesitate to let us know. We are more than happy to continue the discussion and address any concerns you may have.

Best regards,

Authors

Dear Reviewer,

We would like to sincerely thank you for taking the time to review our paper and provide valuable feedback. We are glad that you think our method has a clear motivation and novel. We are thankful to have the chance to address your questions and concerns.

Q1. Further justified of Fig.2(C).

A1. We are grateful for the time and effort you have taken to carefully review our work and raise this concern. We sincerely apologize for the misunderstanding caused. In fact, this misunderstanding was caused by the inappropriate use of colors in Fig. 2(C). We initially displayed the core neurons of 50 sentences using a gradient color to represent different sentences, which made it appear as though some sentences had the same color (though they were actually different sentences). We have updated Fig. 2(C), reducing the number of sentences to 20 and using more distinct colors. We hope this new version helps to minimize any confusion.

Q2. Inappropriate term "zero-cost sparse inference" used in the abstract.

A2. We appreciate your thoughtful suggestions. In the new version, we have revised it to "fast sparse inference".

Q3. Explanation of the acceleration in Figure 7.

A3. We appreciate you raising this question, and we are glad to address your question. In Tab. 3, we presented the experimental results on a low-performance GPU (12GB NVIDIA TITAN XP), while in Fig. 7, we showed the results on a high-performance GPU (100GB NVIDIA A100). For the OPT-6.7b model, the low-performance GPU was memory-bound, and our speedup was achieved by reducing the computational requirements during inference and addressing the memory-bound limitation (similar to what we observed with the 70b model on the high-performance GPU). On the high-performance GPU, the OPT-6.7b model was not memory-bound, so the speedup was purely due to the reduction in computational requirements.

Q4. Clarification of system memory usage.

A4. We thank you raising this question and are more than happy to provide clarification. In Fig. 7, the term "system memory" refers to the CPU memory required during model inference. For larger models such as OPT-66b and Llama2-70b, inference can be memory-bound, meaning that some of the model weights need to be offloaded to the CPU. Larger CPU memory implies greater data transfer requirements between the CPU and GPU, leading to increased latency caused by memory bound. We apologize for the confusion caused by our wording. In the revised version, we have updated it to "CPU Memory".

Q5. The use of hyperparameters may limit practical usefulness.

A5. We understand your concern, and we are happy to provide further clarification regarding the hyperparameters. In Fig. 6, we performed an ablation study on the hyperparameters, and it can be seen that CoreInfer is not particularly sensitive to hyperparameter settings. Good performance can be achieved with a range of hyperparameter values, indicating that our method is robust to hyperparameter selection. Additionally, during the experiments, we used the same hyperparameters for tasks of the same type, and the model performed well across all tasks. This suggests that in practical applications, hyperparameters could potentially be set based on the task type. We acknowledge the potential challenges posed by hyperparameters in practical applications, and we plan to further explore and address this issue in future work. Thanks for your concern.

Dear Reviewer 2DA3,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. In our updated version, we have addressed the areas you highlighted that could cause confusion and have added a comparison with predictor-based methods, as per your suggestion.

We are truly grateful for your valuable suggestions and positive feedback. If you have any further questions, we would be more than happy to provide additional information!

Thank you again for your support.

Sincerely,

The Authors

Q3. Correlate with tokenization.

A3. We greatly appreciate you for raising this meaningful discussion. As mentioned in A2, we conducted experiments and analyses on Mandarin tasks that might experience "caught in the byte-fallback of the tokenizer." In these scenarios, we observed the following:

1. The Number of Core Neurons Required Does Not Increase: In fact, once stability is reached, retaining only 20% of the neurons still allows the model to achieve good generative performance. For instance, in the MultiFieldQA-zh Chinese QA task, CoreInfer retained only 20% of the core neurons, yet the model's F1 score increased from 9.21 to 12.86.
2. The Token Length Required for Stability Increases: Due to the byte-fallback phenomenon, the model requires a longer sequence of tokens to understand the semantics. As shown in Fig. 3, the model requires approximately 500 tokens to achieve stability.

Based on these findings, we conclude that as long as the semantics are well defined, the model only requires a small number of core neurons for inference. However, different tokenization strategies may impact the number of tokens needed to establish clear semantics. We have added a discussion on this interesting phenomenon in Appendix A.2.3.

Again, we would like to thank you for your deep thoughts and support for our work! We truly enjoy discussing these constructive and meaningful questions with you, as it makes our work more complete and robust. If you have any further questions, we would be more than happy to discuss and address them!

Best regards,

Authors

Dear Reviewer,

We would like to sincerely thank you for taking the time to review our paper and provide valuable feedback. We are glad that you think our approach is innovative and has great potential for application on edge devices. We are thankful to have the chance to address your questions.

Q1. Corrections to figures and titles.

A1. We sincerely appreciate your thorough reading of our paper and the valuable suggestions provided. We have revised Tab. 2 and Fig. 6 as suggested, and hope that the updated figures provide clearer insights for the readers.

Q2. Analysis of stability on different tasks.

A2. Thank you for providing this very constructive suggestion! Multilingual and different types of input is an important situation that is well worth exploring. For a deeper analysis, we first examined the stability of core neurons as the input length increased for both Mandarin characters and code inputs. Our analysis results are presented in Appendix 2.3 and Fig. 11. Furthermore, to verify the effectiveness of CoreInfer in this context, we conducted analyses and tests on the lcc/RepoBench-P dataset (code prediction tasks where the goal is to predict the next line of code given a longer code snippet) and Chinese dataset including DuReader, MultiFieldQA-zh, VCSUM, LSHT, PassageRetrieval-zh. We report our experimental results in Appendix 4.1. We also provided examples of the responses in Appendix 5.2.

Based on the experimental results, we have the following findings:

1. Gradual Stabilization of Core Neurons Across Different Tasks. For different tasks, as the input length increases, the core neurons exhibit a gradual stabilization phenomenon. Even for programming tasks with less clearly defined semantics (such as Python, C#, and Java), core neurons gradually stabilize as the length of the input code increases. We speculate that this is because, through training, the model learns to understand the intrinsic semantics of these languages during inference. For these tasks, using stability-guided prediction, CoreInfer demonstrates excellent generative capabilities. For example, on the Lcc and RepoBench-P datasets, CoreInfer experienced less than 3% performance loss even with a 20% neuron sparsity level.
2. Variation in Stabilization Length Across Tasks. Different tasks exhibit varying input lengths required to reach stability. In Appendix Figure 11, we visualize the stabilization process for code and Chinese characters. We found that, compared to typical English text, both code and Chinese characters require a longer token length to achieve stabilization (around 300 tokens for code and 500 for Chinese characters). We speculate that this is due to the unique formatting of code, which often requires more complete code blocks to represent semantics. Meanwhile, Chinese characters, potentially due to byte-fallback effects, require longer token lengths to convey the same meaning as English text.

Dear Reviewer aw95,

We sincerely appreciate your valuable feedback and insightful suggestions on our work. In our updated version, we have incorporated your suggestions by adding evaluations of our method on tasks with different input formats, and we have further explored its relationship with tokenization.

We are truly grateful for your positive feedback, recognizing our approach as innovative. If you have any further questions, we would be more than happy to provide additional information!

Thank you once again for your valuable perspectives.

Sincerely,

The Authors

Q2. Performance comparison with predictor-based methods.

A2. We sincerely appreciate your suggestion. We compared the performance of CoreInfer and PowerInfer on four commonsense reasoning tasks, and included the experimental results in Appendix A.4.2. The experimental results for PowerInfer are taken from the original paper. We observed that both CoreInfer and PowerInfer achieved near-lossless performance on these four tasks. However, PowerInfer requires additional predictors for training and inference, which increases the overall cost. For example, for OPT-30b, PowerInfer requires approximately 10.45GB for the predictors, which accounts for about one-sixth of the total model size.

Q3. Clarification of experimental details.

A3. Calculation of Semantic Similarity. In Fig. 3, the semantic similarity presented is calculated using Sentence-BERT. Specifically, for two sentences, we encode each sentence using Sentence-BERT to obtain their embeddings, and then calculate the cosine similarity between these embeddings as their semantic similarity.

Determining Input Stability. In our experiments, we determined stability based on the task type: for short prompt zero-shot QA and translation tasks, we used similarity-guided prediction, while for other tasks, we used stability-guided prediction. In practical usage, we determine the input stability based on the similarity of the core neurons activated by the last few tokens of the input. If there is a high overlap, the input is considered stable. As shown in Fig. 3(c), core neurons remain almost unchanged when stability is achieved.

We have added additional clarifications to these two sections in the paper to ensure clearer expression.

Once again, we would like to express our sincere gratitude for your valuable suggestions and insightful comments! It was a pleasure discussing our work with you. If you have any other concerns or questions, do not hesitate to let us know. We look forward to your response and are more than happy to address any additional questions you may have.

Best regards,

Authors

Dear Reviewer,

We would like to sincerely thank you for taking the time to review our paper and provide valuable feedback. We are thankful to have the chance to address your questions and concerns.

Concern 1. The general applicability of the proposed method.

Responce 1. Thank you so much for raising this concern and providing your valuable suggestions. We appreciate this opportunity to provide more explanations and address your concerns:

1. We think that the techniques commonly used in real-world LLM applications, such as prefix prompting and input preprocessing, would make user inputs more informative and semantically stable, thereby improving the effectiveness of our method. In practice, large-scale LLM-based systems like ChatGPT often add specific prefix prompts and preprocess user inputs to guide the model in generating more aligned responses $1$

$2$ . These prefix prompts and preprocessing steps are usually carefully designed to provide information such as context, persona, and requirements, as well as to correct grammatical errors and eliminate semantic redundancies. These techniques make the input semantics more stable and informative. In such cases, CoreInfer can more effectively predict the activated neurons. Thus, we think that the stability of core neurons identified by CoreInfer is consistent with the effectiveness of these widely used techniques: the gradual stabilization of core neurons explains why adding prefix prompts and preprocessing inputs helps clarify semantics, which in turn allows CoreInfer to work more effectively in real-world scenarios. 2. We further explore CoreInfer in challenging scenarios to verify its applicability in real-world applications or more challenging questions. To further explore and validate CoreInfer's performance in challenging scenarios, we added experimental results on the LongBench dataset, as shown in Appendix A.4.1 and Table 5. We found that CoreInfer performed excellently across key long-text application scenarios, including single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. This demonstrates that CoreInfer is capable of handling challenging and complex real-world tasks. We provide the experimental results in part 3. 3. In real-world deployment scenarios, we can also implement additional strategies to manage extreme inputs. For instance, we can restart the process or enhance monitoring if significant semantic shifts are detected. To enhance the robustness of our method when applied in real-world settings, we can consider adding extra components or checks to handle extreme input scenarios. For instance, we could adopt a monitor component to track semantic changes and recompute core neurons if there are significant changes in semantics. These approaches can make our method more feasible in engineering applications. In the revised version, we added a discussion on this topic in Appendix A.1 under future work. Thank you for raising this concern. 4. In practical scenarios, certain user habits could also contribute to stabilizing input semantics. For example, users tend to ask questions that are closely related within a single conversation. During the daily use of LLMs, users typically provide context and then ask specific questions based on that context to ensure that LLMs can effectively address their queries. More informative and highly relevant inputs make it more likely for the model to provide satisfactory answers. In practice, this usage habit also leads to more stable input semantics, making CoreInfer more effective.

Reference:

$1$ Achiam, Josh, et al. "Gpt-4 technical report." arXiv preprint arXiv:2303.08774 (2023).

[2] Brown, Tom B. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).

Dear Reviewer 55rr,

We hope this message finds you well. We sincerely appreciate your valuable feedback and insightful suggestions on our work! Given the limited time available for author-reviewer discussions, we would be very grateful if you could let us know whether our responses have sufficiently addressed the issues identified in your second review.

You raised four major questions: (1) the general applicability of the proposed method, (2) the assumption of similarity-guided prediction being too strong, (3) the need for more experiments on long-context tasks, and (4) the need to compare with predictor-based methods.

In our response, we have provided detailed explanations and clarifications regarding your first two concerns. Additionally, we have verified the performance of our method on 21 tasks from the LongBench dataset you suggested, and we have compared our method's performance with other predictor-based methods.

We deeply appreciate the significant time and effort you have devoted to reviewing our work and are grateful for your additional insights. Your comments have been immensely helpful in refining our paper.

Thank you once again for your valuable perspectives. We look forward to any further guidance you may have.

Sincerely,

The Authors