PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS

Thanks for your feedback!

Lack of explanation on how the automated testing is performed end-to-end.

We realized the original description lacked detail about the system design, particularly how our framework performs end-to-end benchmarking. To address this, we updated Figure 1 with more detail and highlighted additional explanations in the main text using blue text. We invite you to review the updated draft.

Furthermore, additional discussions and results have been included in the Appendix.

Unclear how easy it is to support new mobile devices or new LLMs.

Our benchmark framework is designed to be flexible and extensible for additional models and quantization methods. Currently, we use two popular frameworks—llama.cpp and MLC—which provide pre-compiled models and quantization tools to assist users in model quantization. We have tested mainstream models supported by these frameworks, but our benchmark can accommodate more models, quantization schemes, and frameworks as they become available.

Some quantization methods are not yet supported by llama.cpp and MLC, or these frameworks may not support certain models on mobile devices. Once such support is added, our benchmark can seamlessly evaluate these models across devices.

Additionally, if new frameworks emerge for running LLMs on mobile devices, we can adapt them to integrate with our benchmark.

Also, do users need external equipment to fully use the benchmark? For example, in Section 3.3.7, the paper mentions that thermal behavior analysis is done using a FLIR C5 thermal imaging camera and a professional USB power meter.

Yes, we use a USB-based power monitor from Klein Tools to measure energy usage for analyzing power consumption in connected edge devices, particularly for power cable-connected devices like Orange Pi and Jetson Nano. Conversely, FLIR is used exclusively for thermal analysis, providing insights into spatial temperature distribution. However, FLIR is not a requirement for running benchmarks with our framework. While power consumption has a strong, deterministic relationship with temperature, FLIR imaging highlights thermal heterogeneity across the device surface. This is crucial for identifying hotspots that may impact user comfort (e.g., in contact areas like the back of a phone) or hardware reliability.

Modern mobile devices feature a System-on-Chip (SoC) with an integrated Power Management Unit (PMU) to monitor power and battery usage. Profiling tools like Xcode Instruments for iOS and AGI for Android leverage the PMU to provide accurate battery consumption data.

Can we evaluate the LLMs on mobile devices using CPU-only mode, or does the framework require the devices to have GPUs?

llama.cpp and MLC fully support CPU-only model inference; however, our focus is on mobile phones and edge devices with comparable performance. These devices are power-sensitive, and relying on CPU-only inference for LLMs would result in extremely slow performance, making the results and latency measurements less meaningful.

... could we obtain detailed layer-by-layer profiles and analyze what is the bottleneck operation?

Yes. To identify bottlenecks in running LLMs on mobile and edge devices, we added the results of memory breakdown analysis (weights, KV cache, activations and overhead) and layer-by-layer GPU utilization. The memory breakdown highlights the contributions of weights, KV cache, activations, and overhead, influenced by quantization bits and algorithms. GPU profiling across two transformer blocks reveals that self-attention layers achieve over 95% GPU utilization, potentially slowing inference, while FFN layers use significantly less. These insights guide optimizations to enhance model performance.

The figures have been added to Appendix.

Do you have evaluation numbers for prefill and decode throughput respectively?

Yes. Previously, we used a general throughput figure to save page space. Now, Figure 7 presents the prefilling and decoding throughput results for multiple models.

batch size

Maximal batch size of LLM is 80

How is the exact match and F1 score calculated?

Exact Match is calculated as the percentage of predictions that exactly match the ground truth answers. The formula can be expressed as:

Exact Match (EM) Formula

$$\text{EM} = \frac{\text{Number of Exact Matches}}{\text{Total Number of Examples}} \times 100$$

Where:

Exact Match: A prediction is considered an exact match if it exactly matches the ground truth answer after normalization (e.g., removing punctuation, lowercasing, or stripping whitespace, depending on implementation specifics).

Total Number of Examples: The total number of examples in the dataset.

F1 score is a metric used to evaluate a model's performance, particularly in classification tasks. It is the harmonic mean of precision and recall, and it balances the trade-off between the two.

$$F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

Where:

$$\text{Precision} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Positives (FP)}}$$ $$\text{Recall} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Negatives (FN)}}$$

For multi-class or multi-label classification, averaging strategies (e.g., micro, macro, weighted) can be used to calculate the overall F1 score across all classes.

In Section 4.4, it says "Interestingly, the 5-bit and 3-bit models underperformed slightly. K-quant (ggml) 3-bit model produced more hallucinations and toxic content than the 2-bit model." How are hallucination and toxicity observed in Figure 8?

The appendix provides some samples of Hallucination and Toxic outputs, which means lower bit quantization especially 3-bit group-wise (GGUF) produces more contents that are incorrect, randomized, toxic, or not related to the datasets.

This issue is likely attributed to the llama.cpp framework and the specific quantization algorithms employed. Typically, 2-bit and 3-bit quantization underperform when compared to 4-bit quantization. Furthermore, 5-bit and 6-bit quantization, which are supported exclusively by llama.cpp, utilize the group-wise k-quant method.

Some studies have dived into the causes of hallucinations in LLMs [1][2], and researchers continue to explore this area. Notably, [2] shows that hallucinations in calibrated models are unavoidable, although calibrated models exhibit lower toxicity.

[1] Bao, F., Li, M., Luo, R., & Mendelevitch, O. (2024). HHEM-2.1-Open. doi:10.57967/hf/3240 [2] Xu, Z., Jain, S., & Kankanhalli, M.S. (2024). Hallucination is Inevitable: An Innate Limitation of Large Language Models. ArXiv, abs/2401.11817.

Thanks for your valuable feedback.

We noticed that the existing description lacked details about the system design, particularly how our framework performs benchmarks in an end-to-end manner. To address this, we revised Figure 1 and added additional details. We invite you to review our updated draft.

The Benchmark Automation Implementation details are lacking, it's hard to know how PalmBench to process different hardware platform benchmark query.

We have updated Figure 1 to include more details on the workflow and automation. We recognized the need to clarify these system-level implementations and make the revision.

There is already have Benchmark [1] for evaluating LLM on Edge-Device, which should be properly discussed in this paper.

We have discussed the NeurIPS competition starting kit framework in our Related Work section.

Could you provide a detailed description showing how AGI, custom apps, and LLM frameworks interact in your automated system? (e.g.,What's the exact interface between your custom apps and the LLM frameworks (MLC/llama.cpp)?

Yes, we acknowledged that Fig. 1 lacked the necessary hierarchical structure, which may have led to confusion. We have revised it and added details to explain how our framework automation operates.

Our benchmarking framework automates workflows and collects profiling data via multiple interfaces, coordinating programs on both PC hosts and mobile/edge devices. The Evaluator extracts prompt from datasets and evaluate outputs using mainstream benchmark datasets. While frameworks provide limited pre-compiled models, quantizating own models is often required. The Scheduler evaluates quantized models with prompt input from Evaluator, and monitors resource usage through profiling tools like Xcode Instruments for iOS and Android GPU Inspector (AGI) for Android.

We have uploaded some datasets and module testing scripts as supplemental materials, which are not complete code libraries. Additionally, we plan to open source all testing scripts and frameworks, including firmware for edge devices, mobile apps for iPhones, and AGI used on Android devices. Due to upload size limitations, the firmware and mobile app project files could not be included as supplemental materials, but we will make all these files publicly available to ensure the reproducibility of our experiments.

NVTOP was previously used for Nvidia Jetson devices; however, we have transitioned to using jetson-stats and no longer rely on NVTOP.

How you process user's query to evaluate the model? should it must have llama.cpp support, what if my model is not in llama.cpp model zoo?

The frameworks (MLC and llama.cpp) need to be deployed on mobile and edge devices to enable efficient on-device inference. Each device can use one of the two frameworks.

Our benchmark is flexible and extensible, supporting additional models and quantization methods. Currently, it utilizes llama.cpp and MLC, which offer pre-compiled models and tools for quantization. While we have tested mainstream models supported by these frameworks, the benchmark is designed to accommodate more models, quantization schemes, and frameworks as they become available.

Some quantization methods or models are not yet supported by llama.cpp or MLC on mobile devices. As support expands, our benchmark can seamlessly evaluate these models across devices.

Furthermore, we can integrate new frameworks for running LLMs on mobile platforms into our benchmark as they are developed.

Dear Reviewers,

As the author-reviewer discussion phase comes to a close, we have addressed all the questions raised in the rebuttal. If our responses address your concerns, we kindly request you to consider adjusting your score. We are happy to engage in further discussions regarding our paper if needed.

Thank you, The Authors

Thanks for your comments! I really appreciate you reading our paper so carefully.

6. Evaluation

In Figure 3, the GPU utilization across phones is quite large, which comes in contrast with the "memory bounded" workload claim of the authors. I would appreciate some more insights here

Thank you for your thoughtful observation regarding GPU utilization in Figure 3. The "memory-bounded" claim aligns with the community consensus, emphasizing limited memory bandwidth as the primary bottleneck in such workloads. However, our focus on edge and mobile devices uncovers unique hardware constraints and architectural factors, resulting in high device GPU utilization. It is possible to have bottlenecks on edge devices.

We conducted a detailed analysis of memory breakdown and layer-by-layer GPU utilization to identify bottlenecks in running LLMs on mobile and edge platforms. The memory breakdown highlights the contributions of weights, KV cache, activations, and overhead, influenced by quantization bits and algorithms. GPU profiling across two transformer blocks shows that self-attention layers always have over 95% GPU utilization, potentially slowing inference, while FFN layers use significantly less. These findings provide actionable insights for optimizing model performance.

Thank you for raising this concern. Our evaluation focuses on measuring throughput under typical real-world conditions for each device. iPhones demonstrate significantly higher throughput in such scenarios due to their optimized hardware and software integration.

However, for both llama.cpp and MLC, memory usage fluctuations are minimal and nearly consistent. This resulted in very short and indistinct boxes, making the visualization less intuitive, even with varying prompt inputs. This consistency highlights the minimal variations in memory usage during local inference, encompassing both the framework and the model. Therefore, the boxplot may not be the most effective representation in this case.

I believe the reason lies in the large memory footprint required to load the LLM, while the size of the prompt text is relatively small compared to different tasks.

7. Related Work

Thank you for highlighting MELT, which we have also discussed in our related work. The key distinction between our work and MELT lies in our evaluation's broader scope and depth. Our study tested a wider range of models and quantization methods across various devices. Additionally, we explore critical aspects of user experience, such as temperature, hallucination, and text toxicity, which were beyond MELT's focus. While MELT primarily analyzed 3-bit and 4-bit quantizations, our work extends to 8-bit, 6-bit, 5-bit, and 2-bit quantizations, addressing scenarios relevant to future mobile applications.

Palmbench does not integrate high-end edge devices (e.g. Jetson AGX) and different power profiles, which prior work does.

Thank you for bringing up the Jetson AGX. High-end edge devices like the Jetson AGX offer significantly higher performance but also consume considerably more power, making them outliers beyond the scope of this work. Our focus is primarily on devices with relatively low power consumption that are closer in capability to mobile phones. High-end edge devices are more comparable to PCs in terms of performance and energy usage.

Thank you for your valuable feedback! As suggested, we have moved the captions for Tables 1 and 2 above the tables. And added mentioned references!

While some metrics and evaluation tasks in our benchmark paper have been addressed in prior work, a dedicated benchmark framework, combined with results from broader models and quantization methods, provides significant value to the research community. This aligns with the ICLR Call for Papers emphasis under the "Benchmark and Datasets" topic. Our benchmark uniquely addresses a broader range of datasets, metrics, and testing scenarios, focusing on critical yet underexplored aspects such as hallucination and toxicity. These factors are essential for evaluating user experience and are increasingly crucial as LLMs are deployed in real-world applications, where reliability and safety are paramount.

Energy efficiency evaluation goes beyond the software interfaces provided by Google AGI and Apple Xcode. These official profiling tools access battery usage data directly from the Power Management Unit (PMU) in the device's SoC, with mobile phones connected via USB for data transfer. As detailed in the paper, we also utilized professional USB-based power consumption instruments from Klein Tools.

We agree that a more comprehensive background would strengthen the paper, and we have added the references you suggested to the section with the necessary discussion.

4. Motivation We acknowledge that quantized local LLMs may have reduced reliability in accuracy metrics. However, running LLMs locally enhances privacy and reliability by removing dependence on network connectivity and cloud resources. This approach ensures consistent performance, reduces latency, and avoids disruptions from network instability, server downtime, or data transmission errors, thereby improving system robustness.

5. Methodology

We have uploaded datasets and module testing scripts as supplemental materials. Furthermore, we plan to open-source the entire framework code library, including firmware for edge devices, mobile apps for iPhones, and AGI for Android devices. However, due to upload size constraints, the firmware and mobile app project files were not included as supplemental materials. These files will be made publicly available to ensure the reproducibility of our experiments. We will make all project files, including Mobile Apps (IOS, Android), Linux scripts, and interfaces, publicly available to ensure the reproducibility of our experiments.

All tested devices were equipped with the latest OS supported by the devices (IOS 17.6.1, Android 15, Ubuntu 14.04.06 LTS) and firmware versions. We have included these details in the main text, table in Appendix and updated Figure 1 to provide additional clarity.

Experiment repeated: All experiments were conducted 10 times to ensure consistency, with identical settings for both frameworks.

Model Compression and Conversion: The two tested frameworks, MLC and llama.cpp, offer tools such as the MLC quantize command and quantize.cpp to assist users in quantizing models. While many pre-compiled models are available from official repo, some models still need to be compiled by ourselves.

We verified no performance difference when using profiling tools. These profiling tools are non-intrusive, do not interfere with device performance, and use USB only for transferring minimal profiling data.

6. Evaluation

Decoding Strategy: Although llama.cpp and MLC are popular for edge and mobile devices, their decoding parameter configuration is limited. To ensure consistency, we used nucleus decoding with identical hyperparameters: temperature set to 0.2 for output randomness and Top-S set to 0.9, with Top-K left optional. Beam search decoding, which could improve output quality, is not yet supported in llama.cpp.

The average temperature is the average value of surface temperatures recorded by the instrument during model inference.

For Figure 2, we tested 3-bit and 4-bit quantization for MLC, and 2-bit to 5-bit for llama.cpp. Regarding memory bitwidth, all evaluated devices use a 64-bit architecture.

We evaluated the Mistral-7B-3bit and Phi2-4bit models, which are similar in size. They are discussed in the throughput subsection. Models with fewer parameters and higher-bit quantization, such as Phi2-q4f16, decode faster and use less memory, including KV cache and overhead. This efficiency highlights the suitability of smaller models like Phi2 for mobile devices. The results have now been included in the main text and Appendix.

7. Related Work

While MELT[1] presented GPU activity during model inference in Fig. 9, it was limited to only the iPhone with Zephyr-3B (4-bit).

In MELT[1], Tab3 indicates that llama.cpp only run on Android and Linux, not including IOS.

[1] Laskaridis, Stefanos et al. “MELTing point: Mobile Evaluation of Language Transformers.” ArXiv abs/2403.12844 (2024): n. Pag.

Dear authors,

Thank you for your detailed rebuttal. While I appreciate your response, there are pending issues that prevent me from changing my current view on the paper. Concretely:

• On novelty, I continue to believe that the contributions of the paper are limited to the evaluation of various aspects of compression on hallucinations/toxicity, which are indeed important aspects in reliability and safety, and the additional quantization levels. Compared to existing work, this is more incremental than initially presented.
• On the insights of GPU utilization vs. memory-bounded workloads, I am not truly convinced by the provided commentary.
• On energy measurements, I still fail to see how battery draw can be measured from USB interfaces. I would also appreciate if you could provide a reference to the PMU-based software energy measurement?
• On system robustness, the authors should clarify what aspects they are focusing on when discussing on-device robustness in the paper. The relative robustness between cloud and on-device has not been experimentally shown or referenced.

Moreover, could the authors describe how they verified that profiling does not impact the runtime behaviour?

Last, the authors should more carefully represent the details of related work, as they misinterpret aspects of [1]. If I understand correctly, Tab. 3 shows LLMFarm (https://github.com/guinmoon/LLMFarm - a llama.cpp frontend for iOS) which supports llama.cpp GPU/CPU runtime on iOS.

Thank you for your timely response.

1. For energy measurement.

I understand that you may have reservations about the accuracy of power consumption data obtained through power rails or the On-Device Power Monitor (ODPM) from official profiling tools. Let me clarify how power consumption data is measured via power rails, using the example of the Android Pixel 7, which employs (On-Device Power Monitor) ODPM.

Unfortunately I cannot include figures here, I would be happy to provide links to the official Google Android documentation on ODPM and the datasheet for the PMU used in these devices[2]. These references will further validate the accuracy of our power measurement method [1][2][3][4][5]. All of these references are technical documents from official websites.

References [3][4][5] introduce the Power Profiler from the Android Development Kit, which leverages ODPM to obtain precise power consumption data.

ODPM measures power consumption at the device level—not specific to any app. Power Profiler reads power consumption data from the ODPM, which is only available on Pixel 6 and subsequent Pixel devices running Android 10 (API level 29) and higher.

ODPM is essential for accurate power consumption measurement under battery power, as detailed in the official Android documentation on power rails and charging[4]. When connected to a USB cable, devices measure charge flow into the battery (i.e., charging), leading to positive current readings and increased total charge. This distorts power data, reflecting charging activity rather than actual GPU power usage. Moreover, USB connections often trigger high-performance modes, unlike the low-power modes typically used on battery. As a result, USB-based measurements are unsuitable for accurately analyzing LLM inference on GPUs.

As described in the document:

Battery counters measure the charge flowing in and out of the battery. If the device is plugged to a USB cable, you will likely observe a positive instantaneous current and an increase of the total charge, denoting the fact that charge is flowing in the battery (i.e. charging it) rather than out.

This can make measurements in lab settings problematic.

Why ODPM is more accurate?

Recent version of Android introduced the support for more advanced power monitoring at the hardware subsystem level, known as “On-Device Power Rail Monitors” (ODPMs). These counters measure the energy drained by (groups of) hardware units. Unlike the battery counters, they are not affected by the charging/discharging state of the battery, because they measure power downstream of the battery.

ODPM differs from battery counters as it directly uses on device hardware units like the Power Management Unit (PMU) to measure power consumption. Reference [4] specifically mentions the power monitoring chip, which refers to the PMU.

The PMU provides direct and accurate battery power measurements, which external cabled tools cannot replace[2][3][4]. Connecting external devices can alter the phone's power management state or change its power supply behavior, leading to inaccurate results, as indicated in [3].

We use AGI [1] as it offers a more advanced power profiling capability compared to Android Studio, fully leveraging the specifications of ODPM for precise measurements.

Here I provide one widely used PMU[2] datasheet.

[1]Android GPU Inspector (AGI): https://developer.android.com/agi

[2]The datasheet of the PMU used on the device: https://www.nxp.com/docs/en/application-note/AN4199.pdf

[3]Android Power Profiler: https://developer.android.com/studio/profile/power-profiler

[4]Google OPDM: https://android.googlesource.com/platform//external/perfetto/+/b4a6d11ec1c12f391c3f9dab2be4f3b5152994f2/docs/data-sources/battery-counters.md

[5] Android Power Monitor: https://developer.android.com/reference/android/os/PowerMonitor

On the insights of GPU utilization vs. memory-bounded workloads, I am not truly convinced by the provided commentary.

The data we measured comes from Google's official AGI tool and MLC's internal status printing messages(the newest provides detailed information, such as KV cache memory and overhead by default). We believe these are among the most reliable and accurate data sources available. If you are aware of any more convincing alternatives, we would greatly appreciate your suggestions. Thank you for your thoughtful response.

On system robustness, the authors should clarify what aspects they are focusing on when discussing on-device robustness in the paper. The relative robustness between cloud and on-device has not been experimentally shown or referenced.

We understand the importance of comparing robustness between cloud and on-device systems. In our paper, the discussion on on-device robustness primarily focuses on the ability of the device to operate independently of network connectivity, ensuring consistent performance even in offline scenarios, and the reduced risk of data breaches by processing locally.

Unfortunately, we did not have the resources or funding to conduct a large-scale cloud-based benchmarking study to directly compare robustness between cloud and on-device systems. We acknowledge this as a limitation of our work and have revised the statement in the paper to clarify our focus and avoid overgeneralizations. Thank you for pointing this out, and we appreciate your understanding.

However, we cite two references from the systems community that examine LLM serving in cloud and data center environments, highlighting persistent challenges with workload management and latency issues [1][2].

[1] Hu, Q., Ye, Z., Wang, Z., Wang, G., Zhang, M., Chen, Q., … Zhang, T. (2024, April). Characterization of Large Language Model Development in the Datacenter. 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), 709–729.

[2] Wang, Y., Chen, Y., Li, Z., Tang, Z., Guo, R., Wang, X., ... & Chu, X. (2024). Towards Efficient and Reliable LLM Serving: A Real-World Workload Study. arXiv preprint arXiv:2401.17644.

Thank you for your thoughtful comments, particularly on the experimental methods and results.

Reproducibility: Each experiment was repeated 10 times to ensure consistency, with all devices tested on the latest official firmware and system versions. Additionally, all devices were factory reset before testing to maintain a standardized environment. We have added these details in our revised version.

Open-Source: Furthermore, we plan to open-source the entire framework code library, including firmware for edge devices, mobile apps for iPhones, and AGI for Android devices. However, the firmware and mobile app project files were not included as supplemental materials due to upload size constraints. These files will be made publicly available to ensure the reproducibility of our experiments.

Given it was a heavily empirical paper, the authors did not clearly state whether the experiments were repeated multiple times to validate the results, or better, verified on multiple devices of the same family.

The performance of different models on various devices may deviate from expectations due to system and hardware compatibility, as well as algorithm differences. These results are crucial for understanding the practical challenges and considerations of real-world deployment.

We realized that some expressions in the original text might have been unclear and confusing, so we have revised them for clarity. Additionally, we expanded the discussion in the Evaluation section to provide deeper insights, particularly through detailed profiling to identify bottlenecks in on-device LLM inference. Our key finding is that the performance difference between 5-bit and 4-bit quantization in our benchmark is minimal, but 5-bit quantization consumes significantly more resources. Overall, 3-bit performs better, though 2-bit shows superior results in certain cases. This outcome is unlikely due to experimental settings, as all devices were factory reset to the latest system versions, with the temperature parameter set to 0.2. Instead, it could be attributed to the llama.cpp framework and the specific quantization algorithms used. Additionally, differences in quantization algorithms themselves can affect performance. as illustrated in the papers[1][2][3][4]. Suboptimal quantization algorithms may have extra memory cost even in low bit quantizations. Moreover, differences in quantization algorithms and hardware-software compatibility may influence performance, causing deviations from expectations across models and devices. These findings highlight the practical challenges and considerations for real-world deployment of LLMs on edge devices.

[1] Liu, Yijun et al. “Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox.” ArXiv abs/2406.12928 (2024): n. Pag.

[2] Huang, Wei, et al. "How good are low-bit quantized llama3 models? an empirical study." arXiv preprint arXiv:2404.14047 (2024).

[3] Lu, Zhenyan et al. “Small Language Models: Survey, Measurements, and Insights.” ArXiv abs/2409.15790 (2024): n. pag.

[4] Xu, Jiajun et al. “On-Device Language Models: A Comprehensive Review.” ArXiv abs/2409.00088 (2024): n. pag.

Dear Reviewers,

As the author-reviewer discussion phase comes to a close, we have addressed all the questions raised in the rebuttal. If our responses address your concerns, we kindly request you to consider adjusting your score. We are happy to engage in further discussions regarding our paper if needed.

Thank you, The Authors

Our evaluation indicates that 5-bit and 3-bit quantization exhibit a higher possibility to produce irrelevant, arbitrary outputs and hallucinations. This behavior is attributed to the problems of framework and the quantization algorithms. Below are some examples of incorrect outputs generated by 5-bit quantization:

Hallucination

Gemma-3B-q5 llama.cpp

User: Who was the 45th president of the United States, and what were their main achievements in ancient Greece?

AI:The 45th president of the United States was Abraham Lincoln, who was also known for his contributions to ancient Greek philosophy, particularly in mentoring Socrates and introducing democratic reforms in Athens.

Irrelevant Outputs

Llama3-8B-q5 llama.cpp LLMFarm (llama.cpp)

Users: Hi

AI: <end_of_turn>model <start_of_turn>print print("hello world") <end_of_turn> <start_of_turn>with open() <end_of_turn>with open('example.txt', 'w') as f: f.write('goodbye') f.write('goodbye') <end_of_turn> <start_of_turn>print("goodbye world")<end_of_turn> <start_of_turn>print(f"goodbye {i}")<end_of_turn> <end_of_turn> <start_of_turn>for i in range(1,10): <end_of_turn> print(f"goodbye {i}") <end_of_turn> <start_of_turn>for i in range(1,10): print(f"goodbye {i}"<end_of_turn> <end_of_turn> <start_of_turn>with open('example.txt', 'r') as f: <end_of_turn>print(f.read())<end_of_turn> <start_of_turn>print(f.read() + "\n")<end_of_turn> <end_of_turn> <start_of_turn>wget url="https://www.python.org/": <end_of_turn> print(wget)<end_of_turn> <start_of_turn>print(wget) <end_of_turn>...<end_of_turn> anagrams[char] = anagrams.get(char, 0) + 1 return anagrams <end_of_turn> <end_of_turn> <start_of_turn>def is_prime(n): <end_of_turn> if n <= 1: <end_of_turn> return False <end_of_turn> <end_of_turn> <start_of_turn>def get_prime_factors(n): <end_of_turn> factors = [] <end_of_turn> i = 2 <end_of_turn> while n > 1: <end_of_turn> if n % i == 0: <end_of_turn> factors.append(i) <end_of_turn> while i * i <= n n n n n n n n n n n n n n n n n …..n n+zone. in a n^2 some and<|Fun's the, and a with a| and I the. |n the a, the, n the, n't a, isill is the and the the be and., is is C_LIMIT isable is the, the the+ the, is and+ is+ a it the the is the the the is the the the to the the this'< the the the is is the is the of of the a it| the, a the the to the the is the, a is the is the is the the is the the is the is++ the+ thely to and the the is is andly that. the the< isore the it

Thank you for acknowledging the value of our benchmark paper and for your valuable feedback. We added the necessary details in the main text and the Appendix.

Zero-bit quantization[1,2] enables model compression without requiring access to original training data, addressing privacy concerns. Methods like ZeroQ [1] generate synthetic datasets to optimize batch normalization statistics and support mixed-precision quantization with minimal overhead. ZeroQuant [2] combines hardware-friendly INT8 quantization, layer-by-layer knowledge distillation, and optimized backends, achieving significant speedups and memory reductions for large models like GPT-J6B while maintaining accuracy. These approaches provide efficient, scalable solutions for deploying large models on resource-constrained systems.

The approximate thermal time constant for the system, spanning from the power dissipation source (primarily transistors) to the smartphone surface, is typically in the range of 0.6 to 5 minutes, depending on the device's thermal design, materials, and cooling mechanisms. This estimate accounts for the heat conduction through internal components and the dissipation to the external surface. In our setup, we observed stabilization within approximately 2-3 minutes, aligning with typical smartphone thermal dynamics.

This duration falls well within the runtime of a single test case on the phone. Model inference on mobile phones is not that fast.

Although 2-bit quantization generally underperforms compared to 3-bit and 4-bit quantization, certain datasets show 3-bit performance closely with 2-bit, as observed in other research studies[3][4]. This behavior is closely tied to the quantization algorithms used by llama.cpp. Previous work[4] has highlighted that RTN underperforms compared to GPTQ and AWQ, even when using higher-bit quantization. 3-bit quantization performed worse or not better than 2-bit under RTN and group-wise algorithms in some datasets.

[1] Yao, Zhewei et al. “ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers.” ArXiv abs/2206.01861 (2022): n. Pag. [2] Cai, Yaohui et al. “ZeroQ: A Novel Zero Shot Quantization Framework.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 13166-13175. [3] Liu, Yijun et al. “Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox.” ArXiv abs/2406.12928 (2024): n. Pag. [4] Huang, Wei, et al. "How good are low-bit quantized llama3 models? an empirical study." arXiv preprint arXiv:2404.14047 (2024).

Dear Area Chairs,

Thank you for your thoughtful review and for recognizing the contributions of our work on PalmBench. We sincerely appreciate the reviewers’ valuable feedback and the opportunity to improve our submission based on their insights.

We acknowledge the initial concerns regarding system design clarity and comparisons with existing benchmarks. We are glad that our additional explanations, diagrams, and planned open-sourcing efforts have helped clarify PalmBench’s architecture and its role in evaluating compressed LLMs on mobile devices. We also appreciate the recognition of our commitment to addressing hardware-specific challenges and real-world deployment data, which we aim to expand further in future work.

We are honored that PalmBench has been accepted for presentation at ICLR, and we are grateful for the constructive discussions throughout the review process. We will carefully incorporate the final suggestions to further enhance the completeness and impact of our work.

Once again, thank you for your time, support, and detailed assessment. We look forward to sharing PalmBench with the broader research community.

Best regards,