We thank reviewer BPFc for their insightful feedback! Below, we address the comments from the Weaknesses section in detail. We will incorporate your valuable feedback into the revised manuscript.

---

Weakness 1: Clarity on LowRA’s Components

Reviewer Concern: The paper would benefit from more explicit mapping between the three limitations tackled and the different components of the LowRA framework.

Our Response: We will clarify how each component of the LowRA framework aligns with each of the three limitations. Specifically:

1. Limitation 1 (Coarse-Grained Precision Assignment)

   • P2: Precision Assigner
   • T3: Precs

2. Limitation 2 (Discrepancy in Data Distribution)

   • P1: Mapping and Threshold Learner
   • T2: Fine-Grained Mappings and Thresholds
   • P4: Low-Rank Initializer and T5: Intelligently Initialized Low-Rank Tensors
     (the low-rank initialization helps mitigate quantization errors)

3. Limitation 3 (Lack of High-Performance Quantization Primitives)

   • P3: Output-Channelwise Quantize Kernel
   • P5.1: Output-Channelwise Dequantize Kernel

We will include both graphical annotations and supporting text in the paper to further illustrate these mappings.

---

Weakness 2: Fine-Tuning with Data in Embedded Systems

Reviewer Concern: Many approaches favor data-free post-training for generalizability. However, in embedded systems, fine-tuning with data can be more practical. How can LowRA adapt to such scenarios?

Our Response: We interpret your comment as referring to scenarios where there is some prior knowledge of the tasks for which the LoRA base weight will be used. In embedded systems, these base weights might be:

Case 1: Dedicated to a Single Task

• Using Task-Specific Data for Post-Training Quantization (PTQ).
  If the base weights are used solely for one task, that task’s dataset can serve as a calibration set to learn precision assignments, thresholds, mappings, and rounding schemes.

• Quantization-Aware Training (QAT).
  Another possibility is QAT. Preliminary experiments, however, showed that QAT did not outperform PTQ in terms of perplexity/accuracy—likely because LoRA’s low-rank adapters help compensate for quantization errors. Moreover, QAT demands more memory and introduces latency overhead.

Case 2: Shared Across Multiple Tasks

• Representative Calibration Set for PTQ.
  If multiple tasks share certain characteristics, a representative calibration set may be synthesized for PTQ to learn shared precisions, thresholds, mappings, and rounding schemes.

• Possible QAT Approaches.
  While still possible, QAT in multi-task scenarios faces the same drawbacks regarding memory and computational overhead. Moreover, the learned precisions, thresholds, mapping, and rounding schemes may overfit the calibration set and fail to generalize to each individual downstream task.

Other Scenarios and Future Directions

• Encoding vs. Decoding Schemes.
  One strategy is to fix encoding (thresholds) globally and fine-tune only the decoding (mappings) per task. However, preliminary implementations showed significant latency overhead (e.g., scatter-add operations).

• Future Extensions.
  We will continue exploring these ideas, focusing on deeper comparisons between PTQ and QAT under varying deployment constraints and use cases.

---

Conclusion

We appreciate your feedback and have outlined our approach to making LowRA clearer and more applicable to embedded scenarios. Your comments have inspired further investigations and future directions, which we look forward to sharing in subsequent work.

We thank reviewer oGG6 for their thorough review. We are glad that they find our work valuable overall. Below, we address the key concerns. We appreciate your feedback and will integrate it—along with additional experimental findings—into the revised manuscript.

---

Training Overhead

We benchmarked LowRA at various bit widths against QLORA (4 bits) on both an A5000 (24GB VRAM) and A100 (80GB VRAM).

Llama 7B A500, Runtime (ms)

Llama 13B A5000, Runtime (ms)

Llama 7B A100, Runtime (ms)

LowRA introduces minimal overhead (with a maximum of 8.52%) and supports configurations QLORA cannot run without out-of-memory (OOM) errors.

---

Results on Ultra-Large Models

We evaluated LowRA on LLaMA-65B (1.15 bpp), achieving perplexities of 7.49 (WikiText) and 4.97 (OpenAssistant), outperforming LLaMA-33B’s 8.00 and 5.73 under identical hyperparameters. This confirms LowRA’s scalability to ultra-large models.

---

Variance Trends Across Different Layer Types

We will supplement our paper with fine-grained plots motivating output-channelwise quantization.

---

Training Instability

Even at 1.15 bits, we observed no instability or divergence, largely thanks to (1) LoftQ’s alternating SVD-based initialization of low-rank tensors and (2) freezing base weights while only updating low-rank tensors. We will include stability discussions in our paper.

---

Inference implications

Inference implications involve three primary aspects: Loading Latency, Time to First Token (TTFT), and End-to-End Throughput. We discuss each part here.

1. Loading Latency
   This is the time to load model weights into GPU memory. On an RTX A4000 (LoRA branch in float32), LowRA reduces loading latency significantly, as shown in the table:

   Framework	Bits/Param	Loading Latency
   QLoRA	4	220.73 s
   LowRA	4	218.17 s
   LowRA	1.5	157.51 s (28.6% reduction)

2. TTFT
   TTFT is the delay before the first token is produced. While smaller models transfer less data and slightly reduce TTFT, it's primarily compute-bound, so quantization alone provides modest gains.

3. End-to-End Throughput
   Once the model is loaded and past TTFT, throughput (tokens or requests per second) becomes the key factor.

   Llama 7B on 1 RTX 3080 (unit: tokens / sec; float32)

   Prefill Length	Decode Length	1.5 Bit	2 Bit	2.5 Bit	4 Bit	QLoRA (4 Bit)
   100	10	32.16	30.33	24.35	9.19	9.40

   Llama 13B on 1 A4000 (unit: tokens / sec; float32)

   Prefill Length	Decode Length	1.5 Bit	2 Bit	2.5 Bit	4 Bit	QLoRA (4 Bit)
   100	10	16.11	14.46	13.64	12.17	11.56

   Across Llama 7B on an RTX 3080, 1.5-bit LowRA delivers a 3.42× throughput increase over QLoRA (32.16 vs. 9.40 tokens/sec), while on Llama 13B using an RTX A4000, 1.5-bit LowRA still achieves a 1.39× speedup—demonstrating that sub-2-bit quantization offers clear performance gains across varying model sizes.

These results confirm that LowRA provides real-world throughput gains beyond the documented memory savings.

We thank reviewer 7mTX for their thorough and insightful review. We are glad that 7mTX finds our empirical gains convincing and our paper easy to read and follow. We address all feedback below and will incorporate it, along with new experimental findings, into the revised paper.

---

Q1: Inference metrics on consumer-grade hardware

Our experiments target GPUs that are relatively constrained for large language model (LLM) inference—an NVIDIA RTX 3080 with 10GB VRAM and an NVIDIA RTX A4000 with 16GB VRAM—to underscore how LowRA can unlock efficient inference without requiring high-end data-center hardware.

Llama 7B on RTX 3080 (unit: tokens / sec; float32)

• At 1.5 bits, throughput reaches 32.16 tokens/sec, yielding a 3.42× speedup over QLoRA (9.40 tokens/sec).
• Even at 2 bits, LowRA surpasses QLoRA by over 3×, indicating that more aggressive quantization pays off on memory-limited hardware.

Llama 13B on A4000 (unit: tokens / sec; float32)

• For Llama 13B, 1.5-bit LowRA achieves 1.39× speedup over QLoRA, and 2-bit still delivers a 1.25× improvement.
• Though the gains are smaller than on the 7B model—likely due to higher overall computation overhead—LowRA still outperforms QLoRA across all tested bit-widths.

These results confirm that our sub-2-bit LowRA quantization provides real-world throughput gains beyond the documented memory savings.

---

Q2: The tradeoff of model size and compression ratio.

In practice, users often start with a specific model, a fixed hardware memory budget, and particular tasks in mind. LowRA is designed to help maximize performance under those given constraints.

While we do observe that ultra-low-bit quantization can make it feasible to run larger models within limited memory, our current study does not fully investigate the potential trade-off between choosing a bigger model at extremely low bits (e.g., 13B at 1.5 bits) and opting for a smaller model at higher bits (e.g., 7B at 2-4 bits).

We see further exploration of this bits-per-parameter versus performance trade-off - along with automatic selection of the optimal model-compression combination - as an important direction for future research. Moreover, our work serves as a stepstone for future works to further optimize the task performance of models in the lower bit range.

---

Presentation Errors: Graphs, Text, and Algorithms

Thank you for your attention to details! We have fixed all of the presentation errors you pointed out. We will iterate through our draft rigorously to ensure our presentation is accurate.

1. Typographical and Spacing Fixes

   • Corrected spacing for “Data-Free Post-Training Quantization.”
   • Fixed spelling errors (e.g., “Quantzation” → “Quantization,” “Summarization Summarization” → “Summarization”).

2. Figure and Label Updates

   • Corrected the mislabeled memory footprint for Llama 33B in Figure 9.
   • Standardized references to “Llama-33B” (instead of “Llama-30B”).
   • Adjusted the y-axis to a full-scale view in Figures 7 and 8, ensuring a clearer comparison of perplexity scores.

3. Algorithm Clarifications

   We have added clarifications to the algorithm. Specifically, $N$ refers to the total number of output channels in the model being quantized. The values $\{ w^{(1)}, \dots, w^{(K)} \}$ represent the distinct parameter sizes (i.e., the number of parameters) across all output channels. The set $I_k$ contains the channel indices belonging to size group $k$. Each channel $i \in I_k$ has a parameter count of $w^{(k)}$.

   We define:

   $$W_k = \sum_{i \in I_k} w^{(k)},$$

   which is the sum of parameter counts for all channels in partition $I_k$. Finally:

   $$W_{\text{sum}} = \sum_{k=1}^K W_k$$

   is the total parameter count across all partitions (i.e., the total number of parameters in the model to be quantized).