ULPT: Prompt Tuning with Ultra-Low-Dimensional Optimization

We thank the reviewer’s thoughtful comments, especially for recognizing our clear paper writing, effective parameter reduction through sketching, and useful GLUE experiments. We now provide detailed responses to each of the concerns.

“You still need to reconstruct the large matrix $\tilde{P}$ at inference time, so you do not have a memory advantage in this respect.”

We thank the reviewer for the comment. At inference, ULPT does reconstruct the full prompt embeddings, using the same memory as vanilla prompt tuning. However, ULPT targets LLM customization, where an enormous number of customized LLMs are stored but few are active at a time. ULPT significantly reduces the storage for customizations of foundation models, which is a novel use case.

“You need to consider the additional computational requirements of such computations” and “Very little evidence on the computational overhead introduced by the method in terms of time/memory.”

We appreciate this suggestion. To clarify, the computational overhead introduced by ULPT at inference time is minimal compared with the rest of the network, as the reconstruction for the prompt embeddings occurs only once for each loading of the model. We empirically compare the run time of ULPT’s up-projection against vanilla PT (the results are averaged on 100 runs).

Table 2

As seen in Table 2, the embedding up-projection is negligible relative to the decoding time. We will include the runtime analysis in the revised manuscript.

“LoRA weights can be simply merged with the model weights and no overhead at inference time is induced. So it seems to me the only advantage could be that this method requires even fewer trainable parameters at finetuning stage, but the limitations should be made much clearer in the text. ”

We acknowledge that our ULPT is different from LoRA, as we do not keep the original weight matrix. As mentioned in the previous point, the overhead caused by up-projection is negligible compared with the rest of the network (at most 0.3%). The advantages of our work include:

• As recognized by the reviewer, our method has much fewer trainable parameters than LoRA and other methods, which is crucial to the storage of massive customized LLMs.
• In addition to storage saving, our ULPT also combats the overfitting problem and achieves higher performance than full-dimensional prompt tuning and LoRA (Table 1 in our paper).

In the revision, we’ll clarify that we don’t merge the low-rank embeddings.

“many LoRA-like methods that scale better than LoRA have been proposed but not compared in the experiments.”, “VeRA, which is discussed but not compared empirically.” and “more reasoning tasks should be provided … such as Llama for autoregressive tasks”

We thank the reviewer for highlighting additional LoRA variants, and suggesting additional experiments on Llama for autoregressive tasks. In the rebuttal period, we included additional baselines VeRA and FourierFT, and added two generation benchmarks: GSM8K (math reasoning) and MBPP (code generation). We compare ULPT with those baselines using the Llama 3.2 models (1B and 3B).

Results are presented in Table 1 (rebuttal to Reviewer Ycju) due to the rebuttal space limit. We see that ULPT remains highly competitive, outperforming LoRA, VeRA, and FourierFT when the number of parameters is controlled. Importantly, LoRA and VeRA cannot achieve the ultra-low parameter usage as ULPT that is at the level of a few thousand parameters. We will include these comparisons in the revised manuscript.

“Other references include LISA and ReFT.”

Thanks for suggesting additional parameter-efficient fine-tuning methods beyond prompt tuning and LoRA. We will include discussions of LISA and ReFT in the related work section of our revised manuscript.

“Many equations lack a comma at the end”

Thanks for the suggestion. We’ll adopt a better style (including punctuation for equations) in our revision.

---

We believe these clarifications and additional results have addressed the reviewer’s concerns. We are grateful for the reviewer’s feedback, and look forward to your support of our work!

We thank the reviewer for their detailed feedback. We appreciate that the reviewer says “The claims are in general clear and convincing” and that “The experiment design involves multiple fine-tuning tasks”. Below we address each of the comments in detail.

“One issue regarding the claims is the introduction of shift embedding and the scale embedding.” and “it would be helpful if the authors could discuss the necessity of these two new variables theoretically”

Thanks for raising this point. Our empirical analysis (Section 4.3) demonstrates that without scale and shift embeddings, the optimization process becomes significantly more difficult, particularly in the ultra-low-dimensional setting (e.g., 2-dimensional prompts, Figure 3). Additionally, Figure 4 reveals that the learned shift embeddings exhibit high similarities across different r configurations, further justifying our heuristic design of the shift and scale embeddings. Theoretically, our Theorem 3 ensures that these additional embeddings do not negatively impact the optimization process. We’ll provide further explanation in the revision.

“Theorem 3 impose a pretty strong assumption, namely the Polyak-Lojasiewicz inequality… Under such assumption, every local optimal will be gloabl optimal, thus I believe that the theoretical claim should be correct but not very significant.”

We appreciate the reviewer’s insightful comments on our theoretical assumptions. While the Polyak-Lojasiewicz (PL) inequality seems to be a strong condition, recent studies such as [1] demonstrated that over-parameterization often induces optimization landscapes to approximate PL-like conditions. In real-world applications, modern language models are heavily overparameterized, and tend to satisfy the PL* condition (a variant of PL condition) as shown in [1].

We acknowledge the review’s point that the condition may not always hold in practice, but this is the case for almost every theoretical analysis. That being said, our theorem provides a meaningful insight on ULPT in practice (namely, random projection for embeddings does not add to optimization difficulty), which is novel and has not been stated before.

[1] Loss landscapes and optimization in over-parameterized non-linear systems and neural networks, Liu et al. 2020.

“it's interesting to see the result on newer models such as Llama3.2 or Qwen2” and “the Fourier domain parameterization seems to require less parameters than the method proposed in this paper…I think this method is worth comparing with both in theory and in experiments.”

We thank the reviewer for suggesting comparisons with Fourier-based methods and evaluations on newer models. We conducted additional experiments using the Llama3.2 (1B and 3B) models on two generation datasets: GSM8K (math reasoning) and MBPP (code generation). These generation tasks also complement the 21 tasks in our main paper.

Due to space limit, we kindly refer the reviewer to Table 1 of our rebuttal to Reviewer Ycju. As shown, ULPT consistently outperforms FourierFT under controlled parameter budgets. For instance, when controlling parameters at 4.1K (ULPT r=2 vs. FourierFT n=128), our ULPT achieves higher performance on both GSM8K (39.7 vs. 35.8) and MBPP (26.1 vs. 21.5) for Llama 1B, and similar advantages are observed in the 3B setting (66.3 vs 63.1 on GSM8K and 33.9 vs. 21.9 on MBPP). Moreover, ULPT remains competitive or superior to other baselines such as LoRA, VeRA, and vanilla prompt tuning that require significantly more parameters to learn.

Theoretically, both FourierFT and ULPT leverage random matrices to reduce learnable parameters for fine-tuning, but they achieve this through fundamentally different mechanisms. FourierFT compresses weight updates by leveraging random spectral entries in the frequency domain, while ULPT operates in prompt space by parameterizing the embeddings with an up-project random matrix. The projection preserves the embedding distances essential for transformer attention (our Theorem 2). We will discuss these differences in more detail in our revision.

“I think there are some typos but I didn't check all of them carefully. For example, in the statement of Theorem 3, "Polyak–Lojasiewic" seems missing a "z" at the end.”

Thanks for the catch! We’ll fix them in the revision.

---

We hope that our clarifications and additional results address all the concerns. We greatly appreciate the reviewer’s willingness to reevaluate our manuscript! Please let us know if there are any further questions. Thanks!

We appreciate the reviewer for their thorough evaluation and the “strong accept” recommendation! The reviewer fully recognizes the contributions of our work, as well as the comprehensive analysis and clear writing.

“It would have been nice to see how their approach fared in this more challenging setting.”

We thank the reviewer for highlighting the importance of evaluating our approach on tasks involving long generation. We conducted additional experiments on the GSM8K and MBPP datasets for math reasoning and code generation with the maximum generation for a few hundred tokens. We used one of the newest Llama models (3.2), and due to the constraint on time and resource, we considered the 1B and 3B variants.

Table 1: Results from additional experiments. We report accuracy on GSM8K and pass@1 on MBPP. The updated code is available in our anonymous GitHub repo (see footnote 1 in our manuscript for the link).

These results in Table 1 show that, ULPT remains competitive or superior compared with other baselines including LoRA and its recent variants (VeRA and FourierFT), as well as vanilla prompt tuning. In particular, LoRA and VeRA fail to work in the ultra-low parameter setting, as the required parameters are magnitudes more than ours. FourierFT uses fewer parameters, but their performance is much worse than ours (e.g., 35.8 VS 39.7 for GSM8K at 1B scale with 4.1k parameters).

“They could do a better job at explain what Theorem 2 means practically”

Thanks for the suggestion. Practically, since transformers heavily rely on embedding distances during the forward pass to compute attention patterns, Theorem 2 shows that our randomly up-projected low-dimensional prompt embeddings approximately preserve these pairwise distances. This suggests that the model’s attention mechanism operates on embeddings that reflect the same relational structure as the full-dimensional prompts. We will clarify in the revision.

---

Once again, we thank the reviewer for their strong support and valuable feedback!