All You Need is One: Capsule Prompt Tuning with a Single Vector

We sincerely thank reviewer Z96S for the valuable feedback, which is crucial for improving our work.

Major:

Q1.: Deep PT with one prompt.

A1.: “DeepPT with one soft prompt” in Figure 4 and S3 stands for “only tuning the single-vector prompt w/o mean embedding”. We provide a comparison isolating the contribution of the mean embedding in our response to Q2.

To further validate the effectiveness of CaPT, we conduct additional experiments on Deep PT with one prompt. As seen, CaPT consistently outperforms Deep PT with one prompt, demonstrating the effectiveness of leveraging instance-aware info to serve as attention anchor.

Q2: Regarding additional ablation.

A2: We conduct an ablation using only the mean pooled embedding as a capsule across all layers of the frozen T5-Base model. As seen, instance-aware only signals still guide the model effectively—outperforming Deep PT with a single prompt on tasks like BoolQ and WiC. The attention anchor effect also persists in this setting (figures to be included in revision). Notably, CaPT achieves the best performance, validating the benefit of combining instance-aware and task-aware guidance.

Q3: Regarding tasks and datasets.

A3: Sorry for the confusion. We would like to further explain that a “task” refers to a corpus in SuperGLUE, such as COPA. The definition follows common practices [1, 2]. We will revise our paper to name them as “corpus” to distinguish from task categories (NLI, QA, etc) for better clarity.

If you don’t mind, could you kindly clarify which table you are referring to as Table X?

And yes, soft prompts of CaPT are optimized on each corpus of SuperGLUE.

We have included the description of each dataset in section 4.1 and detailed statistics in Appendix S1. As for examples, an instance in the Boolq corpus is a sequence with a fixed structural instruction template: “boolq passage: Powdered sugar, also called confectioners' sugar ... question: Is confectionary sugar the same as powdered sugar?”

Q4: Implementation details.

A4: Each CaPT model is trained on each corpus of SuperGLUE. After careful consideration to ensure maximum clarity, we refine the Eq.(2) which is the core of constructing capsule prompts for both training and inference. There is only one training stage for each corpus.

$S^1 =p^1 + Mean(E)$

$\underline{S}^1, H^1 = L_1({S^1}, {E})$

$S^i = p^i + Mean(\underline{S}^{i-1} \oplus H^{i-1}) \ \ \ \ \ i = 2, 3,\dots,N$

$\underline{S}^i, H^i = L_i({S^i}, \ H^{i-1}) \ \ \ \ \ i = 2, 3,\dots,N$

Each capsule prompt $S^i$ is formed by adding a learnable vector $p^i$ to the mean of the previous capsule $\underline{S}^{i-1}$ and hidden state $H^{i-1}$. $S^i$ is prepended to the sequence at each layer. Vectors $p^i$ are zero-initialized and trained via gradient descent.

(a) For T5 models, we use text-to-text format, following the common practice [2, 3]. For decoder-only models, we preserve the original labels (0, 1, 2) as classification tasks. Therefore, there is no answer in input sequences.

(b) We use cross-entropy loss on a single corpus to update the learnable vectors. For T5, this is generative cross-entropy over the vocabulary; for decoder models, it's classification cross-entropy over class indices.

(c) Mean embedding is added to the soft prompt during both training and inference. Capsule prompts are also constructed in the same way during both training and inference.

Q5: Regarding longer prompts.

A5: Our experimental results on both Finding 2 and ablation indicate that effectiveness of prompts depends not on the amount of info provided, but on how well that info matches the model’s ability to utilize it. This observation is consistent with prior works on prompt pruning [2, 4], which shows that not all prompt tokens contribute positively to performance. Moreover, studies on prompt tuning [5] and P-Tuning v2 [6] have also observed that longer prompts do not necessarily lead to better outcomes, and may even hurt performance by introducing noise or leading to slower convergence.

(a) For longer prompt lengths, we apply adaptive mean pooling to compress embedding into multiple tokens, matching the desired prompt length instead of one copy per prompt token.

Q6: Regarding the attention pattern analysis and why standard prompts fail.

A6: (a) The attention patterns are averaged over a single corpus and there is one task with the same structural template in a corpus. As for the structural template for a corpus, for example, “rte sentence1: premise sentence2: hypothesis” for RTE. Since each corpus is different on data distribution, question type, and structural template, prompts are optimized on each corpus of SuperGLUE.

(b) Standard soft prompts are randomly initialized and designed to be updated via gradient descent, but not specifically designed for interactive attention behavior. In contrast, CaPT directly carries off-the-shelf instance-aware info, resulting in a fundamental difference. Acknowledging the current research on prompt tuning performance analysis remains largely unexplored, we believe the observation of the attention anchor constitutes a foundational contribution.

Minor:

Q1: Regarding terminology.

A1: This is a valuable suggestion! We will revise our paper accordingly. Due to the rebuttal length constraint, we would like to provide an explanation here: Instance-aware describes info tailored to a specific input instance (e.g., a sentence or document). Instead of generic task-aware instructions, instance-aware tokens reflect the feature of each individual input.

Q2: Regarding attention analysis.

A2: We would like to clarify that Fig. 1 shows the averaged attention pattern across all samples of the RTE validation set. When processing each input sample, the prepended instance-aware token is mean-pooled from each sample. We present an averaged result for a high-level observation on the general impact of instance-aware info. This averaging does not conflict with the contribution of instance-aware info. Additionally, we have also provided a concrete example in Appendix S5 which shows the attention behavior of specific heads on one specific instance.

Tokens 1, 5-7 are important because they encode structural info in the input sequence—such as syntactic markers (e.g., "sentence 1:", "question:"). As shown in Fig. 1 (left), these tokens consistently receive high attention from other input tokens, indicating their role as cues that help the model interpret inputs. In prompt tuning, such structural tokens guide the model in segmenting the input into meaningful parts and in anchoring its attention, which improves task understanding and performance. This is supported by prior work showing that templated inputs with explicit structure improve few-shot learning capabilities by reinforcing input semantics [7, 8]. Additionally, prior work [9] indicates that attention heads in LLMs heavily focus on structural and syntactic info during In-Context Identification phase, aiding the model to interpret contextual cues.

As suggested by Q3, we have provided a detailed explanation of tasks, datasets, and instances.

Q3: Explanation of Fig.1 (b)

A3: This is because it’s an experiment based on the prompt-tuned model in Fig. 1 (a).

Q4: Dataset for Table 3.

A4: The average scores are also calculated based on six SuperGLUE corpora.

Regarding the discussion on Limitation

A: We thank the reviewer’s careful attention to our claim about CaPT’s ability on eliminating prompt length grid search. We will add a limitation in the revised version. While our results consistently show successful optimization of CaPT across a broad range of datasets and model scales, we acknowledge that this may not generalize to all models and datasets. During the rebuttal phase, we also extended CaPT to the CSQA, and the results reinforce the robustness of our current single-prompt design.

[1] Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. arXiv 2023

[2] Xprompt: Exploring the extreme of prompt tuning. arXiv 2022

[3] Smop: Towards efficient and effective prompt tuning with sparse mixture-of-prompts. EMNLP 2023

[4] E2vpt: An effective and efficient approach for visual prompt tuning. ICCV 2023

[5] P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. ACL 2022

[6] The power of scale for parameter-efficient prompt tuning. arXiv 2021

[7] Language models are few-shot learners. NeurIPS 2020

[8] Making pre-trained language models better few-shot learners. arXiv 2020

[9] Attention heads of large language models: A survey. arXiv 2024

We hope our response addresses your questions. Please let us know if there are any additional questions, and we will be happy to discuss further.

We sincerely appreciate the time and effort you've devoted to reviewing our work and providing helpful feedback!

Q1: Regarding Finding 2’s generalizability.

A1: Thank you for the question! To assess the generalizability of our Finding 2, we further investigate simply prepending instance-aware information on a decoder-only model (i.e., Llama-3.2-1B) without additional fine-tuning. As shown in the table below, the results on Llama-3.2-1B strengthen our Finding 2 that indicates the significance of instance-aware information.

While CaPT is initially motivated by findings from T5, in our paper we have applied CaPT to decoder-only models (i.e., Llama-3.2-1B and Qwen-2.5-1.5B). Experimental results (see Table 1) show that CaPT can work effectively across different architectures. We have observed a similar attention anchor role of CaPT on Llama-3.2-1B (please see Appendix S7), aligning with our observation on T5-Base.

Q2: Regarding CaPT’s novelty and mean pooling design.

A2: Thank you for raising this question. In section 4.4, we have provided different instance-aware information incorporation strategies, including prepending, 1D CNN feature extraction, and feature projection. Among these different strategies, mean pooling with addition is considered as the most effective and straightforward approach, primarily for three key reasons: First, mean pooling is training-free and requires no additional modules to obtain instance-aware information, ensuring computational and parameter efficiency. Second, mean pooling with addition can demonstrate notable effectiveness with substantially fewer parameter usage (e.g., 17.5x lower than the feature projection strategy). Third, this design consolidates a large set of soft prompt tokens into a single prompt, thereby significantly mitigating the existing training overhead associated with searching for an optimal prompt length.

We would like to further highlight several key contributions of CaPT besides its intuitive and effective design:

$\bullet$ First, our work is the first attempt to analyze the bidirectional attentive interaction between soft prompts and input tokens in the NLP field. We uncover the power of off-the-shelf instance-aware information that can trigger interactive attention anchor.

$\bullet$ Second, CaPT’s superior performance further confirms the significance of employing instance-aware information as part of guidance signal through comprehensive experiments on various language tasks.

$\bullet$ Third, with the incorporation of both instance- and task- aware information in the guidance signals, CaPT eliminates the need for grid search of the optimal soft prompt length. This advantage significantly reduces the overall computational overhead (please see section 4.4).

We will add additional discussions in revision to make it clearer. Thank you!

Q3: Regarding space complexity analysis.

A3: Thank you for the suggestion. We include a space complexity analysis on Figure 3 for a clearer comparison with Deep PT. Let $L$ denote the number of layers with prompts, and $d$ denote the embedding dimension. Given that the number of tokens for Deep PT is $n \gg 1$, the space complexity can be formally expressed as:

As seen, CaPT achieves a significantly lower space complexity of $O(L×d)$, as it requires only a single prompt per layer, enabling CaPT to be more efficient with superior performance.

We sincerely appreciate your thoughtful comments. We hope our response addresses your questions. Please let us know if there are any additional questions, and we will be happy to discuss further.

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. We provide explanations to your questions point-by-point in the following.

Q1.1: CaPT’s performance on CSQA.

A1.1: We include the comparison of P-Tuning V2 and CaPT on the CSQA dataset across different models in the following tables. The results show that CaPT can consistently outperform P-Tuning V2. Specifically, CaPT achieves a notable performance improvement on Llama-3.2-1B, highlighting the effectiveness of CaPT on CSQA. Thank you for the great suggestion, and we'll supplement the results in the revision.

Q1.2: CaPT’s performance on larger models.

A1.2: We conduct experiments on Qwen-2.5-7B to further validate the effectiveness of CaPT on larger models. As seen, CaPT can achieve superior performance compared to P-Tuning V2 across all tested datasets. The results indicate CaPT’s applicability on larger models.

Q1.3: Task suitability

A1.3: Following suggestion, we investigate the specific tasks in which CaPT demonstrates advantages. Empirically, CaPT shows consistently strong performance on the CB and MRC datasets (QA) and remains competitive on RTE and BoolQ (NLI). Our supplementary experiments on the CSQA dataset further highlight CaPT's effectiveness in commonsense reasoning within QA. It is also worth noting that it is generally expected that prompt-based methods may not outperform all baselines across every benchmark, due to varying task characteristics and modeling demands [1, 2, 3, 4].

Q2: Regarding the novelty of CaPT.

A2: This is a great question. CaPT is distinct from prior works from superior parameter efficiency, training time efficiency, and memory efficiency via incorporating instance-aware information. In our study, we also observe the phenomenon “attention anchor”, which suggests a close relation between instance-aware information and input. Considering the limited exploration of prompt tuning inner mechanisms, we believe the discussion of “attention anchor” constitutes a foundational contribution in this community.

In Appendix S3, we have included in-detailed comparison and discussion with other instance-related methods. Between them, LoPA [3] integrates both info, however, it requires additional large-size encoders for extracting instance-aware features, leading to significantly higher memory usage. It also requires grid search of the optimal prompt length. In contrast, CaPT leverages “attention anchor” and enjoys superior efficiency (see table below).

Q3: Regarding Related Work.

A3: We introduce this section to contextualize recent work on attention analysis and attention sinks, which directly motivated our investigation into attention behavior in prompt-based methods. We have included additional in-detailed comparison and discussion with other existing instance-aware related methods in Appendix S3.

We agree the section name might be ambiguous, we will rename and revise the Related Work section to better emphasize the prior works and make clearer distinctions between background material and our contributions.

Q4: Equations and terminology.

A4: Regarding the equations, we follow the standard conventions used in the prompt tuning literature [5, 6]. Based on your suggestion, we carefully review the formatting and typeset to ensure the maximum clarity and provide a refined version of Eq.(2) in the response A5. As for terminology, we revise the manuscript to prevent any inconsistent uses of terminology throughout. If there are any additional suggestions, we are more than welcome to revise our paper accordingly.

Q5: Implementation setting of CaPT.

A5: Thank you for the question. We further elaborate below on the capsule prompt design.

Specifically, the capsule prompt for the first Transformer layer is obtained by combining a learnable vector $p^1$ with the mean-pooled input embedding. For subsequent layers, each capsule prompt $S^i$ is similarly constructed by combining a layer-specific learnable vector $p^i$ with the mean of the processed capsule prompt $\underline{S}^{i-1}$ and the hidden output $H^{i-1}$ from the previous layer. Notably, CaPT does not require a separate training stage—capsule prompts are constructed in the same way during both training and inference.

$S^1 =p^1 + Mean(E)$

$\underline{S}^1, H^1 = L_1({S^1}, {E})$

$S^i = p^i + Mean(\underline{S}^{i-1} \oplus H^{i-1}) \ \ \ \ \ i = 2, 3,\dots,N$

$\underline{S}^i, H^i = L_i({S^i}, \ H^{i-1}) \ \ \ \ \ i = 2, 3,\dots,N$

Q6.1: Regarding the theoretical explanation.

A6.1: Our explanation of CaPT as an “attention anchor” is primarily based on observations—namely, attention heatmaps and performance improvements. While theoretical analysis in the prompt tuning community remains largely limited [7,8], our work takes an initial step toward uncovering potential mechanisms behind prompt effectiveness through the lens of attention pattern.

A possible interpretation is that capsule prompts help preserve salient features by enhancing information flow through stronger and more stable attention interactions (see Eq.(2)). In future work, we plan to complement our findings with additional analyses, including rank collapse [9] and mutual information [10], to better understand the relationship between attention and performance.

Q6.2: Regarding CaPT’s few-shot capability.

A6.2: Following common practice [11], we use the GLUE dataset for pre-training, and then evaluate the few-shot performance using 4 randomly sampled training data from Boolq and CB. Preliminary results on T5-Base show that despite the current single-task design, CaPT can achieve competitive few-shot capability compared to P-Tuning V2. Moreover, we plan to employ Mixture-of-Experts (MoE) architecture [2, 12] for more robust and dynamic capsuling in the future.

Q6.3: Regarding CaPT’s robustness.

A6.3: We further explore the robustness of CaPT under noisy inputs on T5-Base during the rebuttal. Specifically, we add random special characters to random positions of the input sequence of the train sets, simulating input noise. We compared P-Tuning v2 and CaPT under varying noise levels and found that while both degrade with more noise, CaPT consistently performs better. Its superior robustness likely comes from leveraging instance-aware information that benefits generalization under noisy conditions.

[1] Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. ICLR 2024.

[2] Smop: Towards efficient and effective prompt tuning with sparse mixture-of-prompts. EMNLP 2023.

[3] Prompt tuning strikes back: Customizing foundation models with low-rank prompt adaptation. NeurIPS 2024.

[4] Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Product. NAACL 2025.

[5] Visual prompt tuning. ECCV 2022.

[6] M $^ 2$ PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning. EMNLP 2024.

[7] Prompt waywardness: The curious case of discretized interpretation of continuous prompts. NAACL 2022.

[8] On the role of attention in prompt-tuning. ICML 2023.

[9] Attention is not all you need: Pure attention loses rank doubly exponentially with depth. ICML 2021.

[10] Normalized mutual information feature selection. IEEE Transactions on neural networks 20.2 (2009).

[11] Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. EMNLP 2022.

[12] Multimodal instruction tuning with conditional mixture of lora. arXiv 2024.

We hope our response addresses your questions. Please let us know if there are any additional questions, and we will be happy to discuss further.

Dear reviewer. Do you have any feedback to the rebuttal stories have prepared?