Length Representations in Large Language Models

What were the labels for each token? I think I understand now that the th token generated by the model has the label ; notably, this excludes the tokens in the prompt. Maybe clarify this in the paper.

Your understanding is correct. For each token generated by the model during the summary generation process, we assigned a label corresponding to its generation timestep t. We exclude the tokens in the prompt from this labeling.

How were the most important individual neurons found? You say you chose them by R^2; did you train new classifiers to map from single hidden units to the length label? Or did you somehow use the previous classifiers / their R^2?

Your understanding is correct. To figure out the most important individual neurons, we train new classifiers to map from single hidden units to the length label.

You say that you set the value of these neurons to value = value * scale. At which positions of the input did you do this; all of them?

In our experiments, we applied the scaling to the top-k individual neurons in the outputs of the second layer's attention mechanism.

In which settings does neuron scaling genuinely decrease length? From Figure 2, I gather that it works in fine-tuned settings, but only with priming in the non-fine-tuned setting. It would be helpful to plot generation length in an easier to understand way (or explain CR in more detail). For example, you could plot histograms / the distribution of the difference between scaled and baseline generation length, for various levels of scaling. The compression ratio is calculated as the ratio of the length of the generated summary to the length of the source text. The length is counted based on the number of words.

Delta CR is an arithmetical difference between ratios. A delta CR value close to zero indicates that the model-generated summary has a compression ratio similar to the gold summary. The more positive delta CR means the generated summary is longer than the gold summary, while a negative delta CR means it is shorter than the gold summary.

Do the human evaluations actually indicate that scaling neurons actually changes conciseness? In Table 6, conciseness of the up-scaled neuron summaries is 0.00-0.01 higher than that of the baseline, and 0.01-0.02 better than the gold standard. Is this difference in conciseness really (statistically) significant / if so, does an effect of this size really matter? And why is the gold conciseness also no better than the baseline?

We have conducted significant tests to determine whether the differences in conciseness and informativeness scores are statistically significant. We used paired-boot strap resampling with 10,000 samples [1]. We obtained the differences in conciseness and informativeness scores between scaling factors of -10 and +10 are statistically significant (p <0.05). We think this demonstrates that length-related units we found can directly control output sequence length to produce longer summaries with negative scalings or shorter summaries with positive scalings without external prompt changes. Thus, we think we can draw general conclusions that multiplying negative factors to length-related units generate longer summaries while multiplying positive factors generate shorter summaries.

[1] Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

The following is Table 6. Table 6

Hi! Thanks for your responses. Some of them answered my questions, and some did not, so I'll repeat / follow-up on those here:

You say that you set the value of these neurons to value = value * scale. At which positions of the input did you do this; all of them?

In our experiments, we applied the scaling to the top-k individual neurons in the outputs of the second layer's attention mechanism.

Here, I was referring to position in the sentence, not within the transformer. Is the scaling applied at all token positions?

In which settings does neuron scaling genuinely decrease length? ...

Delta CR is an arithmetical difference between ratios. A delta CR value close to zero indicates that the model-generated summary has a compression ratio similar to the gold summary. The more positive delta CR means the generated summary is longer than the gold summary, while a negative delta CR means it is shorter than the gold summary.

Thanks for explaining Delta CR, but I asked about the settings in which neuron scaling genuinely decreases length. I asked this because, in Figure 2, it's not clear that it does work in all scenarios. It seems that either fine-tuning or prompting length-specific prompting is necessary to make it work, which has implications for this method's utility, and the conclusions we should draw about model mechanisms (e.g. do they use different mechanisms when prompted or fine-tuned, yielding causal interventions to only be effective in some cases?).

Do the human evaluations actually indicate that scaling neurons actually changes conciseness?... Is this difference in conciseness really (statistically) significant / if so, does an effect of this size really matter?...

We have conducted significant tests to determine whether the differences in conciseness and informativeness scores are statistically significant. We used paired-boot strap resampling with 10,000 samples [1]. We obtained the differences in conciseness and informativeness scores between scaling factors of -10 and +10 are statistically significant (p <0.05). We think this demonstrates that length-related units we found can directly control output sequence length to produce longer summaries with negative scalings or shorter summaries with positive scalings without external prompt changes. Thus, we think we can draw general conclusions that multiplying negative factors to length-related units generate longer summaries while multiplying positive factors generate shorter summaries.

Thanks for this response! Two things remain slightly unclear to me. First, is it the case that only the difference in conciseness between -10 and +10 scaling is statistically significant (and other pairs, e.g. +0 vs. +10 are not significant)? Second, as I asked originally, and irrespective of statistical significance, what do you think of the effect size? It seems quite small to me, so is this really an impactful intervention?

I'm a bit confused about the formulation given in (1)-(4)...Are equations (1)-(4) wrong? Or did you modify the architectures of the pre-trained models in some way?

We appreciate your careful reading to improve our drafts.

To be clear, equations (1)-(4) are wrong then, and you use a standard pre-LayerNorm architecture?

For all Tables and Figures, we will add error bars with standard deviations. The following is Table 2 including standard deviation. Please note that standard deviations for the numbers in Table 3 are also nearly zero.

3.1: (a) you claim that the “length” prompt instructs the model to provide a summary with a specific desired length. However, that is not what the prompt in Table 1 specifies, it specifies a number of words that should be deleted. Even for that, I am not sure it shaves off the least important words, which I inferred was the purpose. (b) For the “priming” prompt, wouldn’t the two user-provided numbers work against each other? I could specify that I want a summary of, say, 10 words and also specify that I want to delete some number of words; but deleting that many may not yield a sentence with exactly 10 words. (c) In light of these inconsistencies, did you look at whether the model outputs actually adhered to the objectives you intended? (d) Overall, I am not convinced that these particular prompts are sufficient to draw general conclusions about how the instructions given in the prompt interact with the parts of the network that are activated (as is done, e.g., in 5.1).

For the "Length" prompt, we believe it effectively instructs and guides LLMs to control output sequence length to meet desired lengths in that the prompt specifies the number of words to be deleted. LLMs can infer the intended length information through in-context learning. Additionally, previous studies have confirmed that significant improvements in ROUGE scores when using the "Length" prompt compared to "No-constraint" prompts [1]. It demonstrates that the "Length" prompt helps the model to preserve important words while controlling the output length.

We believe that the "priming" prompt is suitable and adheres to the objectives [1]. Specifically, as shown in Table 5, the Base method using the "Priming" prompt with fine-tuning setting consistently generates summaries closer to gold summary lengths in bin-wise analysis. Based on our results, we are confident that used prompts are sufficient to draw general conclusions about how instructions given in the prompt interact with the parts of the networks that are activated. The identified length-related hidden units further support our findings in that multiplying positive factors generates shorter summaries while multiplying negative factors generates longer summaries.

[1] Do, 2024. InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models, Findings of the Association for Computational Linguistics: ACL 2024.

I have some concerns about your method to find evidence of length representations (end of 3.1). (a) You use a neural network, so it is misleading to call it linear regression. The text also lacks details on how you fit the parameters of this model.

Your understanding is correct. However, we think the text itself is detailed for explaining how we fit the parameters of this model as we have described the dimensions with formulas.

(b) It is not clear to me how the response variable “generation time step” is defined. Does it include the prompt? The variable n is also not very well specified.

For each token generated by the model during the summary generation process, we assigned a label corresponding to its generation timestep. We exclude the tokens in the prompt from this labeling.

(c) It looks like you’re predicting the “generation time step” from a matrix of hidden states. However, couldn’t that just be inferred by the attention masking or the positional embeddings?

We appreciate your insightful suggestion. While investigating attention masking or positional embeddings would provide valuable insights, these approaches are beyond the scope of our current study. Moreover, as we conducted experiments using all hidden units in Figure 1, the first layer with attention-residual outputs does not show a higher R² compared to other layers. Thus, simply investigating positional embeddings might not be as suitable as analyzing entire layers in LLMs. In addition, we believe that investigating word embeddings themselves is more appropriate since certain words, such as ``I’’ or periods, are frequently located at the beginning or end of sentences. Thus, they may inherently contain relevant information. In the future, we will explore these aspects including your suggestions to further understanding of LLMs internal mechanisms.

Figure 2: (a) I think some of your claims in 5.2 are not supported by the data. ... differences are within the error margin. This is a more general issue—none of the results in the paper are presented with error bars.

Our goal is to identify the length-related hidden units in LLMs and control output sequence length by manipulating those units. As you commented the Rouge-L has very little positive difference meaning that multiplying length-related hidden units can generate shorter or longer summaries without losing informativeness. We will add the error margin as soon as possible. Thank you very much.

Finetuning methodology I had trouble understanding the motivation and the details for finetuning (after reading Appendix A). What data were the models fine-tuned on?

The models were fine-tuned on the Google sentence summarization dataset, which consists of instruction-based formats, No-constraint, Length, Priming in Table 1 [1]. The fine-tuning process is supervised by incorporating length-specific instructions to enhance the model’s ability to control the output sequence length.

[1] Do, 2024. InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models, Findings of the Association for Computational Linguistics: ACL 2024.

Why do we want to finetune the models? What does this tell us in addition to the zero-shot setting?

By fine–tuning the models with length-specific prompts, we target to reinforce their length-control capabilities. Fine-tuning allows the models to learn length constraints more robustly compared to relying solely on zero-shot settings.

In addition, we aim to investigate how explicit fine-tuning with length constraint prompts affect length-related internal representations within LLMs. By comparing the internal representations before and after fine-tuning, we figured out that using more length specific prompts, particularly the ``Priming’’ prompt, causes the top-k hidden units related to length control to be more highly activated than those in the No-constraint and Length prompt settings.

We also aim to investigate how LLMs process and learn length constraints between explicit fine-tuning and zero-shot in-context learning. Our experimental results in Table 4 show that the same top-k activated length-related hidden units are involved in both the zero-shot and fine-tuning settings when using the more length-specific ``Priming’’ prompt. The consistency in the highly activated units in both settings implies that LLMs interpret in-context learning as a form of implicit fine-tuning.

Evaluation methodology unclear

How were models sampled? E.g. greedy decoding, top-k?

We used greedy decoding for all settings in our experiments to avoid any randomness in the generation process.

As LMs have variable-length inputs, which tokens' representations are being used for the linear regressions?

We apologize for any confusion. In our experiments, we used the representations of the generated output tokens, excluding the input tokens from the prompts. During the summary generation process, the model produces one token at each time step t. For each generated token, we extracted hidden states from the layers of LLM and assigned corresponding time steps.

l402: Please include a summary of how delta CR should be interpreted; please define compression ratio in the text.

The compression ratio is calculated as the ratio of the length of the generated summary to the length of the source text. The length is counted based on the number of words.

Delta CR is an arithmetical difference between ratios. Delta CR value close to zero indicates that the model-generated summary has a compression ratio similar to the gold summary. The more positive delta CR means the generated summary is longer than the gold summary, while a negative delta CR means it is shorter than the gold summary.

Table 4: "The numbers in parentheses indicate an index of hidden units from the second layer of the attention mechanisms." I did not understand this; how many hidden units are there total? Are the reported values for a single hidden unit, or averaged over several individual regressions?

In Llama-2-13b-chat, the total hidden units from the layer is 5,120. Thus, we additionally trained classifiers for single hidden units. The reported values are for a single hidden unit.

Partial results reported in manuscript While Section 3 (Methods) states that experiments are on Llama 2, Llama 3, and Phi, only Section 4 reports results for all models. The rest of the paper only reports results on Llama 2. Please include the full set of results for Section 5.

We appreciate your careful reading and valuable feedback, which have helped improve our draft. We investigated other models and observed that our method for identifying length-related units effectively controls the output sequence length. We will submit the revised draft, incorporating your feedback. Thank you very much.

What fine-tuning objective is actually being used here - is the fine-tuning supervised by output sequence length?

The models were fine-tuned on the Google sentence summarization dataset, which consists of instruction-based formats, No-constraint, Length, Priming in Table 1 [1]. The fine-tuning process is supervised by incorporating length-specific instructions to enhance the model’s ability to control the output sequence length.

[1] Do, 2024. InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models, Findings of the Association for Computational Linguistics: ACL 2024.

How is delta-CR calculated - is it an arithmetical difference between ratios, or e.g. a log ratio of ratios? Are numbers closer to zero better in principle?

The compression ratio is calculated as the ratio of the length of the generated summary to the length of the source text. The length is counted based on the number of words. Delta CR is an arithmetical difference between ratios. Delta CR value close to zero indicates that the model-generated summary has a compression ratio similar to the gold summary. The more positive delta CR means the generated summary is longer than the gold summary, while a negative delta CR means it is shorter than the gold summary.

Is the same gold-standard length used as the norm even when the multiplication factor varies - and would that mean that numbers closer to zero aren't actually "better" if the multiplication factor is being set e.g. to shorten or lengthen outputs?

Your understanding is correct. The same gold-standard length is used as the norm even when the multiplication factor varies in order to demonstrate that the length-related units are actually involved in length-control during generation process. If the goal is to generate summaries that meet the gold summary’s length, then delta CR closer to zero is better. However, when we multiply scaling factors, the gold summary length does not change. Thus, if we aim to generate longer summaries, an increased delta CR indicates success compared to not multiplying scaling factors. If we aim to generate shorter summaries, a decreased delta CR indicates success compared to not multiplying scaling factors.

How should we compare the results in terms of delta-CR and ROUGE compared to alternative approaches e.g. the length priming approach?

Our goal is to demonstrate that we can control the output sequence length by identifying and manipulating length-related units within LLMs. By scaling those units, we aim to show that they indeed have a direct impact on output length during the generation process without changing the prompt. "Base" indicates we did not scale length-related units and solely generated summaries with the given prompts, such as "priming". Thus, we maintained consistent prompts between "priming" and scaling length-related units for fair comparisons. By comparing delta CR and ROUGE scores under consistent settings, we show that multiplication factors to the length-related top-1 unit even control output length without losing informativeness.

The following is Table 5 with the standard deviation. Compared to Base which does not multiply factors, scaling +10 or -10 successfully generates shorter or longer summaries. Dagger indicates the statistical significance (p<0.05) using paired boot-strapping methods with 10,000 samples [1].

[1] Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.