Prompt-Based Length Controlled Generation with Reinforcement Learning

1. The main contributions of this paper are very incremental. For example,

   • 1.1) Controlling the length by input prompts has already been done by (Fan et al., 2018) and CTRLsum (He et al., 2022).
   • 1.2) Applying RL to controlling the length has already been done by (Bian et al., 2019)
   • 1.3) Sample filtering can be considered (I believe) as a weaker version of minimum risk decoding e.g., Freitag et al., 2022
   • 1.4) The relevant references CTRLsum (He et al., 2022) and (Bian et al., 2019) are missing in the paper

2. The paper mentions LLMs (e.g., GPT-4, LLaMA, etc.) which are much larger and more capable than the baseline selected in this work (GPT). So, I’m not sure if the findings in this paper would transfer to those larger models (with emergence properties). I believe these larger models are becoming more accessible to researchers now, so I’m quite surprised about the model choice in this paper. Also, there are other more commonly used models such as BART, T5, and Pegasus which have fine-tuned weights on summarization tasks.

3. This paper doesn’t compare against any existing methods. The authors list some existing approaches in Section 2.2; however, in the experiments, none of them are compared against.

Note that the weaknesses #2 and #3 are minor compared to weakness #1.

References:

• (He et al., 2022) CTRLsum: Towards Generic Controllable Text Summarization
• (Fan et al., 2018) Controllable Abstractive Summarization
• (Bian et al., 2019) Controllable length control neural encoder-decoder via reinforcement learning
• (Freitag et al., 2022) High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics

• While this paper puts emphasis on LLM, the experiments use models with 124M, 355M and 774M, which can be controversial to be claimed as LLM. The behavior of an LM can be significantly different when the size is in Million and Billion scales. Also, which GPT is used as the main model is not specified. Because the paper only mentions the word “GPT”, I will guess it is the GPT1 (Radford et al., 2018) or the GPT2 used for the SPE.
  • Radford, Alec, et al. "Improving language understanding by generative pre-training." (2018).
• Novelty, or writing issue: Subsection 3.4 turns out to be an introduction to PPO instead of a proposed method. The added KLD penalty is also a variation proposed in (Schulman et al., 2017) and similar kinds of KLD penalty has been also added to PPO in prior work, such as (Ziegler et al., 2019). The authors can consider reorganizing the section.
  • Ziegler, Daniel M., et al. "Fine-tuning language models from human preferences." arXiv preprint arXiv:1909.08593 (2019).
• Technical issue: The definition of the advantage function is not conventional here. Specifically, A is often defined as Q(s,a)-V(s) or r + \gamma V(s’) - V(s). But in Section 3.4, the authors say the A is r - V(s,a). First, V is usually used for the state value function, whose input will only have the state. If the input includes both state and action, it is often said to be the state-action (Q) value function. Therefore, I'm wondering that is the V(s,a) should be V(s) here or the advantage used in the experiment is actually r - Q(s,a)?
• The experiments can have one baseline that does NOT use length condition prompts to fine-tune the model. This baseline can help readers understand how an “original” setup model performs on the test sets.
• The experiments miss some important details. I have checked the appendices but haven’t found them.
  • How many samples are generated for the sample filtering?
  • What is the used sampling method in the inference stage, including the hyperparameters?
• More discussion needed:
  • How do the authors view that RL+filter (BERT) receives the best BERTScore in Table 7 and 16?
  • What do the generated examples look like?
  • What kind of errors can happen? Is there a case that the summary is within the given length but the summary is actually not complete?

• It appears that there is a significant improvement in the control settings of 'Equal' and 'Between' when considering the core setting between Prompt and Prompt + RL. However, it remains unclear whether the improvement persists when the method is integrated into larger LMs such as LLaMA. This limits the extent of their contributions, despite the potential practical applicability of the method due to its simplicity.

• The paper does not compare to existing methods, such as LenAtten and LAAM, which could be adapted to the pretrained LMs used in this paper. While I understand some parts of these methods might not directly apply to this study, at least the control target of "equal to" a specific length should be compared.

• The paper exclusively concentrates on the length control ability, rather than enhancing the downstream tasks. It would be beneficial if the reward functions for controllability and preference reward models, such as [1], were combined to enhance both the length control ability and summarization performance simultaneously. Moreover, including human evaluation for the generated summary would be valuable.

  [1] Learning to summarize from human feedback

• The paper lacks a comparison between the modified PPO and the standard PPO.

• Much of the paper is focused on using existing techniques, e.g. training models on the length of existing texts (this was used for length-controlled T5 infilling for instance) and PPO RL
• While the SPE accuracy seems high on held-out templates, all templates were written by the authors and are unlikely to cover the diversity of what humans might use in the wild. It would be useful to find a way to test generalizability to real user inputs. This is particularly important because the authors specifically frame this aspect of the paper as handling diverse inputs, and so truly demonstrating that this component of the pipeline (which is a significant part of the contribution) indeed generalizes to diverse surface forms. Otherwise, it is not completely clear why the authors would not just define a standard format, as the input prompts are all defined by the authors anyway.
• While RL does seem to result in lower constraint error, it also (by inspection, e.g. in table 3) seem to often lower the automatic quality metrics. It would also be useful for the authors to include bold and underline values in all columns, not just constraint error.