Learning to Plan for Language Modeling from Unlabeled Data

Reasons to Reject:

1. Definition of writing actions: Abstract writing actions are equal to the indices of the clusters obtained in the clustering step, which we already clarify in the third paragraph of the introduction and in Figure 1. Intuitively, they correspond to common writing patterns in the dataset, e.g. origin or profession in a biography. We will add further clarification to the abstract.
2. Evaluation: We will add a sentence to Section 4.4 and the caption of Table 1 clarifying that Predicted/Fixed/Oracle actions can be evaluated both at training and test time. This evaluation setup allows for valuable insights such as that training the LM with oracle actions rather than predicted actions leads to worse performance due to the exposure bias.

Questions to Authors:

1. Internal planner: As we refer in Section 4.4. the details of the internal planner (which corresponds to [1] which we reimplement as a baseline) are described in Appendix B. Intuitively, the language model acts as an internal planner that treats writing actions just like every other token. Those tokens are added to the vocabulary and the LM is finetuned on action-inserted text.
2. Model complexity: The complexity (number of computations) of our proposed sentence-based planner (Section 3.2) is $\mathcal{O}( \frac{N^2}{M} )$ for the representation function (computation of $Z_{i-1}$, assuming caching of sentence embeddings during inference) plus $\mathcal{O}(M^3)$ for the prediction function (computation of $\bf{Z’}_{i-1}$), where $N$ is the total number of tokens and $M$ the number of sentences, where we consider a simplified analysis where each sentence has length N / M. Since the representation function (110M parameters vs 6M for the prediction function) represents the vast majority of the complexity, invoking the planner is roughly $M$ times less expensive than the language model (GPT-2: 128M parameters, OLMo: 1B parameters), which has complexity $\mathcal{O}(N^2)$. We will add a more elaborate discussion to the appendix of the paper.

[1] Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, and Alessandro Sordoni. Guiding Language Model Reasoning with Planning Tokens, December 2023b. URL http://arxiv.org/abs/2310.05707. arXiv:2310.05707 [cs].

Reasons to reject:

1. We use a simple unigram classifier for detecting sentence boundaries, which classifies a token to be at the end of a sentence if it appears more often at the end of a sentence in our training dataset than not, which reaches 99.3% accuracy and 88% f1-score, which is invoked during generation.
2. & 3. We now implemented latent PPL (lower=better) with abstract writing actions as latent variables. We trained the same HSMM critic as used for the discourse coherence experiment in the paper you provided, which achieves a test latent PPL of 100.8. Furthermore, we provide MAUVE scores (higher=better).

4. Parameter count: The base language model has ~130M (GPT-2) or ~1B (OLMo) parameters. The adapter parameters are about 14M parameters. The planner consists of a sentence transformer with 110M frozen parameters plus a small Transformer encoder with 6M additional parameters. Having a separate planner with its own sentence representations means we can reuse the same planner for different language models. As a memory-efficient alternative one can reuse the sentence representations of the LM, losing the portability.

We will add all new results and extra information to the final paper.

Questions to Author:

1. We address the impact of P init in Section 5.3 “Model Architecture”, where we state that while it was beneficial in our preliminary experiments on little data, it doesn’t help with more training data.
2. Indeed, since the previous actions are derivable from the previous text, adding the actions explicitly does not add any new information. We suspect adding it might help when training on little data, but would not be beneficial with more data.
3. Exposure bias is indeed the reason for training the LM on predicted actions, as we discuss in Section 5.1 in the paper (“it decreases the train/test mismatch”).
4. Color-coding actions is a nice visualization. We will add it to the final paper.
5. You are correct that Oracle/Oracle performance is likely not achievable since there is no such thing as a perfect planner. Note, however, that the level of leakage of concrete text is small due to the level of abstraction of the actions (millions of sentence embeddings clustered into 1024 actions). We will modify this statement accordingly.

Methodology:

1. We do not agree that the proposed actions are called “next sentence embeddings” as they represent a cluster of sentences at a more abstract level. Consequently, the planning model does not perform “next sentence language modeling” for the same reason.
2. As discussed in Section 3.1, we posit that it is necessary to compute abstractions that correspond to common writing patterns. By nature, planning in natural language involves a form of abstraction to deal with the vast action space. As Figure 2 demonstrates and we discuss in Section 5.3, there is a tradeoff between the granularity of actions (and hence informativeness for generation) and the difficulty of predicting the correct action. In the case that two consecutive sentences are guided by the same action, the language model provides fine-grained control conditioned on previous words and sentences through attention.
3. We have tested your suggested alternative way of using a signal from the language model, namely in our ablation that encodes the entire context using GPT-2 at once rather than encoding each context sentence with a sentence transformer and then fused into one representation with a trained transformer. We observed that this yielded no improvement (see Table 2 second row and third paragraph of Section 3.2). We will state explicitly that this ablation uses GPT-2 in the final paper.

Experiments:

1. It is not true that None is our only baseline. The main baseline to our proposed method (“Predicted/Predicted” in Table 1) is “Fixed/Fixed”, which finetunes the same architecture on the same data using the regular causal LM objective. Our improvement is consistent across 2 models, and more than 1 perplexity point is a moderately sizable difference commonly reported in the LM literature.
2. & 3. We now evaluate generation performance via ROUGE-2 and edit distance on long articles up to 2048 tokens. The table below shows the relative improvements (in %) of our method over the Fixed baseline, cutting off generation at different points.

Literature: Thanks for the suggested controlled generation literature. We will include it in our related work section on controlled text generation, where, in contrast to our method, the control codes are provided by the user.