End-to-end Story Plot Generator

We thank the reviewer for their helpful and insightful comments. Below we respond to the questions respectively.

My major concern is the lack of human evaluation… I hence invite the authors to pair the main evaluations (ie Table 5 and 6) with human evaluation. I will then happily advocate for acceptance.

Thanks for your suggestion. We added human evaluation results, and the details can be found in the global response.

Another minor limitation is that RLHF results in 5 different models, which are each compared against a single SFT model. Could the authors consider combining the different rewards into a single RLHF model and then ask humans to compare it against SFT according to the metrics Q1-6?

Training a unified model that can produce better story plots in multiple aspects simultaneously would definitely be an interesting future direction, which is also studied in a concurrent work [1]. Note that according to our human evaluation results, the model RLPlot_Q1 achieves better performance than E2EPlot consistently in all the aspects, which shows that even if we only use a reward model for a specific aspect, it might generalize well to other aspects since usually different aspects might be correlated.

Q1. The Related Work section is comprehensive. To make the paper’s scope broader and linked to work in the computer vision and multimodal communities, I would recommend adding a brief paragraph on work in story generation for image-to-text [1], text-to-image [2] and text-to-video [3] tasks.

Thanks for your suggestion. We have added a brief paragraph on these three works in the revision.

Q2. Are the results in Table 4 a “validation” of the same RLHF training story plots?

The 7000 story pair preference dataset was split into training and validation sets with a ratio of 9:1 for each one of the five reward models. We have made it clear in the revision.

Q3 .In page 2, when you say “a completion model which accepts a suffix” I was not entirely sure what you meant. It then became clear later throughout the paper. I recommend adding a brief explanation that you mean a model that can do text infilling given a text prefix and a text suffix, and cite [4].

Thanks for your suggestion. We have fixed it in the revision.

Q4. Line 3 of Sec 3 should reference Table 7, not Table 1.

Thanks for catching this! We have corrected it in the revision.

Q5. First line of Sec 3.2 might make it explicit that the RLHF results in 5 models.

Thanks for your suggestion. We explicitly mentioned that RLHF results in 5 models in the revision.

References:

[1] Learning Personalized Story Evaluation, https://aps.arxiv.org/abs/2310.03304.

We thank the reviewer for their helpful and insightful comments. Below we respond to the questions respectively.

For someone who is familiar with DOC and LLMs, the paper does not seem to offer any new insight or new knowledge. Yes, Llama-2 and GPT-3 can do roughly the same things. Yes, we can use a supervisedly finetuned model to replace repeated LLM calls. Yes, we can do RLHF. All these are common knowledge. Hence, it may seem that the paper does not make a real scientific contribution.

As reviewer zk9y mentioned, Our paper “addresses a difficult problem, which has a long history in AI and to which the advent of LLM offers new perspectives”, Also, our work “contains a number of original aspects (experimental approach, evaluation)” and “includes a transparent account of prompt engineering aspects faced during the development of the work, which might be beneficial to readers”. We discussed this and our contribution in the Novelty section in the global response in detail. Also, studying the e2e plot generator instead of the whole story generator provides important insight into the story generation problem. See the Why story plot rather than the entire story section in the global response for more details.

Since the author has spent significant effort to collect human ratings on 7000 story pairs, why not do another 300 pairs? This would create a much more solid evaluation.

Thanks for your suggestion. We added human evaluation results during the rebuttal to make the evaluation more solid. Please see the global response for details.

How is story generation different from other forms of structured text generation, such as poetry or argumentative essays? The paper has used specific prompts to handle aspects of stories such as characters or settings. But it is not explicit that if there is any principle behind the writing of these prompts or if they solely rely on trial and error by the paper authors.

The reason that we use the (premise, settings, characters, outlines) structure of the plot is as follows (also discussed in the response to reviewer zk9y).

• The underlying principle is “coarse-to-fine”, i.e., to progressively provide more and more detailed information. This coarse-to-fine representation makes end2end autoregressive generation easier.
• While short, this representation still contains a good amount of information about the story, which is helpful for multiple perspective evaluations (represented by multiple questions).
• This representation is used by the SOTA story generation work DOC [1] for the planning stage. It can be used to generate high-quality full-length stories. This demonstrates that such a structure can lead to full story generation.
• The plot structure we chose in this paper is text-based, which could greatly benefit from the powerful text-based LLMs that take the plot as input and produce the whole story (usually with multiple steps such as the second stage of DOC [1]).

Other forms of structured text generation, like poetry or argumentative essays, may not require that many levels of hierarchy.

We acknowledge that we haven’t further leveraged the special structure of the story yet (e.g., characters) in our generation process, which will be our future work.

Section 2.1.1 The authors do not define what is meant by "supports suffix". I managed to guess the meaning from the context but it created temporary confusion.

We have fixed it in the revision to avoid confusion. Thanks for your comment.

The authors make several claims about how humans supposedly do things. Humans write stories in a specific way (Page 2, Paragarph 2). Humans write from coarse to fine (Challenge 1). However, these claims are unsubstantiated.

The planning-based method is common in many generation tasks (e.g., [3],[4]). The coarse-to-fine generation has been widely used in previous work (such as event-to-sentence generation in [2], which is mentioned by reviewer zk9y). Also, it is supported by the comment of reviewer G5uN that “The task of story plot generation is interesting and allows to disentangle two major phases of content creation often seen in humans: planning and coarse-to-fine generation”.

What novel or surprising scientific insights or findings are reported by this paper?

We discussed this in detail in the global response (see the Novelty section in the global response).

How is the system dealing with relationships between characters? With the plot/character duality? What is the average length of generated plots (counted in plot units or narrative functions)?

Our e2e model treats the story plot (premise, settings, characters, outlines) as a sequence and does not need to explicitly deal with the relationship between characters, which can be regarded as one of the advantages of the e2e story plot generator, compared to a hand-crafted method.

For the OpenPlot pipeline (or original DOC [1]), after initializing the characters, when generating bulletin points of the outline, for each point, we will detect the characters that appeared in the sentence, and refine the character portraits by adding details in the current sentence to the initial character portrait.

On the other hand, we acknowledge that such a modeling relies on LLM’s capacity to model characters and their relationships/interactions, and in many situations, still yields a shallow and boring plot. In the future, we will think about ways to model their relationships in more detail, not by hand-crafting but by leveraging the power of LLMs.

The average length of the generated plots is ~1000 tokens.

References:

[1] Yang, Kevin, Dan Klein, Nanyun Peng, and Yuandong Tian. "Doc: Improving long story coherence with detailed outline control." arXiv preprint arXiv:2212.10077 (2022).

[2] Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744.

We thank the reviewer for their helpful and insightful comments. Below we respond to the questions respectively.

Although the paper describes generated units as 'plots' it stands in-between substantial previous work of Plan-based plot generation… It is thus unclear whether what is presented in the paper, in particular in Figure 7 are plots or storyboards…

We agree that the terminology ‘plot’ does not have a rigorous definition, and there are different choices of the format of the plot. The most suitable definition of the plot in our paper should be ‘some certain structure that captures most of the essential information of the story, which is often created in the planning stage’.

In our work, we choose the structure of the plot (premise, settings, characters, outlines) mainly due to the following reason:

• The underlying principle is “coarse-to-fine”, i.e., to progressively provide more and more detailed information. This coarse-to-fine representation makes end2end autoregressive generation easier.
• While short, this representation still contains a good amount of information about the story, which is helpful for multiple perspective evaluations (represented by multiple questions).
• This representation is used by the SOTA story generation work DOC [1] for the planning stage. It can be used to generate high-quality full-length stories. This demonstrates that such a structure can lead to full story generation.
• The plot structure we chose in this paper is text-based, which could greatly benefit from the powerful text-based LLMs that take the plot as input and produce the whole story (usually with multiple steps such as the second stage of DOC [1]).

Regarding reward models, there should probably be a discussion of previous approaches in text-based narrative generation, for instance [Ammanabrolu et al., 2019] and [Castricato et al., 2022].

Compared to [Ammanabrolu et al., 2019], where the reward is based on how close the generated event is to a pre-trained goal, our reward model training is more similar to CARP [Castricato et al., 2022], a contrastively-trained preference model as a reward signal in story generation. Note that our reward model is trained on story plots (using cross-entropy loss on the preference label) instead of the whole story, which makes learning and human labeling easier without losing important signals. We added these contents in Section 2.3 in the revision.

Evaluation techniques are somehow underspecified considering previous work in evaluating narrative generation...it seems difficult to rely on GPT-4 with generic evaluation prompts, meaning that plots or storyboards would be better evaluated by industry professionals.

For GPT-4 evaluation, we provided the exact prompt format we used in Appendix B on Page 17. Also, thanks for your suggestion for conducting human evaluation. We conducted human evaluation during the rebuttal session and please see the global response for details. Given the short amount of rebuttal time, we won’t be able to find industry professionals to evaluate the story plots, but will open source all generated stories for future reference.

How has RLHF been performed (user population, instructions, criteria...)?

After reward model training, we use standard RLHF objective (e.g., equation (2) in [2]):

$\mathcal{L}\_\phi = \mathbb{E}\_{(x,y) \sim D\_{\pi\_\phi^{\text{RL}}}} [r\_\theta(x,y) - \beta \log( \pi\_\phi^{\text{RL}} (y|x) / \pi^{\text{SFT}} (y|x))]$,

where $\pi\_\phi^{\text{RL}}$ is the learned policy by RLHF, $r\_\theta$ is the reward model, $\pi^{\text{SFT}}$ is the policy after SFT (our E2EPlot), and $\beta$ is to control the KL divergence. The prompt $x$ (i.e., the premise) is generated using the OpenPlot pipeline. We added the above formula in Section 2.3 in the revision.

Finally, we present the detailed result of human evaluation. Since we have five RLPlot models, we name them RLPlot_Q1, RLPlot_Q3, RLPlot_Q4, RLPlot_Q5, and RLPlot_Q6, respectively. We sent 500 (OpenPlot vs E2EPlot) + 300 * 5 (RLPlot vs E2EPlot) = 2000 story plot pairs, which are the same as what we used for GPT4 evaluation, to prolific, and ~1700 of them are labeled (each pair requires 1 label, each participant are required to label five pairs, and some of the participants labeled part of the five pairs). For each pair, the participants are required to answer seven questions, where Q1-Q6 are the same as in Table 2 in our paper, Q2 is a free text explanation for Q1, and Q7 is ‘Which story is better in overall quality’.

Note that in our original paper, for each story pair, we only asked GPT4 to evaluate one question. For OpenPlot vs E2EPlot, the question is overall quality; for RLPlot vs E2EPlot, the question is the corresponding aspect. For our human evaluation result, we provide results of all six (5+1) questions for each label. For completeness, we also conducted GPT4-evaluation for each of the 2000 pairs on all six questions. Below are the results of the human evaluation. (which can also be found in our revision on page 18-19, table 8-13, appendix C.1)

OpenPlot vs E2EPlot (sent 500, labeled 410)

RLPlot_Q1 vs E2EPlot (sent 300, labeled 248)

RLPlot_Q3 vs E2EPlot (sent 300, labeled 241)

RLPlot_Q4 vs E2EPlot (sent 300, labeled 252)

RLPlot_Q5 vs E2EPlot (sent 300, labeled 251)

RLPlot_Q6 vs E2EPlot (sent 300, labeled 248)