Agents' Room: Narrative Generation through Multi-step Collaboration

We thank the reviewer for their great review and thoughtful feedback. We are delighted by the reviewer's enthusiasm for all the dimensions of our work, motivation, design, evaluation, technical depth! In the following, we elaborate on their comments:

1. About revising: We considered revising and editing agents but we first had to develop an evaluation framework for long-form narratives that was detailed and robust enough to capture subtle differences between narratives. Therefore, this current paper has a strong data and evaluation component. In future work, we would like to explore revising and editing in more depth. We will include some discussion around this in the camera-ready version.

2. Length as a confounder: Our evaluation rubrics (overall, plot, creativity, development, and language use) were specifically formulated to not be a function of the story length. In addition, the human raters were explicitly instructed to not use the length as a discriminating criteria.

3. Diversity of stories in the dataset: We will report a breakdown of the genres in our dataset. The majority of the stories belonged to the genres of science fiction and fantasy. We also included stories from the following genres: horror, drama, comedy, adventure, and folklore. We should also point out that stories often belonged to several genres (e.g., comedic horror or urban fantasy). We will add more discussion in the paper.

We thank the reviewer for their detailed comments and thoughtful feedback. We appreciate that the reviewer found our multi-agent system novel and considers long-form writing an important task for generation with LLMs. The reviewer also highlighted our human-written dataset contribution. In the following, we address their comments and provide answers to their questions.

1. Effectiveness of multi-agent collaboration

In terms of baselines, we would like to highlight that one of the single-agent baselines presented in the paper is actually LoRA-tuned on high-quality data (human-written gold-standard training data), which is a strong baseline. We show that all the multi-agent systems that include writing agents outperform this LoRA-tuned baseline, even when using prompted agents only.

In addition to these baselines, we will provide 4 additional baselines:

• A. Two baselines using detailed prompting (following Yang et al., 2022; Xie et al., 2023):
  • A1. The model is instructed to generate the central conflict, characters, setting, and plot according to detailed guidelines, before generating the story.
  • A2. The model is instructed to think about the central conflict, characters, setting, and plot according to detailed guidelines, before generating the story.

For both A1. and A2., the detailed guidelines are the same as those provided in Appendix B.2.

• B. Two baselines relying on planning strategies (following Yang et al., 2023; Lee et al., 2024):
  • B1. The model is instructed to generate a plan before generating the story in one call.
  • B2. The model is first instructed to generate a plan for the story, followed by a second call in which the model is instructed to generate the story based on the input prompt and the plan.

We will update this thread shortly with the corresponding metrics.

Some of the baselines mentioned by the reviewer can be seen as simpler variants of multi-agent systems, since they involve multiple agent calls with information passed from one agent to the next. As such, they are not purely “single”-agent setups. For instance, one of our ablations, the Agents’ Room zero-shot with only planning agents, corresponds to the prompt-chaining baseline mentioned by the reviewer. Our experiments show that this is the weakest of our multi-agent settings.

As for the turning points (following [1] as suggested by the reviewer), the Agents’ Room framework is modular and allows for varying subtask decompositions. Therefore our proposed writing agents can be replaced with writing agents based on turning points. We actually experimented briefly with turning points, but our structure based on narrative theory (Freytag, 1896; Pavis, 1998) yielded better results.

Overall, we would like to highlight that our proposed approach goes beyond these prompted methods, since we show that we can finetune and specialize the individual agents to improve the generated outputs, i.e. the different agents differ not only by the prompt, but they can be different specialized models.

2. Regarding the use of LLM as an evaluator

Many previous studies [1] have highlighted the challenges associated with evaluating narratives automatically. Metrics based on lexical matching correlate poorly with human judgments ([2], [3]) and do not effectively measure story quality (e.g., is the story original and well-written with plausible characters). Although we consider human-based evaluation our primary means to assess system output, we use the LLM evaluator for system development. Our correlation analysis shows that the LLM can be used for system-level analysis rather than to rate individual stories. Indeed Figure 2 shows that the LLM obtains broadly similar system rankings compared to the humans (although as the reviewer points out, it is biased towards its own output). We will discuss this point further and also make reference to very recent work [4] corroborating our findings. Specifically, Chhun et al. perform a large scale study on the ability of LLMs to evaluate story quality and report results similar to ours: LLMs show very high self-consistency, while correlations with human ratings remain weak but LLMs display some improvement over non-LLM automatic measures.

[1] https://arxiv.org/abs/2408.14622

[2] https://aclanthology.org/2022.coling-1.509/

[3] https://doi.org/10.1145/3613904.3642731

[4] https://aclanthology.org/2024.tacl-1.62/

Answers to questions

1. The participants of the writing workshop all have some type of advanced background in writing. Many have graduate degrees in English, some have professional writing experience, and many are former teachers. They did not use a writing assistant for this task.
2. Cf our discussion about baselines above. We will follow up shortly with the additional baselines’ metrics.

We thank the reviewer for their detailed comments and thoughtful feedback. We are glad to hear that the reviewer found our work thoughtful and solid and that they appreciated the involvement of actual human writers in the data collection and evaluation process. In the following, we address their comments and provide answers to their questions.

• Regarding the general writing framework: We formalized definitions of all components since many of these terms, such as “agents”, tend to be overloaded. While we acknowledge that this paper demonstrates the utility of this framework only for fiction writing, we believe this use case is a strong one because of its broadness and open-endedness. Adding further tasks, along with data, appropriate evaluations, etc. is beyond the scope of a single paper. We will move the discussion around generalization (such as lines 231-233) to the conclusion section.

• Figure 1: We will pick different icons to represent the agents and we will add labels under the individual agents.

• Regarding experimenting with more models: There are some constraints on the choice of model backbone to actually achieve good performance on the creative writing task. Indeed, the model has to have both a long enough context window and high enough writing quality. Most recent work on storytelling using a single model resorts to large, proprietary models such as GPT ([1], [2]) or CLAUDE [3]. This is also the case for multi-agent systems targeting writing which seem to be exclusively relying on GPT-4 ([4], [5]). Nevertheless, we are in the process of obtaining results with Gemma 2 9B to address the reviewer’s point.

[1] https://aclanthology.org/2023.acl-long.190/

[2] https://aclanthology.org/2022.emnlp-main.296/

[3] https://arxiv.org/abs/2309.14556

[4] https://www.ijcai.org/proceedings/2024/0003.pdf

[5] https://arxiv.org/abs/2307.05300

• Examples of model outputs: We are happy to include examples of model outputs. We will re-work Figure 2 to include the beginning of some stories and add complete examples in the appendix.

• About automated writing systems: We will add more reflection on the use of automated systems for complex writing tasks. One of the reasons for developing the Agents’ Room framework was to move away from one-step black-box generation, in order to facilitate integrating a human-in-the-loop via a shared editable scratchpad. Previous work in the context of summarization ([1], [2]) has shown that while users have different modes of interacting with LLMs (desiring more or less input from the AI), they overall expressed wanting more control over the generation process. A multi-step writing workflow with intermediate stages expressed in natural language that users can read and understand is a step in that direction. However, we leave designing an interactive user interface to assess collaborative writing in the context of our storytelling application to future work [3].

[1] https://arxiv.org/abs/2206.06383

[2] https://arxiv.org/abs/2206.14863

[3] https://dl.acm.org/doi/pdf/10.1145/3635636.3656201

Answers to questions

1. The human writers all have some type of advanced background in writing. Many have graduate degrees in English, some have professional writing experience, and many are former teachers.
2. Our human evaluators are better at spotting undesirable repetitions, inconsistencies, or incoherences than the LLM evaluator, so some discrepancies between both are expected. Our human participants found AR_write and AR_plan+write to be very similar when asked to rate the stories in the Overall dimension, whereas AR_write is somewhat better in terms of plot, creativity, and development, but worse in terms of language (see Figure 2). Although we did not elicit feedback on individual dimensions, we did ask participants to comment on the quality of the stories produced by our system, and possibly on aspects our rating system did not cover. We paste some of this feedback below and we will also include it in the appendix of our paper:

"The task was interesting, but over time, I found the language redundant. There seemed to be a go-to vocabulary list utilized in the majority of the stories, phrases used time and again, making the output somewhat predictable."

"It was interesting to see what kind of fictional narrative the model would generate. Most of the stories seemed to be written at a seventh grade level. The stories didn't stray too far from the input and for the most part were grammatically correct. There were at times, instances of repetitiveness, including entire paragraphs, that made me wonder what the model was doing."

"The stories showed some promise, but often fell into the same pitfalls of loops or sudden tone discordance…"