Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection

Thank you for your review. We are glad you found our research direction “innovative” and found our plot hole generation algorithm and evaluations rigorous. Below we try to address your concerns:

---

the benchmark only covers human-written stories and neglects machine-generated stories, which I think are more meaningful since we do not expect LLMs to generate illogical stories

Could you please provide some works supporting your claim that machine generated stories are more meaningful than human authored ones? There is evidence in prior work about LLMs generating narratives with logical inconsistencies [4,5]. Section 6 of our paper also provides evidence towards this phenomenon, where we found LLM generated stories and summaries exhibit a significant rate of detected plot holes.

---

The algorithm relies heavily on LLMs and human filtering

We would really appreciate if you could please elaborate your issue with using LLMs and human filtering for our data generation algorithm? We believe using LLMs with human filtering is actually a strength of our approach -- Using LLMs enables us to automatically generate high quality data for the plot hole detection task, while avoiding the pitfalls of dataset contamination i.e. the stories edited with our approach couldn't be present in LLMs' training data by design. Performing human verification on top of that helps ensure the legitimacy of a story and its corresponding plot hole edited with our algorithm, thus ensuring high quality evaluation data for the benchmark. This approach is also in line with a lot of recent work that uses LLMs to generate synthetic data for building benchmarks [1,2, 3].

---

The benchmark primarily consists of short stories

We believe that using stories with relatively short length is actually a strong point of our benchmark, since we are able to test limitations of LLMs, maintain high quality human filtering since the cognitive load increases significantly with length, and enable evaluation of models regardless of the context window they’re trained on. Our algorithm is length-agnostic and enables generating even longer and complex data as model capabilities improve. We'd like to highlight that it is not our intention to fully represent all human narratives, but rather give a general framework to generate narrative understanding evaluations that are more robust to data leakages.

---

the benchmark's stories are sourced from specific domains such as fairy tales and myths

We would like to clarify that FlawedFictions does include other genres than fairytales and myths, like historical fiction , literary fiction, comedy etc. The detailed statistics of these genres which were tagged using GPT-4o-mini is provided below. While Folklore / Fairy Tale does dominate FlawedFictions with 43% stories followed by Mythology with 10% of the stories, there still are ~50% stories coming from other genres. Similar to the point we made about story lengths, our method is agnostic to the story genre and future work can build upon our work to generate stories with different distribution of these genres or even explore other domains like non-fiction.

---

The paper's evaluation metrics focus on the accuracy of model predictions but may not fully capture the richness and complexity of narrative reasoning

We would like to clarify that we do not just evaluate accuracy metric but also evaluate if the model is able to correctly localize the sentences in the story that have the plot hole as well as the sentences that are contradicted by the plot hole (the metric is named CEEval-Full in the paper). While we initially considered evaluating models’ explanations of the errors as well, those are much harder to evaluate due to their open-ended nature. However, through qualitative checks, we found a strong correspondence between a high CEEval-Full score and the model correctly explaining the plot hole in the story and vice versa.

---

The response continues in the next comment

Thank you for the detailed review. We are glad that you appreciated the new dataset and task in our work as well as found the evaluation “broad” and the analysis “offering practical diagnostic value”. Below we try to address your concerns:

1. Positioning our work with respect to recent work

Thank you for sharing the three papers in your review. In contrast to ours, these works do not introduce a new dataset or benchmark, but rather focus on improving narrative understanding using existing benchmarks (see detailed comparison below). We believe that our work is novel with respect to all the aspects that you mention in comparison to these papers and provide a detailed comparison below:

• Task Novelty: We propose a novel narrative reasoning problem i.e. detecting plot holes in stories, not studied in any of these works, requiring distinct capabilities. The specified papers on the other hand focus on different problems like temporal reasoning, predicting movie tropes, and grounding story events in real world knowledge, which do not dictate how well can models reason for plot holes in stories.

• Algorithm and Benchmark Novelty: We propose a novel algorithm to automatically edit stories to contain plot holes in them: this partial-synthetic data approach avoids the risk of data contamination and aims to disentangle memorization from LLM reasoning capabilities. In contrast,the three papers mentioned do not introduce any new benchmarks or methodologies to curate such datasets, and only consider existing datasets in their evaluations. Jiayang et al. 2024 [1] consider StoryCloze (Mostafazadeh et al. 2016 [5]) and Multiple Choice Narrative Chain (Granroth-Wilding and Clark, 2016 [6]) datasets for their experiments. Su et al. 2024 [2] focuses on trope classification in movie summaries using the existing TiMoS dataset (Chang et al. 2021 [4]) and Zhang et al. 2024 [3] evaluates their prompting technique for the temporal graph generation task using datasets like ProScript (Sakaguchi et al. 2021 [7]).

• Quality of Stories included in FlawedFictions: Our benchmark includes natural human-authored stories, which are edited using our proposed algorithm FlawedFictionsMaker. These include stories with an average of 730 words for FlawedFictions and 2700 words for FlawedFictionsLong. In contrast, Jiayang et al. 2024 [1] and Zhang et al. 2024 [3] work with toy stories containing an enumeration of story events (e.g. Tom was tired, Tom wanted to have fun, Tom bought a movie ticket for Harry Potter) and they are significantly shorter. While Su et al. 2024 [2] focuses on longer stories, which are of similar lengths as our benchmark, these are the synopsis of movies, which potentially have different narrative structures compared to full stories.

• Novel Evaluation Design To evaluate how accurately a model is able to identify a continuity error in the story, we propose a localization task (with its corresponding evaluation metric CEEval-Full) that not only requires the model to correctly predict the story span containing the error but also the span entailing the fact that is contradicted. In contrast, Jiayang et al. 2024 [1] and Su et al. 2024 [2] focus on classification tasks for predicting a story's ending and movie tropes respectively, and Zhang et al. 2024 [3] focuses on a graph prediction problem using the graph edit distance as their evaluation metric.

• Focus on Assessing Quality of LLM Generated Narratives: Besides evaluating the narrative reasoning capabilities of LLMs, we also evaluate the quality of LLM generated stories and summaries in terms of presence / absence of continuity errors in them. The three works focused on the reasoning aspect but not on narrative generation, which also distinguishes our work.

To summarize, we propose a novel task of plot hole detection in our work, propose an automatic algorithm to curate a benchmark for the task, and in addition to evaluating LLMs’ capabilities to reason for plot holes, we also evaluate the quality of LLM generated stories in terms of their narrative consistency. We will make sure to update the related work section to include these works in the final version.

---

The response continues in the next comment

Thank you for the detailed review. We are glad that you found our proposed algorithm novel, representing “practical initial step towards creating controllable story datasets with such flaws”. We are encouraged to know that you found our evaluation “comprehensive” and found our formalization of continuity errors to potentially be “be a significant contribution that inspires further research in this area”. We would like to assure you that we will be releasing our benchmark and code for public use with the final paper version.

Please see the response below where we try to address your concerns:

---

I also wonder if the authors explored the performance of a fine-tuned small language model, in addition to their experiments with large language models.

We did try fine-tuning the Llama-3.1-8b Instruct model using LoRA on a separate dataset generated with FlawedFictionsMaker and found that fine-tuning does improve performance of the base model but overall performance remains low. The results are provided in the table below (along with performance of some other representative models):

The training data used here consisted of ~1300 stories (both positive and negative examples) generated from the summaries of stories included in FairyTaleQA. Note that the training data wasn’t verified by human annotators unlike the data we use for the benchmark. We believe that with better filtering of the training data and using alternate training objectives like Reinforcement Learning with Verifiable Rewards (RLVR) might improve the models’ performance further, but we leave that exploration for future work.

---

the paper's scope appears to be limited to a single type of plot hole

While we agree that we focus on continuity errors in this work, as we explain in the paper (Lines 94-97), continuity errors can entail other forms of plot holes as well like out of character behavior and impossible events. Out of character behavior can be viewed as breaking the continuity of characters' established motivations and characteristics by their behavior later in the story. We have one such examples of FlawedFictionsMaker generating an out of character behavior in the appendix section A.7 (third example), where a character is initially portrayed as courteous and gentle, one who rejoices in success of others, but later the story shows conflict in the court due to his aloofness and self-centered nature. Similarly, impossible events represent events that contradict logic or scientific principles that are consistent with the story’s world. In appendix section A.7 in one of the stories (first example), two characters are shown to leave the house but are present in the house in a later scene, representing a logically impossible event as a character cannot (normally) be in two places at the same time. However, we do acknowledge that these cases are relatively rare. Based on our estimates using GPT-4o ~22% of the continuity errors in the dataset can be interpreted as Out of Character Behavior and 3% as Impossible events. Future work can explore extensions to FlawedFictionsMaker for controlling the specific type and complexity of plot holes to be introduced.

---

I noted that several significant aspects of the paper are located in the appendix.

Due to the limited space we had to move some of the details to the Appendix. However, we tried to ensure that the main paper can be read in a self contained manner. Since CoLM provides an extra page for the final version, we will move results from section A.5 to the main paper, which contains the additional error analysis.

---

The response continues in the next comment

Thank you for taking the time to review our paper and providing detailed feedback. We are glad to know that you found our paper to provide “comprehensive results” and the plot hole detection task to be “challenging and intriguing”. Below we try to answer your questions and concerns:

---

No IRB is mentioned; please describe the level of IRB review

We applied for two IRBs for this work, one for the dataset verification through annotators on prolific and one for the human performance evaluation through college undergraduates. In both cases we got an IRB exemption from the Human Subjects Division which determined our studies as human subjects research that qualifies for exempt status (Category 101). We will share the IRB IDs for both studies in the final version of the paper.

---

I don't fully understand the continuity error definition, and I don't think it's settled by the "extended" version with reader's beliefs.

Thank you for this important question. Let us provide a concrete example to clarify the definition and address the negation vs. opposition concern.

Consider the following story:

• Jack is drinking hot tea in a cup
• Jack gives his tea to Jill
• Jill drinks the cold tea from Jack’s cup

From this we extract propositions: $\phi_1​ =$ "Jack drinks tea", $\phi_2 =$"tea is hot", $\phi_3 =$ "Jack gives tea to Jill", $\phi_4 =$ "Jill drinks tea", and $\phi_5 =$ "tea is cold".

You are absolutely correct that removing $\phi_5$​ from our proposition set does not allow us to conclude $\neg \phi_5$​ ("tea is not cold") from the remaining propositions alone, since "hot" does not logically entail "not cold" without additional assumptions.

This is precisely why we need the extended definition with reader beliefs. A reader's commonsense knowledge includes the belief that "if tea is hot, then tea is not cold" ($\phi_2 \to \neg \phi_5$​). When we include this reader belief in our reasoning, we can derive the contradiction:

$\\{ \phi_2 \to \neg \phi_5, \phi_2\\} \vdash \neg \phi_5$​ (by modus ponens)

This shows that $\phi_5$ is logically inconsistent with the rest of the story when combined with reasonable reader beliefs, making it a continuity error according to our extended definition.

The key insight is that continuity errors in fiction cannot be detected purely from explicit textual propositions—they require the integration of implicit world knowledge that readers bring to their understanding of the story. Thanks again for asking this question. We will include similar examples in section 2, so that the definitions are easier to understand.