Do Language Models Agree with Human Perceptions of Suspense in Stories?

We greatly appreciate your enthusiastic and positive review. Thank you for your detailed feedback and helpful minor suggestions. Your commentary about how thoughtfully we worked to avoid anthropomorphizing the LLMs is sincerely appreciated. We will correct all mentioned typos (lines 290, 232) and add the missing reference promptly. Thank you for suggesting this additional reference for sentiment analysis. We agree it strengthens the theoretical grounding and will include it in our related work section, along with a specific construction of the problem framing to fit the work. Your review significantly enhances the clarity and accuracy of our manuscript, and we appreciate your precise suggestions and support. We hope that you can be a strong advocate for the novelty and impact of our work.

Thank you very much for your supportive review and recognition of our paper's originality and experimental rigor. The Readability of heatmaps and Table 2 is a very valid concern, and we agree that readability is crucial for communication. We fully commit to improving these visualizations in the revised manuscript by clearly labeling axes, reducing visual clutter, and enhancing color contrast using colorblind palletes for improved clarity and interpretability. We appreciate your valuable feedback and aim to incorporate your suggestions fully in our final version, improving readability and enhancing the paper’s impact. We acknowledge that the use of heatmaps are being used in an effort to visualize the data, but we have found that all other alternatives are even more difficult for non-authors to interpret. We would sincerely appreciate any and all feedback as to what you believe is the best approach.

Thank you for your thorough and thoughtful review. We appreciate your acknowledgment of the comprehensive and rigorous nature of our study. Regarding the search for deeper insights into cognitive gaps -- the experiments on adversarial attacks attempt to provide more insight into cognitive limitations of LLMs, notably highlighting their superficial text-processing strategies and limited narrative-context understanding. We will clearly state these cognitive gaps explicitly as limitations and attempt to guide future machine psychology. For example, featuring the use of adversarial attacks to test the validity of the underlying results is a compelling demonstration of how the 'Clever Hans' effect needs to be mitigated among LLM researchers. One of the contributions is a reproducible methodology that others can follow when running their own machine psychology experiments. While it was not our intention to produce a benchmark, experiments on LLMs suspense perception can now be reproduced en masse. One major contribution is providing an example of how to hold to a rigorous standard for machine psychology experiments.. This research establishes a baseline for future LM experiments when measuring the phenomnia of suspense. Psychological literature can be combined with NLP for new benchmarks by using our methodology. We will further highlight this methodological novelty explicitly in the revised paper. Your suggestion to provide concrete guidance for future LLM development is excellent. We agree on the necessity of explicit future directions. Based on our results, we suggest targeted improvements in LLM mid-training to incorporate stronger narrative-context understanding and explicitly reward models during reasoning reinforcement to understand the probabilities and certainties of uncertain (i.e., suspenseful) outcomes. These recommendations will be explicitly stated in our conclusion section. Thank you! Finally, regarding the generalizability to other NLP tasks -- while testing generalization to other narrative and non-narrative NLP tasks exceeds the scope of our current experiments, we acknowledge this as a significant and promising direction for future work. We will explicitly state this limitation and suggest further studies for broader NLP generalization. Also thank you for pointing out the missing reference -- we will fix it. We believe these clarifications and suggested revisions will significantly strengthen the paper and help address your primary concerns.

Thank you for the detailed and insightful feedback. We appreciate your recognition of our methodology and the importance of our research questions. Regarding the simplification of the phenomenon of suspense into a single number, we agree that reducing suspense to numeric values simplifies complex emotional experiences. However, numeric ratings using Likert scales are standard in psychological literature and facilitate clear, reproducible comparisons between human and LM judgments. We will explicitly acknowledge the limitations of numeric measures and suggest how we can integrate more thorough qualitative evaluations of human subject research for future work in machine psychology. To clarify the contextual understanding of LLMs -- our methodology explicitly provided LLMs with cumulative narrative context (prior story segments appended with each rating prompt). Additionally, we tested context-enhanced prompts (e.g., "in relation to the storyline so far"), but this did not significantly alter LM performance, reinforcing our conclusion that current LLMs inherently struggle with deep context integration. We will explicitly state this finding clearly in the manuscript. Our adversarial experiments directly addressed concerns about the analysis of cues used by LLMs. The adversarial experimentation explores how and when an LLM relies on local textual features and superficial cues rather than deeper structural or contextual elements. We will expand the discussion in our revised manuscript to explicitly include these insights. Finally, thank you for highlighting the point about differences in the models. While our current results already offer a comprehensive benchmark across diverse architectures, we commit to adding a brief comparative discussion of the model architectures and training differences explicitly to provide clearer guidance for interpreting model variability in the final manuscript. We hope these clarifications address your concerns and help improve your confidence in our evaluation.