PRobELM: Plausibility Ranking Evaluation for Language Models

Error Analysis with Surface Form Manipulation:

Thank you for this insightful suggestion. We agree that examining the robustness of models' judgments through surface form manipulations, such as paraphrasing and syntactic transformations, would add significant value to our analysis. We are committed to incorporating this additional error analysis into the next version of our paper to better understand the models' robustness and variability in plausibility judgments.

Related Work:

We appreciate the reference to the recent preprint by Kauf et al. (2024) on evaluating LLMs from a psycholinguistic perspective. We find the parallels in their findings and our results particularly intriguing, especially the observation that LLM-derived log-likelihoods can approach human-level distinction for possible versus impossible events but fall short on subtle distinctions. We will include a discussion of this related work in our next version to provide a more comprehensive context for our findings and to highlight relevant research in this area.

Thanks for your review! We hope to clarify all your questions:

Realization of Plausibility:

The approach we take is not directly related to the surprising nature. We consider the most plausible scenarios as immediate next events that, while not recorded in the LLMs’ training data, have since become facts. Imagine traveling back to ancient times when people believed the Earth was flat; the scenario "the Earth is round" should be most plausible based on your knowledge, though it would seem very surprising to them. This shows the importance of plausibility in knowledge discovery, distinct from forecasting future events. We acknowledge that our writing in that section might have been confusing as we talked about "near future events." We will rephrase this to clarify.

Markovian:

Our model doesn't adopt a strictly Markovian approach. While individual timestamps in Wikidata might suggest a single-step view, the nature of world knowledge and historical events captured in LLMs is inherently non-Markovian. This broader context allows for more practical and comprehensive evaluation of plausibility, reflecting real-world applications better.

Bayesian Graph Structure:

Exploring the graph structure of a Bayesian model to create new evaluation cases is a promising future direction. Yet, our current focus is on scenarios that have already occurred but not in the training data.

Comparison with COPA:

COPA focuses on commonsense reasoning without requiring knowledge about the world. An example in COPA, "I tipped a bottle" leading to "The liquid poured out" is self-contained and doesn't challenge the model’s world knowledge. In contrast, our benchmark requires models to leverage extensive world knowledge, making the task more complex and relevant for knowledge discovery.

Hyperparameter for Less Plausible Events:

Regarding why we chose 10 less plausible scenarios, we aimed to make it reasonably difficult for models to score well with random guesses. We can explore with different numbers of less plausible scenarios in future versions of the paper to provide a comprehensive evaluation.

Negative Sampling:

Negative sampling is centered around event relations, and all 71 relations and 213 templates are detailed in the appendix D (the relations are manually selected in the QC process). This ensures a diverse and representative set of scenarios for evaluation. The full list and distribution of event relations are provided to maintain transparency and reproducibility.

Thanks for the review.

Ground Truth Ranking:

We use information from Wikidata added after the training cut-off of the LLMs. This ensures our benchmark evaluates their ability to infer new world knowledge beyond co-occurring words. Our analysis shows that the most frequently co-occurring entity is not always the most plausible entity (>~83% of the cases), illustrating that plausibility often requires reasoning beyond mere co-occurrence.

Generating Ranking Candidates in Real Applications:

Our benchmark can assess the plausibility of selected outcomes but not an exhaustive list. Extending our benchmark to rank a full list of outcomes is a promising future direction and we plan to achieve this with human annotation. This will enhance our benchmark in real-world applications, such as knowledge discovery, where identifying the most plausible hypotheses from a broader set is crucial.

Difference from Beam Search:

Beam search could indeed be used to obtain more plausible hypotheses, and its use is orthogonal to evaluation with our benchmark. However, maximum a posteriori estimates are not always the best way to decode a model, which is why minimum Bayes risk decoding is popular in fields like machine translation. Our method focuses on evaluating the plausibility of scenarios using perplexity, providing a different perspective than beam search.

Deeper Analysis on Key Observations:

Our finding that higher performance on factual accuracy does not necessarily correlate with better plausibility performance can be attributed to the distinct nature of these tasks. Factual accuracy tests a model's ability to recall and verify known facts, while plausibility requires the model to infer and rank scenarios that might not be explicitly present in its training data but are logical extensions of its knowledge. For example, Marcelo being Brazilian is factual, but Marcelo being Spanish is plausible due to his history and Spanish naturalization laws.

The drop in performance with the 2017 dataset is due to the specific characteristics of the data used. The "occupation" relation disproportionately influenced the performance. When “occupation” involves uncommon terms like "valets de chambre," the perplexity is very low across all models, making it the top choice regardless of the subject entity. This occurs because such specific terms might have been less frequently represented in the training data, causing models to assign lower perplexity to these terms when they appear.