TRAM: Benchmarking Temporal Reasoning for Large Language Models

Reviewer PXAF, thank you for providing the insightful suggestions. We hope the following responses address your concerns.

Weakness 1: Difficulty of the benchmark.

We appreciate your concern about the difficulty level of our TRAM benchmark. The purpose of this benchmark is to provide a comprehensive test that covers a wide range of temporal reasoning abilities for LLMs. As detailed in our paper, most tasks consist of multiple subtasks. Although the average SOTA performance is high, among the eight tasks you mentioned, which total 34 subtasks, there are 12 out of 34 subtasks where the GPT-4 (5S, CoT) setting performs at or below 80%. For instance, in this setting, the Ambiguity Resolution (calendar-shift) task achieves 53% accuracy, the Typical Time (comparison) task 77%, and the Arithmetic (date computation) task 78%. These results underscore specific areas that require further refinement, even in SOTA models.

Moreover, the performance gap between GPT-4 and human benchmarks warrants particular attention. While this gap may seem modest (within 5%), achieving even incremental improvements is notably challenging as models approach human-level accuracy. Therefore, these seemingly small gaps are, in fact, substantial indicators of areas where future LLMs can and need to improve.

Meanwhile, to address your concern and challenge the models more rigorously, we have considered some straightforward and effective methods, including introducing additional options and modifying question formats. To test the efficacy of these methods, we conducted case studies on the Causality and Storytelling tasks, which initially showed high performance. Under GPT-4 (5S, CoT), we reevaluated 50 samples from each task that were previously predicted correctly. When the distracting choices were contextually close to the correct answer, the model's performance dropped, with 9 out of 50 incorrect predictions for Causality and 17 out of 50 for Storytelling. In contrast, when distractors were clearly differentiated from the correct answer, the model had 5 out of 50 incorrect predictions for Causality and 13 out of 50 for Storytelling.

Even in a SOTA setting, our deliberate introduction of additional options can decrease model performance by more than 10%. Additionally, converting some tasks to a short-answer format has been shown to affect performance; for example, the SOTA performance on the Arithmetic task dropped from 94.3% to 89.6%. We are dedicated to continuously refining the tasks in our benchmark and to providing a variety of question formats to meet different levels of model testing. Updates to question sets and results will be regularly shared on our GitHub page.

Weakness 2: Temporal vs. general reasoning in model errors.

We designed our temporal reasoning (TeR) benchmark, recognizing the importance in understanding and reasoning about time. Although TeR is a specialized form of general reasoning, our aim is to isolate and evaluate this unique aspect due to its broad use in everyday life. In our error analysis, we distinguish between general reasoning and TeR when evaluating LLM capabilities. For example, in the Frequency task with the question ‘If a person’s job contract has a renewal every 4 years, and they started working in 1857 and renewed it 3 times without gaps, until what year is their current contract valid’, GPT-4’s error was categorized as ‘Contextual Misjudgment’. The model incorrectly assumed that each 4-year contract period concludes at the beginning of the following year, instead of at the end of the fourth year. This error demonstrates a misunderstanding in processing temporal information, rather than a general reasoning flaw. While time-related mistakes may occur within a broader reasoning context, our primary objective is to gauge a model’s progress in TeR.

Question 1: TRAM vs. existing temporal reasoning benchmarks.

The following table provides a comparison between our TRAM and previous temporal reasoning datasets, including task coverage and data size. The order of the prior work listed is from the most recent to the earlier ones.

Reviewer Z4nr, thank you for providing the insightful suggestions. We hope the following responses address your concerns.

Weakness 1: Difficulty of the benchmark.

We appreciate your concern about the difficulty level of our TRAM benchmark. The purpose of this benchmark is to provide a comprehensive test that covers a wide range of temporal reasoning abilities for LLMs. As detailed in our paper, most tasks consist of multiple subtasks. Although the average state-of-the-art (SOTA) performance is high, particularly for 8 out of 10 tasks with accuracies close to or exceeding 90%, there are 12 out of 35 subtasks where the GPT-4 (5S, CoT) setting performs at or below 80%. For instance, in this setting, the Ambiguity Resolution (calendar-shift) task achieves 53% accuracy, the Typical Time (comparison) task 77%, and the Arithmetic (date computation) task 78%. These results underscore specific areas that require further refinement, even in SOTA models.

Meanwhile, to address your concern and challenge the models more rigorously, we have considered some straightforward and effective methods, including introducing additional options and modifying question formats. To test the efficacy of these methods, we conducted case studies on the Causality and Storytelling tasks, which initially showed high performance. Under GPT-4 (5S, CoT), we reevaluated 50 samples from each task that were previously predicted correctly. When the distracting choices were contextually close to the correct answer, the model's performance dropped, with 9 out of 50 incorrect predictions for Causality and 17 out of 50 for Storytelling. In contrast, when distractors were clearly differentiated from the correct answer, the model had 5 out of 50 incorrect predictions for Causality and 13 out of 50 for Storytelling.

Even in a SOTA setting, our deliberate introduction of additional options can decrease model performance by more than 10%. Additionally, converting some tasks to a short-answer format has been shown to affect performance; for example, the SOTA performance on the Arithmetic task dropped from 94.3% to 89.6%. We are dedicated to continuously refining the tasks in our benchmark and to providing a variety of question formats to meet different levels of model testing. Updates to question sets and results will be regularly shared on our GitHub page.

Weakness 2: Questions from other benchmarks.

Thank you for your feedback regarding the use of questions from existing datasets in our benchmark. We acknowledge this as a limitation of our work and understand the potential issues it might pose, particularly in the context of larger benchmark suites where double-counting could lead to skewed conclusions about model performance. Similar to the GLUE and SuperGLUE benchmarks, our adoption of pre-existing datasets is based on their implicit acceptance within the NLP community. For questions sourced from existing datasets, we conducted significant preprocessing and reformulation to align them with the specific objectives and format of TRAM. Our preprocessing involved adapting the questions to effectively test temporal reasoning capabilities and reformatting them to fit the structure of our benchmark. This process included manually adding distracting or confusing options, filtering out irrelevant questions for quality control, and reformulating problems. Due to this extensive manual preprocessing, the overlap with larger benchmark suites like Google’s BigBench should be minimal. To further address your concern and enhance LLM evaluation, we have undertaken the following actions and plans:

1. We have conducted some case studies on tasks in our benchmark, including Causality and Storytelling, by a) adding additional options and b) modifying question formats. The reason for experimenting with a) is to challenge models, potentially confuse them, and test the nuanced reasoning capabilities of LLMs. On the other hand, b) is implemented to avoid the possibility that models can guess instead of truly reasoning. These direct methods have proven effective in our case studies, as demonstrated by the decrease in model performance. Meanwhile, even if models like GPT-4 have been exposed to existing datasets during pretraining, we can still present additional challenges, such as confusing options and challenging problem formats, to test their abilities.

Reviewer v5US, thank you for providing the constructive suggestions. We hope the following responses address your concerns.

Weakness 1: Question format.

We employ multiple-choice questions (MCQs) in TRAM to align with established practices in LLM benchmarks, such as the AI2 Reasoning Challenge (ARC) and Massive Multitask Language Understanding (MMLU), which also utilize MCQs. Additionally, the MCQ format is the most straightforward method for evaluating LLMs. Nevertheless, we recognize the potential limitations of this format. On our GitHub page, we have provided an alternative version of the benchmark in a short-answer format where appropriate, except for tasks like storytelling, ambiguity resolution, and causality. For these three types of questions, I have retained the MCQ format, as other formats could lead to answers that are too open-ended and difficult to evaluate. As part of our ongoing research, we remain open to incorporating more diverse question formats and adding more challenging questions to each task. These updates will be made available on our GitHub page, adapting to the evolving needs of the research community and the developing capabilities of LLMs.

Weakness 2: Imbalanced problems per category.

Our causality task is inspired by the Choice of Plausible Alternatives (COPA) dataset, which itself comprises only 1,000 questions. Given that COPA has set a precedent in the field, particularly for tasks centered around causal reasoning, the number of questions it contains serves as a benchmark for what can be considered sufficient for a robust evaluation. Furthermore, 93.5% of the questions in our original causality task are meticulously crafted by hand. This manual process, while ensuring the precision and applicability of each question, also limits the feasible scale of our dataset. To address the limited size of the causality category, we have added an additional, mirrored instance for each original instance. This expands our causality task to a total of 1,200 questions. Each mirrored instance uses the same alternatives as the corresponding original instance but introduces a new premise that matches the incorrect alternative of the original instance. We have updated the relevant information on this specific task in our revised paper, mainly including updated BERT-based results for causality and the task description. We are committed to continuously refining and potentially expanding this task, with updates to the question set being regularly posted on our GitHub page. This effort is specifically aimed at mitigating issues associated with the pretraining-finetuning paradigms, particularly those arising from limited training samples.

Weakness 3: Texts based on existing datasets.

Thank you for your feedback regarding the use of questions from existing datasets in our benchmark. We acknowledge this as a limitation of our work and understand the issues it might pose, particularly in terms of potential overestimation of model capabilities. Similar to the GLUE and SuperGLUE benchmarks, our adoption of pre-existing datasets is based on their implicit acceptance within the NLP community. For questions sourced from existing datasets, we conducted significant preprocessing and reformulation to align them with the specific objectives and format of TRAM. Our preprocessing involved adapting the questions to effectively test temporal reasoning capabilities and reformatting them to fit the structure of our benchmark. This process included manually adding distracting or confusing options, filtering out irrelevant questions for quality control, and reformulating problems. Due to this extensive manual preprocessing, the overlap with larger benchmark suites like Google’s BigBench should be minimal. To further address your concern and enhance LLM evaluation, we have undertaken the following actions and plans:

1. We have conducted some case studies on tasks in our benchmark, including Causality and Storytelling, by a) adding additional options and b) modifying question formats. The reason for experimenting with a) is to challenge models, potentially confuse them, and test the nuanced reasoning capabilities of LLMs. On the other hand, b) is implemented to avoid the possibility that models can guess instead of truly reasoning. These direct methods have proven effective in our case studies, as demonstrated by the decrease in model performance. Meanwhile, even if models like GPT-4 have been exposed to existing datasets during pretraining, we can still present additional challenges, such as confusing options and challenging problem formats, to test their abilities.

Reviewer rYw6, thank you for providing the detailed suggestions. We hope the following responses address your concerns.

Weakness 1: GitHub page.

Following your suggestion, we have set up an anonymous GitHub page. Here is the link: https://anonymous.4open.science/r/TRAM-Benchmark-596D/.

Weakness 2/Question 2: Overall contribution of the work.

Thank you for your insightful suggestion about clarifying the contribution of our work and the rationale behind the transformation of existing data sources into the TRAM dataset. The following table provides a comparison between our TRAM and previous temporal reasoning datasets, including task coverage and data size. The order of the prior work listed is from the most recent to the earlier ones.

Compared to earlier works, our dataset offers a much larger data size and broader task coverage, encompassing 10 main tasks and 38 subtasks. For instance, in the classic tasks of ‘Duration’ and ‘Frequency’, earlier works mainly cover Facts and Commonsense. In contrast, our work includes 7 subtasks for ‘Duration’ and 6 subtasks for ‘Frequency’, delving into more nuanced aspects of these temporal elements, which are less emphasized by earlier works. Consequently, the extensive data size of TRAM and the intricate design of its subtasks represent a notable change in both the depth and breadth of temporal reasoning (TeR) evaluation, compared to existing benchmarks.

Following your suggestion, we have included examples in Appendix F (pages 23-24) that compare our curated dataset with the source datasets, specifically MCTACO and SQuAD. For information on data quality, please refer to the datasheets available on our anonymous GitHub page. The reformulation of original data sources into the TRAM benchmark offers several advantages:

Increased Complexity and Depth: By converting simple Yes/No questions to multiple-choice format, we inherently increase the complexity of each question. This change demands that models not only identify the correct answer but also discern and reject multiple plausible but incorrect options, thereby enhancing the depth of temporal reasoning evaluation.

Reduction in Guesswork: The multiple-choice format reduces the likelihood of correct guesses compared to Yes/No questions, demanding a more nuanced understanding and reasoning, and thus providing a clearer assessment of a model's temporal reasoning capabilities.

In summary, we have chosen to use existing benchmarks due to their wide acceptance within the NLP community. The multiple-choice format, effective for evaluating LLMs as seen in benchmarks like the AI2 Reasoning Challenge (ARC) and Massive Multitask Language Understanding (MMLU), introduces more complexity for the model and challenges it to discern subtle differences in temporal reasoning contexts. Our benchmark combines the strengths of existing benchmarks with extensive manual work, setting a new standard for assessing temporal reasoning in LLMs. To address the evolving needs of advanced model evaluation, we will regularly share updates to our question sets and results on our GitHub page.

Weakness 3/Question 1: Overall dataset description.

Thank you for providing the insightful advice. Following your suggestions, we have created a datasheet for the TRAM benchmark, which is available on the anonymous GitHub page (datasheet_for_TRAM_benchmark.pdf). Please let us know if there are any additional aspects you think would be worthwhile to incorporate into the datasheet. We are open to suggestions and will gladly update it accordingly.

Weakness 4/Question 3: Performance disparity among LLMs.

Thank you for your feedback regarding the use of questions from existing datasets in our benchmark. We acknowledge this as a limitation of our work and understand the issues it might pose, particularly in terms of performance disparity among different LLMs. Similar to the GLUE and SuperGLUE benchmarks, our adoption of pre-existing datasets is based on their implicit acceptance within the NLP community. For questions sourced from existing datasets, we conducted significant preprocessing and reformulation to align them with the specific objectives and format of TRAM. Our preprocessing involved adapting the questions to effectively test temporal reasoning capabilities and reformatting them to fit the structure of our benchmark. This process included manually adding distracting or confusing options, filtering out irrelevant questions for quality control, and reformulating problems. The performance disparity, especially the superior results of GPT-4, can be attributed to varying architectures, training dataset scales, and sizes of the models. While GPT-4 may have been exposed to these datasets during pretraining, our manual work in reformulating the tasks aims to introduce new dimensions and challenges, thus minimizing any potential advantage. To address both this issue and the need for advanced LLM evaluation, we have undertaken the following actions and plans:

1. We have conducted some case studies on tasks in our benchmark, including Causality and Storytelling, by a) adding additional options and b) modifying question formats. The reason for experimenting with a) is to challenge models, potentially confuse them, and test the nuanced reasoning capabilities of LLMs. On the other hand, b) is implemented to avoid the possibility that models can guess instead of truly reasoning. These direct methods have proven effective in our case studies, as demonstrated by the decrease in model performance. Meanwhile, even if models like GPT-4 have been exposed to existing datasets during pretraining, we can still present additional challenges, such as confusing options and challenging problem formats, to test their abilities.

2. We will prepare and craft more challenging questions for our benchmark, such as reframing existing tasks by introducing novel contexts or counterfactual elements, and introducing a set of novel tasks that require non-linear temporal reasoning. These tasks might include analyzing events in reverse chronological order or understanding cyclical time concepts, which are less likely to be present in standard training datasets. We are dedicated to continuously refining the tasks in our benchmark and providing a variety of question formats to meet different levels of model testing. Updates to question sets and results will be regularly shared on our GitHub page.

Reviewer fFct, thank you for providing the insightful suggestions. We hope the following responses address your concerns.

Weakness 1: Temporal reasoning model comparison.

Thank you for highlighting the need for a comparison with domain-specific models. Following your suggestion, we have conducted additional experiments using a temporal reasoning model (https://huggingface.co/GAIR/rst-temporal-reasoning-11b), referred to as RST-TeR below, where the training and testing sets are the same as those used for BERT-based models. We have also included results from RoBERTa-large and GPT-4 for reference and comparison:

Based on these results, RST-TeR achieves performance on par with the RoBERTa-large model. However, there is a significant gap between this domain-specific model and LLMs like GPT-4, except in the Relation (Rel.) task, where RST-TeR is specifically trained. GPT-4’s extensive training across varied contexts contributes to its robust performance, showcasing its superior generalization capabilities. Overall, LLMs bring depth and versatility to temporal reasoning, enabling them to handle a range of tasks that demand a comprehensive and integrative approach, an aspect where specialized models may be more limited.

Question 1: Impact of length of questions and number of options on model performance.

Thank you for raising the insightful question about how the length of questions and the number of options in MCQs affect model performance. Here are the statistics for the average length of questions, options, and their combined average for each task:

Based on our experiments, we have the following observations:

Length of questions vs. Performance: There is no direct correlation between question/option length and model performance. For instance, tasks with longer questions such as Typical Time and Storytelling have some of the highest accuracies, indicating that the additional context from longer questions can be beneficial. Conversely, shorter tasks like Arithmetic and Ambiguity Resolution do not necessarily result in higher performance, suggesting that brevity, which may reduce context, does not automatically improve performance. For the tasks in our study, as there is no substantial difference in length, the intrinsic task complexity appears to have a more pronounced effect on the final performance rather than the question length.

Length of options vs. Performance: Longer option lengths in tasks like Causality and Storytelling do not lead to uniform decreases in performance, implying that detailed options may provide valuable context that helps the model deduce the correct answer. However, if the added length introduces ambiguity or irrelevant details, it could confuse the model.