SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models

Dear Reviewer rN5c,

We sincerely appreciate your valuable feedback and suggestions, which have significantly contributed to improving our work and broadening our evaluation. Below, we address each of your comments and questions:

About the motivation and importance of SPORTU:

First, we want to reaffirm the motivation behind SPORTU. Sports rules are a critical and indispensable part of sports, designed to protect players, ensure fairness, and enhance the viewing experience. These rules vary significantly across different sports, making them a key aspect of sports understanding. As MLLMs begin to demonstrate reasoning capabilities in various domains, sports remain an under-explored area. MLLMs with advanced reasoning capabilities can help non-experts quickly grasp sports rules and dynamics, enabling them to better enjoy and understand games. Existing works, as discussed in our introduction and related work, either lack depth by not incorporating detailed rule-related questions—an essential component for evaluating sports understanding—or focus on a single sport, limiting their scope. Evaluating MLLMs' capabilities in just one sport is insufficient to demonstrate comprehensive sports understanding across diverse disciplines. This is why we proposed SPORTU, a benchmark designed to address these gaps by covering multiple sports and incorporating tasks that require detailed reasoning. To enhance the dataset's diversity and quality, we introduced slow-motion and multi-camera angle videos. These features are essential for evaluating MLLMs in practical and realistic conditions:

• Slow Motion: In real-world sports broadcasts, fouls and rule violations are often difficult to identify in real-time due to normal-speed, wide-angle shots. Critical details, such as subtle hand or foot movements, are typically captured through slow-motion replays. By including slow-motion clips, we ensure that models are tested on fine-grained action recognition, which is critical for sports reasoning tasks.
• Multi-Camera Angles: In professional sports, rule violations are often reviewed from multiple angles during video replays to ensure accurate judgment. Multi-camera angles in our dataset enable us to test whether models can maintain consistency in reasoning and predictions across different perspectives. This is especially valuable for evaluating MLLMs’ robustness and understanding in dynamic scenarios.

We recognize the need to provide more discussions and visualizations to emphasize these motivations. To address this, we have added examples in Appendix D to demonstrate cases where the same foul yields different model outputs depending on the angle.

Additionally, Section 3 have been updated to explicitly discuss the rationale for including slow-motion and multi-camera videos. We also added a Broader Impacts section in Appendix B to discuss the border impacts of our benchmark.

About missing details in dataset construction:

Our expert annotators manually cropped video clips from replay footage to include multiple perspectives of the same foul. Since such replays are standard in sports broadcasts, this process ensures accuracy while minimizing additional manual work. As for the division between multiple-choice and open-ended questions, this is explained in Sections 3.2 and 3.3, but we have revised these sections to make the process clearer.

• Multiple-Choice Questions: Annotators labeled the category of the ground truth such as foul (e.g., "handball" or "offside") based on the video content. These labels were then used to generate multiple-choice questions with the ground truth as the correct answer and distractors derived from other categories.
• Open-Ended Questions: Annotators provided detailed explanations for the rule violation or foul observed in the video. These explanations, instead of simple labels, were used as the ground truth for generating open-ended questions, ensuring the model is tested on reasoning and explanation generation.

About more advanced evaluation methods:

We greatly appreciate your suggestion to include ST-LLM and Qwen-VL in our evaluations, which helped us gain more insights. We have expanded our experiments to include these two models. We listed these two models' performance below. You can also refer to our Result Section and Appendix G. Additionally, the results are listed in the response of 7Lbr.

Along with feedback from other reviewers, we explored two more prompt settings to analyze the reasoning capabilities of the models further. These supplementary experiments provide an even more comprehensive evaluation and insights into our work. We hope these updates address your concerns and enhance our overall contribution to the field.

About paper writing:

Thank you for your detailed review. We have corrected typos and carefully reviewed the entire paper to address similar issues.

We are grateful for your thoughtful feedback again. Please let us know if there are any additional concerns or points you would like us to address.

Dear Reviewer rN5c,

We kindly remind you that the rebuttal period is about to close, and we have made our best efforts to address your concerns. If they appear satisfactory, we kindly request that you take a moment to revisit your scores and share your updated thoughts. Please feel free to share if you have any follow-up questions, and we will make our best effort to handle them before the rebuttal period concludes.

Thank you,

Authors

Dear Reviewer rN5c,

Thank you for your reply. We would like to clarify that our benchmark has distinctive features that are absent in previous datasets, making significant contributions to evaluating and understanding MLLMs' deep sports understanding capabilities.

• Integration of rules and actions: SPORTU uniquely ties video-based tasks to sports rules, requiring models to reason about how actions in different contexts lead to distinct rule applications or violations, rather than just extending from single-motion scenarios to multi-motion ones. In addition, understanding rules is the most important and foundational part of sports.

• Beyond surface-level questions: Unlike existing benchmarks, SPORTU goes beyond identifying motions or sports and challenges models to answer complex questions such as "Why is this a foul?"—requiring both fine-grained action recognition and rule-based reasoning.

• High-quality human annotations: We have especially added high-quality human annotated explanations in the SPORTU-text part and SPORTU-video open-ended part. This ensures high-quality video ground truth and question answering, which are critical for evaluating fine-grained tasks like foul detection.

• Comprehensive evaluation of sports understanding: SPORTU is the first benchmark explicitly designed to evaluate MLLMs’ sports understanding across multiple sports and aspects, through multi-level difficulty questions, and pushes evaluation in the sports domain further.

As sports continue to captivate people worldwide, MLLMs have the potential to help users engage more deeply with this domain. A benchmark to test MLLMs' understanding of sports is therefore essential. The evaluation of SPORTU clearly highlights the current limitations of MLLMs in deep sports reasoning, emphasizing the need for further advancements in this area.

We hope this clarifies our contributions and look forward to addressing any further questions or concerns.

Dear Reviewer rN5c,

Thank you for your previous response again. As the rebuttal period is approaching its end, we would like to kindly follow up on the clarifications regarding our contributions that we provided in our earlier response. We hope these clarifications effectively highlight the distinctions between existing sports benchmarks and ours.

Please let us know if you have further questions or require clarifications. We are more than happy to address them promptly within the remaining time.

Thank you once again for your valuable insights and for helping us improve our paper.

The Authors

Dear Reviewer gCvg:

We sincerely thank you for your detailed and constructive feedback. Your insights have helped us conduct more comprehensive experiments and bring valuable new perspectives to the paper. Below, we address your concerns and questions in detail.

About the discussion of varying model performance and advice on resolution:

In the Result Section, we have mentioned models that tend to perform well on easy-level questions but show a significant gap in hard-level questions, indicating a lack of domain knowledge, particularly in rule comprehension. We also found that most proprietary models, along with the Qwen2-VL-72B model, achieved higher G-Eval and human rating scores than the 7B open-source models. This indicates that proprietary models exhibit stronger reasoning abilities than the 7B open-source models. However, none of the MLLMs achieved an average G-Eval score above 3, highlighting a gap in the deep domain knowledge required for reasoning in video-based sports understanding. We have added this finding in the result section. Regarding the resolution, We have added a Discussion section in Appendix C. We think better frame extraction strategies need to be developed specifically for sports tasks to ensure that the actions critical for reasoning about the question are clearly provided to the model. A more effective grounding method could also enhance the model’s ability to distinguish actions more accurately. During the experiments, we noticed that while models sometimes captured and described most of the correct movements, they often failed to infer the corresponding rule violations. This highlights a significant gap in connecting recognized actions to specific rules, underscoring the need for models to not only identify movements but also to understand what a rule violation should look like.

About testing prompt strategy in LLM on MLLM:

We appreciate this suggestion. We have tested additional prompt strategies on MLLMs for SPORTU-video. Specifically, we added two prompt strategies, including X → Y , X → RY. The results show that X → Y achieved the highest overall performance in most of the models, outperforming X → Y R and X → RY. For most models, the accuracy follows the order: X → Y > X → Y R > X → RY. However, as task difficulty increases, especially in hard-level questions that require more reasoning, performance drops significantly for X → RY due to hallucinated rationales, which mislead the model’s answer inference. For example, Qwen72b achieved 70.95% accuracy in the X → Y setting but dropped to 69.17% in X → Y R and further to 62.51% in X → RY. On hard-level questions, the drop was particularly significant, from 44.12% (X → Y) to 30.55% (X → RY). This highlights the challenges of reasoning-based prompts in complex tasks. This finding has also been introduced in Zhang's work[1]. We have incorporated these insights into Appendix G and H, including additional examples that demonstrate cases where X → Y succeeded while X → RY failed due to hallucinated rationales. The detailed results are presented in the Results section. Additionally, the results are listed in the response of 7Lbr.

About whether the questions in this dataset can comprehensively detect the models' abilities to understand sports:

Our dataset is the first benchmark that comprehensively evaluates models' fundamental sports understanding across three distinct difficulty levels. Easy-level questions are designed for individuals with general commonsense and no specific sports knowledge, medium-level questions require foundational sports knowledge beyond commonsense, and hard-level questions target those with an in-depth understanding of specific sports, including sport-specific rule comprehension. This tiered structure provides a comprehensive evaluation of MLLMs' sports comprehension across varying levels of complexity.

About the low correlation between human and automatic metrics:

The low Pearson correlation between human evaluations and other metrics highlights the limitations of current evaluation methods for sports-related reasoning tasks. Unlike general domains, sports explanations often rely on precise terminology and detailed action descriptions, where even minor differences can lead to entirely different outcomes. This demonstrates the need for sports-specific evaluation metrics that better capture the nuances of rule-based reasoning. We believe this is an important future direction and a critical step toward developing automatic metrics that align with human judgments.

[1] Z. Zhang et al., “Multimodal chain-of-thought reasoning in language models” TMLR

About your questions:

• How do you split the SPORTU-text questions?

Our expert team manually annotated and labeled the questions into three categories: 1) rule-related questions focus on explicit sports rules; 2) strategy-related questions involve tactical or strategic decisions; 3) scenario-related questions provide specific contexts or player interactions (e.g., "Player A performs action X, and Player B reacts"). While some questions may overlap categories, our experts assigned each question to the most appropriate category based on their domain knowledge.

• What are the results on rule/strategy/scenario, respectively, on sportu-text? The classification results are provided below and have also been included in Appendix E and Appendix F for further reference.

• How is the error analysis in 5.1 conducted? Is there a definition for each error type?

As described in Section 5.1, our expert team selected 20 incorrect examples per sport for both multiple-choice and open-ended questions (20 for each) for each model. Since we have now included two additional models, bringing the total to 14, the total error cases have been updated to 3920, and the paper has been revised accordingly. The detailed definitions for each error type are included in Appendix M , along with representative examples.

We are grateful for your thoughtful feedback, which has helped us improve both the clarity and depth of our work. Please let us know if there are any additional concerns or points you would like us to address.

Dear Reviewer gCvg,

We kindly remind you that the rebuttal period is about to close, and we have made our best efforts to address your concerns. If they appear satisfactory, we kindly request that you take a moment to revisit your scores and share your updated thoughts. Please feel free to share if you have any follow-up questions, and we will make our best effort to handle them before the rebuttal period concludes.

Thank you,

Authors

Dear Reviewer gCvg,

As the rebuttal period is approaching its end, we would like to kindly follow up regarding our responses to your comments. We have tried our best to carefully address your concerns and would greatly appreciate it if you could share any additional feedback or consider adjusting your score.

Please let us know if you have further questions or require clarifications. We are more than happy to address them promptly within the remaining time.

Thank you once again for your valuable insights and for helping us improve our paper.

The Authors

Thank all the authors for addressing my concerns in this rebuttal and it addresses my concerns. After taking other reviews and rebuttals into consideration, I maintain my score.

Dear Reviewer uNov,

Thank you for your valuable feedback. We appreciate your insights and have addressed your concerns as follows:

Yet another vertical dataset for LLM: Our dataset offers unique challenges and insights, making it a critical benchmark for advancing MLLMs in sports. Specifically, sports understanding requires models to integrate fine-grained action recognition with complex rule-based reasoning, a challenge that extends beyond typical domain-specific tasks. SPORTU introduces the first-ever comprehensive benchmark combining text and slow-motion video tasks across various sports. It is designed to evaluate MLLMs' ability to connect visual perception with sports-specific rule comprehension.

About the technical depth: Our primary contribution is to introduce two new sports datasets that are more challenging than existing datasets and contain explanations. We benchmark LLMs on these datasets, demonstrating a significant gap in their performance on hard-level questions that require understanding movement and matching it with corresponding rules. The goal is to facilitate future research on integrating MLLMs for sports. While technical depth is present, it is not the main focus of this paper.

To clarify what current models failed: We conducted additional analyses, including examples of hallucinated rationales and reasoning errors in complex foul scenarios. We also added the results for X-RY and X-Y in the Results section. Our results show that models struggle with reasoning-heavy tasks, where precise action recognition must be linked with domain-specific rules. These examples are included in Appendix H, and a discussion section in Appendix C is added to demonstrate why robust rationale generation is critical in the sports domain. We believe that the insights from these failures are essential for guiding future improvements in MLLM research.

About your questions:

• The video quality for the datasets: The quality of the videos in our dataset has been rigorously controlled through three key aspects:

  • Annotator Expertise: Our annotation team consisted of nine experts, including two intercollegiate student-athletes with over 12 years of experience and seven players with at least five years of training in their respective sports. This ensured deep domain knowledge for accurate annotations.
  • Video Cropping: All videos were manually cropped by our expert annotators to ensure they focus on key actions relevant to the designed questions. This process guarantees that each video provides sufficient clarity and context for detailed analysis.
  • Verification Protocol: Annotators double-reviewed all collected videos to ensure accuracy and quality. We also prioritized the removal of controversial or hard-to-interpret clips to minimize potential mislabeling. These measures collectively ensure that the dataset meets high-quality standards, suitable for detailed action analysis and rule comprehension tasks. And This process is shown in section 3.1.

• For each sports type, the videos are biased to certain views or events?

  • We do not believe the videos are biased toward certain views or events. The dataset includes videos featuring common rule violations that occur across all levels of games and competitions. Furthermore, all videos were sourced from a variety of different competitions to ensure diversity in scenarios and perspectives.

We are grateful for your thoughtful feedback, which has helped us improve both the clarity and depth of our work. Please let us know if there are any additional concerns or points you would like us to address.

Dear Reviewer uNov,

As the rebuttal period is approaching its end, we would like to kindly follow up regarding our responses to your comments. We have tried our best to carefully address your concerns and would greatly appreciate it if you could share any additional feedback or consider adjusting your score.

Please let us know if you have further questions or require clarifications. We are more than happy to address them promptly within the remaining time.

Thank you once again for your valuable insights and for helping us improve our paper.

The Authors

Dear Reviewer 7Lbr,

We really appreciate your constructive feedback, which helped us find more insights. Below, we address your concerns point by point.

About the diversity and coverage of the dataset:

Our dataset is the first to test MLLMs' sports understanding ability, particularly by combining action recognition with rule-based reasoning across different sports and varying difficulty levels. Existing datasets focus solely on action recognition or understanding the deep rules of a single sport. We argue that understanding actions alone does not equate to better sports understanding for MLLMs, as rules play a significant role in games and must also be comprehended. Moreover, performance on a single sport is insufficient to evaluate an MLLM's sports understanding across various sports, as rules differ significantly between sports. In contrast, our dataset is more diverse, covering multiple sports categories and difficulty levels. A comparison table with existing works is provided in the Related Work section. Notably, our dataset is the first to test MLLMs' ability to understand rules across various sports.

We have added more templates and question examples in Appendix O to Q. Additionally, more questions and videos can be found in the anonymous link in the paper.

About implementations:

• The error in Figure 6: We appreciate your attention to the details of our error analysis. This error only appears a few times in the Gemini output, which we think is interesting to show an illustrative example of how templates can occasionally fail when other Gemini outputs work by using the same template. To provide a more comprehensive analysis, we have now added additional error examples in Appendix Q, covering a more comprehensive range of cases and models.

• The reasoning prompt: We really appreciate your concern, which allows us to analyze this issue further. We have conducted supplementary experiments with two additional prompting strategies: X → Y: No-CoT, which directly predicts the answer. X → RY : Reasoning where answer inference is conditioned to the rationale. The results show that X → Y achieved the highest overall performance in most of the models, outperforming X → Y R and X → RY. For most models, the accuracy follows the order: X → Y > X → Y R > X → RY. However, as task difficulty increases, especially in hard-level questions that require more reasoning, performance drops significantly for X → RY due to hallucinated rationales, which mislead the model’s answer inference. For example, Qwen72b achieved 70.95% accuracy in the X → Y setting but dropped to 69.17% in X → Y R and further to 62.51% in X → RY. On hard-level questions, the drop was particularly significant, from 44.12% (X → Y) to 30.55% (X → RY). This highlights the challenges of reasoning-based prompts in complex tasks. This finding aligns with what is shown in Zhang’s work[1]. We have incorporated these insights into Appendix G and H, including additional examples that demonstrate cases where X → Y succeeded while X → RY failed due to hallucinated rationales. We attached the result below.

[1] Z. Zhang et al., “Multimodal chain-of-thought reasoning in language models” TMLR

Three prompting strategies performance of MLLMs on SPORTU-video

The best results are bolded. The results highlight that models perform best with the X → Y prompt (25/56 leading performances), followed by X →YR (21/56), and X → RY(10/56).

• G-eval’s quality with different LLMs as the rater: We appreciate your suggestion. To address this, we supplemented the G-eval evaluation with scores obtained using Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1-405B. The result is shown below：

SPORTU-text Part

SPORTU-video Part

minor issues, we corrected the typo and carefully reviewed the entire paper to ensure similar errors were addressed. Thank you for your careful review and for pointing them out.

We are grateful for your thoughtful feedback, which has helped us improve both the clarity and depth of our work. Please let us know if there are any additional concerns or points you would like us to address.

Dear Reviewer 7Lbr,

We kindly remind you that the rebuttal period is about to close, and we have made our best efforts to address your concerns. If they appear satisfactory, we kindly request that you take a moment to revisit your scores and share your updated thoughts. Please feel free to share if you have any follow-up questions, and we will make our best effort to handle them before the rebuttal period concludes.

Thank you,

Authors

Dear Reviewer 7Lbr,

As the rebuttal period is approaching its end, we would like to kindly follow up regarding our responses to your comments. We have tried our best to carefully address your concerns and would greatly appreciate it if you could share any additional feedback or consider adjusting your score.

Please let us know if you have further questions or require clarifications. We are more than happy to address them promptly within the remaining time.

Thank you once again for your valuable insights and for helping us improve our paper.

The Authors