Clarity questions

• The number of videos is 557 – an earlier version of the work had 656, but we removed a set of actions because they only had one annotated difference each. Thank you for noticing that, and we have updated the text.
• The fig.1 caption refers to two challenges: (i) “identifying alignment between segments” and (ii) “fine-grained image comparison”. This was discussed in the introduction paragraph 2, “Two critical obstacles are precise temporal alignment and the need for fine-grained understanding of action dynamics”, which we then elaborate. Our qualitative results discussion also identifies these as major challenges. However we agree that the clarity could be greatly improved by using more consistent language about these two challenges throughout the text. As such, we’ve updated the text in the introduction, and the figure caption.
• LLMs and VLMs in the method] We have updated in the experiment section that the LLM and VLM are both 'gpt-4o-2024-08-06' (state-of-the-art in both at the time of submission), while the embedding model in the localization module is CLIP “ViT-bigG-14” with checkpoint "laion2b_s39b_b160k".
• [Inverse correlation] You are correct that “squat is lower” and “squat is higher” are equivalent, but where the A/B prediction should be flipped. This is relevant to the open evaluation setting where the model must generate the difference descriptions. Our open evaluation protocol does properly handle this case. First, for the LLM query matching the ground-truth differences to prediction differences – this does match differences that are semantically equivalent but different in sign, and we find many examples in evaluation where this is the case. For each matched difference, we run a second LLM query to check whether the difference string has the opposite meaning, and if it does, then we flip the A/B prediction. Our earlier manuscript version discusses this as an implementation detail in the appendix and shows the prompt (our example of a matching pair was “the arms are more straight” vs “the arms are more bent”). However since this is an important detail that will concern some readers, we have updated the manuscript to discuss this in the task definition and in the open evaluation protocol.

Variations in the dataset – angles, fps, actor’s height

Although there is random sampling used in assigning which pairs to compare, we did manually inspect every video in the sampling set. This allowed us to think carefully about the video attributes, including the very important attributes identified here. We will discuss each particular point below, and we’ll also expand on the writing in the Appendix to be more clear about how the videos are curated. In particular, we’ve explained:

• Camera angles: the change of camera angle perspective does make the task harder. For samples in the ‘Fitness’ category, the camera angle is the same because the source dataset has a fixed camera rig, and we chose to use the same camera angle. For samples in ‘diving’ and ‘surgery’ categories, the camera angle is approximately the same. On the other hand, samples from ‘ballsports’ and ‘music’ categories can change. A related attribute (not mentioned here) is differences in background – similarly the ‘ballsports’ and ‘music’ categories often had different backgrounds as well. Importantly, these attributes were considered when assigning the difficulty splits. This may partly explain why the fitness exercises are all in the easy and medium split.
• FPS: each video pair has the same fps. In case others want to leverage our code with new videos, our code does handle the case where FPS is different. Specifically, the input config has a value for the target FPS for running inference, and we subsample the video to have this FPS. (If the videos cannot be subsampled to have the exact target fps, then a warning is printed).
• Impact of different actor heights: this is a very good observation, and we did address it in our annotation instructions. We clarified that all differences on things like distance should be relative to the actor’s height. We gave the example of “wider foot stance”, saying that if a 5ft actor and a 6ft actor both had their legs 3ft apart, then the shorter actor has a “wider foot stance” relative to their height. This reflects what is commonly understood by descriptions like these in skills coaching.

References

Nagarajan & Torresani 2024, “Step Differences in Instructional Video”

Doughty et al 2018, “Who's better? who's best? pairwise deep ranking for skill determination”

Balakrishnan et al, 2015, “Video diff: Highlighting differences between similar actions in videos”

We’d like to thank reviewer 8a8a for recognising the strength of the work, especially the significance of the new proposed task of Video Action Differencing, and the originality of all three of our contributions – the task, benchmark, and method. We now hope to address each of the raised concerns:

Addressing temporal alignment

Consider the example in Fig.6 row, 2, left, the action is basketball layup, and the difference is “Non-shooting hand guides the ball”. A human would identify the segment in the video where the person is about to release the ball in both videos, and then compare these segments. This is the challenges that we talk about: differencing first requires aligning the sub-actions in the two videos, and after that step, the visual comparison can be done on the two aligned segments. To solve this, our approach performs alignment in a similar way: the ‘frame localizer’ does temporal segmentation in each video, and then the localized segments are both passed to the ‘action differencer’ to compare the visual segments. Having retrieved and aligned the frames, the ‘action differencer’ can work with a smaller segment of video – possibly only a single pair of frames. We have updated the text in our method section to more explicitly say how we have solved the key problem.

Related work

We have added an extended discussion in the appendix about “video comparison datasets”, and summarized it briefly in the main related works section. The high level takeaway is that no other dataset has labels for fine-grained comparison while having a large scale. One relevant prior datasets considers very coarse-grained differences in instructional videos, for example identifying that a different ingredient was used in a cooking recipe (Nagarajan & Torresani 2024). Other works do consider more fine-grained differences, but they either annotate only a single binary variable like “which is more skilled” (EPIC-Skills2018 by Doughty et al 2018), or do not have any annotations (Balakrishnan et al, 2015); both of these dataset examples are small with fewer than 100 pairs.

Scale of benchmarking

In our updated version, we have increased the scale of the benchmark. We’ve added the closed-source Claude-3.5-Sonnet, and the most recent open-source video model, LLaVA-Video-7B. We have updated the main table, and for illustrative purposes, here are is the main table on closed evaluation:

Among closed models, Claude performs worse overall than GPT and Gemini. Among open source models, LLaVA-Video is stronger than Qwen2-VL, becoming the only open source model to achieve statistical significance on the easy split.

Thanks for your comments. These are insightful points that line up quite a bit with discussions we’ve had during the project. We discuss them over two posts.

Point 1:

VidDiff is more computationally efficient than the one-stage GPT-4o baseline while maintaining superior performance in the closed setting and comparable results in the open setting. Despite GPT-4o requiring analysis of 40 frames per video—translating to approximately 12,300 tokens—VidDiff processes only 17 localized frames, reducing the token count to about 4,300, a threefold decrease in computational cost (reducing the API cost for evaluating the whole benchmark from approximately $18 to $6). This efficiency is achieved through an LLM-only Proposer stage and a CLIP-based localization strategy, both of which introduce minimal overhead compared to GPT-4o's visual processing demands.

VidDiff's computational advantages become more pronounced with longer videos, as its cost scales with the number of target differences (processing only 2–6 frames per video pair) rather than linearly with the total frames, as seen in the LMM baseline. Consequently, the efficiency gap widens with longer videos, aligning with recent research exploring efficiency-performance trade-offs, such as Wang et al. (2024).

Our primary contribution is introducing the novel task of Video Action Differencing and developing a comprehensive benchmark to support it. The VidDiff method is a proof-of-concept to demonstrate that the ‘compound’ approach [Zaharia et al., 2024] will work on this problem, and should be explored in future research. The compound approach has two advantages.. First, they will benefit from improvements in zero-shot models for the localization and image understanding. Second, it enables researchers to improve individual stages of the process independently – to facilitate this, VidDiffBench provides stage-wise annotations, giving a robust framework for evaluating performance at each stage.

[Wang et al 2024] “VideoAgent: Long-form Video Understanding with Large Language Model as Agent”

[Zaharia et al 2024] The shift from models to compound ai systems

Question 4,effectiveness of LLM Evaluation

Thank you for your question about trustworthiness of LLM evaluation in the open setting. We’ve added these new experiments and human evaluations to the appendix.

• [Robustness to Multiple Runs] The LLM evaluation is robust to random seed. We repeated the evaluation five times with different random seeds and observed a standard deviation of only 0.7 in the final evaluation score. This indicates that the results are consistent across runs. Although the prompt was specifically engineered for the GPT-4o-2024-08-06 model, we ensured consistency by fixing the model for all evaluations, treating all comparisons under identical conditions.
• [Comparison with Human Evaluation] To measure alignment with humans, we recruited 3 human annotators to perform open evaluation matching, each with 44 video pairs and 347 individual differences. For each video pair, they were provided with a list of ground truth differences, and asked to match each one to a predicted difference from a list, or to suggest no match. We calculated inter-rater agreement across annotators and the automated LLM system. The results are as below. We can see semantic matching proved to be challenging for humans – the mean of pairwise rater agreement from each human to the other humans was 75.7%. Meanwhile, the mean agreement between our automated system and human annotators was 73.9%. Therefore, our LLM-based approach is on par with human annotators, while being completely automatic.

• [Details of Prompt for LLM Evaluation] The LLM prompt was carefully developed using a prompt engineering workflow. We selected a set of four evaluation samples, covering two actions and two models, and iteratively refined the prompt based on performance in individual runs. For example, we added the instruction: "Only match entries if their description strings are visually similar, even if the word choices differ." This adjustment was necessary because the LLM struggled to match equivalent descriptions phrased differently (e.g., “the feet stance is wider” vs. “the legs are spread wider apart”). While this approach achieved satisfactory results, we acknowledge that the prompt could be further optimized using more systematic methods, such as DSPy (https://arxiv.org/abs/2310.03714). Exploring such techniques is a promising direction for future work.

Question 5, Assigning easy/medium/hard splits by LLMs

• Choosing the difficulty splits requires a holistic view of all the actions, so we decided it didn’t make sense for experts to suggest them, since they are only familiar with a few actions each. On the other hand, we didn’t want to rank the splits based on performance of current models since this felt like biasing towards current models; and besides, the performance for many actions in ‘medium’ and ‘hard’ is already random, so it would be hard to differentiate these actions. LLMs are a good candidate because they have a good understanding of the actions and are relatively free of the biases of this paper’s authors Furthermore, human annotators could not do the ranking, because no human annotated all the actions.
• To further support the choice of an LLM, we asked 3 humans to rank the action comparisons from easiest to hardest, and compared against the LLM ranking. We then computed the Spearman’s rank correlation between all ranking sets, and the results are in the below table. The mean of the pairwise correlations between the humans was 0.602, while the mean of pairwise correlations between the LLM and humans was higher at 0.673. This shows (i) that there is non-negligible variability in human rankings, and (ii) that the LLM ranking is reasonable, and actually better correlated with most humans compared to several of the human annotations.

• We added a new appendix table showing the difficulty splits with lists of actions and full descriptions.

We’d like to sincerely thank the reviewer for their comprehensive and very thoughtful evaluation. In response, we’ve completed a number of new experiments and human studies, and we’ve made changes to the manuscript. Firstly, we appreciate your recognition of the overall strength of the contributions (rating 4), in particular that the new task is important, and that the benchmark is valuable.

References for skill determination

Thank you for the recommended papers. We have added all four of these to our Related Work. These four works raise the point that video-comparison is a useful signal in model training, even when the supervision is a sparse ranking.

The remaining text addresses the ‘questions’ section.

Question 1: Are pairs the same action?

Yes, within each pair, the videos are of the same action. We have adjusted the text to make this more explicit.

Question 2: Importance of annotator expertise

• The differences in our taxonomy were designed to be straightforward to evaluate for humans. In general, differences were easy to evaluate without additional filtering, likely because each action contained a small number of differences (so they were distinct from each other) and our criteria required all differences to be visually discernible in video. To ensure this was the case, for this review, we re-checked each action to ensure there is no ambiguity, and did fine 3 actions – 2 in surgery and 1 in music – where the actions were arguably a bit ambiguous, and for the final paper we will remove these 3 differences. They only account for 3 of 147 differences.
• Additionally, to ensure annotation quality, we provided comprehensive instructions with demonstration video pairs for each difference type. As you note, future benchmarks may need to incorporate differences that are even more subtle, and they may require domain expertise. For this benchmark, more discernible actions already lead to a very challenging benchmark.
• The annotators were college-educated, and remunerated $22.19 per hour.

Question 3, part 1 Closed eval – is A/B biased?

• 49.3% of samples are ‘A’ and 50.7% are ‘B’, so there is no significant dataset bias, and we do not require random swapping at inference time. We’ve added this important detail to Section .3.
• Additionally, we test the impact of video order on GPT-4o for the `fitness' category, which has samples in the easy and medium subsets(sample size 193). We test flipping the order of videos which flips the A/B answer. The performance is 54.8% in the original evaluation, and reversing the order of videos gives performance of 55.5%, showing a 0.7% difference. This result suggests that the performance on VidDiffBench is not significantly sensitive to video order.

Question 3, part 2, inclusion of Option ‘C’ for Closed Setting

Thank you for your thoughtful suggestion. Our initial approach to formulating the closed evaluation did include an option ‘C’ for insignificant differences, as you proposed. However, the challenge of calibration made fair evaluation difficult. For example, when comparing two videos of a basketball shot to evaluate stance width, the question arises: how different is “different enough” to be both relevant for skill learning and perceptible? Different annotators may apply varying thresholds for what constitutes a significant difference, leading to inconsistencies. Introducing option ‘C’ further complicates evaluation because it requires calibrating not only the human annotators but also the VLMs, which may have different internal thresholds for perceiving significance. To address these challenges, we adopted the following approach:

• Annotators were instructed to choose either ‘A’ or ‘B’ only when the difference was clearly perceptible.
• We limited the evaluation of VLMs to cases where there was a very clear ground truth answer of either ‘A’ or ‘B.’

This method ensures fairness by focusing on scenarios with unambiguous ground truth, avoiding complications introduced by subjective calibration thresholds. While we briefly discuss this in the section on annotation creation, we recognize that this is a nuanced point. Therefore, we have added a more detailed discussion to the appendix for further clarity.

Thank you for answering my questions and providing more comments about the paper. Because of this, I will increase my rating towards this paper. I believe the extra experiments that were conducted for the validation towards the benchmark to be interesting and informative. I do not agree with the reasoning behind the LLM for the easy/medium/hard split, but do not believe that to be large enough to sway my overall decision of the paper. However, upon reading the updated version of the paper, I discovered some minor spelling/grammatical errors:

Line 141: "... the supervision signal is commonly a binary of which..." -> either "commonly binary", or "commonly a binary one"

Line 144: "... comparison while having a large scale." -> "whilst being large scale."

Line 311: "... the A/B/C ration is ..." -> "... the A/B/C/ ratio is ..."

Line 249: "... key challenge of aligning precise temporal alignment..." align[ing/ment] is repeated here, can aligning be dropped?

We’d like to thank reviewer ThzX for the comprehensive and constructive comments, and for highlighting that the task is well-motivated, that the benchmark is well-constructed, and that the method’s results are interesting. We now address each of the given concerns:

Comparing SOTA models

We have performed a more thorough comparison of the different SOTA LMMs on VidDIffBench, added a small subsection to the results, and a discussion in appendix. Specifically we look at each action, and compare the different LMMs.

• First, we show the correlations in the per-action scores between models, which is interesting:

• The correlations are generally low, but there are 3 clusters of models. LLaVA-Video and Qwen2-VL are in a cluster; they are both open-source, and have the same LLM backbone. Then GPT-4o and Claude-Sonnet cluster together, and Gemini is not similar to any other model. We can speculate that for video data, Claude and GPT have similar training strategies, while Gemini’s is different.
• Next we compare model performance within one action, and this is over two large tables in the Appendix. Specifically we measure ‘relative performance’: the difference between the model score on that action compared to the mean score across all models for the action. The most significant results in the benchmark are on the easy split. Here, the improvement in score is uniform for all models The models are generally close to each other. The ‘relative performance’ is usually less than 10 points – when it is higher, the sample size is very small.
• By comparing models at the level of actions, we are considering smaller sample sizes than in the main results, which compare models at the level of easy/medium/hard splits. There is therefore lower statistical power to identify significant result differences, so the results are less certain. We elected not to compare model performance at the level of action differences, because here the sample sizes are very small, so any correlations would not meet significance thresholds.

LLMs as Difference Proposers

In our work, we introduced two types of evaluation: a closed-set setting and an open-set setting. The closed-set setting relies on human-annotated difference proposals, eliminating the need for LLMs. This setting simulates scenarios where specific differences of interest are already known and serves to evaluate the VLM’s video comparison capabilities directly. In contrast, the open-set setting leverages an LLM-based difference proposer to generate action differences, aiming to closely mimic real-world, end-to-end conditions. This approach not only tests the model’s ability to identify differences in video content but also evaluates its language reasoning capabilities. By incorporating both settings, our framework addresses varying levels of task complexity and grounds the evaluation in both controlled and practical contexts.

• To assess the LLM proposer’s performance, we compare its generated differences with human-annotated differences, reporting matching accuracies across three levels of task difficulty. These results demonstrate that the LLM-based proposer can generate accurate difference proposals more than 60% of the time, making it a viable and useful component in our open-set evaluation.
• Easy: 68.9%
• Medium: 61.9%
• Hard: 62.0%

CONTINUED .....

Error Propagation in Multi-Stage Method

• This is a good question, so thank you for your comment.
• The multi-stage compound system and the single-stage end-to-end system each have their respective strengths and limitations. While the multi-stage approach indeed carries the risk of error propagation, single-stage systems face challenges in performing complex multimodal reasoning effectively. As shown in Tables 2 and 3 in our manuscript, our method demonstrates superior performance compared to single-stage systems (VLMs), highlighting the potential advantages of the multi-stage approach in addressing this task.
• In this work, our primary contribution lies in defining a new task and providing a benchmark to explore it. The multi-stage system presented serves as a baseline to illustrate the potential of this approach. We believe that more advanced single-stage VLMs, particularly those equipped with enhanced internal reasoning capabilities (e.g. GPT4-o1), could further improve performance in this domain.
• Additionally, we have evaluated the performance of the LLM-based Difference Proposer, as detailed in the above provided tables, to address its effectiveness and demonstrate its role in the overall system.

Annotation taxonomy

• Thank you for your constructive comment. Firstly, note that the majority of the candidate differences are clearly visually discernible, and we argue they can be evaluated objectively. You are correct that there are a small number of differences that do require more interpretation. To address this, we do the following
• We performed a manual review of all differences and found 3 cases that are potentially ambiguous –two in surgery, and 1 in music. This is therefore a potential issue for only 3 of the 147 differences.
• Note that we included these actions because we were weighing the objectivity of the difference vs the importance of the difference. Our expert-informed taxonomy generation process identified these as important differences. Furthermore, for surgery and music, the same person did the annotations within a single difference. The annotation instructions emphasized to only mark “A” or “B” if the magnitude of difference is clear, and this was often the case because the datasets have a wide range of skill levels. So we argue that the risk of inconsistency is minimal.
• Having said that, we agree that these differences do add concerns about the objectivity of labels. Since it makes up such a small part of this dataset, we have decided to remove them – we will update the results in the final manuscript, which will impact only the medium and hard results to a small degree (the LMM performance for those actions was already random).

Table number

Thank you, we have corrected to reflect that this is an Appendix table, since it is so long.

We’d like to thank reviewer nLbM for recognising the strength of the work, especially that the proposed Video Action Differencing task is novel and well-motivated, that the benchmark construction process is sound, and that visualizations add understanding to the results. We now hope to address lingering issues:

Impact of fps

We strongly agree that fps is an important consideration for evaluating fine-grained actions.

• While typical video benchmarks like Video-MME sample videos at 1fps, we have sampled at a higher rate depending on category. The categories with shorter videos were sampled at a higher rate: 4fps for ‘fitness’, 5fps for ‘ballsports’, and 6fps for ‘diving’ (they are different so they can be compatible with fps in the source dataset). We chose this relatively higher rate because we are interested in more fine-grained differences, though we did not sample higher due to practical cost constraints of processing too many frames. The longer videos ‘surgery’ and ‘music’ were sampled at 1fps: these are longer videos where differences are discernible at lower sampling rates, and where the longer videos make high-fps sampling impractical.
• To show that our fps is reasonable, we tested the three closed-source models on a range of fps levels on the ‘easy’ subset of closed evaluation. We chose this set because this is where statistically significant differences were clear. The results are in this table:

• Across all models, the sampling rate that we use, 4 fps, has reasonable scores. For all models, the variability is low: GPT’s scores are within 0.8 points of the average; all other models have scores within 2.1 points of the average (except for the low sampling rate of 1fps in Gemini, where it degrades by 5.2 points). Moreover, the optimal fps is different for different models.
• To help explain the results, we refer to the qualitative examples in the main results sections. The only ‘success cases’ for all our models were those having easy localization, and coarse differences. We hypothesize that fps is not important for these cases. Where fps is likely important – fine-grained multiframe reasoning – the current LMMs cannot perform better than random. So although 2fps currently has good performance, we believe that as LMMs improve, they will perform better on subtle motions and using a higher fps will be important.
• We have added these points to the main document, and added the detailed results in the Appendix.

Qwen2-VL poor open performance

This is a good suggestion, and we’ve performed a deeper analysis into the lower scores of Qwen2-VL-7B. We found that a key issue here is that Qwen2-VL-7b was failing to follow the evaluation prompt, while the other compared models did follow it. Below are more evaluation details:

• We sampled 3 video pairs for each action and manually inspected Qwen’s responses, identifying multiple key issues. Below, we list each issue, and provide a quantitative estimate for the prevalence of each issue.
  • (45% of differences) Proposing differences not relevant to how to perform actions, but instead are visual things like “The person in video a is wearing a blue jacket, while the person in video b is wearing a plaid shirt.” We estimated prevalence by using a gpt-4o query that we manually prompt engineered.
  • (26% of differences) Proposing a difference that is actually not a difference, e.g. “The person in video a is performing the exercise with their arms out to the sides, while the person in video b is performing the exercise with their arms out to the sides.” We estimated prevalence by using a gpt-4o query that we manually prompt engineered.
  • (56% of differences) are repeated, meaning when trying to propose multiple differences, it proposes the same difference multiple times. We could directly measure this prevalence exactly.
  • (23% of actions) Proposing only a small number of differences – less then half as many as what is prompted for. We could directly measure this prevalence exactly.
  • (<5% of differences) Proposing vague differences that are harder to interpret visually like “The player in video a has a more versatile and adaptable skill set than the player in video b”. We estimated prevalence by using a gpt-4o query that we manually prompt engineered.
• Overall, only 31.9% of proposed differences by Qwen did not suffer from any of these errors. (Note that some differences suffered from multiple errors at the same time)

CONTINUED .....

.... CONTINUING FROM LAST COMMENT

Frame localization

• We did ablate the importance of the frame localizer in the main Table 5, in which we fix the frame-level action differencing module (the 3rd stage) and only implement the frame localizer module with different methods.
• for the easy split. This shows that choosing a random frame to localize leads to accuracy of 50.1%, our approach gives 65.8%, and using the ground truth frames gives 78.6%.

• We visualize some of the selected frames by the frame localizer compared to the ground-truth frames, and found most of these frames are close to GT frames, we add this visualization to the appendix.

Other experiments

We completed a number of other experiments and paper updates:

• An ablation over frames sampling rate (fps) on baseline LMMs, showing that our choices were reasonable (reviewer nLbM) Further datasets statistics, especially giving insight into video lengths and temporal biases in the retrieval / localization task (reviewer dA39)
• Significant changes to the structure of the paper for improved clarity (reviewer dA39), and smaller changes for clarity (reviewer 8a8a and dA39).
• Longer discussion of related video-pair datasets (reviewer 8a8a), and added new related works on paired datasets (reviewer dA39).
• Analysis into poor results for QwenVL-2 on open evaluation, concluding that it struggled with text instruction following. (nLbM )
• A section on the effectiveness of the Proposer module in our multistage VidDiff (ThzX)
• For the benchmark, we filtered 3 out of the 147 differences to give high confidence that all differences are visually discernible objectively (reviewer dA39 and ThzX)
• Showed that there is no potential bias due to the ordering of A/B in the multiple questions (reviewer dA39) by performing experiments with flipping video order.
• We further justified our design decisions in the closed evaluation setup (dA39), in the multistage design (reviewer ThzX), and in the annotator instructions (reviewer 8a8a)