Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding

We sincerely appreciate your constructive feedback and recognition of our well-written paper, sound VSLS strategy, and comprehensive evaluation. We are very grateful for your positive evaluation, which serves as the highest encouragement for our team. Below, we will address your concerns and incorporate your suggestions.

1. Concerns about the short period of distribution updates. Thank you for your insightful and constructive feedback. We appreciate you pointing out the potential impact of the initial uniform sampling, particularly in the extreme case where keyframes are clustered within a short period. This is a crucial aspect of the method's robustness.

We are confident that VSLS, in its current form, can effectively handle such "short period" cases. This capability stems from two key characteristics of our framework designed for robust exploration:

• First, our distribution update mechanism maintains the potential to explore new frames in every iteration (Sec. 2.3.4). While the algorithm exploits high-confidence regions by increasing their sampling probability, the global distribution update ensures that all frames retain a non-zero chance of being selected. This prevents the search from becoming permanently stuck in a suboptimal region and preserves its ability to discover the critical cluster even if it was missed in the initial samples.
• Second, the search is broadened by observing not just the primary target, but also relevant "Cue Objects" and scenes (Sec. 2.2). If a sampled frame contains a contextual cue, the algorithm increases the sampling probability in that temporal area. Moreover, our scoring mechanism based on relations will further prioritize the nearby frames if they satisfy certain logical relations. Therefore, the final searching results may converge to the frames in a small time window with the highest score. This strategy enhances the probability of exploring the ground truth frames by leveraging related contextual information, guiding the search toward the correct region even before the main target is found.

We agree that a visualization similar to Figure 3 would be the most effective way to demonstrate this. In preparation for this rebuttal, we have analyzed such extreme cases and generated the corresponding visualizations, which confirm this rapid convergence. Unfortunately, as per NeurIPS rebuttal guidelines, we are not permitted to include external links or new figures. Therefore, to illustrate this capability, we would like to present the results in a table format below.

As illustrated in the table, during the initial uniform sampling in iteration 1, only the object “girl” is detected, while the critical object “paper” is missed. Instead of continuing to explore frames uniformly at random, our algorithm leverages an object-relation scoring mechanism to guide the next round of sampling more strategically. This targeted exploration allows our method to focus on promising, previously unvisited frames. As a result, our approach successfully identifies the ground truth keyframe in later iterations, whereas the uniform baseline fails to do so.

Furthermore, we appreciate your forward-thinking question about potential design improvements. To further enhance the robustness of VSLS, we have considered two promising future directions:

1. Hierarchical Sampling: A coarse-to-fine approach where we first identify high-potential video segments before applying our fine-grained iterative frame search.
2. Heuristic Initialization: Using cues like scene-change detection or audio energy to create a weighted, non-uniform initial sampling distribution.

2. Concerns about the semantic logics in specialized applications. Thank you for this excellent question regarding the generalizability of our framework. You have raised a very important point about the need for domain-specific logics in specialized applications like medicine.

We completely agree. The four logics we presented—spatial, temporal, attribute, and causal—are intended as a foundational set designed to cover a broad range of general queries. We see VSLS not as a system with a fixed set of logics, but as an extensible framework that can be adapted to various domains.

Regarding your request for guidelines on deriving new logics for different scenarios, we propose the following systematic, three-step approach:

• Domain Knowledge Elicitation: The first step is to collaborate with domain experts to identify the critical relationships and events they analyze. For a medical application like surgical video analysis, this would involve identifying key instrument-tissue interactions (e.g., ‘cutting’, ‘suturing’, ‘retracting’) which are far more specific than our general ‘causal’ or ‘spatial’ relations. This process translates expert knowledge into a set of target logical relations.
• Operationalization into Verifiable Rules: The conceptual relationship must then be translated into a computable, verifiable rule that the VSLS search can execute. This involves defining the specific visual and temporal evidence required. For example, a new logic like suturing could be operationalized as:
  • (i) Detecting a needle-holder instrument and suture material.
  • (ii) Observing the instrument in periodic contact with tissue edges.
  • (iii) Verifying that the tissue edges become approximated in subsequent frames.
• Modular Integration: Finally, this new, operationalized logic can be integrated as a new module into the VSLS framework. Our design allows new logic-checking functions to be added alongside the existing four. The query parser would be extended to recognize domain-specific keywords (e.g., "suture") and trigger the corresponding verification function during the iterative search process.

This process ensures that the VSLS framework can be effectively adapted to specialized fields, making it more general and powerful. We believe this discussion significantly strengthens our paper's contribution. We will add a section covering the framework's extensibility and these guidelines for deriving new logics to the final version of the paper. Thank you again for this valuable suggestion.

Once again, we would like to express our sincerest gratitude. We will incorporate all of your suggestions into the final manuscript. Thank you for your contributions to the improvement of our paper and the advancement of the field.

We sincerely appreciate your constructive feedback and recognition of our Efficiency Gains and Novel idea. We are very grateful for your positive evaluation, which serves as the highest encouragement for our team. Below, we will address your concerns and incorporate your suggestions:

1. Concerns about the notation in Equation. Thank you very much for your detailed suggestion. We will revise Equation 1 according to your advice. If you have a preferred or better formulation, we would be grateful if you could share it with us. We will then update Equation 1 to use a more standard and explicit format:

$I_s = \text{Sample}(\mathcal{F}, N_s, P)$

where $`Sample`(\cdot, \cdot, \cdot)$ is a function that draws $N_s$ indices from the population $\mathcal{F}$ based on the probabilities specified in $P$.

2. Concerns about the detector affect video understanding. Thank you very much for your insightful and professional suggestion. We fully agree that the choice and configuration of the object detector can influence both the efficiency and performance of long video understanding. Based on your feedback, we summarize the effects into three key aspects:

• Number of frames per detector input (grid size). The number of frames packed into a single grid affects the trade-off between speed and accuracy. As shown below, while using more frames per detector input improves efficiency (reducing search steps), it also degrades QA accuracy due to reduced detection precision:

• Confidence threshold for detection. Higher confidence thresholds improve precision in object detection, which helps QA performance—but also slow down the search process due to fewer confident detections per iteration.

• Type of detector model. We also conducted additional experiments using OWL-ViT, a transformer-based open-world detector, and compared it with the YOLO-OpenWorld model used in our main experiments. While OWL-ViT offers broader concept coverage, its lower detection accuracy slightly affects downstream QA performance, as shown below:

These results demonstrate that while a better detector (e.g., one with higher AP) generally leads to stronger QA performance, the overall effectiveness of long video understanding also depends on balancing detection precision, sampling strategy, and efficiency constraints. We appreciate your suggestion, which helped us deepen the analysis and further confirm the robustness of our framework. Exploring a wider range of detectors remains an exciting future direction.

Once again, we would like to express our sincerest gratitude. We will incorporate all of your suggestions into the final manuscript. Thank you for your contributions to the improvement of our paper and the advancement of the field.

We sincerely appreciate your constructive feedback and recognition of our clear writing style, Remarkable Efficiency Gains, and Advanced Logical Modeling. Your insights have been invaluable in strengthening our work. Below, we will address your concerns and incorporate your suggestions:

1. Additional Open-source Models. Thank you for raising the important point regarding model validation. Initially, our primary focus was on the keyframe search component, and thus, we limited our comparisons to a select set of downstream QA VLMs. However, we fully acknowledge your concern about the generalizability of our proposed VSLS method.

To address this concern, we conducted additional experiments using open-source models, specifically Qwen2-VL and LLaVA-Video, as you recommended. The newly added experiments clearly demonstrate that VSLS consistently improves performance across these diverse VLMs, further validating its effectiveness and generalizability. Our results illustrate that, by employing efficient keyframe selection strategies, VSLS significantly enhances long-video understanding capabilities of various downstream VLMs.

2. Concerns about Limitations in Visual Semantic Logic Detection. Thank you for this excellent point. We agree that relying solely on bounding box overlap for attribute association is a heuristic with logical limitations, and we appreciate the opportunity to clarify our framework's design and scope. The primary role of our Visual Semantic-Logical Search (VSLS) is to function as a highly efficient candidate retrieval mechanism, not as the final reasoning engine. The goal of this heuristic is to rapidly filter a vast number of frames down to a small, manageable set that is highly likely to contain the answer. We then delegate the more complex task of logical disambiguation to a powerful downstream Vision-Language Model (VLM). For instance, given the query "What color is the shirt of the person in the car?", our search might retrieve keyframes of both a person sitting in the car and a person who happens to be walking past the car. Both sets of frames are passed to the VLM, which has the foundational reasoning capability to distinguish the correct context and provide the right answer.

Our method is therefore distinct from, and complementary to, graph-based frame relation modeling. VSLS is designed as an efficient, query-guided temporal search module that serves as a crucial pre-processing step. A more complex method, such as graph-based reasoning, would be a downstream task that operates on the concise set of keyframes our method selects. This two-stage approach allows for both efficiency over long videos and robust reasoning on the retrieved candidates.

To make this clearer, we will add a discussion to the Limitations section acknowledging the potential risks of this heuristic and will further clarify the distinction between our search framework and downstream reasoning tasks in the Related Work section of our revised manuscript.

3. Concerns about the initial detection being an erroneous case. Thank you for raising this important concern. Such concerns are entirely valid, as relying on initial detections can pose a risk of the sampling process falling into a “low-quality frame loop” if critical objects are missed early on. However, we have implemented specific measures in our method to address this issue, as described in the paper:

• Confidence-based exploration control: We introduce a confidence threshold for detected objects. If a detection’s confidence is below the threshold, the corresponding frame’s sampling probability remains unchanged—allowing the algorithm to continue exploring other regions of the video. Additionally, after each sampling step, we apply a temporal suppression window that lowers the scores of neighboring frames. This encourages exploration of untouched video segments and prevents the algorithm from repeatedly focusing on already detected high-score regions. As a result, the system avoids being trapped in a narrow subset of frames, even when early detections are incomplete or noisy.
• Redundancy via cue-object modeling: Our method not only searches for key query objects but also leverages semantically relevant cue objects to guide exploration. This redundancy ensures that even if a key object is initially missed, the presence of related cue objects can still lead the search toward the correct temporal region. Furthermore, distribution updates are not based on a single frame’s score, but are aggregated across sampled frames, which reduces the risk of being misled by outliers.

As illustrated in Figure 3 (bottom), although VSLS assigns low confidence to the second keyframe in iteration 1, the model does not get stuck near the first peak. Instead, it continues exploring and successfully identifies the correct second keyframe near the end of the video—demonstrating the algorithm’s ability to escape from local optima and avoid the low-quality frame loop.

The table below shows another example where initial uniform sampling detects only “girl” but misses “paper.” Thanks to our spline-based smoothing, the algorithm avoids repeatedly sampling similar frames and instead explores new ones. By iteration 3, it successfully finds “paper” and retrieves the correct ground truth frame, avoiding the low-quality frame loop.

4. Concerns about Parameter Sensitivity. We thank the reviewer for this professional and insightful feedback. We agree that analyzing parameter sensitivity is crucial. While several parameters in our framework are manually set, they are designed to be interpretable and correspond to clear, logical trade-offs, allowing users to configure the system based on their specific needs for efficiency versus accuracy.

For example, two key parameters are the confidence threshold and the num of images in grids. The confidence threshold allows a user to balance search cost against performance; as shown in our analysis, increasing the threshold from 0.3 to 0.8 improves performance from 50.2 to 56.4, while the required search cost increases from 18 to 162 steps. Similarly, the num of images in grids parameter controls the granularity of the search. A smaller grid size (e.g., 1) yields the highest performance (56.7) at a high search cost (770 steps), whereas a medium grid size (e.g., 8) minimizes the search cost to just 48 steps, offering a more efficient search. This demonstrates that the parameters offer predictable control over the search behavior rather than being arbitrary settings.

4. Concerns about Readability Decline in Method Section. Thank you for your valuable feedback. We appreciate your positive comments on the Introduction and fully acknowledge your suggestion regarding the Method section. We will carefully revise this part to improve its clarity, structure, and logical flow, ensuring it matches the standard of the Introduction and enhances overall readability for the reader.

Once again, we would like to express our sincerest gratitude. We will incorporate all of your suggestions into the final manuscript. Thank you for your contributions to the improvement of our paper and the advancement of the field.

Thank you so much for your thoughtful engagement and for acknowledging that your concerns have been addressed. We are truly grateful for your detailed review, constructive suggestions, and your willingness to follow up after our rebuttal.

Given that the points you raised have now been clarified and supported with additional experiments and analyses, we kindly and respectfully ask whether you might consider a positive adjustment to your score. As you are aware, even a small change can have a meaningful impact on the outcome of our submission, and your support would be greatly appreciated.

Once again, we are very thankful for your time and contributions to improving our work.

We sincerely appreciate your constructive feedback and recognition of our clear writing style, extensive experimental validation, and novel idea. Your insights have been invaluable in strengthening our work. Below, we will address your concerns and incorporate your suggestions:

1. Concern about the generalizability of the proposed method. Thank you very much for raising these insightful questions. We would like to clarify that the reason we did not report InternVL performance in the 32-frame setting is due to a limitation of the InternVL version available at the time, which does not support 32-frame inputs as it exceeds the maximum token length. Therefore, we included only GPT-4o for that specific comparison.

Regarding generalizability, we have conducted additional comprehensive experiments to more clearly understand the behavior of our method across varying frame counts and VLMs. The results of these experiments are provided below for your reference:

The results clearly demonstrate that our proposed VSLS method consistently achieves better accuracy compared to the baseline (T*) across various benchmarks and settings. Notably, on LVB and VideoMME datasets, VSLS consistently outperforms T* in most settings, especially under the 8-frame configuration, where the improvements are substantial (e.g., 45.83% vs. 41.96% on LVB-Long and 51.36% vs. 48.43% on VidMME-Average with QwenVL). Even under the more challenging 32-frame scenario—where noise often obscures key information—our method still generally achieves superior or competitive accuracy, underscoring its robustness and capability to effectively identify critical semantic-logical frames.

As you observed, while our method sometimes underperforms in the 32-frame setting, particularly on Video-MME, the main reason is that introducing more frames can result in increased noise, overshadowing the few critical frames (typically only 1-3 per query). This issue becomes prominent in models that are sensitive to redundant or noisy information.

The table above presents results from our newly added hierarchical clustering approach, where highly similar frames are removed in selected frames, retaining only 4 diverse and semantically distinct frames for input to the VLM. Despite the significant reduction in total frame count, our method (“+ Ours (cluster)”) achieves notable improvements over the T* baseline—particularly on long videos. For example, on LVB-Long, accuracy improves from 43.9%% to 48.4%, and on VidMME-Long, from 38.9% to 49.8%. These results demonstrate that by eliminating redundant information and focusing on key semantic-logical frames, we can not only maintain but even enhance performance in long video understanding tasks, while significantly reducing computational cost.

Our core optimization objective is to minimize the number of frames required to achieve strong QA performance, thereby reducing computational cost at the VLM inference stage. To further validate this claim, we conducted additional experiments where selected frames were clustered, reducing the frame count significantly (e.g., to only 4 frames). In these more compressed scenarios, VSLS consistently demonstrates more significant improvements over baseline methods, highlighting its effectiveness in efficiently capturing critical visual-semantic information.

The two experiments above demonstrate that our method outperforms the baseline, regardless of the choice of VLMs or the number of input frames. Notably, after removing redundant frames, VSLS with only 4 frames even surpasses the performance of the baseline using 32 frames. We will incorporate these additional findings in the revised manuscript to better demonstrate the generalizability and robustness of VSLS.

2. Concern about the accuracy of extracting the logical relations. Thank you for your professional suggestion. We agree that evaluating the LLM's ability to accurately extract logical relations is a crucial point for validating our framework's reliability.

To quantitatively address this, we conducted a verification study. We randomly sampled 500 query instances from each question category across the Video-MME and Long VideoBench datasets. We then used LLMs to perform the logical relation and object extraction as described in our paper. Subsequently, to ensure the quality of the results, we performed a rigorous manual audit of all extracted logical relations and objects. Our analysis yielded the following high accuracy rates for the extraction task:

• GPT-4o: 92%
• Qwen-VL 72B: 88%

These results demonstrate that state-of-the-art LLMs are highly proficient at this task, confirming that the logic extraction ability does not serve as a performance bottleneck for our framework at this stage. Due to committee guidelines, we are unable to provide a direct link to this data in the rebuttal. However, we are committed to transparency and will release the full set of our manually audited data and LLM outputs as part of our public code release.

3. Benefits of stacking multiple frames into one image as multiple grids. Thank you for raising this important point. Our rationale for this design is that it offers a deliberate trade-off between search efficiency and downstream task accuracy. The grid size is a configurable parameter allowing users to balance these two competing factors based on their needs.The table below shows a supplementary experiment we conducted. It presents the required steps to complete the keyframe search and the performance of the downstream QA task of GPT4o. As demonstrated, increasing the grid size substantially decreases the search cost (from 770 steps down to as few as 48 steps). However, it simultaneously leads to a moderate reduction in performance (from 56.7 down to 52.7). Notably, increasing the number of images in grids leads to a decline in detector accuracy, which in turn increases the overall search cost.

Therefore, this stacking approach presents a clear efficiency-performance trade-off, offering users the flexibility to choose an optimal balance based on their specific requirements or resource constraints. Users prioritizing detection accuracy may prefer smaller grid sizes or individual frames, while those with limited computational resources or stringent runtime constraints may opt for larger grid configurations to achieve greater efficiency.

4. Why need multiple iterations? We thank the reviewer for this insightful question. You are correct in observing that for simpler cases, such as the one in the top plot of Figure 3, our model can identify keyframes in the very first iteration. However, since each iteration only samples 8 frames to estimate the keyframe distribution, a single pass is often insufficient for more complex queries where visual cues are not immediately obvious—a frequent challenge in long video analysis. The bottom plot of Figure 3 illustrates this necessity, showing a scenario where the keyframes are only successfully located after the search distribution is progressively refined over 15 iterations.

We will address the BLOD-related issues in the paper. Once again, we would like to express our sincerest gratitude. We will incorporate all of your suggestions into the final manuscript. Thank you for your contributions to the improvement of our paper and the advancement of the field.