iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning

Thank you for the insightful review of our paper. Below are our responses to your questions. Please do not hesitate to raise further questions.

---

Responses to Questions
Response to Q1. We agree that prompt design plays a crucial role in LLM-based reasoning and clarify the following:

• Prompts are designed based on functional decomposition. As detailed in Sec. 3.1.2, our three-block prompt (Key Explanation, Step Instructions, Peer Response) is motivated by established findings in structured prompting and step-wise LLM reasoning [60–63]. It explicitly aligns with the hierarchical scene representation, guiding the LLM to reason over symbolic inputs.
• Yes, we perform prompt ablations in Table 6 and provide quantitative ablations over prompt components. Removing "Key Explanation" drops MM-AU accuracy by 4.66% (63.39 → 58.73), "Step Instructions" drops performance by 3.33%, and "Peer Instruction" drops performance by 2.77%.
• Prompt design was not searched for optimality. Rather, it is guided by alignment between input structure and reasoning goals, and validated empirically via ablations (in Table 6). For example, if we drop a module from iFinder for ablation, we removed it's corresponding explanation from the main prompt

---

Response to Q2. Based on the reviewer's recommendation, we analyze the LLM choice on iFinder and present the analysis below (MM-AU dataset).

---

Response to Q3.

• Generalization to other domains: Yes, the iFinder framework can be generalized to other domains. This is because the iFinder pipeline is explicitly designed as a modular, training-free framework to facilitate flexibility in incorporating or removing specific vision components. The core idea remains the same: representational abstraction of vision information that enables symbolic reasoning over dynamic scenes in language via LLM. The schema—temporal object-centric data—is domain-structured, not domain-locked. Its modular design allows substitution of perception modules for other domains (e.g., indoor robotics), and the JSON format can accommodate arbitrary scene cues. Our ablations (Table 6) show performance is robust to component removal, indicating generalizable reasoning.
• Adding /Removing modules to iFinder: As shown in our method (Section 3.1) and ablation results (Table 6), each module contributes uniquely to the final reasoning output, and modules can be swapped or extended without retraining. As aforementioned, adding or removing modules would only affect the instructions sent to the LLM to understand the JSON file.

---

Response to Q4. We only need to change the prompt to include/remove information about added/removed modules for the LLM to understand the JSON file. No retraining of LLM is needed.

---

Response to Q5. Yes, ı̇Finder can be adapted to multi-view camera setups. Since our framework is modular and operates on per-frame visual inputs, each view can be processed independently through the same perception modules. The resulting structured cues (e.g., object pose, orientation, depth) from each camera can then be merged into a unified symbolic representation using view-specific metadata (e.g., extrinsics). While our current work focuses on monocular front-view videos, extending the pipeline to multi-view settings involves modifying the scene representation to accommodate cross-view spatial alignment—an area we identify as promising future work for enhancing coverage and reducing occlusion in complex scenarios.

---

---

Responses to Limitations
Response to L1. We thank the reviewer for raising this point. While we explicitly mention that real-time deployment is not a focus of our work, the primary goal of our framework is to enable post-hoc, structured analysis of driving scenes—a task that is fundamentally distinct from real-time inference. Our method is particularly suited for offline applications such as accident investigation, insurance assessment, driver behavior auditing, and fleet-level safety analytics, where interpretability, causal attribution, and verifiable reasoning are critical. As shown in our evaluations on MM-AU, SUTD, and Nexar datasets, our framework excels in post-event reasoning tasks, outperforming both generalist and driving-specific VLMs in accuracy and robustness. For example, On the MM-AU dataset—focused explicitly on accident reasoning—iFinder achieves 63.39% accuracy, significantly outperforming the best generalist V-VLM (VideoLLaMA2 with prompt tuning) at 52.89%, and the best driving-specialized model (DriveMM) at 24.22%.

---

Response to L2.

• We analyzed the impact of error propagation (or effect of compounding errors when the initial module, in our case the object detector, fails to detect objects in the scene) in Table 7 on MM-AU dataset. For example, to imitate this detection failure, we remove >97% of object detections (τ = 0.8) from the JSON files and observed that the overall accuracy still remains 58.27%, surpassing all baselines including VideoLLaMA2 (52.89%) (Table 7). This shows that our symbolic reasoning retains reliability under perceptual degradation. Further, Table 7 also shows that iFinder demonstrates graceful degradation and limited error propagation. Please see L326-333 for a detailed discussion.
• Also based on Reviewer zz3N ‘s suggestion, we performed a different error propagation analysis: replacing modules with their weaker counterparts for the object tracker and the depth estimation on MM-AU dataset. We observed that replacing the tracker with a weaker tracker drops performance slightly to 61.70%, indicating that tracking quality contributes moderately to system accuracy. Similarly, using a weaker depth model causes a minor drop to 62.78%.
• Kindly note that while uncertainty quantification is not explicitly implemented in the current version of iFinder, the underlying LLM can support it. For instance, methods like Monte Carlo dropout, and token-level entropy estimation in the LLM’s outputs could be integrated to estimate uncertainty. This is a promising direction for future work. Further, discussing probabilistic guarantees for iFinder’s outcomes is currently beyond the scope of the paper.

---

Response to L3. While iFinder integrates multiple vision modules, its robustness to degraded perception is quantitatively demonstrated. As shown in Table 5, iFinder outperforms all baselines under adverse conditions—achieving 65.52% in rain, 57.86% in snow, and 63.93% at night, while the best generalist model (VideoLLaMA2) drops to 44.83%, 40.25%, and 50.23%, respectively. These results underscore that iFinder’s structured grounding mitigates failure cascades and ensures stable, domain-shift-tolerant reasoning.

---

Response to L4. Thank you for highlighting this. Owing to space constraints, we included these discussions in the supplementary material. We will incorporate them into the main body of the manuscript in the final version.

---

Thank you for the thoughtful and encouraging review of our paper. Below are our responses to your questions. Please do not hesitate to raise further questions.

---

Response to Weakness 1 & Question 1. The conceptual contribution of ı̇Finder is the idea of a representational abstraction of vision inputs that enables symbolic reasoning over dynamic scenes using LLMs. In other words, rather than passing raw pixel inputs or latent embeddings to the LLM, we transform each dash-cam video into a structured, temporally indexed JSON format. This representation bridges the gap between low-level perception and high-level reasoning, acting as a semantic interface between vision modules and language models.

---

Response to Weakness 2 & Question 2. We agree that ı̇Finder is not optimized for real-time deployment. As stated in the end of Sec. 3.1.2, we emphasize that ı̇Finder targets interpretability-critical scenarios, not embedded real-time inference. To clarify:

• Note that modular design ≠ high memory usage. All modules run sequentially or in parallel without maintaining persistent GPU state. Peak memory remains below 48 GB on an A6000 GPU (see Appendix B). The largest modules—frame undistortion and object attribute extraction—are image-based and not memory-bound.
• Wall-clock runtime is acceptable for post-hoc use. The full pipeline processes a LingoQA video in <4.5 minutes with unoptimized Python (Table 1 in supplementary material), suitable for offline analysis.
• We can achieve scalability via decoupling. Each module can be independently swapped for faster or lower-memory alternatives (e.g., lighter depth or 3D estimators), offering scalability based on deployment needs.

We will clarify this use-case focus in the final version.

---

Response to Weakness 3 & Question 3.

• As iFinder is built on top of correcting peer V-VLM (VLMs designed for high-level reasoning), it incurs higher computational cost relatively. However, this cost is justified by a notable increase in performance, particularly in tasks requiring nuanced understanding and disambiguation. For example in LingoQA dataset (Table 2), VideoLLama2 and VideoLLaVA incur 2.07 sec and 2.04 sec, respectively. However, iFinder (with 165 sec) outperforms them by 8.2% and 23.2%, respectively. Similarly under adversarial conditions (Table 5) in "Foggy", "Rainy", "Snowy" and "Night" conditions, iFinder outperforms them by (8.33%, 20.69%, 17.61%, 13.7%) and (16.67%, 24.14%, 15.09%, 17.35%), respectively. The improved accuracy and robustness validate the extremely significant value of leveraging vision-language grounding offered by iFinder.

• There are two approaches to improve the framework’s efficiency:

  • Limited Compute (e.g., few GPUs): Replace each model with a more efficient alternative to reduce latency.
  • Sufficient Compute (e.g., 8×A6000 GPUs): Host API versions of the models on a GPU node. The iFinder vision pipeline can then query these APIs and retrieve JSON outputs. Additionally, enable parallel execution of independent modules, and aggregate their outputs into a final JSON.

---

Response to Weakness 4 & Question 4. We understand the concern and clarify the following:

• DriveMM [16] and WiseAD [18] are SOTA domain-specialized baselines for ego-centric driving scene understanding. We use the official implementations and pre-trained checkpoints. DriveMM (49%) and WiseAD (50%) on Nexar (Table 3) do not perform worse than chance; the class distribution for the experiment is 50/50 by design (see Appendix C in supplementary material). We investigated this performance further and observed high recall + low precision (e.g., 1.00 recall, 0.50 precision in WiseAD) which indicates bias towards predicting accidents (reported in Table 3).
• Further note that, generalist models outperform driving-specialized models due to massive video pre-training. Models like VideoChat2 and VideoLLaMA2 benefit from scaling, not task specificity. That they outperform DriveMM highlights a gap in domain-adapted VLMs, a key motivation for ı̇Finder’s structured grounding.

---

Thank you for such a thorough and insightful review of our paper. We also appreciate the reviewer's strong support for the paper. Below are our responses to your questions. Please do not hesitate to raise further questions.

---

Response to Weakness 1 "SYSTEM COMPLEXITY" and Q1: We have analyzed the wall-clock run-time of each module in Table 1 in supplementary material. The full pipeline processes a LingoQA video in <4.5 minutes with unoptimized Python (Table 1 in supplementary material), suitable for offline post-hoc video analysis. Further (based on Reviewer zz3N's suggestion) we note the following. On a single A6000 GPU with 48 GB memory, the end-to-end runtime of ı̇Finder for video inputs of 5 seconds, 10 seconds, and 30 seconds (at 10 FPS with image size 640x360) is approximately 2.33 minutes, 3.85 minutes, and 18.75 minutes, respectively.

---

Response to Weakness 2 "RELIANCE ON RULE-BASED STRUCTURE": We agree that ı̇Finder targets the driving domain. However, we highlight the following that would enable generalization:

• Our representation is auto-generated via standard pretrained models (e.g., OWL-V2, SAM, CenterTrack) with no task-specific fine-tuning (Sec. 3.1). This decoupled design generalizes to any domain where object-level cues (position, orientation, interaction) are relevant.
• The robustness under visual noise is empirically evaluated on MM-AU. Even after filtering out >97% of object detections (τ = 0.8), accuracy remains 58.27%, surpassing all baselines including VideoLLaMA2 (52.89%) (Table 7). Our symbolic reasoning retains reliability under perceptual degradation.
• In Table 5, we also show that cross-condition generalization is strong. In adversarial settings (rain, snow, night), ı̇Finder outperforms the best V-VLM by 10–20% across all categories (Table 5), despite no domain adaptation. This suggests high resilience even under covariate shift.

While domain transfer is not the paper’s focus, we believe ı̇Finder offers a template for symbolic grounding that can be instantiated in new domains via different pretrained perception modules. We will include this as a direction for future work.

---

Response to Weakness 3 "UNCLEAR PEER CONTRIBUTION": Note that iFinder improves the peer V-VLM by correcting its initial response with the proposed representational abstraction of vision inputs that enables symbolic reasoning over dynamic scenes using LLMs. We provided the impact of different peers within iFinder on LingoQA dataset in Table below. It can be observed that iFinder can improve enhance the overall performance irrespective of the peer V-VLM involved.

---

Response to [Additional Comments] 1 & Q4:

• We completely agree with the Reviewer that adding a discussion as well as evaluating iFinder on works like reviewer references [1, 2, 3] would demonstrate interesting insights and strengthen iFinder's position. We will add a discussion on these works in the related works section in the revised version of the manuscript.
• In response to the reviewer’s suggestion to evaluate on the "Coool" benchmark [1], we explored the corresponding GitHub repository but found that ground truth annotations were not publicly available as of July 31, 2025. Despite this limitation, we conducted a qualitative analysis using an example to demonstrate the strengths of iFinder in Table below. | video name | Github Baseline response (manually reformatted for better readability in the table) | iFinder response | GT (visually observed) | |----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | video_0002.mp4 | Frame 2-109, 151-360: there is a white and black cat sitting on a table focus and ( gaussian blur ) 144p may 1 0 white fog extreme panoramic zoom see zoom out full width teaser dribbble 8k 8 k matte painting may 6 8 high resolution image untextured without glasses - signature Frame 110-150: there is a deer that is running down the street discord profile picture low quality footage his legs spread apart cryptid captures emotion and movement extremely close shot half dog loosely cropped youtube thumbnail moose dynamic closeup | Animal in same lane as ego vehicle (frames 102-108, 116-119). Animal changing lanes unpredictably (frames 120-125). Car approaching ego vehicle (frames 151-152, 157-159). Debris on wet road (surrounding_info field). | An animal ran from the right side and crossed in front of the ego car to the left. |
• Note that we evaluated iFinder under adversarial conditions (Table 5) in "Foggy", "Rainy", "Snowy" and "Night" conditions where it outperforms baselines (VideoLLama2 and VideoLLaVA) by (8.33%, 20.69%, 17.61%, 13.7%) and (16.67%, 24.14%, 15.09%, 17.35%), respectively. The improved accuracy and robustness validate the extremely significant value of leveraging vision-language grounding offered by iFinder.

---

Response to [Additional Comments] 2: We agree with the reviewer's suggestion. We will add the failure cases of iFinder in the revised manuscript. Please see below two examples from the Nexar dataset, marked by their respective video names (.mp4 files) where iFinder did not produce the expected response. We provide an analysis on possible reasons for these examples as well.

• “01031”: This dash cam video shows a vehicle cutting in from the left side of the ego car to the front. Since there is no obvious steering or vibration caused by a collision, it is even difficult for humans to determine whether a collision actually occurred.
• “00970”: This dash cam video shows a vehicle suddenly braking in front of the ego car. Although it is very close to the ego car, the video also shows the ego car braking, so it is not clear from the video whether an actual collision occurred.

---

Response to [Additional Comments] 3 & Limitations 4 & Q5: Based on reviewer's suggestion, we ran the iFinder module end-to-end with different random seeds on LingoQA dataset and provide the analysis below. Also note that we used fixed temperature for the LLM.

---

Response to comments under [Grammar & Typos & Formating]: Thank you for pointing this out. We agree with the reviewer and will fix this in the final version of the manuscript.

---

Response to Q2: Yes, we plan to extend iFinder toward semi-real-time scenarios (e.g. using quantized version of each module). The major (and exciting) challenge on this will be to maintain the performance of the adopted modules in their quantized form.

---

Response to Q3 & Limitations 2: Yes, iFinder’s pipeline is adaptable to domains beyond driving. This is because its schema—temporal object-centric data—is domain-structured, not domain-locked. Its modular design allows substitution of perception modules for other domains (e.g., indoor robotics), and the JSON format can accommodate arbitrary scene cues. Our ablations (Table 6) show performance is robust to component removal, indicating generalizable reasoning.

---

Thank you for the thoughtful comments and support for our paper. Below are our responses to your questions. Please do not hesitate to raise further questions.

---

• Response to Real-time latency/cost of running eight vision modules is not analysed. This might be a big problem if one or more module take a long time during inference.
  • We have analyzed the wall-clock run-time of each module in Table 1 in supplementary material. The full pipeline processes a LingoQA video in <4.5 minutes with unoptimized Python (Table 1 in supplementary material), suitable for offline post-hoc video analysis.

---

• Response to Besides, It relies exclusively on existing pretrained components. What if some module make mistakes? This is not discuss in the ablation study.
  • We analyzed the impact of error propagation (or effect of compounding errors when the initial module, in our case the object detector, fails to detect objects in the scene) in Table 7 on MM-AU dataset. For example, to imitate this detection failure, we remove >97% of object detections (τ = 0.8) from the JSON files and observed that the overall accuracy still remains 58.27%, surpassing all baselines including VideoLLaMA2 (52.89%) (Table 7). This shows that our symbolic reasoning retains reliability under perceptual degradation. Further, Table 7 also shows that iFinder demonstrates graceful degradation and limited error propagation. Please see L326-333 for a detailed discussion.

---

• Response to It seems that gains come largely from adding extra model-derived priors (3D orientation, depth) rather than novel learning; may be less compelling once future VideoVLMs close the gap.

  • We respectfully clarify that the gains in ı̇Finder result not from simply adding priors, but from a novel structured grounding framework that integrates visual cues into LLM reasoning via symbolic representations. Three quantitative findings support this:
    • Removing 3D orientation reduces MM-AU accuracy from 63.39% → 58.83%, but this is still 5.94% above VideoLLaMA2 (52.89%), showing priors are necessary but not sufficient (Table 6). The largest drop occurs only when structured scene understanding is removed (→ 57.81%).
    • End-to-end VLMs do not close the gap despite strong priors. VideoLLaMA2 and DriveMM have implicit access to visual cues but still lag by 10.5% and 39.17%, respectively, on MM-AU (Table 1). Structured reasoning over priors is critical, not their mere presence.
    • Removing only “Key Explanation” in our prompt drops accuracy by 4.66%, more than removing any single visual input (Table 6). This suggests performance stems from reasoning over structure, not data quantity.

  Our method is modular, zero-shot, and interpretable, offering complementary strengths to future end-to-end VLMs. We hope this clarifies that ı̇Finder is not a wrapper over priors, but a distinct, architectural contribution to structured LLM grounding.

---

• Response to What is the end-to-end runtime of ı̇Finder per 5s/10s/30s/60s video? A breakdown would clarify deployment feasibility.
  • On a single A6000 GPU with 48 GB memory, the end-to-end runtime of ı̇Finder for video inputs of 5 seconds, 10 seconds, and 30 seconds (at 10 FPS with image size 640x360) is approximately 2.33 minutes, 3.85 minutes, and 18.75 minutes, respectively.

---

• Response to If stronger/weaker object trackers or depth estimators are substituted, do the gains persist? Ablation study can be conducted by using weaker object trackers and check their performance with iFinder.
  • Thanks for the suggestion. We replaced the tracker and distance estimation modules with their weaker counterparts and analyzed the performance of iFinder on MM-AU dataset. We observed that replacing the tracker with a weaker tracker (simple IoU based tracker vs ByteTracker) drops performance slightly to 61.70%, indicating that tracking quality contributes moderately to system accuracy. Similarly, using a weaker depth model (Base vs Large) causes a minor drop to 62.78%.

---

• Response to Discuss two weaknesses.
  • We provide two observable weaknesses w.r.t. post-hoc analysis that future works can build upon.
    • The goal of this work is to demonstrate how vision-based modules can effectively ground large language models (LLMs) for robust reasoning, without relying on external sensors like LiDAR or GPS. By focusing solely on visual input, iFinder highlights the potential of vision as a standalone modality for scene understanding and decision-making. While current implementation does not integrate LiDAR or GPS data, this design choice underscores the strength of vision-driven grounding and leaves open the possibility for future multi-modal extensions to further enhance performance in complex scenarios.
    • Being entirely training-free, iFinder cannot self-correct based on prior mistakes or user feedback. Such adaptability can be built-in by adding in-context examples or LLM fine-tuning.

---