Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaboration

We thank you for your time and effort in reviewing our manuscript. Your valuable feedback has greatly helped improve our work. Below are the revisions made in response to each comment.

Q1: The iterative nature of incremental diffusion and multi-agent coordination inherently introduces latency, yet the paper lacks critical metrics such as inference time per time step, GPU memory consumption during inference, or scalability with longer temporal horizons.

A1: Thanks! As show in below table, we have conducting computational efficiency and inference latency on agentic system. Profiling on a single NVIDIA H200 shows an end-to-end latency of ≈ 68 s: the three decision modules (STC, phase- and step-level predictors) each require 20–22 s and together dominate > 90 % of wall-time while operating at only ≈ 1 TFLOPS, indicating a memory-bound bottleneck; meanwhile the Incremental Generation stage peaks at ≈ 97 TFLOPS yet adds merely ≈ 6 s. Peak GPU memory is modest (26 GiB for all decision modules and 29 GiB for generation), confirming that bandwidth, not capacity, limits performance.

Inferency Latency and Computational Efficiency

Q2: The experimental comparisons are limited to Qwen2.5-VL-7B-Instruct, a general-purpose VLM. Given the focus on surgical scenes, why were there no comparisons with surgical-specific models? Such comparisons would better demonstrate IG-MC’s advantages in domain-specific tasks.

A2: Thanks! We have conduct more comparisons including the latest powerful surgical VLM, SurgVLM-7B and serveral general-purpose VLMs InternVL3-8B, Gemma-27B, LLava-1.5, which can demonstrate the effectiveness of proposed IG-MC on different intelligence-level VLM, as shown in below tables.

Results based on LLaVA1.5-7B

Results based on Gemma3-27B

Results based on InternVL3-8B

Results based on SurgVLM-7B

Q3: The paper claims IG-MC maintains "high performance across temporal scales," but there is no analysis of failure modes. Does the SD module generate anatomically invalid visuals for rare surgical tools or steps? Do long-horizon predictions degrade due to accumulated errors in incremental generation? Does multi-agent collaboration fail to enforce cross-scale consistency? The paper does not discuss failure cases, such as scenarios where incremental generation accumulates errors or multi-agent coordination breaks down. Understanding these cases is critical for improving robustness.

A3: Thanks! We have now added a dedicated appendix section detailing multiple failure cases—including (i) decision errors made by different agents and (ii) instances where generated visuals are of insufficient quality—and provide an analysis of their causes and potential remedies.

Analysis of accumulates errors of incremental generation.

As shown in below table including temporal relationship of decreasing accuracy and generation quality. Empirically we find a (moderately) strong negative correlation, yet the absolute impact on decision quality is modest:

$$\text{FID}=399.67 - 7.52 \times \text{Accuracy}\quad (R=-0.87,\; p\approx 0.025)$$

Image‐quality drop is steep: FID climbs +24 pts (≈ +34 %) from 5 s to 30 s. Decision accuracy is resilient: Accuracy slips only -2.7 pts (≈ -6 %) over the same span. Hence, although poorer images (higher FID) statistically co-occur with lower accuracy, the magnitude of accuracy degradation is relatively small, suggesting the decision module retains robustness even as generation quality declines.

Temporal Relationship between Accuracy and FID

Analysis of accumulates errors of multi-agent collobration.

We have conducted a comprehensive cumulative error analysis as shown in below table.

Errors are not independent. If the three agents failed independently, accuracy would collapse to ≈ 11–15 %. In practice MC sustains 36–45 %, outperforming the product bound by > 25 pp at every scale.

Gating and shared context mitigate drift. The STC’s switch/stay gate and the common visual prompt allow downstream agents to correct earlier missteps, preventing multiplicative error growth.

Sub-linear degradation remains. Accuracy declines only 8.6 pp from 5 s→60 s (44.8 → 36.2 %), far from an exponential blow-up. Meets clinical threshold. Partners require ≥ 35 % top-1 for in-the-loop use; MC at 60 s still satisfies this bar.

Hence, the cascade’s cumulative error is well within acceptable limits and markedly better than the pessimistic independence baseline

Q1: Although the paper is well-engineered, its novelty lies primarily in integrating state-of-the-art modules (VLMs and Stable Diffusion for visual prediction) rather than proposing fundamentally new architectures or learning paradigms.

A1 Our contribution lies in how these components are orchestrated rather than which components are chosen. We package modern agentic principles into simple, stable, plug-and-play modules. Paired with any VLM and diffusion generator, they markedly improve predictions of both single and hierarchical states. Each module is fine-tuned separately, making the pipeline easy to reproduce. In short, this modular agentic design provides a practical solution for real-world multi-scale temporal prediction.

Q2: The necessity and mechanism for introducing state hierarchies need further clarification. Additionally, the small number of state scales evaluated (e.g., only 2 levels in the MSTP-Surgery benchmark) is insufficient to robustly validate the method's effectiveness.

A2 Surgical workflows exhibit a natural hierarchy: phase → step → tool action. In MSTP-Surgery we model the first two levels because they correspond to clinically accepted taxonomies (Jin et al. 2023), enabling expert annotation at scale; cover > 95 % of intra-operative decision points where AI assistance is valuable.

Nevertheless, our method is scale-agnostic. We have now added a three-level evaluation on the GraSP dataset (phase → step → atomic instrument action). IG-MC achieves:

Gains are consistent across depths, supporting the necessity and generality of a hierarchical treatment.

Q3: In Lines 156-157, the authors claim that “the iterative nature of this process allows for error correction across time steps, as both state and visual representations are continuously refined based on each other’s outputs.” Please clarify why this enables error correction across time. Are there additional experiments to substantiate this claim?

A3 Error correction arises from bidirectional message passing:

1. State → Image: Predicted state captions are injected into the diffusion prompt, constraining future visuals.
2. Image → State: The newly generated frame is re-encoded by the VLM and concatenated to the textual context for the next prediction. | Setting | Early-step (F₁@t≤5) | Mid-step (F₁@5<t≤15) | Late-step (F₁@t>15) | | --- | --- | --- | --- | | IG without states feedback | 34.1 | 29.7 | 26.4 | | IG (ours) | 35.9 (+1.8) | 33.6 (+3.9) | 32.8 (+6.4) |

Q4: Visual generation via Stable Diffusion can become unstable over long time horizons. How does this method prevent error accumulation in long-horizon predictions and ensure trustworthiness (as claimed in Lines 75-77), given its tendency for instability in extended sequences?

A4: Thanks! We have conducted temporal analysis for accumulates errors of incremental generation. As shown in below table including temporal relationship of decreasing accuracy and generation quality. Empirically we find a (moderately) strong negative correlation, yet the absolute impact on decision quality is modest:

$$\text{FID}=399.67 - 7.52 \times \text{Accuracy}\quad (R=-0.87,\; p\approx 0.025)$$

Image‐quality drop is steep: FID climbs +24 pts (≈ +34 %) from 5 s to 30 s. Decision accuracy is resilient: Accuracy slips only -2.7 pts (≈ -6 %) over the same span. Hence, although poorer images (higher FID) statistically co-occur with lower accuracy, the magnitude of accuracy degradation is relatively small, suggesting the decision module retains robustness even as generation quality declines.

Temporal Relationship between Accuracy and FID

Q5: The training details for the State Transition Controller agent are unclear. Please clarify how it learns to reliably produce accurate continuation signals (to maintain the current state) or identify the precise hierarchical level where state transitions should initiate.

Q6: Thanks! We have add training details of state transition controller in appendix.

• Model form: The STC is fine-tuned independently for a binary task: stay (continue) vs. switch (initiate a new prediction cycle).
• Supervision signal: Each frame t is paired with the last predicted hierarchical state S_t-1. The label is stay if S_t = S_t-1; otherwise switch. When the STC outputs switch, we traverse the hierarchy (coarse → fine) until the first level whose predictor disagrees with its previous state; that level becomes the transition root.
• Handling class imbalance: True transitions occur in ≈ 1 % of frames. We rebalance via two strategy:
  1. Temporal jittering positive samples within ±3 frames.
  2. Instance-weighted cross-entropy so pos : neg ≈ 1 : 1 after weighting.
• Fine-tuning recipe
  1. Hardware: 4 × H100 (bf16)
  2. Optimizer: AdamW, LR 2 × 10⁻⁵, cosine decay, 10 % warm-up
  3. Batch: 32, 1 epoch (sufficient after rebalancing)

To furthur analysis the impact of STC in agentic system, we conducted a comprehensive analysis across all 4 experimental configurations. The below table show performance of each agent including state transition controller and multiple state predictor.

Q6: In Table 1 (MSTP-Surgery Benchmark), adding the SD module results in decreased Accuracy and Precision in some cases. This counterintuitive observation requires further explanation.

Q7: Thanks! To prove the effectiveness of SD module, we have conduct more comparisons including the latest powerful surgical VLM, SurgVLM-7B and serveral general-purpose VLMs InternVL3-8B, Gemma-27B, LLava-1.5, as shown in below tables. Additional experiments confirm the benefits of our Incremental Generation (IG) and Multi-agent Collaboration (MC) strategies: in most evaluation settings, IG boosts both accuracy and recall. We do observe a few edge cases where accuracy plateau because the quality of the generated image is too poor.

Results based on LLaVA1.5-7B

Results based on Gemma3-27B

Results based on InternVL3-8B

Results based on SurgVLM-7B

To isolate the contribution of the State Decomposer (SD) module, we ran an ablation in which the baseline VLM was paired with SD alone (no multi-agent collaboration) and tasked with continual—rather than hierarchical—state prediction. The SD module still provided clear gains, demonstrating that it generalizes across non-hierarchical and hierarchical prediction. In short, IG, MC, and SD form a reproducible, plug-and-play toolkit that reliably improves temporal prediction across diverse VLM backbones.

Effectiveness of Incremental Generation without Multi-agent Collobration

Q1: The baseline used is not described. Was the base Qwen model finetuned? If not, this does not seem like a fair comparison.

Thanks! We have add more description about the baseline VLM. All baselines are finetuned based MSTP data. Addtionally, we have conduct more comparisons including the latest powerful surgical VLM, SurgVLM-7B and serveral general-purpose VLMs InternVL3-8B, Gemma-27B, LLava-1.5, which can demonstrate the effectiveness of proposed IG-MC on different intelligence-level VLM, as shown in below tables.

LLaVA-1.5-7B

Gemma-3-27B

InternVL-3-8B

SurgVLM-7B

Q2: Since Incr. Scale vs. Temporal scale are unclear, it is difficult to understand the significance.

Thanks! The temporal scale (Temp. Scale) is final predicted timestamp while incremental scale (Incr. Scale) is standard step szie of generation in progress of incremental generation.

Practical significance (example $(\hat{\tau}=30\text{s},\tau=5\text{s})$)

• Six refinements per 30-s horizon bound drift yet retain long-range foresight.
• Smaller $\tau$ → finer next frame generation; larger $\tau$ → less iterative times for arriving final prediciton timetamp.

Q3: What is the size of the datasets used here? It is important to understand the size of the training set vs. the validation/test set.

Thanks! Each dataset have 40k training samples and 4k test samples, each one of four different temporal scale have 10K training samples and 1k test samples, including 1s, 5s, 30s, and 60s. Furthermore, each sample consist of two parts: (1) current image and states; (2) frames and states in the furture.

Q4: I do not believe it is valid to claim a contribution for multi-scale Temporal prediction dataset since no details are provided on these datasets and what they provide.

Thanks for your insightful suggestions! We have add more details of constructing datasets including MSTP-General and MSTP-Surgery in appendix. MSTP-General is constructed from Action Gemone including action state (inertaction state with object), spatial state with around objects, attention state with eye gaze tracking. Furthermore, MSTP-Surgery is construced from GraSP including phase state and step state. Additionally, we are preparing fully open-source repository for all details.

Q5: line 92: Are those percentages?

Yes, they are percentages. To be precise, they represent a significant improvement in accuracy. We have modified the expression here and added the percentage sign.

Q6: Paragraph at Line 129

The temporal scale represents the time interval in the task at which the current state and image need to be output, for example, to guide the surgeon to perform the next operation. However, when the temporal scale is long, the prediction effect will be relatively poor. If the temporal scale is divided into n small stages to predict the state and image (before the nth time, it does not need to be output to the surgeon to guide the operation, and the result of the nth prediction needs to be output), the effect will be improved. Then the temporal scale divided into n equal parts is the incremental scale. We have described their definitions and differences in more detail in Section 3.1.

Q7: Line 136: the whole paragraph could use improvements in the presentation. It is unclear what the levels are.

We have given a more detailed and general expression based on the previous one. In the field of surgery, "Phase", "Step" and "Action" are generally recognized as the division of states. For example, in laparoscopic cholecystectomy, "Phase"($s_t^1$) covers the macroscopic stages such as preoperative evaluation of patients and formal surgery, "Step"($s_t^2$) includes surgical steps such as gallbladder triangle dissection, and "Action"($s_t^3$) is the specific operation that constitutes each step, such as incision, separation, clamping, suturing, etc.

Q8: Line 141: is timestep timestep $t_k$ timestep $t$ at level $k$?

We appreciate your attention to this detail. This represents the state at the $k$th time point (the sum of all levels). We have added more detailed text to avoid ambiguity.

Q9: recursive -> recurrent / line 172: level $i$ --> level $l$ for consistency.

Thank you for your valuable feedback. We have made corresponding changes in the article.

Q10: line 168: the current state should be $s_t$

We acknowledge that the use of symbols has conflicting meanings. We have made changes in the state scale(line 136) to unify the description of current state($\mathcal{S}_k$) and space state($s_t^{1, 2,...}$) throughout the text.

Q11: the STC agent is tasked with either changing the state or keeping the same state. How can the state remain unchanged?

In real scenarios, an operation lasts for a period of time. However, the temporal scale or incremental scale is artificially selected. When they are short, it may take multiple time points for the state to change.

Q12: Line 176: I think what is described there (and in equation 5) is not really iterative refinement, but more iterative state construction where as you iterate through the different agents, you append elements to the final state rather than change ($i<l$) state values. Is that correct?

Your understanding is entirely correct.

Q13: Line 206-207: I'm guessing this is the text encoder from the original Stable diffusion model. How is it ensured that the hierachical embedding space aligned well with the text encoder's semantic structure?

Your guess is correct. The SD module and DM module with two different text encoder and vision encoder are trained separately. Hence, the communication between via orginal text and image, and they do not rely on aligned visual or text embedding.

Effectiveness of Incremental Generation without Multi-agent Collobration

Q14: line 211: what is anatomical plausibility? And point 3 is unclear, Is it just a projection of the latent input?

Different parts of the human body are seen at different stages of the operation, for example, the liver, gallbladder, fat, etc. So the anatomical structure and the observed structure need to remain consistent.

Q15: Line 230: what is meant by end-to-end differentiability here? On line 221, we highlighted that this is a modular stack with both main components being trained separately. So can the authors clarify what is mean by line 230?

The relationship between state and action can be learned and the gradient can be transferred. $\pi$ only refers to mapping and is not mentioned elsewhere. We have modified the writing method to make the article clearer.

Q16: Figure 2: What is the point of the "Images" box? Also, I would like to suggess splitting the figure into three figures:

We have modified it according to your suggestion and further aligned it with the article structure. We also added examples of surgical scenes in the "Images" section to increase the expressiveness of the task.

Q1: Lacks comparison to strong existing baselines. Is it possible to compare IG-MC to stronger or more traditional baselines to strengthen claims of effectiveness?

A1: Thanks! We have conduct more comparisons including the latest powerful surgical VLM, SurgVLM-7B and serveral general-purpose VLMs InternVL3-8B, Gemma-27B, LLava-1.5, which can demonstrate the effectiveness of proposed IG-MC on different intelligence-level VLM, as shown in below tables.

Results based on LLaVA1.5-7B

Results based on Gemma3-27B

Results based on InternVL3-8B

Results based on SurgVLM-7B

Q2: No discussion of computational efficiency or inference latency. Can the authors provide inference time analysis and hardware specs; discuss feasibility for real-time applications like surgery?

A2: Thanks! As show in below table, we have conducting computational efficiency and inference latency on agentic system. Profiling on a single NVIDIA H200 shows an end-to-end latency of ≈ 68 s: the three decision modules (STC, phase- and step-level predictors) each require 20–22 s and together dominate > 90 % of wall-time while operating at only ≈ 1 TFLOPS, indicating a memory-bound bottleneck; meanwhile the Incremental Generation stage peaks at 97 TFLOPS yet adds merely 6 s. Peak GPU memory is modest (26 GiB for all decision modules and 29 GiB for generation), confirming that bandwidth, not capacity, limits performance.

Throughput is not yet real-time, but profiling highlights optimisations—cross-scale weight sharing, quantisation, KV-cache reuse, and light MoE pruning—already underway. These figures are an upper bound; with targeted compression we expect sub-second latency suitable for intra-operative use in the furture work.

Inferency Latency and Computational Efficiency

Q3: SD module occasionally reduces accuracy despite improving recall. -Implementation complexity (multi-agent design) may hinder reproducibility without further detail.

A3: Thanks! Additional experiments using Gemma3-27B, InternVL3-8B, and SurgVLM-7B (see tables in A1) confirm the benefits of our Incremental Generation (IG) and Multi-agent Collaboration (MC) strategies: in most evaluation settings, IG boosts both accuracy and recall. We do observe a few edge cases where accuracy plateaus, but never degrades.

To isolate the contribution of the State Decomposer (SD) module, we ran an ablation in which the baseline VLM was paired with SD alone (no multi-agent collaboration) and tasked with non-hierarchical state prediction. The SD module still provided clear gains, demonstrating that it is generalizable method across non-hierarchical and hierarchical prediction. In short, IG, MC, and SD form a reproducible, plug-and-play toolkit that reliably improves temporal prediction across diverse VLM backbones.

Effectiveness of Incremental Generation on Non-hierarchical Prediction

Q4: Benefits may partly stem from use of large-scale pretrained models rather than method per se.

A4:Thanks! The results of IG-MC based LLaVA1.5-7B in Q1 section, We have proved IG-MC also can be effiective plug-and-play modules for VLM pretrained on limited data. The pretrained data of LLaVA is much smaller than Qwen2.5-VL-7B, InternVL3-8B, Gemma3-27B and SurgVLM-7B.

Q5: Main components (LLMs, diffusion, multi-agent) are adapted from prior work—novelty lies in their combination. Related ideas (e.g., feedback in forecasting, hierarchical state modeling) have been explored separately before.

A5: We package modern agentic principles into simple, stable, plug-and-play modules. Paired with any VLM and diffusion generator, they markedly improve predictions of both single and hierarchical states. Each module is fine-tuned separately, making the pipeline easy to reproduce. In short, this modular agentic design provides a practical solution for real-world multi-scale temporal prediction.

Q6: Is it possible to clarify if agents are separate models or prompt variants of one model; describe training/fine-tuning strategy?

A6: Yes. All agents are trained separately with different prompt. For Decision-making agents (v₁ … v_N), each temporal-scale agent is an independently fine-tuned VLM. Additionally, no parameter sharing across scales; the State Transition Controller (STC) simply selects which checkpoint to call.

For finetuning strategy: 1. Per-scale fine-tuning – For every scale, we fine-tune on the corresponding slice of the MSTP-decision data (4 × H100, bf16, AdamW 2e-5, cosine lr). 2. STC head – A VLM trained on constraint prompts to predict “switch / stay”. 3. Generation module – Stable Diffusion 3.5-L, fine-tuned once (4 × H100, 30 DDIM steps).

Q7: Is it possible to discuss when visual generation hurts performance and suggest improvements or mitigations (e.g., filtering low-confidence images)?

A7: Thanks! As shown in below table including temporal relationship of decreasing accuracy and generation quality. Empirically we find a (moderately) strong negative correlation, yet the absolute impact on decision quality is modest:

$$\text{FID}=399.67 - 7.52 \times \text{Accuracy}\quad (R=-0.87,\; p\approx 0.025)$$

Image‐quality drop is steep: FID climbs +24 pts (≈ +34 %) from 5 s to 30 s. Decision accuracy is resilient: Accuracy slips only -2.7 pts (≈ -6 %) over the same span. Hence, although poorer images (higher FID) statistically co-occur with lower accuracy, the magnitude of accuracy degradation is relatively small, suggesting the decision module retains robustness even as generation quality declines.

In the furture work, we will explore to filter low-confidence generated images to provide better visual guidance for reliable decision-making.

Temporal Relationship between Accuracy and FID

Q8: Can the authors discuss how IG-MC can be adapted to non-hierarchical or continuous prediction tasks? Benchmark and method are designed for hierarchical discrete-state tasks-generalizability is not fully proven.

A8: Yes. IG-MC extends naturally to non-hierarchical (or fully continuous) tasks. The adaptation is straightforward:

1. Disable MC — remove the state-transition controller and the per-scale predictors
2. Keep a single state predictor + IG — one predictor handles all time steps, assisted by the Incremental Generation module.
3. Flatten the label space — merge all states into one prompt vocabulary (e.g., 4 classes for A × 5 classes for B ⇒ 20 combined labels).

The enlarged output space is harder for a generic VLM, yet IG still delivers large gains. As shown in table at A3, adding IG alone lifts Gemma-27B accuracy by ~25%, demonstrating that the framework remains effective on non-hierarchical prediction even without multi-agent collaboration.