Dynamic Test-Time Compute Scaling in Control Policy: Difficulty-Aware Stochastic Interpolant Policy

We thank the reviewer for the thoughtful assessment and constructive feedback.

Addressing the Weaknesses

1. "The method has only been evaluated in simulation. Real-world robot validation, though understandably out of scope, would further solidify the impact"

1. To address this concern, we conducted an experiment with the 'real robot pushT' that has publicly available training data. We were able to train a diffusion policy and SIP with linear interpolation and velocity prediction and found a decreasing pattern of training action MSE, revealing that SIP can mimic the expert human demonstrator, which suggests it will be useful when deployed in the real world. Due to time constraints, we were unable to roll out actual robot trajectories.

Train Loss Comparison

Train Action MSE Comparison

2. "The robustness of the method under noisy sensor inputs or misclassified difficulty levels is not fully explored."

1. We conducted an additional experiment where we gradually added same-random-seed noise to the test data images only.

• Training samples per environment: 300
• Test set: 20% of data (stratified split)
• Random seed: 42 (ensures reproducible noise across runs)
• Noise levels: σ ∈ {0.0, 0.05, 0.15, 0.30, 2.0}

Test Accuracy (%) Under Different Noise Levels

This experiment demonstrates that our CNN models still maintain reasonable performance under high noise conditions (σ=0.30) with only 3.06% average accuracy drop. σ=2.0 represents extreme noise conditions beyond typical real-world scenarios.

3. "...the lack of real-world deployment raises questions about generalizability beyond the simulation benchmarks"

1. Simulation-based research has historically driven significant advances in robotics and RL. Notable examples include: DQN (Mnih et al., 2015) and PPO (Schulman et al., 2017), which established foundational RL algorithms entirely through simulated environments before widespread real-world adoption. Similarly, we believe our work introduces an initial exploration of test-time compute scaling for robotic control—a concept that may serve as a starting point for broader applications in real-world systems as the field continues to develop adaptive computation strategies.

2. While sim-to-real transfer remains an important direction for future work, our current results provide valuable insights for the community on integrating human cognitive assessments into robot learning systems.

4. "Some components (e.g., zero-shot and few-shot VLMs for difficulty classification) are based on prior ideas."

1. While we acknowledge that VLMs themselves are established technologies, we believe our work makes important contributions by demonstrating their novel application to adaptive compute in robotic control. Determining task difficulty for dynamic compute allocation in robotic policies is a fundamentally different problem from typical VLM applications. Although VLMs have shown success in image understanding and captioning tasks, their ability to assess manipulation difficulty for compute optimization had not been explored. To our knowledge, our work is the first to show that VLMs can effectively predict the computational requirements of robotic control subtasks, enabling significant efficiency gains.

2. Similar to how successful RL works like CURL [ICML'20], RAD [NeurIPS'20], and DrQ [ICLR'21] made significant contributions by adapting existing CV techniques (contrastive learning, image augmentation) to RL settings, our work demonstrates that VLMs can be repurposed for a novel task—difficulty assessment for adaptive inference. The key contribution lies not in the VLM architecture itself, but in showing that visual-language understanding can guide computational resource allocation in robotic control.

Addressing the Questions

1. "How sensitive is the overall system performance to errors in the difficulty classification stage?"

Table 5 demonstrates the impact of classification accuracy by comparing zero-shot VLM, fine-tuned VLM, and CNN classifiers. While the VLMs achieve lower accuracy than CNN, we included them to address scalability concerns—they require significantly less training data and leverage pre-trained generalization capabilities. Crucially, even with misclassifications, performance never drops below the min-compute baseline because any classification yields at least one inference step. This provides an important safety guarantee: users can set their minimum acceptable performance level through the min-compute configuration, ensuring that classification errors may reduce efficiency but will never compromise this performance floor. Thus, the system degrades gracefully under classification errors rather than failing catastrophically.

2. "Have you conducted experiments where the classifier is intentionally degraded to test robustness? Clarifying this would help evaluate how reliable DA-SIP is under real-world noise or partial observability"

1. We tested both aspects: (1) Degraded classifiers: Table 5 shows that even with lower-accuracy VLMs, performance never drops below min-compute baseline. (2) Noisy sensors: As detailed in Weakness #2 above, we added Gaussian noise to test images and found only 3.06% average accuracy drop at σ=0.30 under realistic noise conditions. Both experiments confirm DA-SIP degrades gracefully, with performance always bounded by the user-configurable min-compute threshold.

3. "What is the runtime overhead of the different classifiers? In high-frequency control scenarios, even lightweight models could become bottlenecks. Providing profiling data would help assess real-time applicability."

1. We re-rolled out the policy for Tool Hang (88 iterations, per-iteration average):

2. Moreover, we tested the impact of increasing number of images as examplars in VLM prompt.

Even though higher number of images would increase the accuracy, we fixed on 3 img/cat as a good balance of performance, time, and memory usage.

4. "Have the authors considered sim-to-real strategies or domains where DA-SIP could already be viable? Any partial evaluation on physical hardware would strengthen the practical impact."

1. Yes, we conducted a partial evaluation by training SIP on 'real robot pushT' data (discussed in Weakness #1). We specifically chose pushT because it directly corresponds to one of our simulation environments. The key finding is that SIP achieves comparable train action MSE error to the proven Diffusion Policy (0.00294 vs 0.00467 at step 4614), demonstrating that SIP can accurately reproduce human demonstrator trajectories. This low MSE indicates SIP learns the mapping from observations to actions as effectively as Diffusion Policy, strongly suggesting it would perform similarly when deployed on the actual robot.

Addressing the Limitations

1. Lack of discussion on societal impacts for resource allocation bias and for safety-critical applications.

1. Our VLM experiments directly address this concern by demonstrating bounded performance degradation under classification errors. As shown in Table 5, even with lower-accuracy VLMs, performance never drops below the min-compute baseline—providing a guaranteed performance floor that users can configure based on their risk tolerance. The worst case (18% drop in Transport task) still maintains acceptable performance. This ensures that classification errors reduce efficiency but never compromise safety thresholds. As VLM capabilities improve, this classifier-policy gap will naturally narrow, making adaptive compute increasingly practical for safety-critical applications.
2. Similar to LLMs that adjust behavior based on context, we envision implementing safety filters that automatically enforce higher compute thresholds when detecting humans nearby or operating near critical objects. The key insight is that adaptive compute should enhance, not replace, existing safety protocols.

We sincerely thank the reviewer for the positive assessment and feedback.

Addressing the Weaknesses

1. "The difficulty categories are too task-specific and may not be directly extended to other more complicated tasks, e.g., bimanual tasks, fine-grained in-hand manipulation tasks, articulated or deformable object manipulation tasks, etc."

1. We appreciate the reviewer raising this important consideration. Through systematic analysis of our results in Table 1, we identified that task failures could be consistently attributed to specific sub-tasks that characterize each main task. While we acknowledge that new categories may emerge for different task types or robot platforms, our extensive grid search of SIP configurations revealed that these six categories comprehensively capture the fundamental challenges for robot arms with limited precision and frequency constraints.

2. Importantly, these six categories successfully explained all sub-task variations across our diverse set of 6 tasks, which we believe provides comprehensive coverage for this robot class. The strong consensus among our 8 human annotators (enabling accurate classifier training) demonstrates that these categories represent genuine, recognizable patterns rather than arbitrary divisions. The classifier's high prediction accuracy further validates that our categorization captures meaningful and consistent difficulty dimensions.

3. Regarding the specific examples of complex tasks mentioned, we note that our evaluation already encompasses both bimanual coordination (Transport task) and fine-grained manipulation (PushT task, which requires precise control despite not being in-hand). The effectiveness of our dynamic difficulty classification in improving performance while reducing completion time for these challenging tasks suggests that our framework can indeed handle sophisticated manipulation scenarios beyond simple pick-and-place operations.

2. "The use of manually labeled data for learning the difficulty model is not scalable. Considering the limitation of the manually defined difficulty categories, the dataset would have to be re-labeled whenever the inputs to the difficulty model are changed or the difficulty categories have to be re-defined."

1. We conducted an additional experiment to investigate the minimal data requirements for effective difficulty classification:

Most tasks achieve near-plateau performance with just 300 labeled images, requiring only ~15 minutes of annotation (3 seconds per image).

2. We hypothesized that after finetuning the VLMs with enough data, they may learn generalization ability that they do not need to be finetuned with new data but are able to make meaningful predictions. We conducted an additional experiment to test generalization:

While there is room for improvement, we believe with the current rate of VLM progress, its generalization ability will only improve in the future.

3. Going further with VLMs, they are used precisely to address the scalability issue in our paper. We only used 3 images per category in the prompt for VLMs and used a total of 1049 training images to finetune VLMs. We conducted an additional experiment to see the impact of differing numbers of images for VLMs:

While VLM classification accuracy remains below that of CNNs, VLMs were still effective enough to show performance improvement and time savings as shown in tables 4 and 5.

4. Comparison to existing approaches: We respectfully note that manual annotation is standard practice in robotics research:
   • Diffusion Policy (Chi et al., 2023) requires extensive human demonstrations
   • RT-1 (Brohan et al., 2022) used 130k human-annotated episodes
   • Our approach requires orders of magnitude less annotation while enabling adaptive efficiency

Addressing the Question

1. In equation 3, could you explain in detail why

$$s(\mathbf{x},t,\mathbf{o})=\sigma_t^{-1}\mathbb{E}[\boldsymbol{\varepsilon}\mid\mathbf{x}_t=\mathbf{x},\mathbf{o}]\ ?$$

1. Forward process From the stochastic interpolant formulation: \mathbf{x}_t=\alpha_t\mathbf{x}_0+\sigma_t\boldsymbol{\varepsilon},\quad \boldsymbol{\varepsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\tag{1}

2. Posterior of the noise Key insight: Due to the linearity of the interpolant (1) in $(\mathbf{x}_0, \boldsymbol{\varepsilon})$, we can express: $\boldsymbol{\varepsilon} = \frac{\mathbf{x}_t - \alpha_t\mathbf{x}_0}{\sigma_t}$

Taking conditional expectations given $\mathbf{x}_t = \mathbf{x}$ and $\mathbf{o}$: \mathbb{E}[\boldsymbol{\varepsilon}\mid\mathbf{x}_t=\mathbf{x},\mathbf{o}] = \frac{1}{\sigma_t}\left(\mathbf{x}-\alpha_t\mathbb{E}[\mathbf{x}_0\mid\mathbf{x}_t=\mathbf{x},\mathbf{o}]\right)\tag{2}

This follows directly from the linearity of the interpolant—no joint Gaussianity assumption needed.

3. Connection to the score By definition, the score function is: $s(\mathbf{x},t,\mathbf{o}) = \nabla_{\mathbf{x}}\log p_t(\mathbf{x}\mid\mathbf{o})$

Using Stein's identity (also known as Gaussian integration by parts), the score can be expressed as: s(\mathbf{x},t,\mathbf{o}) = -\sigma_t^{-1}\mathbb{E}[\boldsymbol{\varepsilon}\mid\mathbf{x}_t=\mathbf{x},\mathbf{o}]\tag{3}

This matches Lemma 3 of Song et al. (2021) and Eq. (38) of the SiT appendix.

4. Sign convention If you define $\boldsymbol{\varepsilon}':=-\boldsymbol{\varepsilon}$ or adopt the alternative convention $\tilde{s}:=-\nabla_{\mathbf{x}}\log p_t$, then: s(\mathbf{x},t,\mathbf{o})=\sigma_t^{-1}\mathbb{E}[\boldsymbol{\varepsilon}\mid\mathbf{x}_t=\mathbf{x},\mathbf{o}]\tag{★}

Addressing the Limitations

1. "The method has yet to be tested on real robots and more complex tasks"

1. To address this concern, we conducted an experiment with the 'real robot pushT' that has publicly available training data. We were able to train a diffusion policy and SIP with linear interpolation and velocity prediction and found a decreasing pattern of training action MSE, revealing that SIP can mimic the expert human demonstrator, which suggests it will be useful when deployed in the real world. Due to time constraints, we were unable to roll out actual robot trajectories.

Train Loss Comparison

Train Action MSE Comparison

The comparable or superior trajectory matching on real robot data provides confidence in sim-to-real transfer potential.

We sincerely thank the reviewer for the thoughtful analysis of our work.

0. "I'm wondering if this is a direct application of stochastic interpolant on robotic policy learning ..."

1. Yes, this is a direct application of the Stochastic Interpolant (SI) framework to robotic policy learning. Our work systematically investigates the flexibility in training and inference configurations of SIP especially applicable for the domain of robotic manipulation.

1. "Could authors justify why max-compute is a good baseline to compare with?"

1. We thank the reviewer for this insightful observation. We selected max-compute as a baseline specifically because it reflects the current practice in the field. To our knowledge, most diffusion/flow matching policy works employ either fixed max-compute configurations (typically 100 steps) or focus on distillation methods using 1-2 steps. Therefore, we adopted the commonly used min and max compute settings with SIP as our baseline comparison. We will include the per‑task ‘best static’ configuration column in the appendix so that readers can directly compare DA‑SIP against both min‑ and best‑static baselines.
2. Initially, we hypothesized that we could identify one single best configuration with max-compute for SIP that would maximize performance across all environments. However, our investigation in Section 4.2 revealed that this assumption was incorrect. We discovered that different tasks indeed require fundamentally different configurations to achieve optimal performance. Through careful analysis of rollout videos across multiple environments, we identified distinct sub-task characteristics that explain why each environment benefits from different configurations.

2. "Table 4,5 illustrate that max-compute achieves very similar performance as min-compute in many tasks (Tool Hang, Square, Lift, Can) but needs substantially more time. In these tasks, DA-SIP seems to use a much longer time than the min-compute baseline."

1. This is true, and this observation actually validates our key finding. In simpler games like Lift and Can, when using linear interpolation with ODE in SIP, an inference step of 1 can achieve a near perfect score (which is not the case with Diffusion Policy as shown in tables 9 and 10). While existing methods typically require secondary distillation to reduce inference steps, our results demonstrate that our linear interpolation with ODE eliminates this need entirely for these tasks.

2. For Square, max-compute yields only marginal improvements (2%) from min-compute while substantially increasing computation. Our dynamic classification approach addresses this inefficiency, reducing runtime from 146.68s to 56.2s using CNN classification while achieving +4% performance improvement.

3. Tool Hang presents a unique case where increased computation fails to improve performance. Our analysis reveals that higher inference steps lead to trajectory oversmoothing and reduced exploratory behavior, both harmful when stochastic attempts are crucial for success. With our VLM classification, we can achieve 8% gain with -2.28x reduced run time.

3. "How is time measured? Is the overhead of the classifier included?"

1. All reported times are wall-clock measurements that include every component of the system:

• Classifier inference time
• SIP inference time
• Simulation execution time
• Data pre/post-processing time

We re-rolled out our policy for Tool Hang (cumulative run of 88 iterations):

2. The theoretical model in Section 3.5 serves to provide analytical understanding of computational complexity rather than predicting actual runtime.

4. "... the current implementation does not seem scalable. It seems that both difficulty categories and their corresponding inference triplets are manually designed with some heuristics."

1. We appreciate the reviewer's concern. The inference triplets in Table 3 were systematically derived from our comprehensive task-level analysis in Table 1. By analyzing failure modes across tasks, we identified that similar sub-task types (e.g., precision placement) benefit from similar computational configurations regardless of the specific task. This insight allowed us to map difficulty categories to appropriate inference settings.

2. The robustness of our categorization is validated by strong consensus among 8 human labelers, and the 6 categories capture fundamental manipulation types (approaching, grasping, precision placement, etc.) that generalize across tasks. The high classification accuracy (Table 2) validates that these categories are meaningful and recognizable, not arbitrary.

3. While the current mapping from categories to configurations is derived from empirical analysis rather than end-to-end learning, this demonstrates the viability of adaptive computation for robotics. Future work can learn these mappings directly, but our contribution—showing that adaptive computation yields 2.6-4.4× speedups—remains valid regardless of how the mapping is determined

4. We also note that human labeled difficulty classifications are very well in line with precedent in influential robotics and RL research. Firstly, Diffusion Policy (Chi et al., 2023) requires extensive human demonstrations, and RT-1 (Brohan et al., 2022) used 130k human-annotated episodes. OpenAI’s Automatic Domain Randomization for the Rubik’s‑Cube hand still begins from expert‑set parameter bounds (OpenAI et al., 2019), and Automatic Curriculum Learning surveys catalogue multiple of works that first discretize task difficulty by hand before learning to select among bins (Portelas et al., 2020). Similar to them, our study provides a simple, interpretable method that enables systematic study and can later be refined or automated once its practical benefits are understood.

5. "... the current implementation does not seem scalable. ... the training data is also labelled by humans."

1. Instead of using total ~20,000 collected images to train CNN, we conducted an additional experiment to see how lower amount of training data affects accuracy:

Most tasks achieve near-plateau performance with just 300 labeled images, requiring only ~15 minutes of annotation (3 seconds per image). The above pattern of improvement also aligns well with Table 2(a).

2. VLMs are used precisely to address the scalability issue in our paper. We only used 3 images per category in the prompt for VLM and used total 1049 training images to finetune VLMs. We conducted an additional experiment to see the impact of differing numbers of images for VLMs:

While VLM classification accuracy remains below that of CNNs, VLMs were still effective enough to show performance improvement and time savings as shown in tables 4 and 5.

3. We hypothesized that after finetuning the VLMs with enough data, they may learn generalization ability that they do not need to be finetuned with new data but are able to make meaningful predictions. We conducted an additional experiment to test generalization:

We believe with the current rate of VLM progress, its generalization ability will only improve in the future.

6. "One merit of test-time scaling in LLMs is that the performance keeps increasing when we scale up the inference budget... I'm also curious about whether such a trend exists for robotics manipulation. In Table 1, it seems that Tool Hang and Multimodal Ant are not using the maximum budget to achieve the optimal result. Does this imply that increasing the computation budget cannot further boost the performance?"

1. This is an exceptionally thoughtful observation that also highlights a crucial finding of our work. Yes, generally, we believe there is a trend where increased compute budget increases performance at least marginally. However, the trend does not seem to apply for Tool Hang and Ant where controlled noise aids intelligent exploration.
2. We show a systematic evaluation across multiple inference steps on Tool Hang, varying inference steps k ∈ {1, 4, 10, 25, 50, 75, 100} while holding solver and dynamics configurations constant.

The data shows success rates increasing from 25% to peak performance of 38.1% at 50 steps. Interestingly, applying additional computation after 50 steps decreases performance, returning to baseline levels at 100 steps. This empirical validation strongly supports our theoretical insight that excessive integration steps over-smooth trajectories, suppressing the stochastic exploration essential for "thread-the-needle" manipulation tasks.

We sincerely thank the reviewer for the constructive review.

#1 Addressing the Weaknesses

1. Exclusive Reliance on Simulation

1. We conducted an experiment with the 'real robot pushT' that has publicly available training data. We were able to train a diffusion policy and SIP with linear interpolation and velocity prediction and found a decreasing pattern of training action MSE, revealing that SIP can mimic the expert human demonstrator, which suggests it will be useful when deployed in the real world. Due to time constraints, we were unable to roll out actual robot trajectories.

Train Loss Comparison

Train Action MSE Comparison

2. Simulation-based research has historically driven significant advances in robotics and RL. Notable examples include: DQN (Mnih et al., 2015) and PPO (Schulman et al., 2017), which established foundational RL algorithms entirely through simulated environments before widespread real-world adoption. Similarly, we believe our work introduces an initial exploration of test-time compute scaling for robotic control—a concept that may serve as a starting point for broader applications in real-world systems as the field continues to develop physical task reasoning capabilities in real world.

2. Manual Design and Generalization Issues

1. We believe this is a valid concern regarding generalization: when the user wants the robot arm to complete a new task, it may require an additional category beyond the 6 categories, and retuning the table. Before we delve into the details, we would like to point out that if there is in fact a new category, this will not require retuning of the entire table but simply adding a new row with a corresponding triplet for the category.
2. We tested the generalization capability of VLM by fine-tuning it without Lift data. Without being exposed to Lift data, VLMs needed to make predictions on Lift:

With the current rate of progress for VLMs, we believe the generalization ability will only improve.

3. Moreover, we want to point out that this is precisely the reason why we introduced VLM instead of CNN as a difficulty classifier. In fact, we only used 3 images per category in the prompt before using the VLM. This means if the user wants the robot arm to behave differently during some sub-task, they only need to take three pictures per category to make that decision. We conducted an additional experiment to see the impact of differing numbers of images for VLMs:

VLMs were still effective enough to show performance improvement and time savings as shown in tables 4,5.

3. Data Annotation Bottleneck

1. In our paper, we used a total of ~20,000 collected images to train CNN. We conducted an additional experiment to see the impact of a lower amount of training data on the same validation data for CNN accuracy:

As you can see, we could achieve accuracy level that tend to plateau around 100-300 images for the above games. If we decide to use only 300 images per task, it will only take ~15 minutes to collect (an average of 3 seconds to label one image), which we believe is a reasonable time. The above pattern of improvement also aligns well with Table 2(a).

4. Lack of Explicit Safety Guarantees

1. The reviewer raised an important point about safety. Our approach provides safety guarantees through its design. As shown in Table 5, misclassifications never result in performance below min-compute levels, since any classification yields at least 1 inference step. Users can set min-compute as their baseline safety threshold for their risk tolerance level.
2. For deployment in high-stakes environments, we recommend the users to benchmark their policy across the full compute spectrum to identify the minimum inference configurations required for safe operation. This threshold becomes the min-compute threshold. Even under worst-case misclassification scenarios, the system maintains safety-critical performance levels.

#2 Addressing the Questions

Q1: "Tweedie Step" in Table 1

1. The Tweedie step refers to a denoising method that directly estimates the clean state from noisy input using Tweedie's formula for posterior mean estimation. In the stochastic interpolant framework, this is implemented as:

x̂₀ = (xₜ + σₜ² · score(xₜ)) / αₜ

Where:

• xₜ is the noisy state at time t
• σₜ is the noise scale
• αₜ is the signal scale
• score(xₜ) is the score function (∇log p(xₜ))

This can also be expressed using the noise prediction formulation as:

x̂₀ = (xₜ - σₜε̂) / αₜ

Where ε̂ is the predicted noise, related to the score by: score ≈ -ε/σₜ

2. For precision tasks like PushT and Block Push, the Tweedie step generates the final action in one computation. It also provides the minimum mean squared error estimate of the clean data. Finally, the direct estimation avoids accumulation of errors from multi-step integration. This makes it an efficient choice when high precision is needed with minimal computational cost, which aligns perfectly with our adaptive computation framework where we aim to match computational resources to task requirements.

Q2: ODE vs SDE Cost Consideration

1. The reviewer correctly identified that our cost model doesn't explicitly account for ODE/SDE differences. In practice, the computational difference is minimal that both require the same neural network forward passes. SDE adds sampling overhead (≈5% additional time). Lastly, the main cost comes from the number of integration steps. We simplified the model for clarity but acknowledge this could be made more precise in future work.

Q3: Lookup Table Design Methodology

1. The lookup table was designed through systematic grid search:

• Step counts: {1, 4, 10, 25, 50, 100, 200}
• Solvers: {Euler, Heun}
• Integration: {ODE, SDE}

We evaluated all combinations on validation trajectories for each task type, selecting configurations that optimized the success-rate/compute-time balance. This one-time offline optimization provides practical deployment guidelines.

Q4: Performance Degradation in Exploratory Tasks

1. To provide a data-driven explanation, we conducted a theoretical experiment of finding a hidden ball.

• True ball position: [5.0, 0.0] (unknown to policy)
• Ball radius: 0.5 units
• Perception noise: σ = 0.5
• Maximum attempts: 10 per episode

    ...
    def simulate_attempt(self, t, observed_hole, attempts):
        ...
        elif self.n_steps == 50:
            last_attempt = attempts[-1]
            if np.random.random() < self.adaptation_rate:
                away_direction = observed_hole - last_attempt
                away_direction /= np.linalg.norm(away_direction) + 1e-6
                return observed_hole + away_direction * self.attempt_std + \
                       np.random.normal(0, self.attempt_std * 0.5, 2)
            else:
                return last_attempt + np.random.normal(0, self.attempt_std, 2)
        else:  
            # for 100 steps
            return observed_hole + np.random.normal(0, self.attempt_std, 2)

Result:

The experiment demonstrates that controlled stochasticity enables intelligent exploration, while 100 steps over-smooth the policy, causing it to repeatedly attempt the same failed approaches.

Q5: Investigating Classifier-Policy Mismatch

1. We investigated this through our VLM experiments, which had lower accuracy than the CNN classifier, allowing us to observe the impact of classification errors. In Table 5, the performance never dropped below the min-compute baseline. This provides an important guarantee: if users can tolerate min-compute performance, they can safely use VLM-based difficulty classification, knowing this represents the worst-case performance floor. The 18% drop in the Transport task represents our most challenging scenario, yet, performance remains acceptable. We believe as VLM capabilities continue to improve, this classifier-policy gap will naturally narrow, making the approach increasingly practical while maintaining bounded performance degradation.

Q6: Generalizability of Difficulty Categories

1. Please check #1.2

#3 Addressing the Limitations

1. Please check #1.1.

2. Please check #1.2.

3. Please check #1.3.

4. Please check #1.4.