InstructFlow: Adaptive Symbolic Constraint-Guided Code Generation for Long-Horizon Planning

Thank you for your thoughtful review and encouraging assessment. We deeply appreciate your support and careful reading of our work. Below, we respond to your comments with detailed clarifications and highlight the revisions incorporated based on your suggestions.

Weakness 1:

“The symbolic inference phase of the constraint generator relies on unverified outputs. If these are incorrectly identified, the plan repair would most likely be incorrect.”

Response:

We appreciate the reviewer’s concern. While symbolic inference can in principle introduce errors, InstructFlow mitigates this via both structured reasoning and semantic validation:

• Grounded diagnostic pipeline: As detailed in Section 3.2 , symbolic constraints are extracted through a four-step diagnostic workflow: from failure entity retrieval to code-level and diagnostic analysis, rather than shallow pattern matching. This structured pipeline mitigate hallucination risk.
• Empirical Support: Ablation result from Table 2(Lines 283-284) shows a 30-40% drop without Constraint Generator; case studies (Appendix B.2.3) and a illustration of constraints discovered in Table 4(Lines 611-612) confirm the alignment of induced constraints with task semantics.

---

Weakness 2:

“Lack of real-world experiments evaluating the sim-to-real gap.”

Response:

We agree that real-robot validation is valuable. While our study is simulation-based, we explicitly model two key real-world uncertainties: sensor noise and feedback ambiguity, to evaluate robustness:

1. Perceptual noise: We inject Gaussian perturbations into object poses, $\sigma\in\{0.005,0.01,0.02\}$. Note that object size is 0.04, the max noise is up to 50% object size. As shown in Table 1, InstructFlow maintains >70% success across tasks even under perceptual noise.

2. Feedback noise: We simulate incorrect and incomplete failure feedbacks. (1) incorrect feedbacks with wrong object references, and (2) incomplete traces with missing object IDs or causes. Table 2 shows InstructFlow remains robust, with only 11–16% average drop despite noisy or missing feedback.

These results show InstructFlow operates reliably without perfect state or supervision，a key requirement for real-world deployment. We view hardware validation as important future work and believe these robustness results provide strong evidence of real-world readiness.

---

Question 1:

“Regarding the limitations of InstructFlow, what analysis has been conducted on its failure cases or the scenarios in which its performance is suboptimal? Furthermore, which component within the framework (say, the Symbolic Constraint Generator) is most frequently the point of failure?”

Response:

We thank the reviewer for raising this important point! Our analysis indicates that InstructFlow performs suboptimally in tasks with high object diversity, irregular geometries, or tight layout constraints, notably Packing-YCB and Stacking-YCB (Fig 5, Lines 636-642 from Appendix B.2.3). These scenarios challenge both constraint induction and plan refinement, due to complex spatial couplings and ambiguous success conditions.

For instance:

1. In Packing-YCB, example shows how constraints that require spatial precision (like placement feasibility) are hard to induce when object shapes are varied. This clarifies why performance degrades in this task, a failure mode tied to constraint induction under geometry noise.
2. In Stacking-YCB, example emphasizes failures from reasoning about geometric compatibility and object stability, again showing a case where symbolic reasoning and plan refinement can fail due to ambiguity in object interactions.

At the component level, the Code Generator emerges as the most common failure point. To support this, we conducted ablation experiments (Table 3, Reviewer-iwsg) by replacing GPT-4o with GPT-3.5-turbo across modules. The steepest performance drop (13%) occurs in the Code Generator, compared to 8% in the Constraint Generator and 4% in the Planner, highlighting the Code Generator’s sensitivity in long-horizon, constraint-heavy tasks.

---

Question 2:

“Have metrics such as wall-clock computation time and the average number of feedback queries been benchmarked for InstructFlow? If so, what are the quantitative improvements in planning efficiency and query complexity relative to the PROC3S baseline?”

Response:

We appreciate the reviewer’s interest in benchmarking efficiency and query complexity. While InstructFlow introduces modular agents, it achieves superior planning efficiency and comparable or improved latency relative to PRoC3S, as shown below.

1. Fewer feedback queries: As summarized in Table 4, InstructFlow reduces feedback-driven repair cycles by ~37% on average. For example, in chellenging tasks like Unstack and Pyramid, InstructFlow completes planning in just 1–3 repair steps, compared to 2–5 full regenerations in PRoC3S. This gain stems from localized plain repair and high-level symbolic guidance.

2. Lower wall-clock latency: Despite its multi-agent design, InstructFlow achieves a ~4.7% reduction in end-to-end latency (see Table 5 below). Symbolic constraints (e.g., Distance(?block_bottom) ∈ [0, 0.04]) help prune infeasible code regions, reducing LLM sampling and retry overhead.

These improvements were obtained without latency-specific tuning, suggesting that modular planning does not incur significant computational penalties. We will report full statistics in the final version and are happy to release detailed logs upon request.

---

Writing issues:

Thank you for spotting these typos. We will thoroughly proofread and fix all such issues in the final version to ensure clarity and polish.

---

Once again, We are deeply appreciative of your generous support and constructive feedback. Your review not only affirmed our direction but also inspired valuable refinements.

We would like to express our sincere respect for your insightful review! In the following, we systematically respond to your concerns with additional analysis and empirical results where appropriate.

Weakness 1:

The multi-agent, graph-guided pipeline incurs extra LLM calls and bookkeeping (planning, code generation, constraint induction), leading to higher computational overhead and latency compared to flat planning methods.

Response:

While our pipeline introduces modular components, InstructFlow achieves lower latency and fewer LLM calls compared to flat planners like PRoC3S[1]:

1. Efficiency: InstructFlow reduces feedback rounds by 37% on average (Table 1), often converging in **1–3 ** vs. **2–5 ** regenerations in PRoC3S across complex tasks.

2. Wall-clock time: As shown in Table 2 below, InstructFlow reduces latency by ~4.7%, benefiting from symbolic constraints that narrow the search space and reduce code sampling needs.

[1] Trust the proc3s: Solving long-horizon robotics problems with llms and constraint satisfaction.(CoRL'2024)

---

Weakness 2:

Setting up the domain requires defining action templates, object/region vocabularies, and API mappings-a nontrivial engineering effort that may slow adoption in new environments.

Response:

InstructFlow follows the same setup convention as PRoC3S, a one-time domain-level definition reused across tasks. Action templates (e.g., pick/place) are generic and map easily to common robotic APIs, making the system modular and portable to new domains with minimal overhead.

---

Weakness 3:

“In scenarios with simple, short-horizon tasks, the marginal benefit of structured repair can be limited, suggesting diminishing returns when failures are rare.”

Response:

We would like to clarify that InstructFlow is designed to adapt to task complexity, and does not incur unnecessary overhead in simple, short-horizon tasks. Specifically:

1. Failure-driven activation: The structured repair mechanism (constraint generator) is only triggered when failures occur. In simple or successful cases, it remains dormant, and behaves like a flat planner, incurring no extra overhead and avoiding diminishing returns.s like a flat planner with negligible overhead.
2. Additional benefits: Even when no failure occurs, InstructFlow’s hierarchical instruction graph provides clear semantic structure to the plan, improving interpretability (what subgoals are being pursued), and debuggability (where and why a plan might go wrong if it ever does). This structured representation facilitates downstream analysis, monitoring, and integration with safety checks，even in short-horizon tasks. In summary, InstructFlow avoids overhead when unnecessary and provides benefits when needed, ensuring that its structural design remains practical across both simple and complex tasks. We will clarify this design behavior more clearly in the revised manuscript.

---

Question 1:

“How sensitive is InstructFlow's performance to the clarity and structure of the user-written instruction scaffold (e.g., decomposition prompts, action templates)? Have you evaluated how variations or errors in those instructions impact success rates and repair effectiveness?”

Response:

To clarify, InstructFlow does not require users to write any instruction scaffold, decomposition prompt, or action template. The “instruction scaffold” in our setting refers to a few-shot prompt provided to the Planner Agent, which contains domain-level decomposition examples. These are fixed across evaluation and selected to be representative of the domain without overlapping with evaluation tasks."

In response to your valuable suggestion, we conduct an ablation study in which the few-shot prompt is changed from same domain tasks to out-of-domain examples (e.g., using examples from Arrange YCB to guide Arrange Block). As shown in Table 3, performance drops moderately but remains competitive, indicating that InstructFlow is reasonably robust to prompt variation.

---

Question 2:

“Constraint Generator assumes clean, structured failure traces (e.g. precise collision logs), but real‐world feedback is often noisy or incomplete, how does InstructFlow handle ambiguous or incorrect feedback data?”

Response:

To directly address this concern, we conducted two sets of robustness experiments:

1. Incorrect Feedback： the structured trace includes deliberately incorrect object references (e.g., reporting a collision with object B when object A was responsible);
2. Incomplete Feedback： key information such as object IDs or failure causes is removed, simulating partial or vague real-world signals.

Results (Table 4) show minimal degradation, with InstructFlow maintaining near-baseline performance. In addition, InstructFlow already handles unstructured semantic feedback via a VLM-based goal checker (Lines 544-566, Appendix B.1.2), enabling symbolic constraint induction from unstructed natural language feedback, making the system robust to real-world ambiguity.

---

Writing issues:

Thank you for carefully pointing out the writing issues. We appreciate your attention to detail and will carefully proofread the full manuscript to ensure clarity and correctness in the final version.

---

We sincerely appreciate the thoughtful and concrete suggestions you raised. Your comments have prompted us to examine several important aspects of the method more carefully, and we believe they have meaningfully strengthened the revised manuscript.

---

We sincerely thank you for your careful review and thoughtful understanding of our work. Below, we provide detailed responses to your comments and describe the corresponding revisions made.

Weakness 1:

“The rationale behind structuring tasks as a symbolic instruction graph is not well-justified. It’s unclear why this representation is fundamentally superior to flat sequence planning or dynamic reactive strategies. The benefit seems incremental.”

Question 1:

“What is the concrete benefit of using an instruction graph over simply sequencing high-level actions with reactive refinement?”

Response:

Unlike flat or reactive planners, our symbolic instruction graph supports structured reasoning and localized repair, which are crucial for long-horizon, constraint-sensitive tasks.

1. Flat plans lack structural insight: Flat/reactive methods re-sample or regenerate plans without understanding why failures occur. InstructFlow induces symbolic constraints and injects reasoning nodes (e.g., spatial analysis, object selection) to refine only the affected subgoals, enabling efficient, interpretable correction. As formalized in Eq. (3), $\text{instr}^{(t)} = \text{Encode}(\{ v^{\text{reason}}\_{\mathcal{T}\_j} \}_{j=1}^{|\mathcal{V}\_{\text{reason}}^{(t)}|},\ v\_{\text{plan}}^{(t)})$, and $\text{code}^{(t)} = \text{LLM}(\text{instr}^{(t)})$ , each code generation prompt is composed by integrating the outputs of upstream reasoning nodes in the instruction graph. This structured composition enables rich, context-aware prompting that flat sequences cannot support, leading to more grounded and accurate code synthesis.

2. The benefit is significant: As shown in Table 2 (lines 299–300), removing the instructflow planner leads to 30–50% drops in complex tasks like Pyramid and Packing, indicating fundamentally different planning behavior.

---

Weakness 2:

“The paper primarily demonstrates simple, linear subgoal sequences. There’s no evidence that the instruction graph needs to go beyond a list of ordered subtasks.”

Question 2:

“Can you provide compelling examples where the graph becomes non-linear (e.g., has branches, merges, loops), and this structure is essential for solving the task?”

Response:

We agree that the final execution plan is often sequential, but the instruction graph is not an execution trace, it is a symbolic reasoning graph that encodes how subgoals are derived, refined, and repaired.

Reasoning nodes often branch to multiple subgoals. When failures occur, we inject new reasoning nodes that modify only the affected subgraph, enabling structural repair, which flat lists cannot express.

---

Weakness 3:

“It is unclear how it would generalize to perceptual uncertainty or sensor noise, common in real-world settings.”

Response:

To address this concern, we added a controlled experiment simulating perceptual uncertainty. We inject Gaussian perturbations into object poses before symbolic abstraction. We test InstructFlow on Arrange and Arrange-YCB tasks across noise levels $σ\in\{0.005, 0.01, 0.02\}$, given that the average block_size is 0.04 (i.e., σ = 0.02 simulates severe, 50% object size noise). This setting reflects real-world errors and allows us to investigate robustness under increasing uncertainty.

---

Weakness 4:

“No attempt is made to show robustness in hardware or even perception-grounded simulation with visual inputs.”

Response:

In robotic manipulation, perception primarily serves to estimate object-level states (e.g., poses, identities), which downstream planners operate on. Our simulation provides these directly, allowing us to focus on planning.

That said, we include a perception-grounded experiment in Appendix B.2.1 (Lines 574-575, Table 3), where symbolic inputs are derived from visual observations via a VLM. This confirms that InstructFlow can operate with visual grounding and improves performance.

Further, as noted in response to Weakness 3, we simulate noisy perception via pose perturbations (Table 1), and InstructFlow remains robust, supporting its potential for real-world deployment.

---

Weakness 5:

“While symbolic constraint induction is novel, the system appears to rely on a hand-designed set of predicates, limiting generality and scalability.”

Response:

InstructFlow does not rely on a fixed or hand-designed predicate set. Instead, symbolic constraints are dynamically induced from execution feedback via spatial analysis, physical diagnostics, and semantic reasoning.

As detailed in Section 3.2, the Constraint Generator constructs symbolic predicates directly from failure contex, without referencing a predefined library. This process is formalized in Equation (4): $\mathcal{C}(\mathcal{E}, \mathcal{R}, \mathcal{F}, \mathcal{B}) = \{ R_i(e_{a_i}, e_{b_i}) \} \cup \{ f_j(\Theta_j) \oplus \tau_j \}$

Here, the constraint set $\mathcal{C}$ is constructed from:

1. Relational constraints $R_i(e_{a_i}, e_{b_i})$, which capture spatial and logical relations between entities ;
2. Physical constraints $f_j(\Theta_j) \oplus \tau_j$, which encode quantitative properties like distance, stability, or alignment thresholds.

Appendix B.2.2 (Lines 611-612, Table 4) further illustrates diverse predicates automatically generated across tasks.

---

Question 3:

“Would a flat plan with dynamic constraint-aware sampling perform similarly, especially if paired with a constraint checker?”

Response:

While flat plans with constraint-aware sampling (e.g., PRoC3S) improve robustness over naïve outputs, they lack the ability to explain failures or perform targeted plan repair. InstructFlow introduces two key enhancements:

1. Symbolic diagnosis: Failures trigger constraint induction (e.g., ClearOf, Aligned) to identify root causes.
2. Graph-guided repair: Only affected subgoals are updated via reasoning nodes, enabling localized and interpretable refinement.

As shown in abaltion study(lines 299–300, Table 2), removing either module leads to 30–50% drops in success across complex tasks, confirming that symbolic abstraction adds essential value beyond flat planning.

---

Question 4:

“If you are using the same test suites in PRoC3S, why are the reported numbers different from the original paper?”

Response:

While we use the same environments and codebase as PRoC3S, its original evaluation protocol considers an execution successful if no simulator-level constraint is violated, even when the task goal is not actually achieved (e.g., misaligned stacks or wrong object placements).

This issue is documented in Appendix B.1.2 Fixing Protocol Limitations in PRoC3S Evaluation, where we clearly provide example cases when PRoC3S consideres success (pass constraints) but actually fail (task goal) in Figure 6 (Lines 560-561). Same issue has been discussed in prior papers such as AHA [2].

To address this gap, we introduce a VLM-based semantic check that verifies whether the final scene satisfies the natural language goal. Figure 7 (Lines 565-566) showcase our VLM-based success detector implementation. This evaluation protocol applies uniformly to all methods.

As a result, our reported success rates may be lower than PRoC3S's original numbers, not due to task changes, but due to stricter, goal-consistent evaluation for fairer comparison.

[1] Trust the proc3s: Solving long-horizon robotics problems with llms and constraint satisfaction. (CoRL'2024).

[2] AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation (ICLR'2025).

---

Question 5:

“Why is the generated constraint reusable? The constraint is specific to a particular task setting/block positioning.”

Response:

By “reusable,” we refer to the symbolic form of constraints (e.g., ClearOf(gripper, ?object), Below(?a, ?b)), which capture task-agnostic physical relations rather than scene-specific instances.

As shown in Appendix Table 4, predicates like ClearOf, PlacementFeasible, and Aligned recur across tasks such as Unstack, Packing, Stacking, and Pyramid, reflecting shared structural demands like collision avoidance, stable placement, and alignment.

---

Question 6:

“How would the proposed system generalize this to real-world tasks that cannot provide systematic feedback in textual format?”

Response:

Our method does not rely on text format per se, but on identifying task-level violations through perceptual or geometric cues. The Constraint Generator uses available feedback to locate relevant code segments, then performs semantic reasoning over object states and task goals to infer symbolic constraints.

As described in Appendix B.1.2 , our evaluation already incorporates VLM-based semantic verification (GPT-4o) that checks goal satisfaction from images. This demonstrates that our system can operate on unstructed perceptual feedback.

---

Thank you again for your detailed and constructive feedback. We greatly appreciate the opportunity to address these points and improve the clarity and robustness of our work.

---

We are truly grateful for your generous evaluation and insightful feedback. We have reviewed your feedback thoroughly and respond below with clarifications and revisions made accordingly.

Weakness 1:

“All results are based on simulations; however, nowadays in robotics, it is standard to have also real-world experiments with actual robots.”

Response:

We agree that real-robot validation is valuable. While our study is simulation-based, we explicitly model two key real-world uncertainties: sensor noise and feedback ambiguity, to evaluate robustness:

1. Perceptual noise: We inject Gaussian perturbations into object poses before symbolic abstraction to simulate sensor inaccuracies. As shown in Table 1, InstructFlow maintains strong performance even under severe noise (σ = 0.02, 50% of object size), with success rates remaining above 70% on most tasks.

2. Feedback-layer noise: We evaluate robustness under incorrect and incomplete failure traces, simulating real-world ambiguity. (1) incorrect feedbacks with wrong object references, and (2) incomplete feedbacks with missing object IDs or causes. Across 10 tasks (Table 2 below), InstructFlow consistently achieves near-baseline performance despite missing or wrong feedback information.

These results show InstructFlow operates reliably without perfect state or supervision，a key requirement for real-world deployment. We view hardware validation as important future work and believe these robustness results provide strong evidence of real-world readiness.

---

Weakness 2:

“Limited kinds of tasks: table top manipulation. ”

Question 1:

“Would InstructFlow generalize across robotic domains beyond tabletop manipulation, e.g., mobile navigation, assembly?”

Response:

We appreciate the reviewer’s question on generality. While our experiments focus on tabletop manipulation for benchmarking consistency (e.g., PRoC3S), InstructFlow is domain-agnostic by design.

InstructFlow operates over textual instructions and symbolic predicates, enabling generalization beyond tabletop setups. Its key components, symbolic instruction graph planning, modular node composition, and failure-driven constraint repair, reason over abstract subgoals without relying on domain-specific assumptions (e.g., fixed workplace or rigid objects).

1. In mobile navigation, pfailures like deviation or blocked paths naturally yield predicates such as PathBlocked or DeviationTooLarge, enabling symbolic route repair.
2. In assembly domains, misalignment or insertion failure can trigger constraints like AlignedForInsertion or GripTorqueInsufficient, driving targeted graph refinement.

These examples illustrate how InstructFlow’s planning-repair cycle generalizes to other robotic domains, and we view broader deployment (e.g., mobile or deformable tasks) as promising future work.

---

Weakness 3:

“No Limitations section (even though some limitations are discussed in the text).”

Response:

Thank you for pointing this out. While we discussed key limitations throughout the paper, we agree that a dedicated section would improve clarity.

• As noted in Lines 365–367, InstructFlow currently assumes structured symbolic inputs and limited sensory grounding. We are actively exploring multimodal extensions with visual grounding and constraint induction from unstructured feedback.
• As discussed in Appendix B.2.3, performance degrades in scenarios with irregular object geometries and tight spatial coupling (e.g., Packing-YCB, Stacking-YCB), due to the increased difficulty of reliable constraint induction.

We will add a dedicated Limitations section in the final version to consolidate these observations.

---

Question 2:

“How much does the method depend on the underlying language model (e.g., GPT-4 vs. smaller models)?”

Response:

Thank you for raising this point. We conducted additional ablation experiments usign GPT-3.5-turbo in place of GPT-4o across all and individual modules.

1. Code generator is model-sensitive: Substituting GPT-4o with GPT-3.5 in this module results in the largest performance drop (~13%), highlighting its reliance on strong LLM reasoning for reliable code synthesis, especially in long-horizon, constraint-sensitive tasks.
2. Planner and Constraint Generator are robust: These modules maintain competitive performance with GPT-3.5 (4–8% drop), demonstrating that symbolic abstraction and structured reasoning mitigate dependence on large-scale models.

These findings suggest that InstructFlow’s core effectiveness stems from its agentic design, not LLM size. It remains effective with smaller models in key components, making it practical for resource-limited deployments.

---

Question 3:

“Can the symbolic constraint vocabulary be learned or adapted online instead of being manually defined?”

Response:

The constraint vocabulary in InstructFlow is not manually defined nor fixed. Instead, symbolic predicates are dynamically induced online from failure traces using spatial analysis, physical diagnostics, and semantic reasoning (Section 3.2).

Formally (Lines 266, Eq. 4), $\mathcal{C}(\mathcal{E}, \mathcal{R}, \mathcal{F}, \mathcal{B}) = \{ R_i(e_{a_i}, e_{b_i}) \} \cup \{ f_j(\Theta_j) \oplus \tau_j \}$, the Constraint Generator constructs task-specific constraints $\mathcal{C}$ from:

1. Relational constraints $R_i(e_{a_i}, e_{b_i})$ over entity pairs (e.g., reachability, proximity)
2. Physical constraints $f_j(\Theta_j) \oplus \tau_j$ capturing measurable task variables (e.g., alignment thresholds, support stability).

These predicates are constructed per execution context without reference to any pre-defined library. As illustrated in Appendix B.2.2 (Lines 611-612, Table 4), InstructFlow generates diverse symbolic constraints across tasks, supporting adaptability and generalization.

We view learning a more generalized or transferable constraint space across tasks as an exciting direction for future work.

---

We sincerely thank you for your encouraging evaluation and thoughtful comments. Your recognition of our work’s contributions is highly motivating, and your suggestions have helped us further strengthen the clarity and rigor of the manuscript.

---