Evolving Symbolic 3D Visual Grounder with Weakly Supervised Reflection

Minor issues:

1. The legend of Figure 1 is too small.

2. The caption of Figure 2 seems not aligned with the diagram.

3. Replace Figure 6 with a high-resolution version.

4. Figure 1 is not referenced.

Major issues:

1. The introduction is very poorly written. (1) the introduction of the significance of the researched problem is missing. (2) The agent-based approaches have not been discussed, which is one of branch of previous work.

2. LLMs are not unnecessarily to every task. It is not advisable to use LLMs for the sake of using LLMs. 50.7% accuracy doesn't looks a good performance. I suggest two possible ways here: (1) You can do a comprehensive analysis to investigate if LLMs are suitable to this task and provide some insights. Or (2) you compare the proposed the method with the state-of-the-art methods to show the effectiveness or efficiency.

3. This work lacks sufficient self-containment and omits essential background information, leading to a structure that compromises readability. Specifically, the discussion of existing work is insufficient, which weakens the persuasiveness of the contribution. For instance, Figure 2 is intended to help readers grasp the narrative, but its impact is limited due to a lack of necessary explanation, making it difficult to understand.

Several weaknesses make this paper further the acceptance threshold:

1. The writing, in particular, seems hasty. A systematic proofread is eagerly anticipated for the introduction and methodology part. The content presented in Fig. 1-3 are not well explained in the corresponding paragraph, making it very hard to correlate the figure content with the text description. In particular, in Fig. 2, the authors attempt to make it clear the difference across different methodologies including their own. However, the presentation of their own method (right most) is too messy to tell the novelty of this paper. The following points are some questions caused by the writing, which confuses me a lot.

2. What is the essential advantage of PLM-based grouding method such as EASE? The performance results in this paper is only comparable with SOTA, while the efficiency proof is missing.

3. Further, what is the motivation of using PLM for grounding, can the authors provide any in-depth elaboration?

4. There might be a overclaim in this paper. The authors have claimed that EASE relieve the human expertise for solving grounding task. However, the prompt design presented in Appendix involves nonnegligible expert-level knowledge to make the framework operatable. For example, the helpful tips in line 718-730. An actionable ablation could be removing such hints and observe the performance.

5. Minor: line 087-089, these two lines seems repetitive.

• Limitations of Interpretability: Although EASE offers a certain level of interpretability, the internal workings of LLMs remain a black box, which may limit the overall interpretability of the framework.
• Strong engineering nature：A significant part of the article is exploring how to make LLMs generate better Python code, which may be highly dependent on the prompt and difficult to evaluate from an academic research perspective.

• I am admittedly not an expert of this domain, and would like to hear my colleague reviewers' opinions.

• Over-reliance on Large Language Models (LLMs): This work appears to lean heavily on LLMs for various stages, from description parsing to relation extraction. This dependency raises questions about the originality of the technical contributions, as it is unclear what innovations are introduced beyond the application of existing LLM-based approaches.

• Performance Concerns Relative to Transcrib3D: The work does not convincingly address why its performance is weaker than Transcrib3D, which also leverages LLMs for solving the ReferIt3D task. A comparative analysis of EASE and Transcrib3D is necessary to clarify their differences in approach and provide insight into why EASE does not achieve comparable results.

• Interpretability Claims: The authors assert that EASE offers enhanced interpretability through program-based relation representation, yet the interpretability is inherently limited by the LLMs it depends on. This approach does not address the challenge of understanding or explaining the inner workings of the LLMs themselves, calling into question the claim of improved interpretability.

• Minor Errors: There are several minor typographical errors in the text, such as "we constraints..." on line 528, which should be corrected for clarity and readability.

• While the symbolic framework is interpretable, handling complex relations or compound instructions may pose limitations without significant modifications.

• Although human annotations are not required, the framework still depends heavily on the capabilities of LLMs for generating initial code, which may introduce variability in output quality.

• The system represents scenes as a list of 3D bounding boxes, omitting detailed attributes like object orientation, texture, or color, which could be vital for more nuanced tasks.

• The feedback and code refinement process may not fully resolve deeper logical errors or misinterpretations made by the LLM in code generation.