Dataflow-Guided Neuro-Symbolic Language Models for Type Inference

We thank the reviewer for the positive feedback.

---

Q1: How much more expensive is Nester inference in terms of time and energy over the naive CL and L3 models?

A1: We conduct additional experiments to evaluate Nester's computational cost in terms of inference time and energy consumption using an NVIDIA RTX 4090 GPU. The results are as follows:

Nester incurs higher computational costs, primarily due to multiple invocations of large LMs during inference. However, when multi-step reasoning is removed, the additional overhead is negligible, making the inference time and energy consumption comparable to those of the naive models. In this case, Nester will only locate the relevant code lines (e.g., identifier definitions) from the source code according to the high-level program. It then leverages LLMs along with predefined rules in a single step to infer types without further iteration, still yielding practical results for type inference. This provides an optional solution with fast type inference time.

---

Q2: Is Nester extensible to compiled languages, or are interpreted languages the only candidate for type inference with Nester?

A2: This is a good point. Nester can be adapted to lower-level compiled languages like bytecode and assembly instructions. This will require adjusting the high-level program prompts for LMs and developing language-specific rules for inference, taking into account the existing type systems of the specific languages. We will add a discussion.

---

Q3: Why not evaluate on ManyTypes4TypeScript as well? It seems that Nester should at least be extensible to JS/TS.

A3: Our current implementation does not support certain JS/TS features due to the limitations of the underlying tools used by Nester. These include JavaScript's dynamic scoping (e.g., eval), prototype-based inheritance, event-driven asynchrony, and TypeScript's hybrid semantics, such as gradual typing and any escape hatches—none of which are present in Python. We will clarify this and discuss how to extend Nester to other languages beyond Python.

---

Q1: The key differences between Nester and TypeGen.

A1: Please see Q2 of Reviewer Q6Mu for a discussion.

---

Q2: Could the author please provide a detailed formalization of high-level programs?

A2: This is provided in Appendix D.1. We will better highlight.

---

Q3: Comparison between the introduced coarse-grained dataflow graph and the original one.

A3: Thanks. This information is provided in Appendix E.2 in the context of CGDG (Coarse-Grained Dataflow Graph). Specifically, for an identifier $x_v$, we represent its three hops using three types of nodes: Def($x_v$), User($x_v$), and Usee($x_v$). In essence, our approach differentiates nodes based on their hop distance from $x_v$, categorizing them into three distinct types. Compared to traditional Concrete Syntax Tree (CST) and Abstract Syntax Tree (AST), CGDG constraints each identifier to at most three hops, resulting in a more concise and structured graph representation. We will explain this approach in more detail.

---

Q4: How does Nester compare to traditional symbolic execution approaches?

A4: Nester is a static analysis framework for type inference and does not rely on symbolic execution—no symbolic execution traces are generated. Unlike traditional symbolic execution, which explores possible input spaces to produce traces, Nester operates purely on static source code information. We will clarify.

---

Q5: What is the coverage of the rules in Nester?

A5: The coverage of rules is reported in Table 3. Our rules support primitive types (int, float, bool, str, bytes) and container types (list, tuple, dict, set), covering the vast majority of type inference scenarios encountered in real-world code.

---

Q1: How does the proposed method perform for complex parametric types and user-defined types?

A1: In response to your concern, we will provide Nester's performance on complex parametric types (depth = 0, e.g., list; depth = 1, e.g., list[str]; depth ≥ 2, e.g., Dict[str, List[int]]) and user-defined types, as follows.

From this table, we can observe that Nester outperforms Typegen for user-defined types in terms of exact match and match to parametric metrics. Furthermore, Nester performs better with complex parametric types in terms of match to parametric metrics. Although Nester does not achieve higher exact match accuracy for depth ≥ 2 and depth=1, this can be attributed to Typegen's use of specially designed chain-of-thought operations for recursive type structures, which Nester has not yet incorporated. We believe that integrating such techniques into Nester could further enhance its performance, and we will explain how this can be done later.

---

Q2: The key differences between Nester and TypeGen (raised in the Claims and Evidence part).

A2: Nester differs from TypeGen in its approach to type inference. Nester integrates static data and control flow analysis with the capability of LLMs. In contrast, TypeGen relies entirely on LLM-generated reasoning, treating code analysis as solely textual processing progress—a less reliable approach and more prone to hallucination. Our design also allows Nester to provide an intuitive, high-level program view, offering greater interpretability during type inference than TypeGen.

---

We thank the reviewer for the positive feedback.

---

Q1: What is the fundamental reason why Nester couldn't just be run with a larger LLM?

A1: Nester can work with LLMs of various sizes. In this work, we use a modest-size LLM as we target scenarios where users prefer running models locally—on a developer PC or cluster—to avoid sending data to the cloud due to security and privacy concerns. In the final version, we will demonstrate how Nester can be integrated with a larger LLM (70B).

---

Q2: Does Nester only work with individual functions and local context, or can it draw from contexts outside the scope of the function?

A2: Our current implementation performs type inference on individual functions and local context. This can be easily extended by - for example, inlining the callee functions. We will provide a discussion.

---

Q3: Other statistics about the functions input to Nester, such as the number of lines of code in each input.

A3: Agree. The distribution of lines of code in our dataset is as follows.

We will provide this information.

---

Q4: Have you thought about evaluating Nester with small language models like Microsoft's Phi family?

A4: Thanks for your comments. Based on your suggestion, we simply evaluate Nester on 100 instances from ManyTypes4Py using top-1 predictions with Phi-3.5-mini-instruct, due to the limited time slot. The results are as follows:

From this table, we can see that Nester outperforms the naive method in overall accuracy. However, it does not surpass the naive method in argument (Arg.) and return(Ret.) type predictions. This is likely due to the small number of return and argument types in the 100-instance dataset—if a single inference fails, it can result in a significant deviation. We leave further investigation of Nester on Microsoft's Phi family for future work.

---

Q5: In Figure 3, for line 6 in the high level program, why is only T2 and T3 being combined?

A5: In the paper, we simplified the presentation by combining T2 and T3 in Figure 3. In actual execution, however, the interpreter processes T1, T2, and T3 separately and checks whether the returned variable is empty. For this example, the runtime check confirms that T1 is unreachable. We will clarify.

---