TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories

Thanks for your valuable feedback and suggestions. We would like to address your remaining concerns with the following responses. We will improve the manuscript accordingly to address them.

LLMs are evaluated file by file. Can the authors discuss the potential limitations of using a single file for type inference and possible future directions in improving LLM's type inference via giving more constructive contexts?

Thanks for the suggestion. We did a preliminary case study and found that a common cause of type inconsistency comes from missing context (defined in other files or imported from third-party libraries). The patterns include but not limited to:

1. Missing base class

class A(B):
    def func(vars):
        super().func(vars)

2. Missing function information

def func(vars):
    return another_func(vars)

3. Missing decorator information

@some_decorator
def func(vars):
    …

The most straightforward way is to provide the whole repository as the context, but limited by LLM's context window and might hurt TypeSim as shown in Table 4.

An alternative way is to copy-paste the necessary context following the import statements but requires engineering efforts. We tried to provide such context manually for the examples we found, and the inconsistency issues were resolved after the context was provided.

Another potential method could be iteratively refining the stub files while using all stub files (and the implementation of the current file) as context. This would require much less context window since the stub files are relatively small.

On the lack of comparison with static type inference tools. From my knowledge, there are other tools than Mypy such as Pyre developed by Meta are doing Python's type analysis. While I am not suggesting additional experiments, could the authors discuss the limitations of relying solely on Mypy as the baseline? How could alternative static analysis tools influence the evaluation results?

Thanks for bringing up this point. While Mypy is imperfect and could produce false positives or incorrect errors, it is still a popular choice for software engineering best practices to improve code quality. Evaluation results may be different when switching to another static type inference tool like Pyre, but since the majority of static checks should be similar, the change in the evaluation results should be small. TypeCheck metric could switch to another static type inference tool if there is a better and well-adopted one.

TYPESIM only relies on the similarity of syntax features. While TYPESIM effectively relaxes exact type matching, it primarily relies on syntax-level features, such as attribute comparisons. Have you considered incorporating other features related to code semantics? For example, the similarity between the control flow graphs of two functions.

Thanks for your question. If our understanding is correct, you are suggesting not only comparing whether two types have the same functions, but also examining the code semantics, like how similar are their implementations for the same function?

We think it is an interesting direction for future work (especially for complex user-defined types), but current TypeSim already reasonably captures the similarity between types with convenient and robust calculations.

Above is a list of type pairs and their TypeSim score. It can be seen that the TypeSim score reflects the compatibility of two types, which roughly matches human's impression on the type similarity.

Thanks for your valuable feedback and suggestions. We would like to address your remaining concerns with the following responses. We will improve the manuscript accordingly to address them.

Have you considered alternative methods for assessing type/argument similarity? (eg, embedding-based approaches?)

Our attribute-based method (L198) can also be viewed as an embedding-based approach. For example, if we regard each attribute as a column (1 means has and 0 means not), then each type gets an embedding representation, then TypeSim is naturally defined as the Jaccard Similarity over such a representation.

For example:

Such embedding representation can naturally work for user-defined types and the similarity is based on the functionality of types rather than the semantic meaning of the name of the types.

It would be helpful to provide error analysis/insights into how/why LLMs struggle with code consistency. This can help to inform mitigation efforts that do not require passing the whole repo in as context.

Thanks for the suggestion. We did a preliminary case study and found that a common cause of type inconsistency comes from missing context (defined in other files or imported from third-party libraries). The patterns include but not limited to:

1. Missing base class

class A(B):
    def func(vars):
        super().func(vars)

2. Missing function information

def func(vars):
    return another_func(vars)

3. Missing decorator information

@some_decorator
def func(vars):
    …

The most straightforward way is to provide the whole repository as the context, but limited by LLM's context window and might hurt TypeSim as shown in Table 4.

An alternative way is to copy-paste the necessary context following the import statements but requires engineering efforts. We tried to provide such context manually for the examples we found, and the inconsistency issues were resolved after the context was provided.

Another potential method could be iteratively refining the stub files while using all stub files (and the implementation of the current file) as context. This would require much less context window since the stub files are relatively small.

It might be helpful to provide technical definitions for consistency and coherence when these terms are used early in the paper.

If possible, it would be helpful to place algorithm blocks after the text that describes them, rather than before.

Thanks for both suggestions. We will improve the manuscript accordingly.

Thanks for your valuable feedback and suggestions. We would like to clarify the metrics design and address your remaining concerns with the following responses. We will improve the manuscript accordingly to address them.

The motivation behind this work is worth discussing. … Why do we need evaluation metrics that are specifically designed for LLMs?

[Q1] First, we want to clarify that our metrics are not only designed for LLMs but all predictions made by any type inference method (including the program-based ones) can be evaluated using these metrics. These metrics are calculated over a set of type predictions.

For TypeSim, as stated in the Introduction (L38), exact matching fails to capture important semantic relationships between types.

[Q3] For match up to parametric type, it is a special case of our TypeSim metric as formulated in L187:

$S(T, T')=\alpha~s(root, root')+(1-\alpha)S_{list}(args(T),args(T'))$

where in TypeSim $\alpha=0.5$ and $s$ is Jaccard on attributes, and in match up to parametric type $\alpha=1$ and $s$ is exact matching (equivalent to taking floor operation on the result of Jaccard Index). Therefore, TypeSim is a more general similarity metric than exact match and match up to parametric type.

For example, for a target type list[str] and the prediction is list, exact matching gives a score of 0 would be too harsh, and match up to parametric type gives a score of 1 is too benign. TypeSim gives a score of $\alpha=0.5$ reflects the similarity between two types.

[Q2] The main goal of the TypeSim metric is not limited to detecting the types that could be used interchangeably (which are illustrative examples) but also providing a better measurement for the semantic similarity between complex types by considering their functional attributes (Section 4.1.1 L209-219) and the structure (Section 4.1.2) within the types.

As shown in the experiments (Section 6.3.1), TypeSim provides a more nuanced semantic-based evaluation compared with exact matching, especially for more nested types (e.g., depth > 3).

Above is a list of similar type pairs. It can be seen that the TypeSim score reasonably exhibits the similarity between the two types.

For TypeCheck, though Mypy is not a perfect type checker and could produce false positives or incorrect error reports, it is currently a well-adopted best practice for software engineering. Repositories passing Mypy check (or having fewer Mypy errors) tend to have fewer bugs in practice, so we introduce this as a good starting point for evaluating type inference results. We will discuss this limitation and leave it as future work to find a better checking mechanism.

We hope these clarifications resolve your questions. Please let us know if you have further questions.

The evaluation lacks comparisons with more diverse models such as GPT-4, CodeLlama, StarCoder, CodeT5, and its variants.

We did not evaluate these LLMs since they are already outdated models compared with the SOTA LLMs we tested (e.g., GPT-4o, Claude-3.5-sonnet). Moreover, SOTA general-purpose models (e.g., GPT-4o) have already surpassed code-focused models in coding tasks. Nonetheless, we tested CodeLlama on the test sets as suggested, which gives similar results as Llama-3-8B:

*: only count on private-gpt

Other questions,

[Q4] Is it reasonable to evaluate the type inference capabilities of LLMs using such simple, naive calculations? Are there more suitable evaluation methods that better align with the way LLMs actually reason about code?

We’d like to clarify that our proposed metrics — TypeSim and TypeCheck — are not designed to mimic or replicate how LLMs internally reason about code or types. Instead, they are metrics measuring the semantic quality and repository-wide consistency of predicted types (regardless of how those predictions are produced, by LLMs or symbolic tools). TypeSim provides an interpretable view of type prediction quality, especially when predictions are partially correct. Its formulation using Jaccard similarity and structural decomposition is a pragmatic and interpretable approach that aligns with how types are composed and used in practice.

I could not find any links to the open-source code or dataset in this paper.

We will open-source the code and dataset when published.

Thanks for your valuable feedback and suggestions. We would like to address your remaining concerns with the following responses. We will improve the manuscript accordingly to address them.

Need to Justify Why We Need LLMs to Do Type Inference

As shown in previous work [1], LLMs already showed better type inference results compared with symbolic tools such as Pyright and Jedi. Moreover, code-completion tools like Github Copilot and Cursor are making progressively better type hints completions in Python. These motivated us to mainly focus on using LLMs for type inference.

the paper should more clearly articulate what specific limitations of symbolic approaches LLMs are intended to address

Thanks for the suggestion. We use Jedi to illustrate the limitation of symbolic approaches.

The original code snipet in the black repo is:

@dataclass
class BracketTracker:
    """Keeps track of brackets on a line."""

    depth: int = 0
    bracket_match: dict[tuple[Depth, NodeType], Leaf] = field(default_factory=dict)
    delimiters: dict[LeafID, Priority] = field(default_factory=dict)
    previous: Optional[Leaf] = None
    _for_loop_depths: list[int] = field(default_factory=list)
    _lambda_argument_depths: list[int] = field(default_factory=list)
    invisible: list[Leaf] = field(default_factory=list)

The predicted types by jedi are:

@dataclass
class BracketTracker:
    depth: int
    bracket_match: dict
    delimiters: dict
    previous: None
    _for_loop_depths: list
    _lambda_argument_depths: list
    invisible: list

The TypeSim score on depth is 1 and 0.5 for others. The overall score for this file is 0.6645.

the predicted types by claude-3.5.sonnet.

class BracketTracker:
    depth: int
    bracket_match: Dict[tuple[int, int], Leaf]
    delimiters: Dict[int, int]
    previous: Optional[Leaf]
    _for_loop_depths: List[int]
    _lambda_argument_depths: List[int]
    invisible: List[Leaf]

The TypeSim scores for these variables are all 1.0 and the overall score of the file is 0.8906.

As shown in the example above, the type predicted by Jedi lost fine-grained information, while Claude recovered most of that with a much higher similarity compared with the original codes.

Usefulness of Applying TypyBench as a Mainstream Evaluation Benchmark

what general model capabilities type inference serves as a proxy for

Thanks for the suggestion. TypyBench mainly introduces the following challenges (see Table 1 for concrete numbers):

1. Code understanding in long input context length (repo-level, could be >1M tokens)
2. Long completion length (each function argument and return value need to be typed)
3. Consistency among lots of outputs (i.e., cases, the predicted types)

As shown in Figure 4 and Table 2, the relative performance of tested LLMs (claude-3.5, gpt-4o, grok2, and deepseek-v3 were top-tier at that time) generally matches the relative performance on other coding tasks like the code completion in LiveCodeBench (claude-3.5: 32, gpt-4o: 30, gpt-4o-mini: 27.7, date range 8/1/2024 - 2/1/2025), indicating it is a meaningful diagnostic benchmark that pressure tests the code understanding ability of long-context LLMs.

More Illustration Regarding the New Metrics

it’s not entirely clear how well it aligns with human judgment of correctness or practical utility in real-world development workflows.

Above is a list of type pairs that are similar but not exactly the same. It can be seen that the TypeSim score reasonably exhibits the similarity between the two types.

it would be helpful to better justify why this is a comprehensive proxy for type quality.

Mypy is a widely adopted best practice in software engineering to improve code quality and detect potential bugs with static checks. For many well-maintained Python open-source repositories, the number of Mypy errors should be almost 0. Therefore, the number of Mypy errors reflects the developer's efforts to fix the inconsistency after using type inference methods.

[1] Shivarpatna Venkatesh, Ashwin Prasad, et al. "Typeevalpy: A micro-benchmarking framework for python type inference tools." Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 2024.