CRABS: A syntactic-semantic pincer strategy for bounding LLM interpretation of Python notebooks

Thank you for your valuable comments and suggestions! We address your questions below. Please let us know if you have any follow-up questions.

Answer to Q1: Please refer to the "clarification of notebook selection" section in the general responses.

Answer to Q2: Our comparisons are limited to two ablation studies: (1) cell-by-cell prompting without providing any hints from syntactic parsers; and (2) a single prompt for the whole notebook with hints for all cells from the syntactic parsers. Table 3 shows the percentage-point decreases of 7 for (1) and 16 for (2) in F1 score. We agree with the value of looking into other prompting strategies. For example, in future work we plan to try an approach of including in the prompts a sliding window of preceding and succeeding code cells.

Answer to Q3: Please refer to the "clarification of the role of syntactic parsers" section in the general responses. Our syntactic parsers employ the Python built-in ast module to access and traverse the abstract syntax tree for each cell, a widely employed technique. Via the traversals of the AST we produce lower and upper estimates of inter-cell I/O sets that bound the ground truth represented by our notebooks. We believe this covers most of the cases in practice. However, we acknowledge that the validity of our syntactic parsers are not thoroughly evaluated, and that a practical implementation of CRABS will require comprehensive validation and testing.

Answer to Q4: We agree that LLMs have the potential to significantly transform data science roles and education. Our current focus is on the notebook understanding task and the CRABS strategy for performing it, but we plan to explore the broader implications of CRABS and related approaches for enhancing the transparency of notebooks and the scientific research that depend on them.

Answer to Q5 & Q6: Thank you for these suggestions. We will certainly pursue these opportunities in our future work.

Thank you for your valuable feedback! We address your comments below. Please let us know if you have any follow-up questions.

Answer to W1: Please refer to the "clarification of notebook selection" section in the general responses.

Answer to W2 & Q1 & Q2: Yes, the annotation result is unique, and there is only one correct dependency graph per notebook.

a) Multiple code cells can each be considered the execution dependency of a downstream code cell. Taking Figure 3 as an example, cell 3 uses the phone data from cell 1 and the ec string from cell 2. There is only one unique information flow graph (Figure 3) and one unique cell execution dependency graph (Figure 4) for the notebook (Figure 2). We annotate this example notebook by inputs and outputs per code cell (see below), which can be used to construct the corresponding graphs.

[
  {"id": 1, "outputs": ["phone", "survey"]},
  {"id": 2, "outputs": ["ec"]},
  {"id": 3, "inputs": ["phone", "ec"]},
  ...
]

b) Annotation accuracy is ensured through careful validation. Ground truth is annotated by a skilled data scientist, who is confident in the majority of cell I/O sets. For uncertain cases, she compared the program state before and after cell execution to determine whether the data should be considered an output candidate.

c) Yes, annotations will be made available together with code via GitHub.

Answer to W3: Given the high performance of GPT-4o on this task (correctly resolving 98% of the ambiguities), we don’t think there is a lot of room for improvement. However, we do see the potential for using these techniques (e.g., few-shot learning and LoRA) to help improve the performance of smaller and cheaper LLMs (e.g., Qwen3 8B).

Answer to W4: Thank you for highlighting the issue of computational cost. Pure AST approaches do not represent baselines but instead provide lower and upper estimates that bound the ground truth. Their metrics shown in Table 2 are not intended for evaluation, but for reflecting the characteristics of our notebooks. I will clarify this in the next version of the paper and use lower estimate and upper estimate to replace MinIOParser and MaxIOParser.

Table R2 shows the computation cost per notebook (in seconds) using the baseline and CRABS approaches, and will be added to the appendix of the paper. The baseline approach is a single prompt for the whole notebook. Compared to baseline, we observe that (1) for notebooks with no (e.g., ID#1) or very few (e.g., ID#31) ambiguities left by syntactic parsers, CRABS requires less time and achieves comparable performance; (2) for complicated and lengthy notebooks with many ambiguities (e.g., ID#10), CRABS takes more time than baseline but yields better results; (3) for medium-complexity notebooks with around 10 ambiguities (e.g., ID#49, #50), CRABS requires similar time to baseline but also achieves better results.

Table R2: Computation cost per notebook (in seconds).

Thank you for your valuable feedback! We address your comments below. Please let us know if you have any follow-up questions.

For W1, W2, and W3, please check the "clarification of the role of syntactic parsers" section in the general responses first.

Answer to W1: With respect to global variables, the specific limitation of the current implementation of CRABS is that it does not identify information flows that arise from accesses or updates of global variables within function calls, i.e. flows via variables not mentioned explicitly at the function and method call sites. We fully agree that a practical implementation of CRABS must handle flows arising from global variable accesses (global variables are commonly employed to implement notebook-wide constants) and global variable updates (which may be considered poor practice in some communities but are still, for this reason, very important to handle in a real-world implementation) within functions.

Answer to W2: Our current implementation can in fact handle notebooks with multiple workflows. We exclude these notebooks from our dataset because such notebooks are not the focus of our work.

Answer to W3: Our current implementation detects hidden modifications by direct assignment (e.g., x=y) or by inclusion in collections (i.e., mutable objects stored in a list, a tuple, or a dictionary). This covers all instances of shared references in our dataset; we will make this clear in the next version of the paper.

Answer to W4: Take the same case (Figure 14) using the CRABS input prompt template (Figure 16) as an example. Given the following prompt, the LLM returns: No, “count” is not an input.

Given a Python program block, determine if an object is an input. An input is a variable that is used in the program block but not defined within it, or variables used before being reassigned. 

Important Cases: 
1. Conditional Statement Within a Loop A variable may be initialized during an early iteration of a loop and then utilized in subsequent iterations. In such cases, the variable is not considered an input, as its value originates from the loop's execution rather than external sources. 

Question: 
In the following Python program block, is "count" an input? Return the final answer in the form "Yes(No), "{x}" is (not) an input" where "{x}" is the actual name of the variable.

for i in range(3):
    if i ==0:
        count = 0
    else:
        count = count + 1

Answer:

Answer to W5: Due to time and cost limitations, we tested GPT-4o-mini, Qwen3-8B, and Qwen2.5-Coder-7B-Instruct models. Table R1 (under Reviewer dByL) validates the effectiveness of CRABS over smaller, open-source, general and code language models.

Answer to Q1: The low value (74%) of exact match (EM) is reasonable given its strict definition, i.e., the percentage of notebooks for which the entire information flow graph for the notebook (i.e. every one of the individual information flows in that notebook) exactly matches the ground truth. Take the notebook ID#27 as an example. It has 41 code cells and 286 information flows. An LLM correctly resolved 59 of 60 ambiguities that remain after analyzing the syntactic structure of this notebook. However, the single incorrect answer for one ambiguity leads to at least one wrong information flow, disqualifying this notebook from being included in the EM numerator. According to our results, all of the questions in each of 37 notebooks (74%) were answered correctly. Only a few questions (sometimes just one) were answered incorrectly in the remaining 13 notebooks (26%).

Thank you for your valuable feedback! We address your comments below. Please let us know if you have any follow-up questions.

Answer to W1: (1) Importance of notebook understanding without re-execution. We acknowledge that executing a notebook cell-by-cell to understand its function is common practice and can be highly effective in particular settings, e.g., when authoring and debugging one’s own notebooks or when adapting others’ notebooks for your own purposes. On the other hand, running notebooks that one discovers on various online resources and wish to evaluate as candidates for reuse or repurposing can require significant time and computational resources, especially if the dataset associated with the notebook is large or the notebook has complex software dependencies (as is often the case in scientific applications). So while we obtained the notebooks for our dataset from Kaggle, we expect that solutions to the problems we address will have broader applicability. Pimentel et al. (2019) studied 1.4 million GitHub notebooks and found that only 24.11% could be executed without errors, highlighting the value of avoiding re-execution. We will add these details to the introduction section of our paper. (2) The benchmark is still challenging for smaller, cheaper, and faster models that would be more suitable for practical applications of CRABS and that we plan to employ as part of our larger research agenda. Conducting the same experiments using GPT-4o-mini, Qwen3-8B, and Qwen2.5-Coder-7B-Instruct models achieves F1 scores of 83%, 75%, and 77% for identifying information flows respectively. Detailed results can be found in Table R1 which will be added to the appendix. In addition, we expect that this benchmark will serve us well as we employ the broad range of Qwen 3 model sizes in our planned efforts to identify the information and skills employed by LLMs to solve the various sub-problems comprising the overall notebook-understanding task.

Answer to W2: While combining symbolic tools and LLMs is a common practice in prompt engineering generally, researchers have explored diverse approaches and frameworks for doing this. Dater (Ye et al., 2023) leverages LLMs to decompose a task into logical sub-questions, where each sub-question can be answered independently using an SQL query; Binder (Cheng et al., 2023) maps a question into a program; and Maieutic Prompting (Jung et al., 2022) generates a maieutic tree using an LLM and employs MAX-SAT to infer the True/False answer. Among them, Dater (Ye et al., 2023) is the most similar to our work. But while Dater uses an LLM to decompose the task and a SQL query to solve a sub-task, CRABS uses syntactic parsers to decompose the task and an LLM to solve a sub-task. Furthermore, our CRABS is a pincer strategy that places bounds both on the total workload assigned to the LLM, and on the individual responses that it can return.

Answer to Q1: We have tested the baseline approach (see Figure 15 in the Appendix for an example prompt) using OpenAI o3 with high reasoning mode. It achieved an average F1 score of 94% identifying information flows across our 50 notebooks. We observe that the LLM still fails to understand one of these notebooks, returning the wrong number of cells and earning zero values for all metrics. Furthermore, one of these notebooks (ID#15) takes around 15 minutes to get the answer—about 10 times longer using CRABS and GPT-4o.

Answer to Q2: Thank you for referring us to the works related to code execution. CRUXEval (Gu et al., 2024) provides examples of successes and failures of GPT-4 for input and output predictions of Python functions. CodeI/O (Li et al., 2025) trains LLMs on code input-output predictions to improve their general reasoning ability. While LLMs may employ similar skills to solve these various tasks, our focus is information flows and the CRABS strategy for identifying them. We currently are investigating what information is used and what skills an LLM employs to solve such tasks, and the papers you cite will provide a valuable context for that work. We agree it would be extremely interesting to compare the information and skills employed by these different code understanding tasks.