Can Large Language Models Understand Intermediate Representations in Compilers?

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we address the key concerns:

W1: The paper only uses the HumanEval, which contains code functions with an average of fewer than 10 lines.

A1: Though HumanEval consists of relatively short functions, our results reveal fundamental limitations in LLMs’ IR comprehension that extend beyond dataset scale from three aspects: (1) Controlled Complexity: Even small programs require precise reasoning about IR constructs; (2) Task Difficulty Beyond Code Size: Despite their brevity, small programs exhibit complex low-level semantics.; (3) Systemic Failures: Consistent errors across tasks—such as CFG Construction and Execution Reasoning—indicate systemic deficiencies in IR understanding.

---

W2: The paper evaluates GPT4, Gemma, and so on. More SOTA LLMs, such as GPT-4o, DeepSeek-V3 (or R1), should be included.
&
W4: It is not specified whether the results are obtained in multiple turns.

A2: We agree that including SOTA models is essential. We supplemented our experiments with DeepSeek R1.

A4: We fully agree that multi-turn evaluations using metrics like pass@k enhance robustness. We ran DeepSeek R1 on all four tasks three times (R1–R3) to ensure stability.

T1: CFG Construction

T2: IR Decompilation

T3: Code Summarization

T4: Execution Reasoning

DeepSeek R1 performs comparably to LLaMA 3.1 on T2–T4 but excels in T1 (CFG Construction) due to its integrated chain-of-thought mechanism that helps identify basic blocks and critical control flow instructions (e.g., "br" and "jmp").

Revised Finding 1: LLMs generally struggle with detecting basic blocks and constructing control flow edges; however, chain-of-thought prompting modestly enhances the recognition of key control flow instructions.

A4: Notably, the three runs yielded consistent results, demonstrating the reliability of DeepSeek R1’s performance. We will include these multi-turn results in the revised manuscript.

In the final version, we will include comprehensive analyses, additional experiments, and concrete examples to support these findings. In future work, we will systematically explore additional prompting techniques to further validate and extend these findings.

---

W3: There exist non-neural developed tools for IR tasks in common software testing applications that could help highlight the dis/advantages of LLM-based methods.

A3: We appreciate the reviewer’s point that established non-neural methods serve as valuable baselines. Tools such as Ghidra, various IDE tools, and symbolic execution engines like KLEE excel at CFG construction, decompilation, and execution reasoning through extensive domain-specific training. In contrast, our study uses these tasks solely as diagnostic benchmarks to evaluate the raw, untuned IR comprehension of LLMs. Our goal is not to outperform specialized tools but to reveal intrinsic LLM limitations—such as in control flow inference, granular semantic understanding, and loop handling—that can guide future optimizations. In the final manuscript, we will include a detailed discussion comparing our results with these non-neural baselines.

---

W4 & A4 are included in A2.

We thank the reviewer for the thoughtful feedback and recognition of our novel application of LLMs to compiler IRs. We also appreciate the acknowledgement of our focus on task-specific analysis. We address the key concerns below.

Q1 & W1: Are there any non-LLM approaches that would be applicable to the introduced tasks, e.g., fine-tuning or training a classifier on top of IR2vec / Bert-style models and/or applying graph representation learning methods? Or is there any particular reason that those are not included/applicable
&
W3: The paper does not include non-LLM baselines.

A1: We appreciate the reviewer’s point that established non-LLM methods can serve as valuable baselines. For instance, pre-trained models such as FAIR [1] have shown impressive results in semantic summarization. However, while non-LLM approaches (e.g., GNNs, fine-tuned BERT-style models) excel on specific tasks, they require extensive dataset-specific training and are typically applied to higher-level applications.

In contrast, our study focuses on the raw, untuned IR comprehension of general-purpose LLMs. By using tasks such as CFG reconstruction, IR decompilation, code summarization, and execution reasoning, we establish diagnostic benchmarks that evaluate the zero/few-shot and chain-of-thought (CoT) prompting capabilities of current LLMs without any task-specific fine-tuning. This approach not only reveals the inherent limitations of existing LLMs in understanding intermediate representations but also provides actionable insights for future improvements.

Moreover, as detailed in A2, we will incorporate additional CoT prompting in the revised manuscript. In future work, we plan to explore non-LLM baselines and a broader range of prompting techniques to further enhance our analysis.

---

W2 & W3: Evaluation is limited to zero-shot experiments; few-shot and basic CoT prompting have not been explored.

A2: Our evaluation is not strictly zero-shot. We use zero-shot prompting for CFG Construction and IR Decompilation, few-shot examples for Code Summarization, and chain-of-thought prompting for Execution Reasoning. Although these techniques improve performance, fundamental challenges in IR understanding—particularly in control flow and execution semantics—persist, with model rankings remaining unchanged. To assess the benefits of chain-of-thought prompting, we conducted supplementary experiments using DeepSeek R1. We selected DeepSeek R1 because (1) it is among the state-of-the-art LLMs, and (2) it features an inherent chain-of-thought mechanism that decomposes complex IR tasks into intermediate reasoning steps, showing potential advantages for our IR tasks.

We ran DeepSeek R1 on all four tasks three times to ensure stability. Our preliminary results (one as an example) are as follows:

T1: CFG Construction

T2: IR Decompilation

T3: Code Summarization

T4: Execution Reasoning

DeepSeek R1 performs comparably to LLaMA 3.1 on T2–T4 but excels in T1 (CFG Construction) due to its integrated chain-of-thought mechanism that helps identify basic blocks and critical control flow instructions (e.g., "br" and "jmp").

Revised Finding 1: LLMs generally struggle with detecting basic blocks and constructing control flow edges; however, chain-of-thought prompting modestly enhances the recognition of key control flow instructions.

In the final version, we will include comprehensive analyses, additional experiments, and concrete examples to support these findings. In future work, we will systematically explore additional prompting techniques to further validate and extend these findings.

[1] Niu, Changan, et al. "Fair: flow type-aware pre-training of compiler intermediate representations." ICSE’24.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate your recognition of the novelty of applying LLMs to compiler IRs and the value of our in-depth evaluation. Below, we address the key concerns:

Q1: Can you provide more details on the level of prompting?

A1: In our paper, we have adopted two prompting strategies: (1) Task-Specific Expert Prompts. For each IR task, prompts assign a role (e.g., “CFG analyzer for IRs”) and clearly define the input, task objective, and required output format (e.g., DOT files with nodes and edges); (2) Advanced Prompting Techniques. We also applied the Zero/Few-Shot Prompting and Chain-of-Thought (CoT) Prompting in Tasks 1–3 and Task 4, respectively.

Note that the main text summarizes our approach, and complete prompt templates are provided in Appendix B (extended examples omitted for brevity).

---

Q2: Can you provide the statistics of the IR humaneval programs?

A2: Key statistics of 164 IR programs across optimization levels are as follows:

At -O0, minimal optimizations yield verbose IR with duplicate functions to preserve debugging (‘linkonce_odr’). -O1 and -O2 remove redundant elements, reducing size but slightly increasing loop counts, while -O3 further simplifies the structure at the cost of more loops. These trends explain why decompilation performs best at -O1/O2. We will include a detailed description and a summary table in the revised manuscript, along with data-driven explanations to substantiate our findings.

---

W1: Choice of HumanEval problems used for constructing tasks likely challenges the generalizability.

AW1: We acknowledge that the HumanEval dataset may limit generalizability. However, our results reveal “general limitations” in LLMs’ IR comprehension that extend beyond dataset scale or diversity from two aspects: (1) Controlled Complexity: Although the programs are small, the extracted IRs involve common operations—including nested loops, conditional statements, and function calls—that pose significant challenges; (2) Systemic Failures: Consistent errors across tasks, such as CFG reconstruction and execution reasoning, indicate systemic deficiencies in IR understanding.

These issues represent fundamental “general challenges” that LLMs have yet to overcome for most programs. Once these preliminary obstacles are addressed, we plan to expand our evaluations to larger, more diverse datasets to continuously enhance generalizability. We will also discuss future directions for incorporating more diverse IR samples in the revised manuscript.

---

W2: It is unclear how to calibrate the findings once we develop models explicitly optimized for IR programs

AW2: We agree that fine-tuning on IR-specific datasets may improve performance. Our study is primarily exploratory, positioned to assess the untuned performance of current LLMs in understanding IRs without relying on benchmark-specific fine-tuning. We acknowledge that fine-tuning LLMs on specialized datasets could boost their performance on IR-related downstream tasks. However, by focusing on their untuned performance, our work establishes a clear baseline that exposes key challenges. Our findings pinpoint areas where current LLMs fall short—such as control flow comprehension (Finding 1), granular semantic understanding (Findings 4–5), and loop handling—while also offering guidance for targeted fine-tuning strategies. We are committed to pursuing fine-tuning as a central focus of our future research, leveraging our insights to significantly enhance models' abilities to tackle more complex IR-related tasks.

---

W3: The related work for intermediate/assembly language is well provided; however, a rich body of work about code understanding is not discussed.

AW3: We appreciate the suggestion to contextualize our work further within the broader code understanding literature. Although our manuscript comprehensively covers IR-level work, we agree that expanding the discussion of static/dynamic analysis methods (e.g., CRUXEval [2] and CFG path analysis in [3]) would enrich our related work section. In the revised manuscript, we will incorporate these references to better position our study within the broader landscape of code understanding research.

[1] Chris C., et al. LLM Compiler: Foundation Language Models for Compiler Optimization. CC’25

[2] CRUXEval: Code Reasoning, Understanding, and Execution Evaluation

[3] LLMs: Understanding Code Syntax and Semantics for Code Analysis

Thank you very much for your detailed review and positive feedback on our work. We greatly appreciate your recognition of the novelty and significance of our study, as well as your thorough evaluation of our experimental setup. We would like to address your comments as follows:

W1: The study does not consider fine-tuning on a benchmark-specific dataset, which limits its ability to reflect true benchmarking metrics.

AW1: We appreciate the reviewer’s comment. Our study is primarily exploratory, positioned to assess the untuned performance of current LLMs (including zero/few-shot and chain-of-thought prompting) in understanding intermediate representations (IRs) without relying on benchmark-specific fine-tuning. This approach enables us to use tasks such as CFG reconstruction, IR decompilation, code summarization, and execution reasoning as benchmarks for evaluating LLMs' understanding of intermediate representations. Moreover, our findings reveal inherent limitations in how these models understand and process IRs. We acknowledge that fine-tuning LLMs on specialized datasets could boost their performance on IR-related downstream tasks. However, by focusing on their untuned performance, our work establishes a clear baseline that exposes key challenges. Our findings pinpoint areas where current LLMs fall short—such as control flow comprehension (Finding 1), granular semantic understanding (Findings 4–5), and loop handling—while also offering guidance for targeted fine-tuning strategies. We are committed to pursuing fine-tuning as a central focus of our future research, leveraging our insights to significantly enhance models' abilities to tackle more complex IR-related tasks.

Typos and Formatting Issues We appreciate you highlighting some spelling and formatting errors. We have reviewed the manuscript carefully and will correct these issues in the final version.

Again, thank you for your constructive comments.