Large Language Models are Interpretable Learners

---

[Original review]

W1,W2 “I found the manuscript somewhat challenging to follow due to the absence of consistent examples within the main text. Currently, major examples are relegated to the supplementary materials. To enhance clarity, I recommend incorporating one or two examples directly into the main manuscript and referencing them throughout.” “Moreover, a clear problem statement would significantly improve comprehension. Specifically, outlining the input and output parameters before discussing the methodology would be beneficial.”

[Response]

Thank you for these valuable suggestions! To improve clarity and readability, we have updated the main figure (Figure 2) to illustrate the entire pipeline using a simple visual classification task as an example. We hope this addresses your concern, and we would greatly appreciate any additional feedback on this update or further suggestions for improvement.

---

[Original review]

Minor W4-7 “The structure of the document feels somewhat convoluted. There is an overuse of emphasized text formats (bold, italic, etc.), particularly noticeable on Page 3, which detracts from the core ideas being presented. I recommend retaining emphasis only on the most critical sections and reducing the use of bold and italic text.

Equation 3 could be better explained with a figure that includes a concrete example and details like α and yi.

The text size in Figure 8 (f) makes it difficult to interpret. Consider increasing the text size for better readability.

Furthermore, the code is not submitted, making it challenging to assess reproducibility.”

[Response]

Thank you for the detailed and valuable suggestions! We have incorporated your feedback into our revisions to improve the paper’s readability and presentation. Specifically, we have made the following changes:

1. Reduced emphasis formatting: We have minimized the use of bold and italic text throughout the paper, reserving it for highlighting only the most critical findings and conclusions.
2. Enhanced explanation of Equation 3: We updated the main pipeline figure to include Equation 3, a concrete example task, and a clear problem statement to better illustrate the entire learning process.
3. Improved Figure 8(f): We switched to a different visualization of the learned programs in Figure 8(f), which enabled us to use a much larger font for better readability.
4. Code submission: While we plan to release both the code and benchmarks upon acceptance, we provide a demo of the LSP training and testing process (though not yet organized) in the supplementary material.

We hope these changes address your concerns and improve the overall clarity of the paper.

---

[Questions]

Q1. “I was not fully following the reasoning in line 268 -- 286. Could you elaborate the following two arguments?

Therefore, with a fixed LLM, the set of natural language prompts, ... provides a massive set of interpretable neural network modules for the task.

Why the prompts can be trivially seen as interpretable neural modules?

... the resulting model is inherently interpretable, as the prompt s is expressed in natural language.

Why the resulting model can be seen inherently interpretable?”

[Answer]

Thank you for the question. We are happy to elaborate:

1. Why can prompts be seen as interpretable neural modules? Pretrained (M)LLMs are powerful conditional probability models of p(y∣s,x)p(y \mid s, x)p(y∣s,x), where sss is a prompt, xxx is the input, and yyy is the output generated by the LLM based on that prompt. This allows us to implement a wide variety of neural network modules (or specialist models) simply by defining different prompts. For example:
   1. Sentiment analysis model can be implemented by LLM(y|s=”Classify the following text into one of these categories: [positive, negative, neutral]”, x=“I like Pizzas”)
   2. Shape detector model can be implemented as MLLM(x|s=”What is the shape of the object in this image?”, x=<image of a ball>)
2. Why is the resulting model inherently interpretable? Unlike traditional neural network modules in NSP, which are defined by numerical parameters, LLM modules are defined through natural language prompts. This makes the prompts inherently more interpretable because they are understandable to humans without requiring technical expertise in neural network representations.
   1. To validate this claim, we conducted an empirical evaluation where human raters labeled test data by reviewing the learned rules/prompts (Table 2). The results demonstrated high “transferability” of these learned rules to humans, further supporting their interpretability.

We hope this explanation helps clarifying these arguments. If you have additional questions or suggestions, we would be delighted to discuss them further!

---

---

[Original review]

W1 “As the authors say, LLM's responsibility is to make decisions in each LLM module in their llm-symbolic programs. Thus, generating a correct LSP relies on the LLM's ability. I am curious whether the proposed method could correspond with other open-source LLMs that could be deployed locally.”

[Response]

Thank you for raising this point. With the recent advancements in open-source LLMs, we have observed that LSP performs well even when using smaller, open-source models as the learner LLM. To explore this, we conducted experiments using the same tasks as our ablation study. The results, summarized in the table below, show that LSP maintains strong performance compared to other prompt optimization methods, even with open-source LLMs.

This robustness is largely attributed to the divide-and-conquer strategy of LSP:

• If an LLM generates an incorrect rule at a node, the error can often be corrected by the child nodes.
• Additionally, as the tree structure is traversed, the data distribution is divided among the nodes. This means each node is responsible for predicting only a subset of the data, simplifying the task of generating correct rules.

We believe these characteristics make LSP particularly well-suited for deployment with open-source LLMs.

(Due to license policy, we are not able to use llama in our work, instead, we adopt Gemma-2-9b-it as the learner LLM, the inference LLM is still GPT3.5.)

---

[Remaining Questions]

Q2. “Are datasets from AIPS or/and AlphaGeometry, which contains IMO level mathematical problems, able to be IL benchmark? The authors may discuss the difference between the proposed dataset and the datasets like AIPS or AlphaGeometry in terms of measuring interpretable learning ability.”

[Answer]

Thank you for introducing these datasets! Similar to other math datasets like GSM8K, both AIPS and AlphaGeometry consist of tasks that are zero-shot solvable. This is primarily because:

• Their inputs (i.e., the problem statements) already provide all the information necessary for an LLM to solve the tasks.
• As a result, LLMs can solve these problems without needing to learn any rules from labeled data.

While the methods used in these datasets are inspiring, they differ from LSP in that they do not focus on learning interpretable predictive rules from labeled data. As the reviewer may know, Symbolic AI encompasses a broad range of problems and methodologies. For example:

• Some approaches, such as those referenced in the review, use search-based structural decoding for mathematical reasoning.
• Others equip LLMs with symbolic solvers (109).

However, these methods are designed for zero-shot solvable tasks and are not directly compatible with interpretable learning (IL). We’ve added a reference to these works in Section 2 (line 99), where we discuss why certain benchmarks and approaches are not suitable for IL due to zero-shot solvability.

Q3. “Instead of constructing LSP, does SFT or some RL could be applied to LLM to improve the LLM's IL ability?”

[Answer]

This is a great point! We agree that there are opportunities to enhance the IL ability of LLMs through methods like SFT or RLHF. Our observations suggest:

• Similar to other abilities, stronger LLMs generally perform better on IL tasks.
• As you mentioned, incorporating IL datasets during SFT could further improve IL capabilities, much like how LLMs are fine-tuned on math datasets to enhance their reasoning skills.

We believe this is a promising direction for future exploration.

---

---

[Original review]

W4 “Generality of the findings : the IL-Bench should be more motivated. It is not clear for me what a symbolic task is exactly. Moreover, a large part of the task are classification tasks. What about other tasks regarding the proposed approach ?”

[Response]

Thank you for highlighting this point. We provide an overview of all tasks in Appendix Table 8. The symbolic tasks are decision-making problems where all variables, values, and labels are represented using abstract symbols (e.g., x1, A1, foo, bar). This design has several advantages:

1. Ensures non-zero-shot solvability: Since the tasks use abstract symbols, LLMs cannot rely on prior knowledge to make predictions. For instance, if the task had real-world context, such as predicting rainfall based on humidity, the LLM can make predictions even without learning any underlying rules from data.
2. Scalability: Symbolic tasks can be automatically generated and scaled to any size or complexity.
3. Known oracle rules: The underlying rules used for data generation are explicitly known, which facilitates model diagnostics. For example, in Appendix A.3, we are able to compare the learned LSP to the oracle rules.

Consistent with traditional XAI approaches, particularly those focused on learning interpretable decision-making models (e.g., NSP), our work primarily focuses on predictive tasks. However, the versatility of the LSP framework, powered by LLM modules, allows it to be extended to any task solvable by LLMs. We consider this an exciting direction for future exploration.

---

[Original review]

W5 “What is Dist in equation 1 ?”

[Response]

Thank you for bringing it up. Dist(a1, a2) is the distance function between a1 and a2.

---

[Remaining Questions]

Q3 “In the following approach, is the decision making process limited to conditional branching and tree-based models ? Is it a strong limitation ?”

[Answer]

Thank you for raising this point! The resulting program adopts a tree structure, not due to a constraint of conditional branching, but because trees are a widely used representation for programs (e.g., any code can be represented into a syntax tree). This alignment also explains why prior NSP methods naturally discover tree-structured programs.

Q4 “Can hallucinations can occur for RuleSum ?”

[Answer]

Yes! we have observed that the learner LLM does not always summarize the rules correctly. However, we also discover the following:

• Stronger models (e.g., GPT-4) tend to exhibit significantly fewer hallucinations compared to weaker models (e.g., GPT-3.5 or gemini-medium).
• The LSP framework, with its divide-and-conquer approach, is more tolerant to errors compared to vanilla prompt optimization algorithms. This is because errors in parent rules can often be corrected by child branches.

We believe that fine-tuning LLMs to improve rule summarization is a promising direction for future research.

---

---

[Original review]

W1,W2,Q1 “a first small concerns is about the title of the paper which is for me not really aligned on the content. The authors use LLMs as learnable building blocks for NSPs but are not interpretable as their own. The title of the paper should be slightly changed to better reflect the true claims of the paper. lack of clear definition of the notion of interpretability and of the underlying assumptions : as the authors tackle the trade-off between expressiveness and interpretability, a clear definition of them should be given. For instance for me, using a minimalist DSL means less expressivity since there are only two operators. The assumption that relying on natural language brings interpretability should also be discussed. The interpretability seems to be linked to the interpretability of the prompts.”

[Response]

We appreciate the reviewer’s insights, and also agree that interpretability can take on multiple forms in the XAI literature. In this paper, we adopt a strong form of interpretability: whether a human with no prior domain knowledge can understand and apply the decision rules learned by the model (line 035).

• This definition emphasizes the transfer of knowledge from AI to humans.
• In the LSP framework, one LLM acts as the learner (optimizer), summarizing rules from data. These rules, presented in natural language, are inherently interpretable to humans. For this reason, we view LLMs as interpretable learners.
• We empirically evaluated the interpretability of LSP by asking human raters to label test data based on the learned rules. The results showed high “transferability” of the learned rules to humans (Table 2, Last Group).

We add this as a formal definition block in the revision. Please feel free to let us know your thoughts on the updated framing or if there are further suggestions for improving it!

Moreover, since LLM module in the DSL can implement a wide range of operators with different prompts, it makes the minimalist DSL much more expressive than NSP. In fact, this is an advantage of LSP:

• In NSP, the functions in DSL needs to be pre-defined by humans, whereas the LLM modules in LSP are fully learned from data, making the DSL more flexible and expressive.
• Having a minimalistic DSL also makes the program structure search much more efficient, as each node has fewer functions to choose from.

---

[Original review]

W3,Q2 “a better positioning to recent approach on XAI, in particular concept-based XAI, mentioned in appendix A.1 and Mechanistic Interpretability.” “How this idea of using LLMs to transforms signals into high-level concepts, how the approach compared to concept induction approach ? To Concept-based Explanations for LLMs “

[Response]

Thank you for the suggestion! As you noted, the related work provides a literature review of concept-based methods; Following your suggestion, we further clarify how LSP compares to concept-based XAI:

• Connection: Both concept-based methods and LSP aim to provide insights into the decision-making process.
• Differences: Intrinsic concept-based methods add a “concept layer” (typically a linear transformation) to the neural network, requiring the model to make predictions based on predefined concepts. Post-hoc methods align human-defined concepts with a pretrained model. Both approaches share the same limitations: (1) they rely on human-defined high-level concepts, and (2) concepts do not fully represent the decision rules. In contrast, LSP directly learns all interpretable modules (LLM prompts) from data without relying on human prior knowledge. Furthermore, LSP fully reveals its decision process through learned prompts and program structure, while concept-based methods only partially expose the decision process.

We’ve incorporated the above discussion in the Appendix A.1.

---

Dear Reviewers,

Thank you again for reviewing our paper and leaving insightful suggestions! As the deadline approaches, we want to reach out to see if our rebuttal addresses your concerns? If you have any further questions, please feel free to let us know. We are happy to address any questions you might have!

Best, LSP Authors