SymTex: A New Benchmark for Non-monotonic Reasoning Capability of Large Language Models

We greatly appreciate the time and effort you have invested. In response to your concerns, we have provided clarifications here.

W1:The tasks designed in this benchmark seem highly artificial and do not reflect real-world non-monotonic reasoning challenges. For instance, the use of random strings or words as predicates (as in Figure 2) seems unrealistic for reasoning scenarios that LLMs encounter in natural language tasks. This limits the generalizability of the results. Real-world non-monotonic reasoning involves handling contextual and dynamic information, such as conversational changes, rather than simple logical queries over synthetic facts and rules. In other words, it would be necessary for the authors to justify the relevance of the proposed benchmark to real-world scenarios.

Response (W1):

(1): For the relation between the our focused task and natural language tasks.

We would like to clarify that the focus of this work is different from the work given in [1] and [2], which consider non-monotonic reasoning in natural language driven by common-sense knowledge. In contrast, our work considers evaluate the ability of LLMs on symbolic non-monotonic reasoning which is the main-stream of non-monotonic reasoning [3][4].

(2): For artifical data.

The purpose of this benchmark to use a synthetic dataset is due to the pure and controlled environment provided by this setting. It can directly evaluate the non-monotonic reasoning ability of models and exclude the influence of semantics for evaluation results.

(3): For the relation to real-world scenarios.

In symbolic non-monotonic reasoning, scenarios consist of facts and rules, and the process of dealing with given scenarios can be seen as dealing with dynamic information in the real world. Moreover, we conducted experiments to assess whether LLMs can correctly modify the previous conclusion when given a new contradictory fact, as shown in the paper's Figure 4.

To better align with the reasoning habits of LLMs in real-world scenarios, we also provide pairs of samples featuring symbolic and textual settings respectively.

W2: The two proposed tasks—tri-state boolean querying and answer set computation—are not clearly defined in practical terms. The descriptions of both tasks lack details on how the LLMs are supposed to process the input, leading to confusion about what is being evaluated. For example, it is unclear how the "M" label is determined in Tri-State Boolean Querying, and why this label significantly complicates reasoning tasks. The evaluation metrics for the tasks (wF1, Acc, etc.) are briefly mentioned but not adequately explained. This weakens the reader's ability to assess the relevance of the tasks. More specific descriptions for the tasks should be provided, such as step-by-step examples of how an LLM should process a sample input for each task.

Response (W2):

(1): Definition of tasks.

The two tasks—tri-state boolean querying and answer set computation—are both about solving an ASP (Answer Set Programming) problem. Tri-State Boolean Querying is a classification task that assigns a label to a query based on the facts and rules, while Answer Set Computation is a generation task that generates all possible conclusions from the given facts and rules. The label definition can be found in Section 4.2.2 ("ANSWER SET GENERATION").

(2): Prompts for LLMs.

The definition of prompts are shown in Appendix C ("PROMPTS FOR TASKS"), and we reported case study in Appendix D.4 ("ERROR CASE ANALYSIS").

(3): Analysis of the label "Maybe".

From the error case analysis in Appendix D.4 ("ERROR CASE ANALYSIS"), we discuss that LLMs always want to give a certain state to a conclusion. However, in non-monotonic reasoning, the conclusion may be in an uncertain state ("M"), which leads to error prediction from LLMs.

(4): For metrics.

In Section 5.1.2 ("METRICS"), we describe the definition and computation methods of the metrics. These metrics are commonly used in classification and question-answering tasks, so we provide only a brief description here. We plan to optimize them in subsequent versions.

We greatly appreciate the time and effort you have invested. In response to your concerns, we have provided clarifications here.

W1:The motivation for testing LLMs' ability in the symbolic format requires further clarification. There are well-developed rule-based solvers for ASP problems that can solve ASP tasks effectively. Therefore, why focus on testing LLMs' ability on the ASP format instead of relying on rule-based solvers? In the analysis, you mention that improving LLMs' understanding of symbolic structures could enhance their ability to translate natural language into symbolic formats and leverage external solvers. However, solving ASP questions is not equivalent to translating natural language into ASP. Please correct me if my understanding is incorrect.

Response (W1):

(1): The reason to evaluate the symbolic programs.

By evaluating the LLMs' ability in symbolic programs, we can gain a deeper understanding of the internal reasoning ability of LLMs. Moreover, the logic programs, such as the ASP programs in our benchmark, is also a type of code, which is highly related to code generation or code understanding tasks on LLMs $1$

$2$ , and non-monotonic reasoning has a well-established and mature symbolic foundation $3$

$4$ .

(2): Benefits when LLMs can execute ASP programs reliably.

It's inconvenient to use ASP solvers in daily life, due to their complicated installation and strict grammar requirements. If LLMs are reliable enough to infer on logic programs, they can be used as ASP solvers conveniently to deal with ASP problems, due to their tolerance for minor grammar and expression errors.

(3): Complement between symbolic and textual samples.

Table Supplement.2 suggests that if both formats are used together for reasoning, LLMs can potentially produce more accurate answers. This also underscores the necessity of symbolic logic samples.

W2: Further details are needed in Stage 1: Generation. How are the rules generated in this stage? There are various types of symbolic rules, so how do you determine which specific rule to use? Additionally, it is unclear why the modification is necessary. What purpose does modification serve when building the benchmark?

Response (W2):

(1): Rule generation.

In Stage 1: Generation, symbolic rules are generated by randomly selecting predicates from a set of existing predicates to form the body of a new rule. A new predicate is then created to serve as the head of the rule.

(2): There are various types of symbolic rules, so how do you determine which specific rule to use?

We use the syntax of a state-of-the-art ASP (Answer Set Programming) solver, dlv2.

(3): Purpose of modification.

To avoid generating ASP programs with the same logical structure, we first create "templates" using only positive predicates. These templates provide a basic structure for the rules. Then, we modify them by conducting modification operations. This ensures the rules are more diverse and cover a wider range of scenarios.

W3: More details are needed for Stages 2 and 3. In Stage 2 (Predicate Modification), while you modify the predicate, it appears from Figure 2 that the argument (e.g., Tom, Jack) is also modified. However, in Stage 3 (Textualization), those specific arguments revert back to symbols (e.g., Tom becomes name_0). This process is somewhat confusing, and further elaboration is needed in the methodology to clarify the reasoning behind these steps.

Q3: Why is the argument in stage 2 modified to Tom and Jack but changed back to symbols at stage 3 according to Fig 2?

Response (W3, Q3):

Stage 3 will retain all the modifications from Stage 2. The description of the images was somewhat unclear, and we are preparing to revise it. Specific examples are shown in Appendix B ("SYMTEX Examples").

We greatly appreciate the time and effort you have invested. In response to your concerns, we have provided clarifications here.

W1:As far as I understood, existing work (and mentioned in the related work) has focussed on more text-based non-monotonic reasoning capabilities and the main contribution of the proposed dataset is to evaluate if the LLMs can process symbolic representations. What were the results from the earlier work, how does this new work add/modify our views of LLM capabilities? In general, I felt like placing the new dataset in the context of what is known about LLMs using existing work and what new knowledge the dataset and results are adding is important in an empirical paper such as this one.

Response (W1):

(1): The results from earlier works.

Earlier studies have demonstrated that LLMs face challenges when dealing with non-monotonic reasoning problems. However, the samples in these works suffer from relatively simple construction or driven by common-sense knowledge, which can not fully or purely evaluate LLMs' capability on non-monotonic reasoning.

(2): New views from our benchmark.

From the results in Table Supplement.2, we can suggest that there is a complement between symbolic and textual samples.

W2: In general, why would it make sense for an LLM (built primarily for language) to be evaluated with a symbolic program was not something that was very clear. Related to this, is there a real-world use-case that motivates this usage of LLMs? (There are a couple of sentences mentioned but a more clear motivation would be useful)

Response (W2):

(1): Reason to evaluate the symbolic programs on LLMs.

The logic program, such as the ASP programs in our benchmark, is also a type of code, which is highly related to code generation or code understanding tasks on LLMs $1$

$2$ , and non-monotonic reasoning has a well-established and mature symbolic foundation $3$

$4$ . The LLMs' capability of understanding and solving the logic programs can represent their ability for logic reasoning and instruction following.

(2) The usage of symbolic programs.

Table Supplement.2 suggests that if both formats are used together for reasoning, LLMs can potentially produce more accurate answers. This also underscores the necessity of symbolic logic samples.

W3: One of the steps in the proposed approach also performs textualization of the program, how sensitive is the LLM to the quality of going from symbolic models to an equivalent text representation?

Response (W3):

(1): For data quality.

This work focuses on scenarios that the non-monotonic reasoning logic programs can describe, especially by answer set programming (ASP). In this case, we can evaluate the correctness of generated samples by executing the programs on ASP solvers.

(2): For equivalent between symbolic and textual samples.

In practice, textual samples are often converted back into symbolic samples. For example, both "X eats Y" and "X and Y are friends" can be represented symbolically as a(X, Y). To ensure equivalence between symbolic and textual samples, we employ a template-based approach that facilitates a consistent transformation between the two, preserving their logical integrity.

We greatly appreciate the time and effort you have invested. In response to your concerns, we have provided clarifications here.

W1:The authors note that LLMs perform better on textual representations than on symbolic ones, but the paper lacks specific solutions or insights into how to address this gap. An in-depth exploration of architectural or methodological changes that could help LLMs better handle symbolic reasoning would add value.

Response (W1): In Section 6 ('Conclusion'), we provide some preliminary insights and suggests future directions, including potential approaches to bridge the gap between symbolic reasoning and the capabilities of LLMs.

W2: One area for potential improvement in the benchmark's symbolic setting is the integration of reasoning interpreters (e.g., ASP solvers) alongside LLMs. While evaluating LLMs independently provides an insightful baseline of their inherent reasoning capabilities, it might also be valuable to explore how these models perform in combination with interpreters. This would reflect a practical, real-world scenario where LLMs often rely on auxiliary tools for enhanced reasoning.

Response (W2): We agree that combining reasoning tools like ASP solvers with LLMs could be a valuable direction. This aligns with our future research plans, and we're already exploring how such collaboration can improve the reasoning abilities of both models.

W3: Potential Dataset Bias: As with any generated dataset, SymTex may contain unintentional biases due to the template-based generation process. Without rigorous diversity checks, the dataset might fail to capture the full spectrum of possible reasoning challenges, leading models to overfit specific structures. Providing a bias analysis or diversity evaluation for the generated samples would help ensure a robust assessment of LLMs’ reasoning abilities.

Response (W3):

(1): Diversity of samples.

To ensure diverse sample generation, we adjust various parameters, such as the number of facts and rules, as well as the maximum arity of rules and predicates.

(2): Multi-run results.

Biases in logical program datasets can be difficult to define and identify. To mitigate this issue, we performed three independent runs for each experiment and reported the average results.

For dataset bias

Regarding your concern, SymTex may contain unintentional biases, as different types of samples might affect model performance differently. To address this, we conduct experiments in Section 5.3 in the revised paper, titled "Variable Impact Analysis on Results", where we analyze the effects of varying query arity, related facts and rules, as well as noisy facts and rules on F1 scores.

Revised Paper

The revised paper has been submitted.

We sincerely appreciate the valuable feedback provided by the reviewers and have submitted the revised paper accordingly. We hope the revisions could address the reviewers' concerns, and we welcome any further questions or suggestions that may arise.