Logically Consistent Language Models via Neuro-Symbolic Integration

We thank the reviewer for the feedback and for appreciating how our approach manages to efficiently improve consistency and quality of responses without needing an external solver. We address below all the concerns they raised.

LoCo-LLMs assumes that facts are conditionally independent

Good point. This limitation is actually shared by most works in Neuro-Symbolic AI, including circuit-based solutions [1]. This does not have a dramatic impact on performance, however, as highlighted by our experiments. We will consider relaxing this assumption in future work.

[1] van Krieken et al. "On the Independence Assumption in Neurosymbolic Learning." Forty-first International Conference on Machine Learning. (2024)

efficiency and scalability may lag behind approaches [like] RAG and knowledge editing.

Concerning efficiency, LoCo-LMs only require fine-tuning using a light-weight loss term and have no inference-time cost. At training time, our loss leverages circuits to avoid having to enumerate truth assignments, and allow us to compute exact probabilities of satisfying constraints into an operation that takes time linear in the size fo the circuit. (see lines 174-177 for more refs). The compilation step is also extremely fast, taking only ~2.5 milliseconds to compile a constraint and compute the loss on BeliefBank. Moreover, many data points will share the same constraint during training, enabling caching. At inference time, our approach has no overhead. For reference, ConCord takes ~3669 seconds to perform inference on BeliefBank (silver + calibration sets) for Macaw-large, whereas LoCo applied to the same model only requires only ~2405 seconds. We will add these results at the end of Appendix A.

While RAG has been used to mitigate factuality hallucinations [Lewis et al., 2020], it is no silver bullet, as LLMs occasionally ignore retrieved information [Xu et al., 2024, Jin et al., 2024], rather over relying on their learned knowledge. LoCo-LLMs are designed to avoid this.

[Lewis et al., 2020] Lewis et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." Advances in Neural Information Processing Systems 33 (2020).

[Xu et al., 2024] Xu et al. "Knowledge conflicts for llms: A survey." arXiv:2403.08319 (2024).

[Jin et al., 2024] Jin et al. "Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models." arXiv preprint arXiv:2402.14409 (2024).

We believe LoCo-LMs, which are more self-consistent and factual compared to regular LLMs, could benefit knowledge editing, in the sense that 1) it makes it less likely we need to edit LLMs to achieve the same objectives, allowing us to focus on updating their knowledge instead, and 2) updating a more self-consistent model can be less likely to produce non-logical “ripple effects”. Intuitively, the “ripple effects” left could be of a “logical kind” and thus making it easier to identify and correct them using logical consistency. This is a very interesting research question to investigate as future work.

LoCo-LLMs may be vulnerable to attacks, such as just-in-time injection.

We think LoCo-LLMs are no more vulnerable to prompt injection attacks than regular LLMs. Please let us know if you have further pointers that can suggest the opposite, we are happy to investigate this further.

Can LOCO-LMS be adapted for more complex, multi-level, or nonlinear logical reasoning scenarios?

We stress that the Semantic Loss term is applicable to logic formulas with an arbitrary number of connectives and logic variables. We showcased this property in our EntailmentBank experiment (Section 5.3), where the goal is to ensure consistency wrt entailment trees with up to 10+ logic facts. (The number of facts ranges from 1 to 5, see Figure 2 of [Dalvi et al., 2022] for the precise distribution.)

We are not sure about the second point: could you please clarify what you mean by nonlinear logical reasoning?

[Dalvi et al., 2022] Dalvi et al. Explaining answers with entailment trees. EMNLP 2022.

We thank the reviewer for the feedback and for appreciating how our approach allows us to side-step external solvers. We address below all the concerns they raised.

Macaw-Large [...] is quite old already. Even Llama-2 [...] is much less capable [...] compared to the current Llama-3.2. This raises questions [of applicability].

This was a forced choice to enable comparison against ConCord, which relies on Macaw and does not scale to newer and larger LLMs. In Table 2 we now also report the performance of Llama3.1 8B and show that, while the model is supposed to be more capable at reasoning than Llama 2, it falls short on BeliefBank in the very same way Llama2 7B with Few Shot. We are in the process of running the finetuning of LoCo-Llama3.1 and expect it to show similar improvements under different constraints.

there really should be some additional baselines with newer methods that also use model updating. For example, there is a whole library of papers focussing on updating specific facts in language models using targeted fine-tuning.

We’d be glad to compare against additional baselines, provided our computational budget allows it. What other approaches do you think we should consider?

It would be very useful to see how these changes impact the model performance in practical applications.

We are open to run additional experiments if the reviewer provides concrete suggestions for concrete applications that we can execute during the rebuttal.

Asking the LLM “Is an albatross not an organism?” is a very unnatural phrasing, whereas LMs are trained to predict natural continuations. I suspect that may be negatively affecting the performance for LMs.

We’d be glad to test additional prompts. Please let us know what you think we should test. We have updated our scores in Table 2 (and Tables 6-14 in the Appendix) with other syntactical variations of prompts, see also the new Appendix F and our answer to reviewer jLBW.

The method relies on collecting the probabilities for specific tokens to estimate the yes/no probabilties. How much is this going to be affected by the label bias of the LLMs? https://openreview.net/forum?id=shr9PXz7T0 https://arxiv.org/pdf/2402.09910

That’s a good point, but our LoCo-LMs are as susceptible as other LLMs to this phenomenon. We note that there is no a priori selection bias (as referred in the paper you linked) in the constraints defined in BeliefBank and EntailmentBank, therefore we do not believe that the semantic loss is affected in this sense.

We thank the reviewer for the feedback and for appreciating that our approach tackles an important problem and that, while simple and efficient, yields benefits beyond the training set. We address below all the concerns they raised.

[too much space] explaining logical constraints

Our goal was that of equipping NLP researchers, which might not be overly familiar with the topic, with the necessary preliminaries. We find this is essential for understanding LoCo-LMs and the problem they aim to solve: In our interactions with NLP researchers, that was the hardest part to understand for them.

details on the actual method is limited

Thank you for pointing this out. We have added an overview of the overall pipeline in Figure 1 and detailed how the circuit is constructed in Appendix A.

Please let us know if there are any other details that you find are unclear, we’ll be glad to detail them further in the revised manuscript.

the experiments mix unimportant and important details

We are happy to restructure it, if you can specify which details you would prefer to be moved to the appendix.

line 242: I don’t understand if the method handles facts that can be inferred from \alpha and the KB but require more than one hop?

The loss enforces constraints on given facts from a fixed-size knowledge base. We are not proposing a way to augment knowledge bases via deduction (i.e., generating new facts).

That being said, if we are given a multi-hop reasoning constraint (see our EntailmentBank experiments and Appendix D), we can enforce logical consistency over multiple reasoning steps. I.e., the formula $\alpha$ can reference arbitrarily many given logical facts; the semantic loss term into which $\alpha$ is compiled will constrain exactly those facts. E.g., for $\alpha$ is “(albatross is a bird) AND (birds are animals) => (albatross are animals)” the value of the SL depends on the probability that the model assigns to all these facts holding.

section 3: \mathcal{D}_c = {alpha_1, \dots, \alpha_m}, but the structure of \alpha is not clearly defined.

$\alpha$ refers to formulas like Eq. (Imp), (Neg), (F-Imp) and (2) in Section 2. We had mentioned this in line 186. We have made this more explicit.

z ~p_\theta(z) is confusing.

The individual probabilities $p_\theta(z)$ are obtained using Eq. (1). We have added a backref in the text.

We stress that in LoCo-LMs the circuit computes the probability that a constraint holds for the model (i.e., that expectation) exactly, no sampling is required.

the only baseline is ConCord.

We found only ConCord as a sensible baseline and we point out that maieutic prompting is just a variant of ConCord. In fact, maieutic prompting implements the same algorithm: exactly like ConCord, it extracts raw fact probabilities from the LLM and refines them so as to be as consistent to each other as possible using a MAX-SAT solver. We have clarified this in line 317. Furthermore, besides ConCord, we compare also against Chain-of-Thought (Section 5.2).

what are the examples in the few-shot examples?

Thanks for pointing this out, we added them to Appendix F.3.

For ConCord, it seems that the authors use ROBERTA-ANLI as an inference model. [...] this is unfair towards ConCORD. Do the two methods use the same models and same constraints?

We use ConCORD as originally proposed, and we use the same MACAW models. Hard constraints are enforced by a MaxSAT solver, therefore they are guaranteed to hold ultimately.

The difference is that ConCORD uses ROBERTA-ANLI to propose facts to ground the constraints, while we just maximise the probability of the constraints given the data. At inference time, LoCo-LMs do not guarantee that constraints are satisfied.

Therefore, one could use LoCo-LMs as a loss at training time and then combine it with a MaxSAT solver at inference time.

The difference in performace reflects a difference in approach: ConCORD attempts to rectify post-hoc the predictions of a potentially non-factual, inconsistent model, but this at best can help with self-consistency, not with factuality. LoCo-LMs on the other hand make the model and its answers both more self-consistent and factual.

*Line 192: the authors claim that they expect transfer from albatross to cockerel since they are similar - but there is no definition of what is similarity, and how should the model know when things are similar enough to conclude new facts about entities and when not.

The idea is that entities that map to similar embeddings (via, e.g., cosine similarity) will yield similar activations. Hence, applying the SL to one entity is likely to yield benefits for entities similar to it. Empirically, this is what happens in our tests. We were careful not to claim this will happen with guarantees

Line 469 - where are the resutls? are they in Table 3?

Correct, we amended the text.

We thank the reviewer for the feedback and for appreciating that our work is novel and interesting, that our approach can handle complex constraints and is empirically promising. We address below all the concerns they raised.

other components, such as circuits and sentential decision diagrams, are not discussed in detail

Thank you for pointing this out, we (wrongly) assumed that the theory of knowledge compilation was consolidated in the neuro-symbolic community. We introduced a new Appendix A to revise the background on circuits and compilation.

Note that for our experiments, we use standard compilation tools from the knowledge compilation literature to obtain a circuit starting from a propositional logical formula in conjunctive normal form. Specifically, we use PySDD2 [x], a python compiler that converts logical formulas into Sentential Decision Diagrams (SDDs) [y, z]. For example, given a formula such as (albatross => bird), the compiler instantiates two nodes for each variable encoding, in this case, whether albatross holds, albatross does not hold, bird holds, and bird does not hold, respectively. These nodes store the probabilities of these events. The compiler then adds sum and product nodes – which, very intuitively, compute the sum and product of their inputs – to the SDD, which is structured such that bottom-up evaluation of the circuit yields the probability that the formula holds given the probabilities of the input events. A more detailed step-by-step example is shown in Appendix A.

[x] (2017). Pysdd. In Recent Trends in Knowledge Compilation, Report from Dagstuhl Seminar 17381.

[y] Choi, A. and Darwiche, A. (2013). Dynamic minimization of sentential decision diagrams. AAAI.

[z] Darwiche, A. (2011). SDD: A new canonical representation of propositional knowledge bases. IJCAI.

4 tokens max; unclear how well the proposed method supports generating longer [...] responses.

We would like to point out that there is a difference between the number of logical variables appearing in a formula and the number of tokens produced by the model.

The former ranges from a minimum of 1 to $2^D$ where $D$ is the depth of the implication trees in EntailmentBank [Dalvi et al., 2022]. Please see Figure 2 in the Appendix for one example.

The number of tokens used to evaluate the probability of every fact is instead 1. See also our prompts in Appendix F.

[Dalvi et al., 2022] Dalvi et al. Explaining answers with entailment trees. EMNLP 2022.

do the scores of baselines in Tables 1 and 2 improve with greedy decoding?

In Tables 1 and 2, we used the default decoding strategy for Llama. We re-run our evaluation on LoCo-SUPER using greedy decoding, and found that performance is essentially the same. We reported the scores for all constraints in Table 18 in the Appendix.

We thank the reviewer for the feedback and for appreciating that our work is novel and that our approach is flexible, empirically promising and significant for applications. We answer below all the questions they asked.

the paper could improve by incorporating evaluations on multi-hop reasoning tasks / more sophisticated logical constraints.

We remark we already used constraints involving more than one implication: This analysis can be found in Section 5.3, where we evaluated LoCo-LMs on EntailmentBank. This consists of entailment trees involving multiple inference steps across multiple entities/logical variables; see Appendix D, Figure 2 for a visualization for an implication tree. The number of steps ranges from 1 to 5, see Figure 2 of [Dalvi et al., 2022] for the precise distribution.

We have made sure to clarify this point at the beginning of Section 5.4. We are happy to discuss this further.

[Dalvi et al., 2022] Dalvi et al. Explaining answers with entailment trees. EMNLP 2022.

Adding downstream tasks (e.g, question answering, reading comprehension, mathematical reasoning, etc.) [...] to assess whether the proposed fine-tuning approach [...] maintains the language capabilities of the original model.

This is a good idea! Unfortunately, our computational resources are limited and we might not be able to provide results for these additional tasks during the discussion period. We will try our best to integrate a such an evaluation.

improves generalization only within the same type of constraints, and it even hurts performance when the constraints differ

We remark this is expected and common when doing multi-objective optimization. Optimizing one constraint might not always benefit all others as much as it benefits its own kind.

Note that, however, the great majority are cases of positive transfer, i.e., optimizing for one constraint also benefits others. For example, optimizing for NEG improves all columns of Table 2 wrt the baseline (C-FAC: +19%, C-IMP: +20%, C-REV: +42%, SC-REV: +35%) but self-consistency IMP, and optimizing F-IMP only degrades self-consistency REV and NEG (C-FAC: +74%, C-REV: +8%), as it rightly does not consider negation, while delivering much better performance over all cases than using XENT. We will highlight these relative improvements in the Table for the camera-ready version.

We have integrated this discussion in the paper in Section 5.2.

This limitation could be especially pronounced in smaller models, so testing on larger models [is warranted] (e.g., llama 2 vs llama 3).

In Table 2 we also report the performance of Llama3.1 8B and show that, while the model is supposed to be more capable at reasoning, it falls short on BeliefBank as Llama2 7B with Few Shot. We are in the process of running the finetuning of LoCo-Llama3.1 and expect it to show similar improvements under different constraints.

gains in consistency might be partially due to prompt selection -> test other prompts or prompting methods

This is a good question. We note that we already performed experiments on two prompts (as shown in Appendix F). We have now extended our analysis to two more prompts (using as syntactic variations correct/incorrect and right/wrong).

We can observe that the performance we previously reported remains stable and our claims still hold: LoCo-LMs improve consistency for the different constraints even on alternative prompts.