From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question-Answering

Thank you for highlighting the innovativeness of our distillation method and the comprehensiveness of our evaluation!

1. Dependence on Training Questions and Answer Options:

This dependency restricts its applicability to new corpora that lack such structured data, making the approach challenging to transfer across domains or to corpora where questions and answers aren't predefined.

In fact, our method could also be easily applied to a corpus without predefined QA pairs, as numerous methods exist to generate QA pairs from sentences. In this case, those QA pairs would similarly drive the LLM to articulate underlying facts required to prove the answers, from which microtheories can be distilled. We will note this in the paper. It’s also worth noting that almost all learning systems require training QA data. In this sense, this work is no different from those learning systems (and perhaps even more flexible, as such training data can be automatically generated from a corpus).

2. Clarifications on filtering process

The paper could indeed improve by including more details on the filtering process in Section 3.2.2. Specifically, it doesn’t clarify how many microtheories were ultimately filtered out after applying the redundancy reduction techniques.

To clarify our terminology: in our paper, a “microtheory” is a collection of facts (not a single fact). Our filtering techniques take a large microtheory and filter it down to a smaller microtheory of the most powerful facts. Again, to clarify: a “10-Mt” is a 10-fact microtheory filtered from a larger pool, not 10 microtheories. The distillation process doesn’t filter out microtheories; rather it filters out candidate sentences for the microtheory that is being distilled.

Regarding how many of those facts get filtered out: Figure 3 might help answer your question. The reduction techniques in S3.2.2 filter the raw fact pool $\mathcal{F}$ down to condensed pool $\mathcal{C}$ (see line 185). As we show in Figure 3, the size of the initial fact pools, $\mathcal{F}$ is 20K facts for ARC and 11K for MedQA, while the size of the condensed fact pools (from which a microtheory is then extracted), $\mathcal{C}$, is 8.7K for ARC and 8.1K for MedQA. In other words, the redundancy reduction techniques filter around 55% of the ARC facts and 25% of the MedQA facts. We have added this to our updated pdf.

Please let us know if there is anything else about the filtering process that is unclear to you.

3. Performance drop for smaller microtheories

In Figure 7, adding 10 random microtheories to the base corpus could make the performance drops a lot, could you please explain it?

Again just to be clear, Mt-10 is a single microtheory of 10 sentences, not 10 microtheories. Indeed if we just select 10 sentences at random for a microtheory, they are unlikely to be helpful for test questions (and may even confuse the entailment engine, which relies on the facts to guide its search) given the sentence pool contains irrelevant and idiosyncratic facts. Hence the low score for Rd (random). This is precisely the problem that the distillation process is intended to solve, namely distilling a small set of relevant, useful facts. The higher scores for the three ways of distilling facts (Us, QC, PC) suggest this has been achieved.

We have added a line to S5.2 in our updated pdf explaining that this happened.

5. Comparison of multiple models on the same dataset

Although the paper claims to generate microtheory from both GPT4 and Mixtral to show that their method not only works on closed models, there's no comparison of performance and discussion on how different models may affect microtheory generation. The paper also does not explain why they use GPT4 for ARC and Mixtral for MedQA.

Thank you for pointing this out. We’ve updated the paper to include using GPT-4 for MedQA so that the results between models are directly comparable. We find building and using Mts with Mixtral to be as effective as using GPT4, in terms of grounding and relevance, showing our method is not restricted to GPT4 only. We also re-ran our human evaluation using the GPT4-generated microtheories.

6. Clarifications of training scenario

I am confused about the "training questions", do you mean questions in the original training set of ARC and MedQA?

Almost, except we created our own train/test split for topical subsets of ARC and MedQA (see lines 258–264 and Appendix B). This was because we wanted train/test questions about the same topic (e.g. all the ARC questions that target knowledge of mechanical physics), so that the test set is a proper evaluation of whether the Mt has captured knowledge about that topic.

Also, the word "training" appeared in this paper so many times in different contexts, "training questions" "training hypothesis" "training set" "training proofs"... but it seems like you are not training the QA module at all?

Correct, we are not training the QA module directly, e.g. modifying parameters of a neural net. However, there is some learning going on: The performance of the (fixed) QA module depends on the input knowledge (e.g., Wikipedia, a microtheory) that it takes as input, and hence by improving those resources (i.e., building a good microtheory), the performance of the overall system improves (grounding, QA). Hence, the train/test paradigm: training is used to help build the knowledge resource, and a test set is used to evaluate how successful that has been.

Thank you for your careful analysis of our experiments! We appreciate you highlighting the importance of our considered problem.

1. Entailment Engine I/O

The format of the input to the entailment engine is unclear. The engine takes in a mixture of original corpus such as Wikipedia and microtheories, but how these external information is structured is not clear.

The input to the entailment engine is a flat index of documents, where each document is typically one or several sentences, and a hypothesis to prove. The output is an entailment tree showing how the hypothesis follows from a subset of those documents via entailment (or null, if no tree can be found).

In our case, we break the Wikipedia corpus into paragraph-sized chunks (“documents”) – specifically, the 2021-01-20 version of Wikipedia with standard 100-word chunks. The microtheory is structured similarly, as a set of individual fact statements (each sentence is a “document”). Until they are used to compose new entailment trees at test time, the individual statements in the microtheory are not tied together in any way; it is just a large list of statements.

When the search is given both the microtheory and Wikipedia, we concatenate the results in that order– microtheory statements first. The search then uses the concatenated set for entailment lookups and hypothesis decomposition. We’ve updated the appendix to include this information.

2. Fact Usage Rates

5000/8000 of facts for both datasets are never used in the proof.

To clarify: In the raw fact pool, before we start distilling any microtheory, only 3000 of 8000 (i.e., about 40%) statements are used in the training proofs. This highlights the problem that our method aims to solve: simply gathering relevant facts produces a significant amount of unnecessary, duplicate, and/or unhelpful information. In contrast, essentially all (~100%) sentences in the distilled microtheory play a role in proving the training questions by virtue of the microtheory construction process, reflecting the Mt’s concentration of core, important facts, as well as helping ground test questions more effectively. This is a key contribution of the work.

3. Whether the findings support that the microtheory selection finds the most relevant facts

The fact that only adding more than 1000 facts can increase QA performance shows that current microtheory selection is unable to get the most relevant facts.

Note that our primary goal is to distill core, relevant facts that can help ground answers in multiple settings while not hurting QA performance (and, of course, any performance gains are an added bonus). We do see evidence that this distillation goal has been achieved even for the small Mts in multiple experiments. For example, for the 100-Mts and 500-Mts, Figure 6 (S5.1) shows that they provide more hypothesis grounding over the random baselines of the same size. Figure 8 (S5.3.1) shows that they are more relevant under expert human evaluation. We've also updated Figure 9 (S5.3.2.) to show the difference in "at least one relevant fact" rate between a distillation method and the random baseline. This suggests the more relevant facts have been distilled for the smaller sizes. Note that there is no guarantee that a good Mt will improve QA, as the entailment engine may then just ground to the large corpus instead (Wikipedia). But this does nothing to distill the LLM’s core knowledge about the topic, which is the goal of this work.

4. Discussion of required compute

the compute needed to retrieve a large enough fact corpus to improve QA performance, and the trade off between selection comprehensiveness and QA performance.

Yes, this is important. This is what motivated our new metric, $p$-relevance, in Section 6.

For a fact corpus to be relevant to a given question, it should contain at least one fact that is critical to answering the question correctly. We found in S5.3.2 that the rate at which the microtheory corpus contains at least one critical fact per question is proportional to the raw size of the corpus, hence there is a clear trade-off between storage and coverage. You’re right that compute is another critical consideration, since creating the distilled microtheories requires running the entailment engine on each training question, which can be costly. We’ve added language to section 6 discussing this.

We also found that even the raw fact pool F doesn’t contain at least one relevant fact for 100% of the questions. The $p$-relevance metric introduced in S6 measures how many training questions we’d need to extract facts for - hence the additional compute needed - in order to be relevant to questions some $p\%$ of the time, hence helping characterize the compute required.

Thank you for your careful review, highlighting the novelty of our generation method and the comprehensiveness of our evaluation!

1. Differences with Chain-of-Thought

what's the inherent difference of your method compared to chain/tree/graph-of-thought, since you are both distilling knowledge from LLM as intermediate steps

How do you detach the improvements brought by microtheory and that possibly brought by seemingly useful LLM-generated chain-of-thought?

The goals of chain-of-thought and of microtheories are quite different. Chain/tree/graph-of-thought methods show transient fragments of knowledge for individual questions, but they don’t reveal the "big picture": a single, concise store of discrete knowledge that is generalizable as a source of argument grounding for multiple questions in a topic or dataset (see Figure 1). Our goal is to attempt to build this store: e.g., what is (say) GPT-4’s core knowledge about physics or medicine? This is a non-trivial task since just unioning many machine-generated sentences together results in duplicates, near-duplicates, over-generalizations, under-generalizations, unhelpful idiosyncratic facts, etc.

So, the heart of our work is a careful distillation process to condense these into a parsimonious, most-useful theory (which you have noted in your Strength #2). This is the first time such theories have been generated from LMs, allowing people to see the LM's underlying "picture" and improving grounding and accuracy.To our knowledge, this is the first system to achieve this.

In addition, the context of our work is to produce well-defined entailment arguments (“proofs”) that go from facts in an inspectable corpus (e.g., a microtheory, Wikipedia) to an answer, i.e., grounded proofs. This is different from chain/tree/graph of thought, which generates informal arguments supporting an answer and which are ungrounded (i.e., not connected to an external corpus).

2. Regarding a formal definition of textual entailment

Line 74: The entailment in NLP typically is typically referred to as textual entailment, which surely has a formal definition.

In fact, entailment doesn’t have a fully formal definition in the literature due to its reference to human understanding. The standard definition used by the NLP community, which we adopt here also, is:

“We say that T entails H if the meaning of H can be inferred from the meaning of T, as would typically be interpreted by people. This somewhat informal definition is based on (and assumes) common human understanding of language as well as common background knowledge” – Dagan et al (2005)

Despite this semi-formative approach, recent works have produced reasonably reliable entailment engines from large amounts of training data, including the off-the-shelf one that we used in this work.

3. Grounding Metric

How is the grounding metric in section 5.1 calculated in detail?

We take a dataset of questions and measure the fraction of correct answers that can be grounded, i.e., “proved” via entailment from sentences in a discrete, authoritative corpus. We have added a clarification for this in S5.1 of the new pdf.

The engine converts each correct answer to a hypothesis and runs an entailment tree search that returns {grounded} or {not grounded}. In the {grounded} case, it also returns one or more entailment trees. The leaves of the trees are either facts in the base corpus, or facts from the microtheory. The solid-colored bars in Figure 6 show the fraction of all tree leaves that came from the microtheory, while the striped-colored bars show the fraction that came from the base corpus.

If the prover finds multiple trees, we count the leaves in the one with the highest fraction of microtheory leaves (footnote 8).