Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving

We thank the reviewer for their detailed review.

Multiple skills per question

The reviewer has asked for preliminary experiments showcasing multiple skill labels. We refer the reviewer to the common rebuttal for this experiment. We show decent improvements on the MATH dataset with a multi-skill approach.

Cases where categorization of skills is difficult

The reviewer has questioned how the proposed approach would work for cases where skill categorization is difficult. We address this by showing that the proposed approach can be utilized for skill labeling in many different domains beyond MATH. In the common rebuttal, we have used the proposed approach can be used to assign skill labels to a variety of domains spanned by the alpaca, just-eval, hh-rlhf datasets such as journalism, art, literature, coding, engineering, etc. Therefore, we can conclude that as long as a given domain is represented in the LLM training data, we can obtain decent skills for that domain.

On curation of the labelled dataset being resource intensive

The reviewer has mentioned that it may be very resource intensive to create the skill exemplar repository. We find that the computational complexity of creating and storing the repository is O(N) each where N is the size of the dataset since it requires one pass over the dataset. This is not very intensive as it is linear in the dataset size.

On the flexibility or generalizability of the labeled skills

The reviewer has pointed that given the skill exemplar repository, it may be difficult to generalize to tasks that require different capabilities from those represented in the repository. We address this by pointing to experiments which highlight the generalizability and flexibility of the skill exemplar repository.

1. In Table 6, we have transferred the skills derived from the GSM8K dataset to 6 other datasets which encompass various problem types and formats that are not present in the original GSM8K dataset. For example, SVAMP [1] presents problems with certain linguistic variations which are specifically designed to confuse the LLMs. Such kind of confusing problems are not present in the GSM8K datasets.
2. In Table 4, we have shown that even though the skill exemplar repository does not contain any code generation examples, sourcing skill-based exemplars from it improves code generation performance of LLMs.

On complex hierarchical skill definitions

The reviewer has asked how the proposed approach can handle hiearchical skill definitions such as the compositions mentioned in [1].

Handling of skill hierarchies is a complicated problem therefore it is beyond the scope of the current paper. At the moment, our framework supports labeling questions with single and multiple skills as we have shown in this rebuttal. As future work, we hope to extend support to skill hierarchies.

[1] A theory of Emergence of Complex Skills in Language Models

On Skill-Mix paper

We have cited skill-mix as citation number [31] in our work. Skill-mix introduces an evaluation benchmark which evaluates the ability of a LLM to generate text consisting of K skills. On the other hand, this work focuses on extracting 1 or more mathematical skills from text and using them to improve mathematical reasoning in LLMs.

Figure 1 uses two alternative notations

Skill 2 refers to the pre-clustering skill and Skill II refers to the post-clustering skill. The diagram indicates that the 5 pre-clustering skills were clustered to 3 aggregate skills.

How have you determined how many coarse-grained categories to have for the skills — and how do more or less skill categories affect performance?

There are no hyperparameters used in the skill labelling, skill clustering, or the skill relabelling phases. We only use the prompts given in App. Figure 2 and let the LLM decide the granularity of skills and the number of clusters.

Why are results not shown separately for CoT in Table 1? Could CoT not be applied separately to these kinds of questions.

CoT can be applied separately to the question in MATH. However, we took the CoT result from [1] which does not report the per-topic performance of CoT.

[1] Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models https://arxiv.org/abs/2308.00304

How would the results be (let’s say in Table 7), if we identify multiple skills as being relevant to the question at hand [instead of just one].

We have presented the modified solutions when using multiple skill labels per example in the pdf provided in the common rebuttal. Both questions are now solved correctly.

it is a bit of a stretch to call the ability to categorize as a metacognitive ability of the LLM

We appreciate the reviewer's succinct definition of metacognition, which corresponds with our understanding. The comment sparked a discussion among authors, leading us to mostly agree with the reviewer's "stretch" comment. While metacognitive knowledge can be in many forms, in this work we are interested in the specific knowledge about the catalog of skills required to solve math problems. We think of skills as pointers to knowledge, not the knowledge itself. We could switch to using the more generic term 'metaknowledge'. We welcome the reviewer's further thoughts on naming.

We find that LLMs can categorize mathematical data into skills and use them to improve their own reasoning. That being said, the proposed approach cannot extract the various other forms of metacognitive knowledge as of now and we hope to explore that in future work.

Metacognitive knowledge vs theory of mind

Metacognition involves awareness and control over one’s own cognitive processes, while theory of mind involves understanding others' thoughts and beliefs. Our work focuses on metacognitive knowledge, specifically the ability for an LLM to recognize, label, and apply appropriate skills to tasks to improve its own reasoning.

Dear Reviewer,

We thank the reviewer for acknowledging our rebuttal.

The reviewer had also asked us to perform an experiment during the rebuttal stage to extend the proposed methodology to multiple skills per datapoint. We have extended the framework proposed in the paper to label each datapoint with multiple skills. We have shown that this approach indeed leads to impressive performance gains across the MATH dataset (see common rebuttal).

The reviewer also expressed concerns about the difficulty of categorization of skills in different tasks. We have shown in the common rebuttal that the proposed categorization of skills can be obtained for many domains beyond math such as literature, journalism, economics, ethics etc.

We wanted to ask the reviewer whether they feel our new experimental results have addressed the concerns they expressed to us, and if not, can they give us guidance for future versions of the paper?

We hope that the reviewer will account for the new experiment while deciding the final rating for the paper.

We thank the reviewer for their detailed review.

Comparison to RAG

The reviewer has mentioned that the proposed method is similar to RAG. While we agree that the proposed approach has a similar flavor as RAG and can be considered as one instatiation of RAG, we would like to point that there is one major difference:

• In RAG, the current query is mapped to an embedding space and then similar documents or in-context examples are fecthed based on a similarity metric in embedding space.
• In contrast, the approach presented here makes explicit the skill or concept required to solve the question (which we refer to as metacognitive knowledge) by labelling the question with a skill name. Then, similar in-context examples are fetched based on the skill name.

Moreover, we argue that fetching relevant in-context examples based on embedding similarity may not be ideal as 1) these methods usually use an aggregated embedding of the query which may capture unwanted information which may hinder its ability to fetch truly relevant examples, and 2) Computing the embedding similarity of a given query with all documents during inference is very expensive. On the other hand, fetching examples based on metacognitive knowledge such as skill name always ensures that the fetched in-context examples demonstrates the primary skill required to solve the question. Also, during inference our approach requires constant lookup time since we organize our skill exemplar repository as a dictionary or a hashtable with skill name as the hash function.

To further illustrate our methods effectiveness over RAG, we perform comparisons to methods which fetch relevant in-context examples based on similarity in the embedding space similar to RAG.

We consider the following baselines:

• Embedding Similarity - This is a baseline proposed by us. We retrieve in-context examples based on similarity to the query in the BERT embedding space.
• Retrival RSD [1] - They also have a notion of skills like we do. However, their skills are represented as embeddings. During inference, a query is mapped to the skill embedding space and in-context examples corresponding to the closest skill in the skill embedding space based on euclidean distance are fetched.
• Skill-KNN [2] - This approach also has a notion of skills. For inference, they map a query to a set of skill decription which describe the competencies required for the query. The in-context examples are fetched from an example bank based on embedding similarity to the skill descriptions.

Therefore, all these methods fetch relevant example based on some embedding similarity metric.

We compare the proposed approach with the first two methods on the GSM8K dataset. We use the GPT-3.5-turbo backbone for this experiment which is the backbone used in [1]. For all models, we use 4 in-context examples. Here are the results -

We can see that the proposed skill-based approach outperforms both the embedding similarity based baselines.

For comparison to the Skill-KNN [2] baseline, we refer the reader to the common rebuttal.

[1] Xu, Zifan, et al. "Latent Skill Discovery for Chain-of-Thought Reasoning." arXiv preprint arXiv:2312.04684 (2023).

[2] An, Shengnan, et al. "Skill-Based Few-Shot Selection for In-Context Learning." arXiv preprint arXiv:2305.14210 (2023).

Do LLMs have metacognitive ability?

The reviewer states that we have not discussed 'metacognitivity' extensively. We agree that more clarity and precision is necessary in the introduction to advance the notion of metacognitive knowledge. Metacognitive knowledge refers to knowledge about one's own cognitive process. While this knowledge can be in many forms, in this work we are interested in the specific knowledge about the catalog of skills required to solve math problems. We think of skills as pointers to knowledge, not the knowledge itself. By analogy, a student may know where in the library to find a book that will answer a question, without knowing what the answer is.

We find that LLMs can categorize mathematical data into skills and use them to improve their own reasoning. That being said, the proposed approach cannot extract the various other forms of metacognitive knowledge as of now and we hope to explore that in future work. Moreover, we have also shown from discussion in App. Section 9.4 that we need strong LLMs such as GPT-3.5 or GPT-4 to extract this kind of knowledge. But once extracted, this knowledge can improve reasoning of various LLMs including much smaller LLMs like Mixtral 8x7B.

On alternative experiment

The reviewer has suggested an alternative experiment for evaluation of metacognitive abilities. The reviewer suggests to ask the LLM to recall similar examples of a particular skill and use those to solve the question. We thank the reviewer for this very interesting suggestion. First, we would like to clarify that the reviewer is asking for the LLM to retrieve questions and answers from its own memory i.e. to generate questions and answers pertaining to a skill.

If we are right about this interpretation of the reviewers question, we would like to point that generating sound questions and answers for a particular skill can be very challenging and often requires meticulously engineered prompts [1].

Hence, while we would be very curious to try this approach, we think it would require more time than the current length of the rebuttal period.

[1] AI-Assisted Generation of Difficult Math Questions https://arxiv.org/abs/2407.21009

We would also like to direct the reviewer to the general response, which includes additional experiments demonstrating how we extend the proposed framework to assign multiple skills per question and also apply it to other domains.

"For this paper, the process of extracting skills and recall examples are coded and defined outside of the LLM model itself."

If by outside the LLM, the reviewer means that skill extraction is done as a separate process different from inference, then the reviewer is right. However, we would like to clarify that the process extracting skill is done entirely using a LLM and not outside it. The skill labels come from the LLM_A and the clustering is also done by the LLM_A (and not by any external clustering procedure). These skills capture useful aspects of problem solving as evidenced by the improved performance in in-context mathematical reasoning for various other LLMs (LLM-B). We are only suggesting that LLM_A has some metacognitive capability.

We interpret your remark that "Your experiments shows, with metacognitive, LLM can perform better, but it's insufficient to prove LLM already have metacognitive ability." as being similar to remarks raised by reviewer ADDk, who stated that metacognition can come in many forms most traditionally having to do with a model's ability to reason about its own cognitive processes. The particular kind of metacognition that we explore is the knowledge about how domain-specific skills are labeled and grouped. Perhaps you are guiding us to avoid making broader claims about LLMs and metacognition, which is a fair point, and we will incorporate into the next draft of our manuscript.

We thank the reviewer for their detailed review and appreciating the novelty of our method.

On the hyperparameters required for skill clustering and analysis of granularity of skill names

There are no hyperparameters used in the skill labelling, skill clustering, or the skill relabelling phases. We only use the prompts given in App. Figure 2 and let the LLM decide the granularity of skills and the number of clusters.

Does variations in QA pair across different datasets hinder generalization

The reviewer has asked whether presence of a particular type of questions in the skill exemplar repository would hinder its generalization to other types of examples and also what happens if we apply the skill exemplar repository of MATH to GSM8K.

We would like point that we have already done the following experiments which demonstrate the generalizability and flexibility of the skill exemplar repository.

1. In Table 6, we have transferred the skills derived from the GSM8K dataset to 6 other datasets which encompass various problem types and formats that are not present in the original GSM8K dataset. For example, SVAMP [1] presents problem with certain linguistic variations which are specifically designed to confuse the LLMs. Such kind of confusing problems are not present in the GSM8K datasets. Similarly each of the 5 other datasets also contain distinct problem which may not be in the GSM8k dataset. Across all these datasets, we show improved performance by using skills derived from GSM8K.
2. In Table 4, we have shown that even though the skill exemplar repository does not contain any code generation examples, sourcing skill-based examplars from it improves code generation performance of LLMs. Therefore, we have shown that applying the exemplars to a completely different domain of code generation also works well.
3. Furthermore, we applied the skill exemplar repository from MATH dataset to GSM8K and found that it achieved a performance of 94.76% which is slightly higher than the 94.31% achieved by using the skill exemplars sourced from the GSM8K dataset. This slightly higher performance can be attributed to the higher complexity and more detailed reasoning traces of the MATH dataset.

[1] Are NLP Models really able to Solve Simple Math Word Problems? https://arxiv.org/abs/2103.07191

We would also like to direct the reviewer to the general response, which includes additional experiments demonstrating how we extend the proposed framework to assign multiple skills per question and also apply it to other domains.

We thank the reviewer for their detailed review.

On Assigning multiple skills per question

We would like to point the reviewer to the common rebuttal for the experiment on multiple skills per question.

Discussion on overfitting

The reviewer has raised the question of overfitting on the training data when creating the skill exemplar repository. We thank the reviewer for raising the question and we believe that it is an interesting point.

Firstly, we would like to point that overfitting here is different from the general notion of overfitting in machine learning. In machine learning, overfitting is usually considered in the context of updating parameters when the model's training loss goes down while the validation loss goes up often as a consequence of the model memorizing training data points.

Here, we are operating in a different setting, where we do not update any parameters. In this case, we could extend the definition of overfitting to mean multiple things:

• Given a skill-exemplar repository curated using exemplars in a dataset D, the in-context examples may not lead to good performance on another dataset D'.
  • In table 6, we have shown that skills curated from GSM8K dataset transfer to 6 other datasets each comprising of various different types of questions than seen in GSM8K data set.
• Given a skill exemplar repository containing examples from domain D, the examples don't help in domain D'. For example, text-based examples do not help in generating code-based solutions.
  • We have already shown in Table 4, that providing skill-based in-context examples which do not include any code also improves the code generation ability of LLMs.

Therefore, we do not find any evidence as of now that the approach is overfitting to the exemplars represented in the skill exemplar repository.

On the method being domain-agnostic

We would like to point the reviewer to the common rebuttal for the experiments in different domains.

On comparison with ToRA

The reviewer has asked for discusion and comparison to ToRA [1]. ToRA is an LLM which has been trained to solve questions through tool use.

The methodology proposed in this paper is agnostic to and independent of any improvements in the core capabilities of the underlying LLMs. We have shown in Table 4 that the proposed methodology improves code generation in GPT-4 even though the skill exemplars contain only text. Code generation is also one of the main tool use capabilities of ToRA. Therefore, we can assume that the proposed methodology would also improve reasoning in ToRA.

[1] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

On code release

We apologise for not providing code with the submission. Neurips submission does not allow to update the submission, hence we will release it after the review cycle.

On multiple skills and whether only single skill assignment inhibit the model to generalize

We have presented an experiment where we have shown that multi-skill labeling improves performance over single skill labeling.

However, we would like to point that labeling with single skills is not always prohibitive. First, we have seen that the proposed framework is not always restricted to single skill assignments as some skill names can comprise multiple skills, e.g., counting_and_number_theory. Second, from the plots presented in Appendix Fig. 4, we have shown that providing examples based on a primary skill improves success rates on secondary skills which may not be represented in the in-context examples.

Therefore, the current framework does not necessarily lead to bad performance in cases where multiple skills are required. Of course, there are some cases which we have highlighted in Appendix Table 13 where having multiple skill annotations per data point would be useful (as we have already shown in the common pdf). But, in general we find that the proposed methodology of labelling questions with 1 skill is powerful and leads to strong performance. For example, in Table 7, second column, the question also requires the skill "celing_function" but our methodology does not identify that skill for the question but nonetheless the LLM can solve it correctly.

Effect of size of dataset and computational complexity

The computationally heavy part of the proposed methodology are the following 3 steps:

1. Skill labelling: This step requires one pass over the entire dataset. Therefore its computational complexity is O(N) where N is the size of the dataset.
2. Skill Clustering: This step takes constant time where the LLM is given all the skills identified in the labelling step and it is asked to output a clustering of the provided skills. Therefore, its computational complexity is O(1).
3. Skill Relabelling: This step also requires one pass through the dataset to relabel dataset examples with one of the clustered skills. Hence its computational complexity is O(N).

Therefore, the total computational complexity for creating the skill exemplar repository is O(N) in theory. In practice one issue with increasing dataset size is the increase in the number of unique skills in the first phase (skill labeling). This can be problematic when clustering examples using an LLM as an LLM has finite context length. Therefore, one optimization that we are working towards as future work is to have a clustering method which does not use LLMs or is not affected by its limited context length. We don't have any concrete solutions for this yet. During inference, the retrieval of relevant in-context examples is implemented using a dictionary or hashtable which takes constant lookup time hence the complexity during inference is O(1).