Explaining the Complex Task Reasoning of Large Language Models with Template-Content Structure

Thank you for your thoughtful and comprehensive review. Your feedback provides us with clear direction on how to improve our article. We summarize your concerns and suggestions as well as our responses as follows：

1. "The authors frequently relegate crucial details to the appendix, disrupting the continuity of the reading experience."

   Thank you for your valuable feedback. In our endeavor to present a completely novel perspective on LLMS, we encountered the challenge of incorporating motivation, examples, mathematical deduction, and experimental evidence within the confines of a succinct 9-page document. This limitation unavoidably constrained our capacity to delve into comprehensive details in the main body of the text. To facilitate readers in quickly comprehending our t-c structure, we gave precedence to elucidating the motivation and intuition. As a result, more intricate details were placed in the appendix, catering to readers with specific interests.

   In the final version of our paper, we intend to refine the organization further. We are particularly grateful for any feedback, and we highly value your insights regarding the elements that should receive greater emphasis in the main text. Your suggestions are instrumental in enhancing the quality of our work.

2. Related work

   "Related papers:

   Explaining Agent Behavior with Large Language Models - Link: https://arxiv.org/pdf/2309.10346.pdf

   iPrompt: Explaining Data Patterns in Natural Language via Interpretable Autoprompting - Link: https://arxiv.org/pdf/2210.01848.pdf"

   Thank you for your kind reminder. These papers indeed offer a valuable addition to our current section on related work. Contrary to the works we have cited that focus on elucidating LLMs themselves, these studies explore a distinct facet of LLM interpretability: employing LLMs to explain external systems. We will incorporate these references into the related work section of our final version.

   • The first paper "Explaining Agent Behavior with Large Language Models" introduces a methodology for generating model-agnostic explanations for agent behaviors using LLMs. This approach involves translating an agent's decision-making process into a decision tree, which is then inputted into an LLM to produce explanations. Their crowd experiments show that LLMS generate better explanations than previous work.
   • The second work "iPrompt" propose to find the pattern in the dataset, i.e. utilizing LLMs to elucidate the dataset itself. This method involves feeding "input-output" pairs ${(x_1,y_1),\dots,(x_n,y_n)}$ into an LLM, resulting in the generation of semantic explanations for the relationships between $x$ and $y$ in natural language. The explanations thus generated have been highly rated in terms of human evaluative metrics.

   Both works show that LLMs can summarize and explain the input, including dataset patterns and the output of external systems. It suggests that LLMs learn a "concept" or "task" that is more abstract than specific data, which is evidence for our point.

Dear Reviewer KnhL,

We trust this message finds you in good health. The author response period for our submission has been extended until the end of December 1st, due to the paper not having received three reviews yet.

As we are nearing the end of this extension period, we note with concern that we have not yet received your feedback. In preparation for the rebuttal, we have exerted substantial effort to enhance our submission, specifically focusing on:

1. Engaging in a thorough discussion of some pertinent related work.
2. Conducting a detailed analysis of the misclassification error in the t-c classification.
3. Making a comprehensive comparison between our t-c structure and other explanations in the field.

We would greatly appreciate it if you could inform us whether our revisions and responses have adequately addressed your concerns, or if you have any further questions or points for clarification.

Your time and dedication in reviewing our work are greatly valued and appreciated.

Sincerely,

Authors

3. Experimental results to show the importance of t-c decomposition

   There is no experimental result showing such decomposition benefits the tasks either in computation efforts or task performances.

   Thank you for your kind reminder. We are glad to explain our experimental results further and add some experimental evidence to show the importance of t-c decomposition.

   1. First, our experimental results in section 6 show the correlation between the capacity of LM to handle t-c structure with their reasoning performance, which means for current LMs, a significant t-c structure is a condition of complex reasoning ability. To show the relation more clearly, we choose the t-c classification AUC-ROC as the t-c classification accuracy. We also report the difference of the mean-variance (DMV) $\sum_{s\in C} V_{s} / N_C-\sum_{s\in T} V_{s} / N_T$ where $N_T$ ($N_C$) is the number of template (content) tokens, and $V_s$ is the variance of the output probability at position $s$. DMV can measure how confident the model is about their classification results because the larger the DMV is, the more significant the different behaviors of the model for T and C. We compare them to their complex reasoning (CoT) ability and accuracy on GSM8k as the measure of their complex reasoning ability.

      Model	DMV	AUC-ROC	CoT ability	Accuracy on GSM8k***
      GPT2-335m	0.15	0.87	$\times$	0.0
      GPT2-774m	0.28	0.87	$\times$	0.0
      GPT2-1.5b	0.27	0.97	$\times$	0.0
      OPT-1.3b	0.29	0.96	$\times$*	0.0
      OPT-13b	0.42	0.99	$\times$*	0.0
      OPT-30b	0.46	1.00	$\times$*	0.0
      Llama2-7b	0.72	0.99	$\checkmark$**	14.6
      Llama2-13b	0.80	1.00	$\checkmark$**	28.7
      Llama2-70b	0.81	1.00	$\checkmark$**	56.8

      * The OPT models have trouble following the instructions or simulating the examples of prompts; therefore, no CoT ability is shown. See [1] as a reference.

      ** Llama-2 models have CoT ability to do some complex reasoning, such as GSM8k or MATH. See [2] for reference.

      *** Test in the 8-shot CoT setting to align with our framework. The CoT accuracy of the GPT2 and OPT models without CoT ability is 0.

      The results show that the CoT ability, i.e., the complex reasoning ability, is available only for models that handle significant t-c structures. It shows that the t-c structure is the key to LLMs' complex reasoning (or CoT ability). For the model with less t-c classification ability, such as GPT2s, generating a reasonable CoT is too hard because just one wrong token will lead to failure under autoregressive generation, so the accuracy is 0.

   2. The t-c structure can help the reasoning performance of LMs. For example, the template-content structure can be used as data augmentation, where we can replace the content in the training data to instruct our LMs to learn the decomposition and better reasoning ability. Here, we collect 100 examples of the 'chicken and rabbit problem' to form a dataset. For each question, we collect five answers generated by GPT-4. We use these question-answer pairs to finetune a Llama2-7b model. As a comparison, we train another Llama2-7b model with the same dataset and other settings but randomly replacing the content (denoted as cr) in each question-answer pair as data augmentation. In order to ensure that our content replacement does not just mitigate overfitting due to the introduction of data perturbation, we also conducted experiments on models with general data augmentation. Here, we selected EDA[3] as the baseline, a data augmentation method based on synonym replacement, random word deletion, and addition. The efficacy of these approaches is depicted in our Figure, where the training steps are plotted along the x-axis, and the test accuracy (reported as pass@3) is represented on the y-axis. The results clearly demonstrate that content replacement enhances model performance. Specifically, the maximum performance increased from 75% (vanilla model, blue curve) to 85% (cr, orange curve), and from 85% (eda, green curve) to 95% (eda+cr, red curve) in the models incorporating cr augmentation. This experiment demonstrates that teaching the model the T-C structure through content replacement indeed helps the model to reason better, showcasing the importance of the T-C structure in reasoning.

Thank you for your insightful comments. We address your concerns as follows:

1. The importance of the existence of template-content decomposition:

   "It is not clear the implication of the theorem (the existence of template transformers and content transformers). The importance of the existence of template-content decomposition is unknown."

   The t-c structure leads to the reasoning ability of modern LLMs, and we prove it in proposition 1 (section 4.4). The skeleton of our paper is to prove that (1) the t-c structure leads to reasoning ability, i.e., generating answers for complex reasoning problems, and (2) the transformers can learn the t-c structure. Combining these two parts, we can explain the reasoning ability of LLMs. Below is a detailed explanation.

   Intuitively, one key point of complex reasoning lies in learning the correct combination of the multiple small steps, i.e., the skeleton of a solving process. This observation can be easily verified by examining answers generated by humans or LLMs. Taking math problems as an example, the process of human-generated answers typically consists of two parts: the overall problem-solving skeleton and the specific solution of each step. At the same time, powerful LLMs like chatGPT also show similar behavior (as shown in Figure 1). The skeleton determines the type of the problem and the solution method (such as a linear equation system composed of two variables in our example in Figure 1) while the specific steps perform concrete math operations (for example, determine the sum of two variables x + y is 35). In our t-c structure, the skeleton tokens are defined as template and the specific step tokens are defined as content. The template should stay the same once the problem type is determined, while the content can vary according to concrete problems.

   In Figure 1, we show such an important example to help understand the importance of the t-c structure. In the classic rabbit and chicken problem, even if we change rabbit and chicken to correct and wrong answers (and also change the number of feet to the number of marks), the LLM uses the same template (of linear equation system) to solve the problem. In other words, the template remains the same for all problems of the same type, while the content is different for each problem. Therefore, we emphasize that if an LLM can generalize on various problems of the same type, rather than just remembering individual data points that appear in the training set, it must be able to learn the template (invariant part) of the task. Otherwise, if we modify the problem slightly, it will fail. So, the existence of the t-c structure is the key to the reasoning ability of LLMs.

   Our theoretical analysis (sections 4.3 and 4.4) is to express the above intuition formally. Section 4.3 proves the existence of such a t-c transformer. Then, Proposition 1 (in section 4.4) shows the t-c transformers can generate the answer for a complex reasoning problem as long as they have seen a similar problem of the same task (even though its content may be completely different from the target problem), which paves the way for LLMs' generalizable reasoning ability.

   In summary, template-content decomposition is important because it explains where LLMs' strong reasoning ability comes from and shows that the transformer architecture can learn such a t-c structure. We illustrate it intuitively and theoretically in our paper. And we further verify it with additional experimental evidence in our response to the third point.

2. The complexity of the template (or content) transformers:

   "The template (and content) Transformers function could be much more complex than a single transformer model... Can $f_T$ and $f_C$ be more complex than a single transformer model? In other words, to decompose tasks that can be handled by a LLAMA 7B model, will the $f_T$ and $f_C$ be an LLAMA 70b function?"

   No. Suppose a model can indeed handle the task, which means that any changes to the content of the problem will not affect its solution process. In this case, this model has successfully learned the t-c structure because our t-c structure is precisely defined by the independence of the template from variations in content. That is, if a Llama 7B model handles a task, it must have achieved the decomposition within its 7B parameters---a small model that can solve a type of problem indicates that this type of problem is not complex.

4. Other task

   "Will such decomposition work for more general NLP tasks such as summarization, translation, or LLM benchmark tasks including HELM or lm-evaluation-harness?"

   Yes. While primarily demonstrated on complex reasoning tasks like arithmetic and symbolic reasoning, our template-content decomposition approach can apply to more conventional NLP tasks, provided these tasks incorporate relatively standardized processes. Take, for example, the task of summarizing a lengthy article. This task can be approached by initially extracting key information from each paragraph and then synthesizing these elements into a coherent summary. A possible template for this process might be: "The first paragraph discusses [...]. The second paragraph addresses [...]," and so forth. This method can be further granularized to summarizing individual sentences within a paragraph, using sub-templates like: "Based on the sentence <sentence>, it appears to focus on <subject>."

   Another example could be commonsense reasoning such as OpenbookQA[4] dataset. Here we show an answer generated by GPT-4.

   Question: Which of these would let the most heat travel through? A) a new pair of jeans. B) a steel spoon in a cafeteria. C) a cotton candy at a store. D) a calvin klein cotton hat.

   ChatGPT:

   To determine which of these items would allow the most heat to travel through, we can analyze each option based on the thermal conductivity of the materials involved:

   A) A new pair of jeans: Jeans are typically made from denim, a material woven from cotton fibers. Cotton is a poor conductor of heat, making it a good insulator. This means that jeans would not allow much heat to travel through them.

   B) A steel spoon in a cafeteria: Steel is a metal and metals are generally good conductors of heat. This means that a steel spoon would allow heat to travel through it quite efficiently.

   C) A cotton candy at a store: Cotton candy is made from sugar that is heated and spun into thin strands. While sugar can absorb some heat, the airy structure of cotton candy makes it a poor conductor of heat. It would not allow much heat to travel through it.

   D) A Calvin Klein cotton hat: Similar to the jeans, a hat made of cotton would also be a poor conductor of heat, as cotton is a good insulating material.

   Based on this analysis, option B) a steel spoon in a cafeteria, would let the most heat travel through due to the high thermal conductivity of steel.

   This answer contains four steps that analyze each of the choices and a result. Each analyzing step can be decomposed into: 1) The index, 2) the object, 3) the property of this object, 4) the relation between the property and the question. LMs can use the same process to answer similar questions.

   For these tasks, our t-c structure can also be applied. However, for some tasks like translation, the process is not fixed and the template is unclear, which is beyond our coverage.

[1] Susan Zhang, Stephen Roller, et al: OPT: Open Pre-trained Transformer Language Models, 2022. https://arxiv.org/pdf/2205.01068.pdf.

[2] Hugo Touvron, Louis Martin, et al: Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. https://arxiv.org/pdf/2307.09288.pdf.

[3] Jason Wei, Kai Zou: EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks, 2019. https://arxiv.org/abs/1901.11196.

[4] Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering, EMNLP 2018. https://arxiv.org/abs/1809.02789.

Dear Reviewer KmxN,

We trust this message finds you in good health. The author response period for our submission has been extended until the end of December 1st, as it has not yet received three reviews.

As we approach the end of this extension period, we note with concern that your feedback has not been received. In preparation for the rebuttal, we have dedicated considerable effort to address key aspects of our work, including:

1. Elucidating the importance of the t-c decomposition, that the decomposition leads to the reasoning ability.
2. Providing a explanation of the complexity in each component of our model.
3. Presenting additional experimental results that reinforce the validate of the t-c structure.
4. Offering a explanation of the specific tasks for which our methodology is most appropriate.

We would be immensely grateful if you could inform us whether our responses have satisfactorily addressed your concerns, or if there are any further queries you might have.

Your engagement and thorough evaluation of our work are highly valued and appreciated.

Kind regards,

Authors

3. "How can one (define template/content) and apply this (t-c classification model) in general for any tasks?"

   We are not aiming to define the t-c structure for any tasks. Instead, we aim to explain the complex reasoning ability of LLMs, i.e., the reasoning involving a relative fixed logical structures and steps. Most arithmetic reasoning (such as singleEQ, GSM8k, ...), most symbolic reasoning (last-letter-concatenation), some commonsense reasoning, and some classic NLP tasks (such as summarization) can be explained by our t-c structure.

   Because we have shown some examples of arithmetic and symbolic reasoning in our paper. Here we give more examples of other tasks. Take summarization as an example. This task can be approached by initially extracting key information from each paragraph and then synthesizing these elements into a coherent summary. A possible template for this process might be: "The first paragraph discusses [...]. The second paragraph addresses [...]," and so forth. This method can be further granularized to summarizing individual sentences within a paragraph, using sub-templates like: "Based on the sentence <sentence>, it appears to focus on <subject>."

   Commonsense reasoning such as OpenbookQA[3] dataset could be also with a template. Here we show an answer generated by GPT-4.

   Question: Which of these would let the most heat travel through? A) a new pair of jeans. B) a steel spoon in a cafeteria. C) a cotton candy at a store. D) a calvin klein cotton hat.

   ChatGPT:

   To determine which of these items would allow the most heat to travel through, we can analyze each option based on the thermal conductivity of the materials involved:

   A) A new pair of jeans: Jeans are typically made from denim, a material woven from cotton fibers. Cotton is a poor conductor of heat, making it a good insulator. This means that jeans would not allow much heat to travel through them.

   B) A steel spoon in a cafeteria: Steel is a metal and metals are generally good conductors of heat. This means that a steel spoon would allow heat to travel through it quite efficiently.

   C) A cotton candy at a store: Cotton candy is made from sugar that is heated and spun into thin strands. While sugar can absorb some heat, the airy structure of cotton candy makes it a poor conductor of heat. It would not allow much heat to travel through it.

   D) A Calvin Klein cotton hat: Similar to the jeans, a hat made of cotton would also be a poor conductor of heat, as cotton is a good insulating material.

   Based on this analysis, option B) a steel spoon in a cafeteria, would let the most heat travel through due to the high thermal conductivity of steel.

   This answer contains four steps that analyze each of the choices and a result. Each analyzing step can be decomposed into: 1) The index, 2) the object, 3) the property of this object, 4) the relation between the property and the question. LMs can use the same process to answer similar questions.

   However, for some tasks like translation, the process is not so fixed, and the template is unclear, which is beyond our coverage.

4. "Missing references, the following paper also discusses how text (similar to the "content" part defined in this paper) and patterns (similar to the "template" part defined in this paper) affect reasoning performance in language models: Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango. Madaan et al. 2022."

   Thank you for your kind reminder. It is a fascinating work focusing on explaining the CoT ability of LLMs, which is indeed related to our work. We have some similar inspirations. For example, their symbol can be seen as part of our content, and they say the symbol replacement will not degrade the performance, which is consistent with our Proposition 1. We are glad to add it to our related work. Unsurprisingly, some work have similar motivations to us because it's so natural. But these two papers are still different enough, as follows:

   1. With a similar idea to decompose different roles in the CoT, their pattern, symbol and text cannot directly correspond to our template and content. For example, in their mathematical examples, the equation "5+4=9" is a pattern, which is similar to our "content" because the number and the operator depend on the concrete problem. However, in their date example, their pattern consists of implicit steps: <restate>, <intermediate results> and <output>, which corresponds to our template.
   2. They only give some example tasks and do not formally define the symbol, pattern, and text in general tasks. In our work, one main contribution is providing a formal definition and theoretical analyzing tool.
   3. As other work from in-context learning to explain the reasoning ability, their work focuses on how the few-shot exemplars motivate models to reason, while our work is not limited to few-shot. The examples in the paper demonstrate that our t-c structure can explain the reasoning ability under zero-shot (instruction).
   4. Because of different decomposition, their conclusion differs from ours. They focus on the mutual help between pattern and text, but we emphasize the uni-direction dependence between templates and content (or templates at different levels).

[1] Susan Zhang, Stephen Roller, et al: OPT: Open Pre-trained Transformer Language Models, 2022. https://arxiv.org/pdf/2205.01068.pdf.

[2] Hugo Touvron, Louis Martin, et al: Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. https://arxiv.org/pdf/2307.09288.pdf.

[3] Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering, EMNLP 2018. https://arxiv.org/abs/1809.02789.

2. Experiment: (contd.)

"In Figure 5, even on the simple SingleEQ task, the proposed classification already conflicts with the human intuition, so I'm not sure if this can be used in practice, especially for more complex reasoning tasks." In our shown example in SingleEQ, only three "misclassifications" exist among the 77 tokens. Ambiguous or misclassified results tend to occur in scenarios where two semantically identical language usages result in different classifications. For instance, consider the sentence "John gave 43 seashells to Sam." (in Figure 5, right), where LMs give different classification on tokens 'John', 'to' and 'Sam': - Recall that our classification model cannot see the current word but only the previous words to classify the current position to be generated. For the position "John", the LM is not sure whether a vocabulary like "Therefore" should be generated here. There is a mixture of two kinds of generation --- "John" or "Therefore, John". So it gives a relatively small variance (a value between obvious content and template). - Similarly, the phrases "gave 43 seashells to Sam." and "gave Sam 43 seashells.\n" are semantically equivalent, but correspond to "TCCTCT" and "TCCCTT" respectively. It leads to the misclassification of the words "to" and "Sam", which are the fourth and fifth tokens. - Such "misclassification" actually contains more inspiration than just "error", such as the "habit" of an LM. For example, the above example shows that the LM actually prefers to add more adverbs or conjunctions at the beginning of sentences to clearly indicate the logical structure.

We acknowledge this classification model can be improved. A three-class model can distinct these ambiguous positions and label them as "ambiguous" instead of "template" or "content" to check them further manually or using other models. However, because we use this classification model as a tool to show the reasonability of our t-c structure in this paper, we leave it as future work.

> "Also on this task, the authors define content as "names, objections, and Arabic numbers", this also seems very ad-hoc and how can one apply this in general for any tasks?"

Thank you for your reminder. We do not expect our framework can apply for any task. Instead, we focus only on reasoning tasks that have relatively obvious fixed processes. Here, we choose "names, objections and Arabic numbers" as the typical content in some arithmetic problems. These contents are most commonly replaced to generate similar problems. For example, these two questions in GSM8k are the same except for different names, verbs (but the same mathematical meaning), objections, and Arabic numbers.

> Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
> 
> Salvadore earned $1956. Santo earned half of what Salvadore earned. How much money did the two men earn together?

And these problems could be further extended to more complex problems in GSM8k:

> Hash has nine more than half as many toys as Bill has. If Bill has 60 toys, how many total toys do the boys have?
> 
> Caleb has 3 dozen jellybeans. Sophie has half as many jellybeans as Caleb. How many jellybeans do they have in total?
> 
> Randy has 60 mango trees on his farm. He also has 5 less than half as many coconut trees as mango trees. How many trees does Randy have in all on his farm?
> 
> Lizzie has half as many crayons as Bobbie. Bobbie has three times as many crayons as Billie. If Billie has 18 crayons, how many crayons does Lizzie have?

In summary, we acknowledge that these are only part of all the possible content, but they are still representative for some tasks.

2. Experiment:

   "The two tasks used are very simple tasks, last-letter-concat and SingleEQ. In order to support the authors' claim on the flexibility and generalizability of this framework, a slightly more complex task should be presented as well. For example, on slightly more difficult algorithmic reasoning tasks like GSM8K or AQuA, can the hierarchical T-C be used to explain model's reasoning?"

   We provide an intuitive example above to show that the hierarchical t-c structure can be used to explain the reasoning ability on GSM8k. However, it is challenging to design an experiment to show the practical models' behavior on such a complex task, because there could be too many levels and too many possible reasonable decompositions which prevent us from making a clear intuitive judgment. In fact, mostly focusing on providing a formal and theoretical explanation, our experiments aim to support our framework, at least on the most typical t/c tokens. We acknowledge our experiments are preliminary. As we say in the limitation section, We expect this theoretical explanation to be mutually reinforcing with newer visualization work.

   "The results on T/C classification is a bit ad-hoc. Based on Figure 4 right, 1) as the authors mentioned, some green bars are too short to see, so it is unclear which model "exhibits the clearest T/C distinction" "2) Can the authors provide more quantitative analysis on the ratio between the green/blue bars? from the 2nd figure and the 3rd figure, the ratios seem not very significantly different, so I'm not sure if one can indeed judge a model has a better "reasoning capability" from those ratios .... Can the authors provide more quantitative analysis on the ratio between the green/blue bars in Figure 4?"

   Yes, we explain the results in Figure 4 right further here. Firstly, we should compare them by the difference instead of the ratio because these variances are based on (normalized) probabilities, i.e., the values after softmax. For any model, the variance at each position has the same range: $[0, (N-1)/N]$ where the $N$ is the number of disturbances. (Due to our y-axis range, it looks like some of the bars exceed 1 incorrectly. Actually, they are just very close to 1. We will modify this in the final version to make it more intuitive.) We no longer need to eliminate the impact of scale by calculate the ratio. On the contrary, since the variance of the template is often close to 0, the ratio would excessively amplify the differences between insignificant small quantities. For example, there is actually not much difference between $10^{-3}$ and $10^{-4}$.

   Here, we focus on whether a threshold can separate them. So we give an AUC-ROC score (Appendix D.2) and the difference of the mean-variance (DMV) $\sum_{s\in C} V_{s} / N_C-\sum_{s\in T} V_{s} / N_T$ where $N_T$ ($N_C$) is the number of template (content) tokens, and $V_s$ is the variance of the output probability at position $s$. AUC-ROC score measures the t-c classification accuracy, and the DMV further measures how confident models are about their classification. We also report the test accuracy on GSM8k as measures of their reasoning ability.

   Model	DMV	AUC-ROC	CoT ability	Accuracy on GSM8k***
   GPT2-335m	0.15	0.87	$\times$	0.0
   GPT2-774m	0.28	0.87	$\times$	0.0
   GPT2-1.5b	0.27	0.97	$\times$	0.0
   OPT-1.3b	0.29	0.96	$\times$*	0.0
   OPT-13b	0.42	0.99	$\times$*	0.0
   OPT-30b	0.46	1.00	$\times$*	0.0
   Llama2-7b	0.72	0.99	$\checkmark$**	14.6
   Llama2-13b	0.80	1.00	$\checkmark$**	28.7
   Llama2-70b	0.81	1.00	$\checkmark$**	56.8

   * The OPT models have trouble following the instructions or simulating the examples of prompts; therefore, no CoT ability is shown. See [1] as a reference.

   ** Llama-2 models have CoT ability to do some complex reasoning, such as GSM8k or MATH. See [2] for reference.

   *** Test in the 8-shot CoT setting to align with our framework. The CoT accuracy of the GPT2 and OPT models without CoT ability is 0.

   These results show the evident tendency that the significant t-c structure is the prerequisite for strong reasoning ability.

Thank you for your insightful comments and suggestions. We appreciate the opportunity to elucidate and clarify our article further. Below, we provide a summary of your questions and suggestions:

1. Reasonability of the assumptions:

   "This paper made many simplified (possibly unrealistic) assumptions in its proposed theoretical framework and misses the connection to real-world tasks. For example, the paper assumes a model is "well trained", meaning the model can memorize all answers perfectly. The authors claimed that this assumption is "not challenging for the prevailing LLMs with a huge amount of parameters", but many existing work has shown that even the largest models could struggle with tail knowledge [1]."

   As you rightly pointed out, it is crucial for a theoretical analysis to pay attention to whether the assumptions are reasonable, and we pay close attention to this. We do not assume that the model can memorize all answers perfectly. We only need the LM to memorize just one answer, which can provide the template for the target task. For example, to answer an arithmetic problem, the model learns the template (or sub-template in our hierarchical setting) from anyone correct solution in its training set. Therefore, our assumption is much weaker and more practical than "learning tail knowledge". Also, our t-c structure is proposed to explain whether and how LMs can handle generalized reasoning. Here, "generalized" denotes the ability of models to not only recall but also to process problems of a similar nature with varying content. Suppose a model cannot memory any problem of a certain task in the training set. In that case, it is almost impossible to reason on this task, so our explanation model is naturally not suitable for such a model --- "explanation" is valid only for those who do have the ability to reason.

   "The hierarchical extension assumes a model can decompose complex tasks well into sub-template and sub-content. But as the sub-template and sub-content can have many different combinations (especially if they are in slightly different forms as defined in Appendix A.1, based on label consistency), it is unrealistic to assume that the model is able to search in this combinatorial space efficiently and find the right template/content combination, especially when the task is very complex. Thus the presented theory in its current form, doesn't seem to be sufficient in explaining model's ability to solve very complex tasks."

   No, our t-c framework does not involve search. Our theory does not require models to search or find a combination, but emphasizes that natural language itself has the property to recombine fragments from different texts to form new sequences. Because of the composability of natural language, models can generate the correct combined answer to solve the task, as long as these examples which can be combined into an answer exist, as our Proposition 2 says.

   Here, we explain the composability more intuitively. Please consider this arithmetic problem in GSM8k:

   Randy has 60 mango trees on his farm. He also has 5 less than half as many coconut trees as mango trees. How many trees does Randy have in all on his farm?

   To solve the problem, LMs need first solve the half of mango tress numbers: 60/2=30, which can be learned from a sample that contains the template about half: "half ... /2=" (which translates the word 'half' into an equation '.../2') and another one contains "60/2=30" (solving the equation). As long as the intermediate result 30 has been generated, the template '<num a> less than <num b>... <num b> - <num a>'' with the content 5 and 30 in this problem leads to the equation '30-5' where the answer 25 is learned from any sample contains the equation, no matter what this sample is originally about. The following steps are similar. From this process, we find that the answer to this problem can be split into different template fragments, each of which can come from different training samples. It is this property (separability or composability) that ensures that our model can naturally generate this answer by learning these samples separately. For the model itself, we have no requirements other than remembering these samples.

Dear Reviewer oieR,

We trust this communication finds you in good health. The author response deadline for this submission has been extended until December 1st end-of-day, due to the paper not receiving three reviews.

As the extended deadline approaches, we have yet to receive your valuable feedback. We have invested great effort in our rebuttal, which includes:

1. Addressing and clarifying two critical misunderstandings regarding our assumptions, along with a explanation about their rationality.
2. Providing additional experimental data and analyses to elucidate the t-c classification outcomes.
3. Elucidating the tasks for which our approach is most suited.
4. Engaging in a discourse on a recent piece of related work.

We would greatly appreciate it if you could inform us whether these responses have addressed your concerns, or if there are any further queries you wish to raise.

Your time and effort in reviewing our work are deeply appreciated.

Kind regards, The Authors