TADIS: Steering Models for Deep-Thinking about Demonstration Examples

• The paper is motivated by the findings of Kung and Peng (2023) that analyzed instruction-tuning models that learn from examples. They found that models seem to learn the format of the examples, rather than the mapping in those examples. This results in models achieving similar performance whether they are provided with accurate or inaccurate examples. While the paper aims to tackle this problem, it seems that TADIS exhibits the same issue. In particular, Table 3 shows that the models trained with true or flipped thinking results achieve comparable performance with only slight performance drops. Furthermore, across all models, the performance is comparable between receiving correct, flipped, or random thinking results. The only major drop happens for the larger model trained with random thinking results. The paper explains this result by saying that TADIS helps more during training than testing, however, if the model is relying on the "thinking" result, it should be affected at inference time as well. As a result, it is unclear to me that the problem identified in the paper has been addressed.

• Following on the previous point, Kung and Peng (2023) showed that models perform comparably when provided with accurate or delusive instructions, however, the comparsions done in Table 3 is when the thinking results is flipped. It is unclear if the model is ever evaluated on delusive examples.

• The paper shows a correlation between judged accuracy of statements and the generated score, and uses that to support the claim that forcing models to "judge the correctness of [the] provided examples could steer them towards deep thinking." However, it is completely expected that those two values would be correlated even if there was no causal link between learning to judge the correctness of the statements and learning to generate more accurate responses. To support the claim that judging accuracy improves model performance, one needs to evaluate a setting where you evaluate the correlation between those two numbers without having the model learn to judge the accuracy. For example, getting the model to judge the accuracy of the statements without training or doing it after the model generates the answers.

• The paper uses the findings of Kruger and Dunning (1999) to motivate their problem by connecting it to the findings of Kung & Peng (2023). Kruger and Dunning (1999) studied the relationship between skill and self-assessment of performance, and found they people over-estimated their abilities. Kruger and Dunning argued that poor performance and miscalibrated assessment were both caused by low competency; ie, one's ability to perform a task affects their ability to assess how well that task is done. This is very different from the findings of Kung & Peng (2023) where the model simply learns a different thing from what is learned. Furthermore, there is no self-assessment or estimation of the accuracy of one's predictions for both the proposed model or in Kung & Peng (2023). While the "thinking" stage does include an assessment of the accuracy of the inputs, this assessment is of the inputs not the model's predictions or overall model capability. As a result, I do not see how Kruger and Dunning is relevant to this work.

• The paper includes a lot of anthropomorphizing of different components of the model (eg, thinking, seeing, acting) which sometimes does not indicate different model behaviors or capabilities. I provide two concrete examples of this below. I think this is problematic because it (a) ascribes behaviors, capabilities, and agency to the model which do not seem to be there; (b) drowns the technical explanations with metaphors which confuses the point for no clear reason. One possibility is that those terms are all aliases/synonyms for something more technical. In that case, I think the paper woould be much stronger and clearer if the exact capability is described.

  • The method name is an example of this as it implies the model is thinking instead of seeing, however, it is unclear why standard in-context learning is only seeing, while an extra prompt should be categorized as thinking.
  • While I understand why it might be good to differentiate the "thinking" and "answering" losses as they deal with two different things (generating accuracy labels for the inputs vs. generating the answer), the method section is full of words that indicate the same thing, those include "deep thinking", "thinking", "answering", and "generating actions."

• Typo in the second paragraph on page 8: The paper references Figure 4 as an example of separating the two stages, when it should be figure 5.

• The main claim of the paper is that the proposed method promotes "thinking" on the examples, which informs subsequent "answering". More precisely, "thinking" amounts to classifying the examples as positive or negative, and the way "thinking" influences "answering" is through conditioning the task output on the classification labels. However, the results in Table 3 challenge this claim. At inference time, whether or not an example is correctly labelled has absolutely no impact on model performance. That is, despite being fed the "thinking" result, the model chooses to ignore it when predicting the task output. This leads me to think that the proposed method does not work as the authors claim, and it does not solve the problem that initiated the work.

• I encourage the authors to further study the source of performance gain. According to Table 3 and Figure 3, my hypothesis is that example classification is an effective auxiliary task even though it does not encourage an LLM to follow instructions. Further, I suspect that the specific instruction format (especially with the action text) boosts performance (Table 6 provides some evidence).

• Comparing results in Table 2 (first row) and Table 4, it is clear that LLMs fine-tuned using the proposed method on synthetic examples perform worse than zero-shot SuperNI (i.e., without using examples). This challenges the second contribution of the paper, which says the method can scale up instruction tuning by synthesizing examples for a large number of tasks. In the case where examples are lacking, one would rather choose not using examples (i.e., zero shot) than synthesizing examples for few-shot instruction tuning.

Overall the paper is a bit hard to follow. At a high level, this is a relatively incremental change to an existing approach, evaluated on a single dataset. The results are mixed: generally a small improvement -- though not always.

Presentation

The paper is a bit hard to follow. I try my best to ignore syntax -- many of the best researchers in the field are nonnative english speakers. The presentation is hard to follow for other reasons -- e.g. section 2.1 is a single paragraph that takes up more than half a page.

Some parts of the method are not defined, e.g. Actions $A_e$ in section 3. What exactly are the actions?

Experiments

1. Evaluation is only on two relatively small variants of T5. The results seem mixed for the smaller model, though a bit better for the 3B model. It seems like the way to evaluate this would be to train larger models
2. The lack of error bars makes it hard to judge the significance of the performance deltas
3. I found the ablation study a bit hard to understand -- so if (w/o TR) is just removing that one line, why is the drop-off so much larger than removing the whole thinking section? Shouldn't the model learn to compensate for the lack of prompt? Or is this evaluation removing the prompt at inference for a model trained with the full prompt (this taking the model OOD)

• Limited novelty over recent prompting work: the proposed methods is conceptually similar to Chain-of-Thought prompting, falling under the same theme of adding intermediate reasoning steps before making a final prediction. Nevertheless, the proposed method might be more widely available to existing data given that it does not have additional requirement beyond truthfulness labels for the examples

• Requires re-training of the concerned models: which could be costly given the targeted models are LLMs, when compared to the few-shot prompting paradigm.

• Marginal improvement over baselines: Only few-point improvement in ROUGE-L score is observed over baselines, in which a larger model may demonstrate greater improvements for the proposed method.

• Some overly broad claims in the presentation of the paper: statements such as steering models to perform 'deep-thinking' might require more extensive evidence and prior literature to support that the LLMs are actually 'thinking'.