Revisiting the Superficial Alignment Hypothesis

We are glad that the reviewers found our work and analysis insightful. We address the comments and questions as follows:

• GPT-based Evaluation: We agree that naively using GPT-based analysis might introduce biases. Before running the GPT-based prompts on the data, we initially collected 100 sample results from a finetuned model and manually annotated the errors using two human reviewers to get gold-standard annotations. We then refined the prompts until the GPT-based evaluation achieved an agreement of $>0.8$ with the human annotations. We have described this in the updated manuscript in Appendix A.4 and added examples of model generations and their corresponding annotations.

• Missing Citation Thank you for pointing this out, we have updated the manuscript with an expanded literature review.

• How does the model's performance change when it is post-trained on datasets with diverse task formats, beyond those used here? This is an interesting area for future work. Our error analysis experiments used models that followed step-by-step reasoning before providing the final answer. As shown in Figure 3, formatting errors are resolved with relatively little post-training data, and while we expect this pattern to hold across diverse formats, it warrants further investigation in future work.

• What are the underlying mechanisms that allow LLMs to integrate new knowledge effectively during post-training? We hypothesize that posttraining helps a model understand how to effectively utilize available knowledge, as demonstrated during finetuning on pre-cutoff data. Subsequently, it can better use new knowledge received through further finetuning/in-context learning. We verify this in our ablation study in Appendix A.2. Other works like [1] and [2] have also motivated studying the effects of post-training on new knowledge.

• Does it risk overfitting to specific tasks or styles, potentially diminishing its generalization capabilities? This is a critical consideration for future work. We need better descriptions of how post-training causes catastrophic forgetting and overfitting. While the common approach is to post-train on large, diverse datasets (as shown in the llama3 technical report), more work is needed to understand the implications in the low data regime. Our work focuses on establishing that new knowledge is learned during posttraining, proposing posttraining scaling laws, and highlighting proper evaluation using relevant benchmarks. We look forward to future work extending these ideas to questions of broader, multitask generalization.

References:

• [1] Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
• [2] Instruction-tuned Language Models are Better Knowledge Learners

Dear Reviewer NfiD,

We sincerely thank you for your time and expertise in providing your review.

Based on your valuable feedback, we have added a new section that details the rigorous treatment that GPT-based evaluation underwent, to prevent biases. We have also provided example generations that illustrate the annotations from the evaluation.

As the discussion period is ending, we would greatly appreciate any further inputs or comments from you, to ensure that we fully addressed your questions and weaknesses.

Thank you again for your efforts.

We are glad that the reviewers found our work and analysis insightful. We address the weaknesses as follows:

• ...Missing URIAL prompt [1] as a training-free alignment baseline URIAL prompt primarily consists of two components - system prompt and K-shot In-Context Learning (ICL) over the base model. In our evaluation setting, all base and fine-tuned models included a system prompt. While evaluation was done in 0-shot, we also conducted 5-shot evaluation of math models in the lm-eval GSM8k task:

The shape of the scaling curve remained consistent, with only an upward shift in evaluation points. This suggests URIAL prompting benefits both base and finetuned models, making it more about prompt optimization study than finetuning performance scaling.

• ...The authors should also make the investigation/claim more systematic or formal... We apologize for any lack of clarity. Taking mathematics as an example: the Superficial Alignment Hypothesis would suggest that mathematical ability is fully developed in the base model and would saturate GSM8k benchmark performance with just a few hundred examples. However, we demonstrate continued improvement across thousands of examples for all model families and sizes (Fig 1 and 2). Our error analysis shows that with 100 examples, models learn to mimic GSM8k response style but make numerous mistakes. Similar patterns emerge in multi-hop reasoning, where initial attempts at reasoning are incorrect, improving only with more data.

• ...Also, how does the overall accuracy change? Are the number of incorrect responses decreasing? Figure 3's "Total Mistakes on the test set" shows overall accuracy. Incorrect responses decrease with larger datasets, correlating more strongly with improved reasoning than improved response style.

• ...Here the same observation of "making improvement" are interpreted in two completely different standpoints, without good rationalization. We apologize for any unclear writing. From Figure 3, we wish to drive home the point that the users care about improvement in the final task-specific evaluation metric. From this lens, the rapid “improvement in style” followed by stagnation doesn’t translate to the “improvement in the end goal of the task”, as evidenced by the still-high “total mistakes” columns. They are mimicking the response style. However, the “improvement in reasoning” is more gradual with more data, which correlates well with the end user task. We have added specific examples of generations for further evidence in Appendix A.5 in the revised manuscript.

• It sounds more like the authors are building their own instruction-tuned model instead of using an off-the-shelf one... We didn’t use an off-the-shelf-instruction-tuned model because of the reason highlighted in footnote 2. Llama-3-8b-instruct is strongly post-trained to refuse answering questions beyond their knowledge cutoff to prevent hallucination. Attempts to break this behavior by strong finetuning hampers its general capabilities.

• ...reasoning is a broad term here, and authors only test one kind of reasoning Since we evaluate each task independently, we define reasoning as a correct sequence of steps that helps in arriving at the final answer. In addition, we check for adjacent capabilities, for instance, arithmetic calculations in Figure 3 or hallucinations in Figure 5, and label them differently, instead of considering them as another reasoning capability.

• ...The author also didn't do a thorough survey for related work. We agree that the literature survey could be more thorough and we have updated the manuscript accordingly.

• ...in the Llama 3 technical report [1], the importance of improving performance for specific capabilities such as code/math/reasoning/tool use, etc., during post-training is emphasized. This suggests that the re-evaluation presented in the paper may not be as novel as claimed. We provide several novel insights including: (1) systematic scaling law treatment of performance changes across model families and sizes, (2) in-depth qualitative and quantitative analysis of model behavior at different finetuning levels, showing how generations evolve from incorrect to correct responses, and (3) a hand-curated set of new facts and multihop question-answers (to be open-sourced) with experiments on learning new knowledge alongside new capabilities. We note that our study was well-underway when the llama-3 technical report was released, and their reported large-scale post-training dataset actually supports our findings.

• ...However, the current submission only mentions multi-task settings as a potential future direction... While [2] presents useful analysis on post-training task data mixtures, our work focuses on establishing post-training scaling laws for LLMs and directly challenging the superficial alignment hypothesis. To limit interacting variables, we studied how specific capabilities improve through post-training by measuring performance against the number of finetuning examples. Multitask settings introduce additional variables (data mixtures, combined metrics, cross-task transfer) that were beyond our scope.

• ...Consequently, the exclusive focus on the scaling of post-training may be overly restrictive. While we agree this distinction isn't absolute for LLM Alignment, it remains a useful framework, as evidenced by technical reports like Llama 3 that clearly separate pre-training and post-training approaches. We define pre-training as developing general language modeling capability (evaluated via NLL loss) and post-training as improving specific human-valued tasks like math, coding, and instruction following. These stages have distinct data formats and collection methods. While high-quality pre-training remains mostly opaque within industry labs, post-training offers opportunities for controlled, observable experiments. We plan to explore other stages like mid-training or continual learning in future work.

• ...It is recommended that the authors consider augmenting the dataset through the use of synthetic data or by constructing a larger, more comprehensive dataset to study the scaling law... We have conducted additional experiments using synthetic data to verify our results using synthetic data. Please see our unified response above.

References:

• [1] Llama 3: Technical Report
• [2] Analysis of Post-training Task Data Mixtures

Dear Reviewer 7xz1,

We sincerely thank you for your time and expertise in providing your review.

Based on your valuable feedback, we have added the additional experiments and discussion that extend the results of the paper to training sets that are orders of magnitude larger using synthetic datasets. We hope that we also provided enough evidence to motivate this study as well as its impact and novelty.

As the discussion period is ending, we would greatly appreciate any further inputs or comments from you, to ensure that we fully addressed your questions and weaknesses.

Thank you again for your efforts.

We are glad that the reviewers found our work and analysis insightful. We address the comments and questions as follows:

• However, the training set is still small for building a reliable scaling law, One solution is to verify the idea of synthetic tasks. We have conducted additional experiments using synthetic data to verify this. Please see our unified response above.

• ...The contribution of this work is more like to be a verification and provide more evidence. We agree that some of the key insights are intuitive, but a rigorous treatment of this behavior is either conducted in closed industry research or is anecdotal evidence. As the reviewer pointed out, we have done several in-depth analyses in terms of model behavior at various scales, for different tasks, at different model sizes and dataset scales, using open-source datasets and models. This would be valuable information for the vast majority of researchers who intend to collect data for their use cases, as well as smaller industry teams that want to improve LLMs for a particular task. We also provide a detailed study of learning new knowledge during post-training with novel insights on evaluation and hallucination.

• ...this work uses the term 'post-training' frequently, but the actual experiments are limited to SFT Please refer to the unified response to this comment above.

• ...the C3 and C2 at L055 are in the wrong order. Thank you for pointing this out, we have fixed this.

Dear Reviewer pyTF,

We sincerely thank you for your time and expertise in providing your review.

Based on your valuable feedback, we have added the additional experiments and discussion that extend the results of the paper to training sets that are orders of magnitude larger using synthetic datasets. We have also improved the writing of the paper to clarify the use of our terms.

As the discussion period is ending, we would greatly appreciate any further inputs or comments from you, to ensure that we fully addressed your concerns and weaknesses.

Thank you again for your efforts.

We thank the reviewer for their comments and suggestions, which have improved the work.