PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization

Dear Reviewer,

We sincerely thank you for your insightful comments and valuable suggestions on our manuscript. Your expertise and detailed review have been instrumental in enhancing the quality and clarity of our work. In response to your comments, we have carefully examined and addressed each issue you raised. For your convenience, we have marked all the related revisions in blue within the manuscript, ensuring easy identification of the changes made. We welcome any further inquiries or suggestions you might have, as your input is invaluable in refining our research. Thank you for your continued guidance and support in this process.

Weakness 1:

Even though this model is still better than GPT-4 which is great, still the low accuracy/F1 scores leave one desiring a better model. Can the authors propose a higher-bound on the expected F1/accuracy scores? For example, can we compare this to the percentage of human evaluators usually that agree with each other on the same task?

Answer:

Our test set was annotated by three individuals with inter-annotator agreement (IAA) exceeding 0.85. To refine the model's performance assessment compared to human evaluators, we can use the inter-annotator agreement (IAA) of 0.85 as a benchmark. If our model exceeds this, it indicates strong performance. However, setting a realistic target slightly above this human IAA, say around 0.90, offers a challenging yet achievable goal. We have added this analysis in Section 4 in our revised paper.

Weakness 2:

In Table 3, PandaLM-7B is shown to have dangerously low numbers: 47% for LSAT and 51% for BioASQ. Assuming that chance accuracy is 50%, this highly questions the validity of this version of the model. I understand that the judge for these results has been GPT-4, however, that doesn't resolve the issue. I highly suggest to not simply dismiss the results, and further validate why. If it turns out that human evaluators actually do prefer this model, then all the better. Otherwise, the 7B version of the model might not be fit to the task of evaluating results outside the domain in which it was trained on.

Answer:

We're addressing a three-category classification task (win/lose/tie), where a random guess would lead to around 33% in precision, recall, and F1 score. PandaLM-7B's results of 47% and 51% are notably above this level. To further validate PandaLM-7B, we conducted a human evaluation with 30 samples on the BioASQ evaluation. The human evaluation showed that both PandaLM-7B and GPT-4 tended to favor Vicuna over Alpaca, indicating a consistent trend in their evaluations. We have added this analysis in Section 4 in our revised paper.

Question 1:

In Table 2, I believe GPT-4's Recall should be highlighted as it's higher than PandaLM-70B's (68% vs 66%)

Answer:

Thank you for highlighting this detail. We have updated our paper accordingly, marking the revision in blue font for clarity and emphasis.

Question 2:

In the conclusion, the authors mention that "based on PandaLM evaluations", the superior models are the ones that have been trained using PandaLM. Then they claim that "this order emphasizes the efficacy of PandaLM ...". I believe this claim is incorrect since by definition a model that the PandaLM preferred in hyperparameter tuning, is already preferred to the base model. Hence, one cannot make any claim based on the results. If I misunderstood, then please clarify. Otherwise I believe this sentence should be ommitted.

Answer:

We acknowledge the potential oversight in our initial interpretation of the evaluation results. To address this and provide a more accurate representation, we have revised the section in the paper as follows:

"Besides, as shown in Appendix B, based on PandaLM's evaluation, the model demonstrating superior performance is LLaMA-PandaLM. Note that the base foundation model's characteristics can be a significant factor in performance, as evidenced by LLaMA models securing the top two positions. The ranking pattern observed aligns closely with the base model rankings presented in Figure 4."

Weakness 4:

There seems to be a gap in the generalized and robustness evaluation of PandaLM. Although the authors have assessed PandaLM in the context of legal and biological domains (indicative of domain shift), a broader spectrum of evaluations, especially with unseen models like LLaMA-2 (indicative of model shift), would enhance its credibility.

Answer:

We provide a detailed comparison of PandaLM's performance against human benchmarks and in the context of different versions of instruction tuned LLaMA models. Note that llama1-13b, llama1-65b and llama2 is indicative of model shift. The results clearly demonstrate that PandaLM aligns closely with human, consistently showing a preference for the LLama-2 model. This alignment is in line with expectations, as LLama-2 benefits from more pre-training data. We have added this analysis in Appendix K in our paper.

Weakness 5:

While evaluating LLMs is undeniably vital, solely focusing on outcome evaluation might be resource-intensive. A deeper dive into behavior prediction could provide a more comprehensive understanding and potentially be more resource-efficient.

Answer:

We agree with your point about the importance of evaluating large language models (LLMs) beyond just outcome-based measures. Currently, our emphasis is primarily on outcome-based evaluation, which is indeed resource-intensive. However, integrating behavior prediction into evaluation framework could offer a more comprehensive understanding of LLM performance. For instance, analyzing and evaluating the extended text outputs of an untuned LLM can help predict how a tuned version might behave in various scenarios. This approach could provide a more efficient and insightful way to balance resource-heavy outcome assessments. We have added this in the Limitation Section of the revised paper.

Question 1:

How does a model optimized by loss or perplexity fare in evaluation results? Can you shed light on the advantages of PandaLM over automatic metrics in hyperparameter optimization? The statement, "... Perplexity ... does not effectively guide the choice of hyperparameters", prompts questions regarding the mechanism by which PandaLM aids in hyperparameter selection. Is there a direct correlation between PandaLM's results and the choice of hyperparameters? Additionally, could you share the results for models selected based on optimal perplexity?

Answer:

The experimental results in response to Weakness 2 indicate that model optimized by loss or perplexity might not always correlate with high performance in practical tasks. While perplexity measures predictive ability during pretraining, it doesn't encompass all aspects of language understanding crucial in practical applications. Therefore, PandaLM's broader assessment is more effective for hyperparameter optimization, as it evaluates models in complex, instruction-based scenarios. We have included the analysis of perplexity in Appendix C in the revised paper.

Question 2:

Is there an implementation of the early stopping strategy? If so, how does this impact overall performance?

Answer:

We implemented an early stopping strategy using Pandalm. Due to time constraints, we focus specifically on LLaMA. We use Pandalm to determine if the current training checkpoint surpassed the previous one. If the current checkpoint was found to be inferior, the training process was stopped. It turned out that LLaMA training stopped at epoch 5, aligning with our earlier observations from a comprehensive pairwise comparison of all tuned LLaMA checkpoints, which also considered variations in learning rate, optimizer, and scheduler. The early stopping strategy proved effective in reducing the time required to identify the best epochs for the model. However, it's important to note that this approach still necessitates experimentation with different learning rates, optimizers, and schedulers to optimize performance fully. We have added the analysis of early stopping in Appendix J of the revised paper.

Dear Reviewer,

We sincerely thank you for your insightful comments and valuable suggestions on our manuscript. Your expertise and detailed review have been instrumental in enhancing the quality and clarity of our work. In response to your comments, we have carefully examined and addressed each issue you raised. For your convenience, we have marked all the related revisions in blue within the manuscript, ensuring easy identification of the changes made. We welcome any further inquiries or suggestions you might have, as your input is invaluable in refining our research. Thank you for your continued guidance and support in this process.

Weakness 1:

While the authors utilize PandaLM to optimize hyperparameters, there's an evident absence of evaluation results concerning different hyperparameter settings. Specifically, insights into how performance varies with adjustments in learning rate or epochs would be beneficial.

Answer:

Thank you for your insightful comments regarding our utilization of PandaLM for hyperparameter optimization. We have added visualization and analysis in Appendix J of the revised paper. In our hyperparameter searching process, we explored a range of learning rates, epochs, optimizers, and schedulers. The learning rates tested varied from 2e-6 to 2e-4, with model checkpoints saved at the end of each epoch. Performance was rigorously assessed through pairwise comparisons between checkpoints, counting the win rounds for each model. Our analysis, as depicted in the figure, suggests a tendency towards a learning rate of 2e-5, although this preference was not uniformly clear across all models. The figure also demonstrates the variability in the optimal number of epochs, with a trend showing that peak performance often occurs around the fourth or fifth epoch. This evidence points to the complex interplay of hyperparameters with model performance, which is further influenced by data distribution, optimizer, and scheduler choices. The findings from our hyperparameter optimization process highlight that there is no universally optimal setting for different models and training setups. While a pattern emerged suggesting that a learning rate around 2e-5 and an epoch count near 4 might be beneficial in some cases, these results are not conclusive. This reinforces the need for specific hyperparameter searches for different models, as demonstrated in our visualizations. A tailored approach to hyperparameter optimization is essential, as it allows for a more nuanced understanding of model performance across various scenarios.

Weakness 2:

The presented evaluation results aren't juxtaposed with learning metrics like loss and perplexity. Despite the authors' assertion that perplexity might offer optimization directions, this comparison is conspicuously missing.

Answer:

Thanks for your valuable suggestions, we have included the analysis of perplexity in Appendix C in the revised paper. We report the win rate of LLaMA-Alpaca and LLaMA-PandaLM over 170 samples. The evaluation results show that lower perplexity, indicating better predictive ability in pretraining, does not always mean better overall performance of instruction tuned models. For example, LLaMA-PandaLM has a higher perplexity than LLaMA-Alpaca but outperforms it in both pairwise comparison (PandaLM,GPT,Human) and traditional tasks (lm-eval). In such contexts, lower perplexity might even suggest potential overfitting, thereby diminishing the model's generalizability.

Weakness 3:

The hyperparameter settings presented appear incomplete, particularly the omission of early stopping.

Answer:

Thanks for pointing out, we implemented an early stopping strategy using Pandalm. We focus specifically on LLaMA. Our experiments showed that in some cases, a model's performance at epoch 3 was inferior to that at epoch 2. However, subsequent epochs demonstrated performance improvements. This indicates that early stopping may not always be suitable for large model fine-tuning, as it could prematurely halt training before reaching optimal performance. We have added this analysis in Appendix J of the revised paper.

Dear Reviewer,

We sincerely thank you for your insightful comments and valuable suggestions on our manuscript. Your expertise and detailed review have been instrumental in enhancing the quality and clarity of our work. In response to your comments, we have carefully examined and addressed each issue you raised. For your convenience, we have marked all the related revisions in blue within the manuscript, ensuring easy identification of the changes made. We welcome any further inquiries or suggestions you might have, as your input is invaluable in refining our research. Thank you for your continued guidance and support in this process.

Weakness 1:

In the abstract and introduction sections, the authors should make clear the claim point and avoid overclaiming the contricution. For example, in the abstract, rather than simply claiming that PandaLM-7B offers a performance comparable to both GPT-3.5 and GPT-4. Impressively, PandaLM-70B surpasses their performance, the authors should point out it is on the evaluation of the dataset collected by the authors. In the introduction section, instead of claiming that Tuning models with PandaLM-selected hyperparameters yields more substantial performance enhancements, it should point out specifically the performance compared to the LLM searched by Alpaca.

Answer:

Thank you for your valuable feedback. We have revised the abstract and introduction sections to provide a clearer context for our contributions.

Weakness 2:

In Table 1, it is better to add the performance of LLaMA as a judge. By doing so, it can rule out the possibility of the performance of PandaLM is caused by the backbone model, rather than the proposed LLM tuning framework.

Answer:

We have acknowledged the significance of incorporating LLaMA's performance as a benchmark. In fact, Appendix G of our manuscript already provides evidence indicating that tailored tuning significantly affects performance in both zero and few-shot scenarios. Our results reveal that a smaller model, when well-tuned, can surpass a larger model that hasn't been fine-tuned. To improve clarity, we have updated Section 5 of our manuscript to provide a detailed explanation of the findings in Appendix G. Apart from utilizing pretrained LLaMA for evaluation, we also conducted experiments with finetuned LLaMA (Vicuna). These experiments showed that the evaluation capabilities of instruction-tuned models can be substantially improved. It's important to note that since Vicuna doesn't always produce evaluation responses in accordance with instructions, we compared the log-likelihood of next token(win/tie/lose) to conduct our experiments.