ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models

We appreciate the constructive feedback and thoughtful suggestions. We address the concerns as follows:

One of the core innovations of the paper is the tool ClimaGen that automates the construction of ClimaQA, which involves the QA principle and prompt assembler. From line345 it appears to be a direct reference to other articles, which the authors do not describe in detail. However, this part directly affects the quality of the input prompt, which determines the quality of the generated QA pairs.

We understand the importance of the QA principles and prompt assembler in ensuring the quality of the generated QA pairs. The assembled prompt, which plays a critical role in the generation process, is indeed explained in detail in Appendix 4. Hope that this provides clarity in the matter.

This paper proposes a benchmark to assess the quality and scientific validity of the output of climate base models, but there is a lack of experimental validation that the QA pairs in the benchmark are scientifically meaningful, i.e., how to prove that the constructed dataset is able to help future research on

To ensure the scientific validity of the benchmark, we selected textbooks that cover a broad range of critical topics in climate science, reflecting both technical and qualitative aspects. These textbooks were chosen to provide a comprehensive understanding of the field, from foundational concepts to advanced research topics. Please refer to the newly added Appendix 1 for more details. Moreover, domain experts were involved in validating the generated QA pairs to confirm their scientific accuracy. This expert validation ensures that the dataset is scientifically meaningful and aligns with established climate science knowledge. As a result, we are confident that this dataset will contribute to advancing future research on climate foundation models, improving the quality and scientific validity of their outputs.

Lack of detailed information about the data source. The authors emphasize the data sources as 18 graduate-level textbooks, but give no detailed information about the data sources to prove it.

We have now added detailed information about the data sources in Appendix 1, including the specific textbooks used for question generation. This additional information provides clarity on the selection and scope of the textbooks, helping to substantiate the validity of our dataset.

The experimental section lacks a case study. There is no visual comparison between the actual performance of the existing LLMs on these tasks and the difference between the groundtruth.

Thanks for bringing this up. We have added a new case study in Appendix 2.3, where we compare and analyze the outputs of different models along with their corresponding evaluation scores. This case study provides a visual comparison between the actual performance of existing LLMs on the tasks and the ground truth where we demonstrate how the newly proposed metrics provide a reliable and robust evaluation.

Writing flaws: lack of citations for some important ideas, such as in line32 and line35; inconsistency in font size of illustrations, such as Figure 6; lack of necessary explanations for the letters of the formulas, such as the equations about factual accuracy and phrase similarity.

Thank you for the feedback, we have fixed these.

We appreciate the constructive feedback and thoughtful suggestions. We address the concerns as follows:

The framework relies on a relatively small set of graduate-level climate textbooks, which may limit the diversity of contextual information and scientific perspectives in the generated questions.

The QA pairs in ClimaQA cover a broad range of climate science knowledge through the careful selection of textbooks that span key areas such as atmospheric science, aerosols, clouds in the climate system, geostatistics, and technical aspects of climate science. These textbooks were chosen to reflect both breadth and depth across critical topics in the field. We have included a statistical analysis in the newly added Appendix 1, which demonstrates the distribution of topics covered by the QA pairs.

Combining an expert-validated gold-standard dataset with a large-scale synthetic dataset provides comprehensive resources for training and evaluating foundation models, supporting both high-quality benchmarking and scalability.

We agree that the relatively small size of the ClimaQA-Gold dataset limits its scalability and generalizability. This is a recognized limitation, and we plan to expand the dataset and improve the annotation process in future work. We acknowledge that experiments would need to be repeated for each domain to ensure broader applicability.

Although improvements were noted, the models often struggle with reasoning-based questions, suggesting that further work is needed to refine reasoning capabilities in foundation models.

We consider this observation a strength of our work, as it highlights the limitations of current foundation models. By capturing these challenges, the benchmark provides valuable insights that can guide future improvements in model reasoning capabilities.

The BLEU and BERTScore metrics show some bias towards the model used for question generation (GPT-3.5-turbo), which may affect the objectivity of the evaluation.

We agree with this observation which can be explained by noting that BLEU is not immune to paraphrasing, and BERTScore does not validate scientific accuracy. This reinforces the motivation behind our proposed Factual Accuracy metric, which evaluates factual and contextual correctness for a more robust assessment. We can see from Figure 5 that Factual Accuracy does not show this bias and this is concluded in the paper. Kindly refer to the newly added case study provided in Appendix 2.3 for a detailed analysis of how the proposed metrics prove to be robust.

Implement a tiered evaluation system that emphasizes model performance on reasoning and hypothetical questions separately, allowing for a deeper analysis of model strengths and weaknesses in scientific reasoning.

This is a great suggestion. We agree that implementing a tiered evaluation system focusing on reasoning and hypothetical questions separately could provide a deeper analysis of model strengths and weaknesses. This is an interesting direction for future work, and we will consider it to enhance the granularity of the evaluation.

Further experiment with retrieval-augmented generation using a wider array of sources or explore weighted retrieval to reduce distractors and improve information relevance.

We agree that using a wider array of sources or weighted retrieval could help reduce distractors and improve information relevance in retrieval-augmented generation. This is a promising direction for future work to enhance the quality and reliability of generated outputs and the ClimaQA benchmark will be instrumental in evaluating these strategies.

Introduce additional objective metrics to balance any biases introduced by BLEU or BERTScore, potentially incorporating domain-specific metrics tailored to scientific language accuracy and precision.

We concur that domain-specific metrics would enhance evaluation. While developing them is challenging, we see it as an exciting opportunity to refine scientific accuracy assessments with collaboration from domain experts in future work.

We appreciate the constructive feedback and thoughtful suggestions. We address the concerns as follows:

The domain that the authors focus on can be deemed as a very niche one. The authors could have tried to do a better job at providing concrete arguments in the introduction that underline the significance of this domain. Specifically, it would be good to explain why we need LLMs to be good at this specific domain. If they become better at this domain, which applications would this unlock?

Climate change is one of the most pressing global challenges today, with profound impacts on ecosystems, economies, and societies. Ensuring that LLMs perform well in this domain can unlock applications in climate policy analysis, environmental decision-making, and public education. By improving LLMs' understanding of climate science, we can empower stakeholders to make informed decisions, develop actionable solutions, and foster broader awareness of climate issues. We have modified the introduction to reflect this strongly.

(1) the authors use the term "foundation models" but probably just want to say either "large language models" or "question-answering models". Foundation models are pre-trained on large corpora and used mainly for transfer learning across different tasks, but many of them are not LLMs. It would help to use a more specific term.

Modified the occurrences accordingly

(2) Formulas presented in section 3 use some symbols that were not explicitly defined (e.g. ps, pr).

Thanks for the feedback. These are fixed.

The numbers in Figure 4 seem to keep going up (albeit at a reducing pace), and yet the authors conclude that a "context window of size 4 most effectively differentiates between correct and incorrect answers". From the figure, it is unclear to me why this context window is optimal.

The choice of a context window size of 4 is based on its ability to yield the largest difference in mean cosine similarities between correct and incorrect answers while still maintaining sufficiently high scores for correct answers. That said, we acknowledge that this conclusion is empirical and may vary depending on the specific dataset and domain.

This is a minor weakness, but the provided corpus is relatively modest, compared to other ones. I don't think this is too much of an issue given that it is probably large enough to sufficiently evaluate a QA model, and a small size makes it more manageable. That said, it would be good to provide some sense of how well the provided corpus covers the actual domain (e.g. by giving the readers a sense of the subdomains and the number of questions from each subdomain).

The QA pairs in ClimaQA cover a broad range of climate science knowledge through the careful selection of textbooks that span key areas such as atmospheric science, aerosols, clouds in the climate system, geostatistics, and technical aspects of climate science. These textbooks were chosen to reflect both breadth and depth across critical topics in the field. We have included a statistical analysis in the newly added Appendix 1, which demonstrates the distribution of topics covered by the QA pairs.

In section 5.2 you mention that there are 5 books used to derive the corpus of questions, and an additional 13 books that are used for the RAG model as a retrieval corpus. Do we know if all relevant information between these two sets of books overlaps? In other words, do we know if the information in the 13 books is sufficient to support all the questions that were produced by using the 5 books?

There is very little overlap between the information in the 5 books used for question generation and the 13 books used as the retrieval corpus. This is reflected in the retrieval-augmented generation (RAG) scores, as some questions lack sufficient supporting information in the retrieval corpus. Addressing this is part of our future work, where we plan to expand and refine the retrieval corpus to ensure better alignment with the question-generation sources.

In the introduction you mention "Creation of publicly releasable datasets". Could you please clarify what you mean by a "releasable" dataset?

By "releasable" dataset, we mean that the dataset will be properly cleaned and organized, and made publicly available for use. Currently, it is provided as supplementary files, and we plan to release it online and make it easy to access.

I very much appreciate the authors' thorough responses and provide any additional comments below:

• (W1) - The update to the intro that the authors have made helps bolster the motivation.

• (W2) (1 and 2) Thank you for making the fixes! (3) I appreciate the clarification but suggest that it would be better if these were plotted in the figure in order to make the tradeoff more clear.

• (W3) I appreciate the explanation and the breakdown provided by the authors!

• (Q1) I appreciate the explanation.

• (Q2) Thank you for clarifying!

Finally, I have updated my score a little but in support of this paper getting accepted.

We appreciate the constructive feedback and thoughtful suggestions. We address the concerns as follows:

Exclusive use of proprietary models (GPT3.5 for question generation and GPT4o-mini for freeform judgment) introduces potential bias favoring models from the same provider or those trained/fine-tuned on similar model outputs.

We acknowledge the concern regarding the exclusive use of proprietary models. Our choice was driven by the remarkable generalization capabilities of OpenAI models, which have consistently demonstrated strong performance across diverse domains and tasks without extensive domain-specific fine-tuning. While this introduces reliance on a single provider, it allows us to focus on the inherent strengths of these models in handling varied tasks. Future work can incorporate a broader range of models to address potential biases.

The proposed metrics demonstrate an inconsistent correlation with expected language model performance. For instance, GPT3.5 exhibits superior or comparable performance against newer models, including those from the same family (GPT4o), on BLEU and BERTScore metrics as shown in Figure 5, highlighting potential metric reliability concerns in model evaluation.

We agree that BLEU and BERTScore may favor the generator model (GPT-3.5) due to the reference answers being generated by it, which could explain the observed performance discrepancies. However, BLEU is not immune to paraphrasing, and BERTScore does not validate scientific accuracy. This reinforces the motivation behind our proposed Factual Accuracy metric, which evaluates factual and contextual correctness for a more robust assessment. We can see from Figure 5 that Factual Accuracy does not show this bias and this is concluded in the paper. Kindly refer to the newly added case study provided in Appendix 2.3 for a detailed analysis of how the proposed metric proves to be robust.

The benchmark's reliance on textbook knowledge may introduce distribution bias in the evaluation process.

We believe that reliance on textbook knowledge is a strength rather than a limitation, as textbooks represent foundational knowledge, consensus among experts, and strong coverage of basic concepts. This ensures that our benchmark evaluates models on well-established and widely accepted information, providing a stable and reliable basis for assessment.

The research provides a limited analysis of the expert validation process and common question generation errors, which could offer valuable insights into potential biases in the synthetic benchmark version.

Thank you for your valuable feedback. We have conducted further analysis of the expert validation and included more details in the newly added Appendix 5.1. During the validation of the ClimaQA-Gold dataset, experts identified several common reasons for rejecting generated QA pairs, including wrong answer, bad context, question doesn't make sense, not enough info, and other factors. While most of these errors are attributed to limitations in the generator LLM, the bad context category highlights issues with the seed context, suggesting that improvements in preprocessing could help reduce such errors. This is an area we plan to address in future work to enhance the quality and reliability of the benchmark.

We appreciate the constructive feedback and thoughtful suggestions. We address the concerns as follows:

Complexity of Generated Questions: While the QA generation pipeline is well-designed, some questions in ClimaQA-Silver may lack the complexity required to thoroughly test advanced climate models. Supplementing the dataset with more nuanced, multi-layered questions could improve its utility.

We agree that creating a dataset with more nuanced, multi-layered questions is a valuable suggestion, and we will consider it to enhance the dataset's ability to better evaluate model performance in future iterations.

Dependence on Manual Annotation for ClimaQA-Gold: Although ClimaQA-Gold is an expertly annotated dataset, its reliance on manual validation limits scalability. Developing a semi-automated validation process or implementing peer review from multiple experts could reduce bias and improve reproducibility.

This is an insightful suggestion. We agree that using multiple experts in the validation process is a valuable approach to reduce bias and improve reproducibility. Implementing a semi-automated validation process involving peer review from multiple experts is an excellent idea and something we would love to explore in future work.

What is the potential for scaling the ClimaQA benchmark to other scientific domains? Could ClimaGen be adapted for other domains like environmental science or epidemiology, and what would be the main challenges?

The ClimaQA benchmark has significant potential for scaling to other scientific domains, as long as we have access to relevant data sources. The main challenges in adapting ClimaGen for these domains would be collecting appropriate data sources and securing domain experts for validation. Once these resources are in place, the framework could be effectively extended to evaluate model performance across different scientific fields.

Have you considered additional metrics for freeform QA evaluation? While factual accuracy is addressed, additional metrics like coherence and logical flow might further enhance the assessment of model responses on freeform tasks.

We agree that metrics like coherence and logical flow would enhance freeform QA evaluation. In addition, we are considering a tiered evaluation system for a more nuanced analysis. These are promising directions for future work, and we plan to explore them further.