From Queries to Criteria: Understanding How Astronomers Evaluate LLMs

Thank you so much for your helpful feedback and appreciation of our work. Regarding reasons to reject (RtR):

• RtR 1: We completely agree that evaluation criteria should involve multiple dimensions, as we specify in Table 2. As our work focuses on understanding human evaluation practice, we do not fully implement automated metrics for all of our proposed criteria, though we offer suggestions of how to do so in Section 5, which we encourage future work to implement
• RtR 2: User judgments were highly correlated: out of 222 queries that received a reaction, there were only 5 cases where one user gave a thumbs up and another user gave a thumbs down. As our primary goal is not evaluating this specific deployed system, but rather understanding user evaluation practice, we do not provide finer-grained evaluations of this system. Instead we propose user-grounded guidelines for future evaluations.
• RtR3: Thank you for this suggestion. We agree that comparing outputs of different systems is generally useful. Our work suggests that users should rate outputs along a range of criteria rather than just selecting which one is preferred.

Thank you for the minor comments, we will address them in the revisions.

Questions:

• Q1: 7 participants received PhDs in 2015-2020 and 4 received PhDs after 2020.
• Q2: 2 of the annotators are senior astronomers (10+ years post-PhD), 3 of them are early career astronomers (postdocs or assistant professors), and 2 of them are senior astronomy PhD students. They currently hold primary appointments at 3 different institutions, and all of them have secondary appointments and/or completed training at additional institutions. They focus on a variety of subfields within astronomy (e.g., we recruited an additional annotator when we realized none of our original 6 focused on exoplanets). One of the annotators is a women and 6 are men.
• Q3: One of the astronomer authors on our paper read through all of the gold responses and validated that they were reasonable. They did observe that the responses varied in style, which is likely reflective of varying user preferences

Thank you for your review of our paper. We address your Reasons to Reject (RtR) and Questions to the Authors in a combined manner as follows:

• RtR 1: Our study is based on a deep analysis of 367 queries from 35 users with 11 follow-up interviews all of whom were recruited with expertise in astronomy. If we expanded the scope, we would need to reduce the depth of analysis, which currently includes manual coding of user queries, follow-up interviews, and construction of gold query-answer pairs. Furthermore, our study matches or exceeds the scope of similar human-centric studies, where a sample size of 6-20 users is common (e.g. Mirowski et al. 2023 similarly studying LLM usage with 15 users total; Lazar et al. 2017 discuss methodology more generally; other example studies include Feng et al. 2025, Zhao et al. 2022 and Liao et al. 2020). As noted by other reviewers, the scope of our study is sufficient for identifying interesting trends in user evaluation practice that can help improve LLM evaluation and can be further validated in follow-up replication studies.
• RtR 1, RtR 3 and Q1 about “outdated tech stack”: Our deployment and data collection was conducted in June 2024. Thus the underlying set of papers ending in July 2023 excludes only the most recent year of research. Astronomy research moves more slowly than ML, and the exclusion of the most recent year of data is extremely minor compared to the large volume of literature. In our review of user queries, we identified only a very small number (~5) of queries that targeted topics too recent for our system. Furthermore, our work focuses on analyzing human evaluation practice: what types of queries users posed and how they judged responses, not evaluating or optimizing the performance of the specific system we deployed. Our system was sufficient for this use case. There is also clear evidence that users found the system useful. In follow-up interviews, they appreciated that our setup enabled providing specific references (Sec 4.2) and they continued querying it throughout the duration of the study (Fig 6; Appendix F).
• RtR 2/Q2: We would like to clarify a misconception: our study does not assume that users were genuinely using the system for research. We assume that users were primarily seeking to test and evaluate the bot. This assumption is validated through follow-up interviews, where almost all interviewees described probing the bot about things they knew (Sec 4). A smaller subset of users did try out the bot on actual use cases, but primarily ones they were also approaching through other means (Sec 4). The assumption that most queries were about evaluation rather than usage is integrated throughout our work, including its title. The Hawthorne effect is primarily a concern for researchers studying behaviors that are not socially desirable, which is not the case with our work, especially since prior studies of LLM usage suggest users do not modulate their queries (e.g. they disclose private information Mireshghallah et al. 2024). Attempting to mitigate any potential Hawthorne effect by not informing users that we are logging their queries would be unethical.
• RtR 4/Q3: Sec 5.2 details the construction of the benchmark and validates its usefulness for evaluating LLMs. We show that the automatically estimated closeness between model generated responses and gold responses are highly correlated with human judgements (Pearson correlation = 0.8239). Could you please clarify what details about the benchmark you expected that are not already included in 5.2, or update your review if you originally missed this section? We also have used the gold benchmark data to compare our original chatbot system with an updated system constructed after the study’s completion. We did not include these results in the original submission as we considered them beyond the scope of this work, but we will add them to the appendix.
• Q4: We greatly appreciate your interest in the results and ways this work can contribute to a broader understanding of LLM usage in its current scope. We report the frequency of queries by number of years since PhD completion (a reasonable proxy for career stage) in Fig 4 in Appendix D. We do not see consistent patterns in how total volume of usage varied. There is some variance in query type: stress testing queries were asked more frequently by people who completed their PhDs after 2010. There is a striking pattern in ratings of bot responses: there are more positive ratings on responses to queries asked by younger participants who completed PhDs later (Fig 5, Appendix D, discussed in main Sec 6). This result may indicate unequal adoption and greater familiarity with AI among earlier career researchers, though we do not further validate this result as it is not the focus of our work.

Thank you very much for your helpful feedback and your appreciation of the value of our work.

Regarding reasons to reject (RtR):

• RtR 1: We discuss comparisons with existing evaluation benchmarks in Section 2 and 5, and in revisions we will add more detailed comparisons with Gao et al. 2023 and Asai et al. 2024 (Asai et al. 2024 was released after our data collection and analysis had been completed). These two papers propose methods to automate evaluation criteria, whereas our work focuses on discovering what criteria users care about and how they enact them. Thus our work is complimentary in that it validates the usefulness of metrics like correctness and coverage, uncovers additional criteria like specificity and clarity, and further shows that certain types of queries need to be included in benchmarks to surface these properties.

• RtR 2: Our full analysis data set consists of 367 queries from 35 users with diverse backgrounds (appendix B). While we agree that expanded scope would be valuable, the level of depth that we conduct would not be feasible with a substantially larger scope. We would gladly provide support for follow-up replication studies that repeat our work in a different domain (to facilitate this, the code for our slackbot is already public, though not linked in the paper for anonymity), and we also agree that our work in its current form offers substantial contributions that are of interest to LLM researchers.

Question to authors:

• We initially conducted the human/LLM judgement correlation experiment with gpt-4o-mini because as an inexpensive model, it is more practically usable by researchers conducting multiple rounds of evaluations. As we already saw high correlation with human judgments (0.8239) with gpt-4o-mini, we did not conduct further experiments with more powerful models.

Thank you for your helpful feedback and appreciation of the strengths of our paper, including its fit for the conference, that its “investigations and findings are very interesting”, and has high quality writing, which facilitates reproducibility and follow-up work.

Regarding reasons to reject (RtR):

• RtR 1: Our study matches or exceeds the scope of similar human-centric studies that focus on deep analysis of user behavior. Our study contains 35 users with 11 follow-up interviews, while a sample size of 6-20 users is common (e.g., Mirowski et al. 2023 similarly studying LLM usage with 15 users total; Lazar et al. 2017 discuss methodology more generally; other example studies include Feng et al. 2025, Zhao et al. 2022 and Liao et al. 2020). While our analysis focuses on one domain and a limited set of users, it is extremely deep and multi-faceted. Expanding scope would force us to reduce the depth of analysis (we already had to drastically cut insights from interviews to fit the page limit; we would not be able to manually analyze all queries). We agree that replicating our study with a different user group or investigating how specific aspects scale to larger data are great directions for follow-up work. In its current form, our work already offers valuable insights (e.g. as noted by Reviewer aurw and Sraf) that would be useful to other researchers if published.

  We do not compare against a baseline because our project focuses on exploratory analysis of user behavior, rather than testing performance of a specific system. It is unclear to use how comparison against a baseline could improve our understanding of evaluation practice.

• RtR 2: Our study participants are diverse in terms of race, age, career track, and phase of career (Appendix B, Table 3), and they have varying specialities within astronomy (exoplanets, galaxy clusters etc.) as demonstrated through their queries. Thus, our users reflect a broad range of types of astronomers. Additionally, the dispersion of query types across users (Fig. 2) and overlap in content in follow-up interviews (Section 4) with diminishing findings from new interviewees suggest that our user pool was sufficient to achieve saturation.

  There is one main limitation of our study population: that researchers entirely disinterested in AI (with neither strong positive nor negative opinions) likely did not choose to participate. However, this is a fundamental limitation of research studies conducted ethically with informed consent. Other limitations include our focus on professional astronomers who already have PhDs, rather than students, and people currently in the U.S. (though many participants are not originally from the U.S.), but we do expect our population to be representative within these criteria. We will include these limitations in the main text of the paper.

• RtR 3: We primarily discuss comparisons with other studies in Related Work (Section 2). We will expand this section, including incorporating the references in RtR 1 and integrating them in 3.2 and 4. In general we follow standard best practices for inductive data coding and interview studies. While some prior work has considered aspects of LLMs to evaluate (4.2), less work has examined what types of queries users pose in order to probe for this criteria (4.1). Because our work is novel in this aspect, there are fewer relevant works for this section.

Questions to authors:

• Q1: Thank you for the references to professional search. We will incorporate them.

• Q2: arXiv is heavily used by astronomers. Surveys have shown 93-94% of physicists and astronomers use arXiv with 100% usage in astrophysics (Narayana and M.K. Bhandi 2022). For this reason, arXiv has also been used as a data source in constructing AI for astronomy in prior work (Perkowski et al., 2024).