AutoBencher: Towards Declarative Benchmark Construction

We thank reviewer vFMj for their time and review. Reviewer vFMj primarily asks for ablation studies justifying the impact of different evaluator LM, and clarifications on the qualitative analysis. We have added additional discussions in the paper to address these, summarized below. Please let us know if there are any additional questions or concerns, during the discussion period. Thank you!

“AutoBencher uses GPT-4 as the evaluator, which may introduce potential bias in datasets that could favor LLMs from the same family (e.g., OpenAI's LLMs).”

For all experiments in the paper, we use GPT-4-turbo as the evaluator LM. We notice that GPT-4-turbo generated questions induce the following model ranking: claude-3-opus-20240229 > gpt-4-turbo-2024-04-09 > claude-3-sonnet-20240229 > gpt-3.5-turbo. Since claude-3 is ranked the highest, it suggests that GPT-4 is not exactly biasing towards models in its family. To further justify this point, we set the evaluator LM as Claude-3.5-sonnet.

We find that the dataset discovered by Claude reveals the same relative ranking of the GPT and Claude families, suggesting that the evaluator LM is not significantly biased towards models in its family. Moreover, we report the novelty, separability, and difficulty score of AutoBencher-claude, it’s similar to AutoBencher-GPT4 in novelty and separability, slightly better in difficulty, and preserves the trend compared with HumanBench.

For the safety evaluation setting, we set the evaluator LM as LLaMA 3.1-405B, and we report the ASR (attack success rate) below. We find that AutoBencher (LLaMA-3.1-405B) attains a similar ASR as AutoBencher (gpt-4-turbo).

“How does AutoBencher differ from previous work on automatic benchmarking, such as TaskBench?”

TaskBench proposes a method for generating tool-use benchmarks. Specifically, they (i) construct a full tool graph, which captures all the dependencies between various tools; (ii) they sample a subgraph and use back-instruct to generate instructions that correspond to the subgraph. (iii) they verify the generated instructions and filter low-quality ones.

AutoBencher offers a broader framing than TaskBench and they are complementary. TaskBench step (ii, iii) could be plugged into AutoBencehr as a module of automatic dataset generation for tool-use domain. In AutoBencher terminology, the back-instruct from the subgraph is privileged information that makes the question generation easier for the evaluator LM. This module would be parallel to our automatic dataset generation for knowledge intensive / math QA domains.

We can use the optimization loop in AutoBencher to augment TaskBench by (i) brainstorming more tools and proposing a diverse set of tool graphs. For example, AutoBencher can construct tool graphs based on task descriptions, and it could form new graphs that combine “Huggingface” tools and “Multimedia” tools. (ii) search for systematic failures in tool-use. For example, AutoBencher could discover that the model fails for the chain length exceeding 5; or maybe the model fails when it involves audio-generation followed by retrieval.

We thank reviewer oakz for their time and review. Reviewer oakz primarily asks for clarifications on the salience criteria. We have added additional discussion in the paper to address these, summarized below. Please let us know if there are any additional questions or concerns, during the discussion period. Thank you!

“how is the math/science salience set constructed? The approach to the math and science categories suggests an open vocabulary problem that is not clearly tackled.”

For science, we still use the number of page views of the corresponding Wikipedia article. For math, we evaluate the categories by including humans in the loop to judge and prune the non-salient categories. Specifically, we didn’t do open card sorting, we manually classify each description into two categories {salient, non-salient}. Empirically, we observe that the rejection rate for math categories is close to 0, because prompted GPT-4 for math categories yields highly salient math categories.

“Low-resource languages are both less likely to be tackled by LLMs and by translation tools”

We agree with the reviewer that low-resource languages are a problem that’s hard to address, and we have added this discussion in the limitation section.

"hyper-/hyponymy graph of Wikipedia topics. Did you consider using such a graph, or a similar topic relationship graph to further guide the adaptive search?"

Yes, we have considered a similar idea of using categories and subcategories in Wikipedia in a preliminary experiment. Empirically, for each category (e.g., physical science), there is a large number of subcategories (on a scale of thousands) at different granular levels. We considered concatenating all these category names in the context of the language models to guide the selection of next topics, but this is computationally expensive. Further, this doesn’t help improve dataset desiderata, as the language model often ends up picking some non salient subcategories.

“why multi-objective optimisation was not employed and instead the loss is linearised by adding hyperparameters? (evolutionary/population-based methods?)”

Thanks for pointing out alternative methods of multi-objective optimization. In our use case, linearizing the multiple losses is a good simplification, because we don’t need to construct the full pareto frontier, instead we are just looking for one dataset that maximizes the specified desiderata. The evolutionary algorithms could still be a viable alternative to iteratively optimize our linearized objective.

We thank reviewer bP3L for their time and review. Reviewer bP3L primarily asks for ablation studies of AutoBencher to understand the impact of (i) setting the temperature to 1, (ii) privileged information, (iii) different evaluator LMs; and clarifications on the practicality/compute of AutoBencher. We have added several new experiments and additional discussion in the paper to address these, summarized below. Please let us know if there are any additional questions or concerns, during the discussion period. Thank you!

“language models (LMs) can exhibit high variance across different runs. A robust scoring criterion should account for this issue by incorporating variance between different runs, both for generating questions and answers. Introducing a statistical method to handle this variance would strengthen the contribution.”

In AutoBencher, there are two components that are subject to randomness, (1) dataset description proposal (2) (question, answer) generation. For all experiments in the paper, we set the decoding temperature for the evaluator LM to 0, which yields a deterministic response. We believe that the reviewer made a good point about building a better understanding of the variance involved in AutoBencher, therefore we added ablation studies to answer this question. We included this analysis in appendix G.

First, we set temperature to 1 for the generation of (question, answer) pairs. This means that conditioned on the dataset description and privileged information, we could draw different QA pairs for the distribution. We report the Pearson correlation between the accuracy vectors below:

Pearson Correlation Matrix:

The Pearson correlation across different random seeds is close to 1, which means that the model accuracies are very similar across datasets generated with different random seeds. This suggests that our dataset generator is low variance and robust.

Then, we extend this setting, and set temperature to 1 for proposing the dataset description. We find that randomness here leads to the discovery of different dataset descriptions. The new AutoBencher run generates the description: “International Trade disputes on rare-earth elements”. Although this description is new, we attain similar levels of novelty and difficulty as our initial run, as seen below.

These additional experiments support the argument that AutoBencher is robust to variations in decoding temperature, as well as run-to-run variability when sampling.

Variance analysis

Thanks for the clarification -- I did not know that the temperature was set to 0 for the experiments. I don't remember if this was mentioned in the main paper or not, but if not it should be clarified.

Also thank you for running additional experiments. I am not too concerned about the randomness with the proposing as I am with the scoring of generations.

With regards generation, what is the variance of the (1) separability, (2) difficulty, and (3) novelty of different generational runs? How many (question, answer) pairs do you need to generate to get an accurate estimate of the criteria? Why was $|\mathcal{D}_c| = 50$ chosen as the sample size for the experiments?

Variance Analysis

Thanks for your suggestion! We added a new analysis and have edited the paper to include this.

In the rebuttal, we performed three different generation runs conditioned on the same dataset description with three different random seeds at temperature 1. Now, we bootstrap from them to create different datasets of size N. We plot the standard deviation for novelty, separability and difficulty as a function of N.

See the plot in the paper (appendix G, Figure 3), and also the anonymous link below. https://openreview.net/notes/edits/attachment?id=K4VGVZoJcK&name=supplementary_material Empirically, we observe that the curve looks proportional to $1 / \sqrt{n}$.

Our final generated dataset contains at least 300 examples, and the standard deviation is (0.035, 0.016, 0.019) for novelty, separability and difficulty respectively.

According to this plot, the standard deviation at 50 samples is roughly (0.095, 0.022, 0.039) for novelty, separability and difficulty respectively. This standard deviation defines an interval that excludes the HumanBench’s metrics in novelty, separability and difficulty. Specifically, both novelty and difficulty metrics are less than $\mu - 2 \sigma$. Therefore, 50 samples is roughly the lowest number of samples that we can get meaningful results compared with the human baseline. Once we selected the best dataset description, we run generation again to gather 300-500 examples, which brings our standard deviation down to (0.035, 0.016, 0.019), yielding a robust estimate for the three metrics.

Computational Cost Analysis

Thanks a lot for the suggestion, and we appreciate that you like the cost-effectiveness of AutoBencher!

Specifically, the hyperparameter that will impact computational cost is A = the number of proposed datasets, B = the number of examples per dataset, and C = the number of candidate LMs we evaluate.

In the question generation phase, only the first two matters. The computational cost is roughly proportional to the total number of questions generated (A * B). Note that the context lengths of adaptive search will include historical metrics, which will be larger for each iteration. However, this delta of token cost (around 400 for each new iteration) is negligible compared with the cost of generating the full dataset.

Based on our empirical estimation, AutoBencher takes around 380 * A * B tokens (which costs 0.0075 * A * B dollars) to generate A * B questions.

In the model evaluation phase, the computational cost is proportional to A * B * C, because we need to evaluate every question on every model. In particular, since different candidate LMs have different parameter counts, the cost will also depend on the model size. In AutoBencher, we run inference of 7B models in an A100, and use API calls for larger models. For simplicity, I will use the price of GPT-4-turbo as an upper bound for API cost estimation, because it’s more expensive than or on par with most other models.

The computational cost of evaluating a 7B model on 500 questions is 0.13 GPU hours. So, the total GPU hour would scale based on (0.13 * A * B * C / 500).

For inferring larger models (e.g., Claude, Gemini, GPT), the token cost also scales linearly with ABC. The average length of question+response is around 80 tokens for knowledge intensive questions. The API token count is A * B*C * 80, and the cost is around A * B * C * 0.0016 dollars.

For judging the correctness of model responses (LLM-as-judge), it also scales linearly with A * B * C. Each LLM-as-judge query+response length is roughly 120 tokens. The API token count isA * B * C * 120, and the cost is around A * B * C * 0.0024 dollars.

In summary, Let C1 be the number of 7B models we evaluated, and C2 be the number of API models we evaluated, and C = C1 + C2.

GPU-hours: 0.00026 * A * B * C1

API-credits: A * B * C2 * 0.0016 dollars

API-credits (LLM-as-judge) = A * B * C * 0.0024 dollars

Note that the wall clock time will not scale linearly with A * B * C, because we can significantly parallelize these computations. All the API queries are independent, therefore they can all be parallelized, so adding more models or more data will not slow down the API calls, and we are not even close to the rate limit.

Harmfulness

For clarification, in the safety setting, the goal is to jailbreak the model, which entails finding the harmful queries that the models fail to reject [1, 2]. Therefore, we are explicitly trying to ensure the generation of harmful questions, and trying to minimize the occurrence of non-harmful questions.

[1] Red Teaming Language Models with Language Models https://aclanthology.org/2022.emnlp-main.225.pdf

[2] Universal and Transferable Adversarial Attacks on Aligned Language Models https://arxiv.org/pdf/2307.15043

Thank you for your detailed response. I am largely satisfied with the variance and cost analysis.

Can the authors update the paper with a revised abstract and introduction?

Also, I believe this question from the original review was never fully answered:

Are there any ablation experiments on the following: the criteria for evaluating dataset descriptions

We thank reviewer A8iq for their time and review. Reviewer A8iq primarily asked for incorporating AutoBencher into popular evaluation frameworks, and for a discussion on the role of privileged information. We addressed these comments below. Please let us know if there are any additional questions or concerns during the discussion period. Thank you!

“The paper would be stronger if it integrated AutoBencher into existing popular evaluation frameworks like Stanford's HELM or HuggingFace's Open LLM”'

We thank the reviewer for this suggestion to make AutoBencher more impactful. We have submitted pull requests to add AutoBencher-Capabilities and AutoBencher-Safety to the HELM repository, and they were accepted and incorporated into HELM.

“how AutoBencher could be integrated into a mainstream framework for safety evaluation?”

AutoBencher is adaptive, meaning that the datasets are generated conditioned on an LM under evaluation to reveal its safety vulnerabilities. Most safety evaluation framework is about red teaming, which is aligned with AutoBencher-safety’s goal and adaptiveness, so they can be easily incorporated. There are also some evaluation frameworks such as HELM that operate with static datasets. In order to produce a static dataset that is compatible with HELM, we combine the AutoBencher datasets for a set of candidate LMs to produce a static dataset that reveals model safety failures across the set of LMs we care about.

“The paper does not make a convincing case why future LMs should be expected to have such capabilities without using tools themselves.”

The key of automatic dataset construction is asymmetry, which doesn’t have to be in the form of tool use. For example, one form of asymmetry is providing more test-time compute to the evaluator LM. As shown by o1’s test time scaling result, more test-time compute can lead to better performance, leading to a stronger evaluator. Asymmetry could also rely on the task structure where forward is easier than backward. For example, randomly browsing the web to observe information is easier than actively seeking information [1]. We can leverage this gap to make the evaluator LMs generate questions that are hard to answer by the candidate LMs.

Additionally, our constructed datasets are a useful intrinsic test of LMs capabilities. Even if we expect realistic downstream LMs applications to use RAG, knowledge-intensive benchmarks such as MMLU and TriviaQA still remain highly useful as capabilities tests of language models.

We have added this discussion in the revised paper (Section E).

“I would expect a benchmark/evaluation focussed paper to be more comprehensive and derive more insights than those currently presented. For instance, AutoBencher currently only generates one-turn (question, answer) pairs; it would be interesting to see it extend to multi-turn data, chain-of-thought, and multi-modality.”

The reviewer made good suggestions about extending AutoBencher to multi-turn QA and multi-modality. We believe these are interesting potential extensions in future work. In this paper, we focused on single-turn QA in order to provide more extensive experiments on capabilities and safety. In this simple setting, we have already discovered meaningful insights, including the knowledge gap of Gemini and the safety vulnerabilities of GPT-4. Some of these automatically discovered insights echo human findings, which is very meaningful.

[1] https://arxiv.org/abs/2403.08140