CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

Thank you for your helpful comments and recognition of our CS-Bench's support for multiple languages and various types, in-depth evaluations, intriguing scaling laws, and extensive error analysis. We will address your concerns point by point below:

Weakness 1 & Question 1 & Question 2: CS-Bench is static and openly available online. This may lead to the benchmark being trained on, resulting in data contamination. What do the authors suggest to avoid the issue of contamination, and have the authors considered using some methods for dynamic benchmarking?

It is important to protect the test set from contamination. Your suggestions are very enlightening to us.

In line with your suggestions, on one hand, we prevent the most basic form of test set cheating by randomizing the position of the correct option. On the other hand, considering the diversity and complexity of CS-Bench instances, we find that not all questions are suitable for dynamic reconstruction. Therefore, we carefully select a subset of data suitable for dynamic generation and maintain it as a private data subset. Through this approach, we can dynamically generate test instances to detect whether a model has been privately trained on this bench, thereby mitigating data contamination.

Specifically, we dynamically generate new test instances based on three strategies:

1. Random Dynamic Numbers

For calculation-based questions, such as: "How many different permutations of elements are there when {n} distinct elements are pushed onto a stack and then popped off?", we manually define calculation rules like "ans = Catalan number C(n, 2n)/(n+1)". A system sets a predefined range for random numbers, such as n ∈ (5, 10), and dynamically generates a valid number n, which is then substituted into the formula to compute the correct answer. This ensures that the generated question and its answer are both consistent and reasonable.

2. Statement Rewriting

For open-ended questions, we use GPT-4 to rewrite the questions multiple times while preserving their original meaning. These rewritten versions are manually validated to form a dynamic question pool. For instance:

• Original question:"In DMA mode, what mechanism is typically triggered to issue an interrupt request after data transfer is complete?"
• Rewritten version:"What mechanism usually initiates an interrupt request following the completion of data transfer in DMA mode?"

During usage, the system randomly selects one of the rewritten versions as the dynamic question.

3. Random Semantic Negation

For Assertions questions, GPT-4 generates semantically negated versions of the original question and reverses the answer, with manual validation. For example:

• Original question:“Mailbox communication is a direct method of communication.”(Answer: False)
• Negated version:“Mailbox communication is an indirect method of communication.”(Answer: True)

Weakness 2 & Question 3: What is the utility of CS-Bench when compared to approaches in which LLMs are situated in an environment with access to a terminal and tasked to solve problems?

As mentioned in the "Related Work" section (lines 509-515), a primary motivation for CS-Bench is that while there have been some works exploring the application of LLMs in specific scenarios such as software engineering and terminal environments, a comprehensive assessment of LLMs' more fundamental knowledge and reasoning abilities in computer science has been consistently lacking. Therefore, CS-Bench focuses on comprehensively evaluating LLMs' knowledge and reasoning abilities in computer science, rather than being limited to specific application scenarios.

We believe this is both a strength and a bias of CS-Bench. Combining CS-Bench with studies targeting specific application scenarios can provide a richer perspective. We elaborated on this point in detail in Appendix A: Limitations and Dataset Bias.

Thank you for your valuable comments and recognition of our work in proposing a comprehensive CS benchmark and conducting an extensive evaluation and analysis. We will address your concerns point by point below:

Weakness 1 & Question 5: Some writing and charts are occasionally dense, especially the radar charts, which lack simplicity and clarity.

Thank you for your suggestions on our writing and visualization. We recognize that while Radar Chart 3 in the main text aimed to compare different models' capabilities across various aspects, it indeed lacked clarity. In our new manuscript, we've replaced it with a bar chart, keeping the original radar chart only in the appendix. We'll also continue to refine our writing to enhance overall quality.

Weakness 2 & Question 2: While the paper shows correlations between CS, math, and coding abilities, deeper analysis is needed to understand the underlying factors driving these relationships, such as data characteristics and internal representations.

To analyze the reasons behind capability correlations, we first explore the relationships between different abilities of general models at the representation level. We extract the last hidden layer of Llama3-8B-instruct and obtain representations for different domains through average pooling. We then calculate the average cosine distance between CS-Bench reasoning-type representations and other data representations, where smaller distances indicate greater similarity. The results are as follows:

We find that the similarity of representations between different abilities/domains generally aligns with the observed capability correlations in Figure 9 , especially in the areas of code and math. This explains the correlation between CS skills and math and coding abilities at the representation level. Additionally, due to BBH containing various reasoning data, the distance between CS-Bench and BBH is relatively small, which is consistent with the analysis in lines 465-467 of the manuscript.

Next, we analyze from the perspective of training data characteristics. We intuitively find that some questions in CS-Bench can be understood as mathematical problems in a computer science context. For example:

Q1: If the data part is 3800B and the maximum fragment size is 1420B, what is the total length in bytes of the third fragment?
Q2: The height h of a binary tree with 1025 nodes is ().

, while the main role of code is to supplement the model's knowledge of data structures and logical reasoning abilities. This data-level correlation explains the expert model experimental results in Tables 3 and 4, specifically why math and code expert models help improve CS capabilities in certain areas, even when their general abilities are weakened.

Finally, we analyze the relationship between general models and math & code expert models in computer science representations. We select the chat model Llama2-7b-Chat, math expert model MAmmoTH, and code expert model CodeLlama, all based on Llama2-7b-base, to observe their average cosine distances for the same data points in CS-Bench.

Results show that expert models' representations shift to some extent compared to the chat model, especially the code model, which can be attributed to Code's unique data patterns. Further representation visualization in Figure 19 of manuscript reveals that in the general dataset chatbot-arena, the features of the three models do not significantly separate. However, in math and csbench datasets, the code model forms a distinct feature cluster, while math and chat models' representations remain similar, as we stated: some questions in CS can be viewed as mathematical problems in a CS context. Lastly, in the MBPP dataset, the representations of the three models clearly differentiate and form their own clusters.

Dear Reviewer g1yj,

Thank you for your insightful suggestions. We have done our best to address your concerns. Since the rebuttal period is closing very soon, could you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you for your time and consideration, the authors.

Thank you for your valuable comments and recognition of our work in data quality and thorough review. Next we will now address your concerns one by one.

Weakness 1: It is not clear what the benchmark aims to achieve or improve. LLMs are already good at MCQ, open-ended, FITB and T/F questions. There is no coding in the benchmark. The authors need to discuss more clearly why this is pivotal and important to the field.

Thank you for your question. We will address it in four parts: the motivation and importance of CS-Bench, goals of CS-Bench, LLMs are already good at multiple questions, and why programming questions were not included.

Motivation and Importance of CS-Bench:

CS-Bench is motivated by two key factors: "understanding LLMs' performance in the CS field" and "CS can effectively support cross-capability analysis".

First, CS is a critical and rapidly evolving field. Given that LLMs have shown promising results in other scientific domains like chemistry and biology, as well as in specific CS scenarios such as code completion and cybersecurity, we believe applying LLMs effectively in CS can drive innovation, solve complex problems, and improve efficiency (lines 35-43, 501-515). To achieve this, it is essential to deeply understand the strengths and limitations of LLMs in CS. This understanding can guide their application in appropriate scenarios and inform further improvement and optimization of LLMs (lines 45-50). Empirical studies on LLM performance in CS are among the most effective ways to achieve this goal.

Secondly, considering the intersection of CS with coding, mathematics, and reasoning abilities, we have reason to believe that computer science can effectively support cross-capability analysis and research, addressing the community's shortcomings in this area (lines 50-80).

Goals of CS-Bench:

Building on the motivation above, CS-Bench focuses on two key objectives, as outlined in RQ1 and RQ2 in the introduction:

1. To empirically explore the performance of LLMs in CS, identifying their challenges and potential improvement directions.
2. To investigate the relationships between LLM capabilities in CS, mathematics, and coding, supporting cross-capability analysis.

LLMs are already good at multiple questions:

Although LLMs perform well in tasks like knowledge-based questions and common-sense reasoning, we cannot directly infer that they are equally proficient in the field of CS. One of the goals of our work is to empirically analyze this question (lines 45-50). Within the CS field, CS-Bench includes various question types to simulate different scenarios in CS. The experimental results in Table 2 confirms that even GPT-4 scores only 72.29 overall on CS-Bench, with a reasoning score of just 64.15. For weaker models like Llama3-8B and Llama2-70B, the scores are close to 50, falling far short of being "good."

Exclusion of Programming Questions:

Considering the extensive research already conducted by the LLM community on coding abilities, we view programming as a separate focus area. Therefore, CS-Bench does not include programming questions but instead focuses on the relationship between CS capabilities and coding abilities.

Weakness 2: Scoring on the benchmark appears to be ad-hoc with regular expression matching, instead of measuring with existing techniques.

CS-Bench employs a hybrid scoring approach combining regular expressions and LLM evaluation. As described in Section 3.1 (lines 183-195), regular expressions are used for comprehension tasks (MC and Assertions), while GPT-4 scores generative tasks (FITB and Open-ended questions) based on reference answers.

The motivation for this hybrid approach is as follows: for comprehension tasks, regular expressions reduce the evaluation complexity and enhance result stability. However, for generative tasks, the diversity of valid answers makes regular matching inadequate for accurate evaluation. Regular expressions only assess text at the surface level, while LLM-based evaluation, grounded in semantic understanding and multidimensional criteria, aligns better with human judgment. Appendix D2 validates GPT-4's scoring reliability through consistency with human ratings (Table 11) and examines the impact of using different scoring models (Table 12).

We further provide the Pearson correlation between regular expression-based or traditional scoring metrics and human ratings for generative tasks. The results are as follows:

The results show that the Pearson correlation coefficients for these metrics do not exceed 0.60, indicating poor consistency with human annotations. This further supports the rationale for adopting a hybrid evaluation approach combining regular expressions and LLM scoring.

Dear Reviewer 7dpp,

Thank you for your insightful suggestions. We have done our best to address your concerns. Since the rebuttal period is closing very soon, could you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you for your time and consideration, the authors.

Thank you for your helpful comments and recognition of our work in providing a valuable dataset and thorough analysis. Additionally, based on your suggestions, we will accurately explain the source of the issues in the final version. We will address your concerns point by point below:

Weakness 1: Some sentences need to be toned down, avoiding over-philosophizing discussions.

We greatly appreciate your suggestions and have made initial revisions to the relevant sentences in the manuscript. We will continue to refine other writing expressions to further improve the overall quality.

Weakness 2: The generative questions that require autorating to score are less convincing than the regex-based ones, leading to the consideration of whether the generative questions could be downgraded even further.

We understand your concerns.

First, generative tasks comprise only a small portion of CS-Bench (below 20%). These tasks don't significantly impact the overall score but greatly enrich task types and case studies. To demonstrate the effectiveness of LLM scoring, Table 11 in Appendix D2 showed high consistency between GPT-4 and human scoring, while Table 12 also explored the impact of different scoring models.

Combining suggestions from other reviewers, we further explore scoring generative tasks using rule-based metrics or Regex and calculate their Pearson correlation with human scores. The results are as follows:

The results show that all the Pearson correlation coefficients of these metrics do not exceed 0.60, indicating poor consistency with human annotations. We believe this is because regex methods can only perform surface-level matching and score calculation, while LLM and human evaluation standards can involve deeper semantic and multi-dimensional understanding. We hope the above experiments and analysis can alleviate your concerns.

Question 1: What does your dataset not include for testing competency in CS and what level does this test?

We address your question in two points below:

Aspects not included in CS-Bench:

On one hand, CS-Bench focuses on comprehensively evaluating LLMs' fundamental knowledge and reasoning abilities in computer science, thus excluding assessments of LLMs in specific scenarios. This is because a key motivation for CS-Bench is that while some studies have explored LLMs' specific applications in scenarios like cybersecurity and software engineering, a comprehensive assessment of LLMs' more basic knowledge and reasoning capabilities in computer science has been lacking (lines 509-515).

On the other hand, considering the extensive research already conducted by the LLM community on coding abilities, we view programming as a separate focus area. Therefore, CS-Bench does not include programming questions but instead focuses on the relationship between CS capabilities and coding abilities.

The test level of CS-Bench:

In Appendix A Limitations, we described the overall testing level of CS-Bench as university-level difficulty. We believe this is an appropriate difficulty to differentiate between LLMs, as reflected in the score range from 39.86% to 72.29%. Moreover, this level of difficulty also poses a challenge to the best-performing existing models (GPT-4o only achieves 64.15 points in reasoning-type questions).

The level is primarily influenced by our data sources and the tendencies of designated annotators during data collection. While CS-Bench maintains an overall university-level difficulty, the questions exhibit a diverse range of complexity, spanning from simple data structure definitions to complex computer network application problems.

We have expanded Appendix A from "Limitations" to "Limitations and Dataset Bias" in the new manuscript, providing further supplementation and modifications.

Sure thing. Taking a look!