MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

1. Contribution and novelty

Thank you for your constructive feedback. We agree that our benchmark offers a more fine-grained evaluation, which provides valuable insights into the strengths and weaknesses of LLMs. As we mention in Appendix J, we have indeed conducted further analysis on error patterns to highlight model performance issues across different categories. We believe this analysis helps us identify specific weaknesses in model performance that would be hard to detect using more general benchmarks. Additionally, we also discuss potential strategies for improving LLMs in Appendix F, where we explore targeted improvements based on the errors observed in specific challenge categories.

Regarding the novelty, we believe that the fine-grained categorization of problems is one of the main contributions of our work. It allows us to focus on particular problem types, which helps guide more targeted improvements without the need for extensive error case analysis. This approach saves significant human effort, as analyzing every individual error case in large-scale code datasets would be highly time-consuming and inefficient.

We understand the concern about the contribution being perceived as incremental, but we believe that this granularity of analysis and the potential for actionable improvements in model architectures make it a meaningful addition to existing research.

2. Automated generation could lead to biases

Thank you for raising this point. While we acknowledge that our benchmark relies on manual curation, we intentionally chose this approach over automated problem generation. Automated generation could lead to significant biases, as the generated problems are often ones that models are already good at solving. This would limit the diversity and real-world applicability of the problems. Furthermore, if automated systems were to generate problems, they could potentially introduce a bias where certain models that have been trained on similar datasets might perform disproportionately well on those problems.

In contrast, our manual curation process ensures a more careful and thoughtful selection of problems. We have statistics demonstrating the diversity of the problems we include, as we examine a wide range of problem types. Additionally, we employ meta annotators to ensure that the problems do not overlap or duplicate, which sets our dataset apart from others, such as HumanEval, where there are even highly similar problems. While comprehensive coverage of all possible problem types may not be feasible, we have designed our benchmark with different categories to ensure broad coverage. We do not believe that a "perfect" dataset can cover all possible programming challenges, but we aim to cover a representative variety of problem types to test model performance across different aspects of code generation.

3. Curation and evaluation pipelines ensure diversity and reproducibility

Our approach prioritizes high-quality, human-curated problems to ensure that the problems are challenging and not trivially solvable by current models. We follow a reliable process during dataset construction, starting with analyzing existing datasets and identifying key challenges. After generating problems, we refine and adjust them multiple times to ensure their quality. This manual curation process helps us create diverse, robust problem categories that represent a wide range of coding challenges.

Regarding reproducibility, we have established a clear evaluation pipeline that guarantees reproducibility of results, and the benchmark has been in use for an extended period with consistent results. This pipeline has been tested and verified, ensuring that future use of the benchmark will provide consistent and reliable evaluations.

In summary, we believe that manual curation is essential to prevent bias and overfitting to models, and our well-established, reproducible process ensures the quality and diversity of the dataset.

1. Contribution and Novelty

Thank you for your comment.

• One of the main contributions of our work is the introduction of many challenging, clean code problems, which differ from those in other benchmarks that are mostly derived from publicly available data. The new problems we include require significant manual effort to curate, and we have also established a robust evaluation pipeline for testing. Unlike other datasets, where answers can be easily found, our approach minimizes the risk of data contamination, which is crucial for ensuring the integrity of future evaluations.

• Another primary contribution of our work lies in the more detailed categorization of problem types, which enables us to observe how models perform across a range of challenges. This fine-grained classification helps identify performance gaps, which can guide targeted improvements in models without the need for extensive error case analysis. Unlike existing benchmarks, which typically provide broad categories, our approach allows for deeper insights into specific areas of model weakness and the potential for improvement.

2. Number of tests

We acknowledge that EvalPlus has more tests per problem. However, we believe that as long as coverage is sufficient to test core model capabilities, additional tests may add redundancy. Our coverage tests confirm that we strike a balance by keeping a smaller but diverse set of problems, reducing unnecessary repetition. Additionally, referring to the ClassEval benchmark, we observed from Table 1 in its paper that our manually curated dataset already contains a sufficiently large number of test cases compared to other manually created code datasets. This supports our approach of prioritizing quality over sheer quantity.

3. Challenging for CodeLMs and Domain Limitations

We understand your concerns, but we believe that our benchmark is addressing a complementary need. While BigCodeBench, ClassEval, and SWE-Bench focus on different aspects (e.g., utilizing function calls as tools, generating a class of multiple interdependent methods or software engineering issues), our dataset emphasizes code reasoning and logical problem-solving. This aspect is currently underrepresented in existing benchmarks, making our contribution distinct and necessary.

For example, our Shortcut category tests models' ability to apply concepts like game theory, which is not a focus of the other benchmarks. This provides a new dimension to model evaluation that has been largely absent from existing datasets.

4. Python-Specific Nature of the Benchmark

Thank you for your comment. While many of the problems in our benchmark are implemented in Python, we want to emphasize that the majority of the tasks are not Python-specific; they focus on algorithmic concepts and logical problem-solving, which can be translated to other programming languages.

For example, as shown in the Appendix D of the paper, we have already translated 140 problems into different languages. Additionally, during the rebuttal period, we completed a full translation of the entire 210-problem dataset into C++. The performance of o1-mini on this C++ dataset is shown in Rebuttal Table 3. While performance remains suboptimal across languages, this demonstrates the flexibility of our benchmark to be applied to non-Python languages as well.

We will further explore and expand the benchmark to include more language versions in the future.

Rebuttal Table 3: Results for o1-mini on MHCP Dataset

While I acknowledge the upgrade this benchmark represents in terms of difficulty viz-a-viz existing ones like HumanEval, I still think this benchmark doesn't bring anything new compared to tough benchmarks like BigCodeBench-Hard which have been shown to be challenging and also have low overlap with pre-training corpora. With this in view, I have decided to retain my scores.

1. Benchmark Size and Comparisons to BigCodeBench

We appreciate the reviewer pointing out the comparison to BigCodeBench. However, we emphasize that BigCodeBench is fundamentally different from our benchmark in terms of design and focus. We provide a comparison of examples below. BigCodeBench is not artificially constructed and centers on real-world scenarios without algorithmic problems, which aligns with a different category of problem types. In contrast, our benchmark focuses specifically on algorithmic problem-solving, targeting a direct comparison with HumanEval. This deliberate focus allows us to evaluate algorithmic reasoning capabilities more effectively.

Furthermore, our benchmark is artificially constructed, which reduces the likelihood of data leakage—a common challenge in real-world datasets. Additionally, this approach minimizes biases in problem distribution, ensuring a balanced evaluation of algorithmic understanding. While the size of the dataset may appear smaller than BigCodeBench, we conducted extensive variance analysis to demonstrate that the dataset size is sufficient to ensure reliable and stable performance evaluation.

# example in BigCodeBench
import http.client
import socket
import ssl

def task_func(server_name, server_port, path):
   """
   Makes an HTTPS GET request to a specified
   server and path, and retrieves the response.
   Parameters: ...
   Returns: ...
   Raises:...
   Requirements:...
   Examples:...
   """

# example in MHPP
from typing import List

def generate_string_permutation(raw_data: List[str], replaced_str: str) -> int:
   """Generate all permutations of raw_data elements, excluding one element per permutation.
   Replace up to three occurrences of replaced_str in each permutation with an empty string.
   Return the number of unique strings.
   >>> generate_string_permutation(["a", "b", "c"], "a")
   4
   >>> generate_string_permutation(["hello", "world", "rock", "great"], "or")
   24
   """

2. Data Contamination Concerns

We understand the concern about data contamination and have taken significant steps to mitigate this issue in our benchmark. Specifically:

• Limited Test Set Exposure: Only a minimal portion of the test set (3 test cases for each problem) has been made publicly available, ensuring the vast majority of test problems remain hidden.
• Evaluation Process: We have designed an evaluation pipeline where participants can only get a result report by submitting model outputs using API without knowing any further test case or canonical solution.

We are confident that this pipeline effectively mitigates data contamination risks, as our problems were made available on GitHub early on, and after applying the contamination detector mentioned in Section 3.2, the current contamination rate remains 0%.

3. Additional Results for o1-preview

Thank you for the suggestion. We agree that including o1-preview results strengthens our argument. As shown in Rebuttal Table 1, the evaluation results confirm that o1-preview performs lower than other benchmarks (e.g., 89.0 on Codeforces), highlighting the ongoing challenges faced by current LLMs in mastering function-level code generation. We will include these results in the revised paper to further emphasize this point.

Rebuttal Tabel 1: o1-preview and o1-mini performance on MHPP

1. Benchmark Size and Confidence Intervals

We acknowledge the concern regarding the limited size of the benchmark. To address this, we re-examined the noise levels using a bootstrapping-based approach, similar to methods employed in Eval-Arena and CruxEval. Specifically, we sampled 25 problems from each category (totaling 175 problems) and performed 1000 iterations to estimate the variability across the dataset. The results for GPT-4o, shown in Rebuttal Table 2, indicate that for the entire dataset, the noise levels are manageable. However, for certain categories, the noise approaches 10%, which may impact the reliability of results.

To mitigate this issue in future work, we plan to expand the dataset to include over 1,000 problems and apply Eval-Arena's methodology for evaluating noise more rigorously. This will help reduce noise and improve the robustness of our results.

We also appreciate the reviewer’s observation about the confidence interval analysis in Section 5.1. While the current analysis primarily addresses noise introduced by temperature sampling, it does not fully account for variability due to the benchmark size, which we now acknowledge as a larger source of noise.

Rebuttal Table 2: Noise Levels for GPT-4o with Bootstrapping Sampling.

2. Comparison to EvoEval and Related Work

Thank you for pointing out the similarities to EvoEval. While there are some overlapping categories (e.g., Tool Use in EvoEval vs. Codesense in our work), the two benchmarks differ significantly in scope and objectives. Our Codesense tests a model's ability to call external functions and libraries, while EvoEval’s Tool Use focuses on function decomposition, which is a simpler task (we provide a comparison of examples below.).

#example in EvoEval
def check_vowel(s):
   """helper function"""

def frequency_count(s):
   """Given a string s, count the frequency of each vowel in the string. Return the results as a dictionary. """

#example in MHPP
def get_specific_permutation(N: int, M: int) -> str:
   """Given a integer N (1 <= N <= 9), there are n! different permutations that can be formed using digits less than or equal to N. If we arrange these permutations in ascending order, for example, when n=3, the 1st result is 123, and the 6th result is 321. Please find the Mth permutation when given the values of N and the positive integer M.
   >>> get_specific_permutation(3, 6)
   "321"
   >>> get_specific_permutation(4, 9)
   "2314"
   """

More importantly, EvoEval does not introduce new problem categories, unlike our work, which creates novel categories such as Shortcut, designed to test advanced knowledge like game theory—something EvoEval struggles to address. This distinction allows us to better evaluate models on a wider range of abilities.

We also have concerns that automated generation could introduce significant biases, as the problems generated are often ones that models are already adept at solving. This would limit the diversity and real-world applicability of the problems. Additionally, if automated systems were used to generate problems, they could introduce a bias where models trained on similar datasets might perform disproportionately well on those specific problems.

We will expand on these differences in the related work section and plan to conduct a detailed comparison in future revisions. We also believe that combining EvoEval’s problem-generation approach with our dataset could further enhance our benchmark.

3. Novelty and Contribution

Thank you for your feedback.

• One of the main contributions of our work is the introduction of many challenging, clean code problems, which differ from those in other benchmarks that are mostly derived from publicly available data or generated by models. The new problems we include require significant manual effort to curate, and we have also established a robust evaluation pipeline for testing. Unlike other datasets, where answers can be easily found, our approach minimizes the risk of data contamination, which is crucial for ensuring the integrity of future evaluations.
• The main novelty of our work also lies in the more detailed categorization of problems, which allows us to observe how models perform across different types of challenges. This granularity helps identify performance gaps, guiding targeted improvements without the need for extensive error case analysis.

Although our benchmark does not directly challenge existing performance trends, it offers valuable insights, especially in light of the saturation in benchmarks like HumanEval. We believe this more nuanced approach will help drive future model improvements.