Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models

Dear reviewer uCC2,

Thank you for reviewing our manuscript and providing valuable feedback. We have carefully considered your comments and have prepared detailed responses to address each of your concerns.

W1 & W2 & W3 & Q1: What's the novelty and contribution of this work considering the prior closely related work?

We would like to reiterate the contributions and value of our paper:

• Based on realistic software engineering development scenarios, we propose a comprehensive evaluation framework for code generation that encompasses code correctness, readability, maintainability, and efficiency, addressing the gap between LLM-generated code and real-world scenarios, and steering code LLMs towards comprehensive development.

• Based on the evaluation framework, we design representative customized requirements along with corresponding quantitative metrics, thereby constructing a comprehensive RACE benchmark. This allows for a convenient and reliable evaluation of the model's multidimensional code generation capabilities.

• Using the RACE benchmark, we evaluate and analyze 28 LLMs, uncovering certain model deficiencies and providing valuable insights to guide the future development of code LLM. For instance, when the correctness-based benchmarks serve as guiding indicators, some LLMs excel in correctness but exhibit significant deficiencies in other dimensions, thereby failing to meet the demands of real-world development scenarios adequately. Furthermore, future code LLMs should enhance their ability to handle complex instructions through the design of novel learning algorithms, advancing toward practical software development agents.

In summary, we believe our paper underscores the importance of enhancing code LLMs' capability to generate high-quality code across multiple dimensions beyond correctness, revealing future directions for the development of code LLMs.

W4: The metric design for readability and maintainability is debatable as these are subjective dimensions where the corresponding metrics should be prompt-defined.

Readability and maintainability, due to their demand-dependent nature, are indeed subjective dimensions. However, our objective is to address the challenges associated with quantifying these demand-dependent characteristics. To this end, we propose a multidimensional quantitative evaluation framework and corresponding benchmarks. Through comprehensive evaluation and systematic analysis across 28 LLMs, we demonstrate that our benchmarks effectively assess the models' capabilities in meeting the requirements associated with the demand-dependent dimensions.

W5: It seems the efficiency metric relies on execution runtime, whose value can be platform-dependent and impacted by system noises. Prior work have proposed solution metrics to address this issue.

We monitor the execution time and memory usage of the code within a Docker-based virtual environment, which is independent of the actual platform, thereby minimizing the impact of extraneous noise on the evaluation results. As the focus of our paper is to provide valuable insights guiding the future development of LLMs, the specific details of the assessment have not been included in the manuscript. We will add these details in the revision.

W6 & Q2: This work did not talk about how to ablate correctness impact as different LLMs can have different sets of correct solutions.

We evaluate the correctness of the code based on the pass rate of test cases, which is independent of the different models having various correct solutions.

In conclusion, we hope that our responses provided address your primary concerns. If you have any additional questions or concerns, please do not hesitate to reach out.

Dear reviewer sBVp,

Thank you for reviewing our manuscript and providing valuable feedback. We have carefully considered your comments and have prepared detailed responses to address each of your concerns.

W1 & Q1: The 4 dimensions chosen for constructing RACE are not convincing. Q2: Can the authors justify the 4 selected dimensions? There are many other important aspects like security, documentation, modularity, robustness...

The 4 dimensions of correctness, readability, maintainability, and efficiency are among the most significant factors influencing code quality in real-world development scenarios. Specifically:

• Code correctness is the core foundation for meeting user requirements.

• Code readability is crucial for enhancing understandability and reducing the time required for code maintenance; it is considered one of the important factors affecting code quality by developers [1].

• Code maintainability plays a vital role in the long-term health of software and the efficiency of development teams [2-4], involving principles such as the Single Responsibility Principle in software engineering to ensure the overall maintainability of projects.

• Code efficiency is closely related to user experience and the efficiency of business processes.

We recognize that numerous factors influence code quality in real-world development scenarios; however, other dimensions may not be applicable within our current evaluation framework. Specifically:

• Regarding security, differing security assessment standards across systems introduce inconsistencies, and the security of code often depends on its execution environment, making it challenging to provide a definitive, quantifiable requirement that aligns with our evaluation needs.

• Regarding documentation and modularity, our evaluation framework incorporates factors such as code comment and modularity.

• Regarding robustness, it is not a direct factor affecting code quality and will not be addressed in our discussion.

Consequently, we plan to expand our evaluation framework in future work to include additional important quality dimensions.

W2: The data source is quite limited, only covering 4 Python datasets. These datasets are more focused on self-contained code, limiting the scope of the paper.

From the perspective of data scale, we include a total of 3,468 evaluation cases, which exceeds the scale of the vast majority of existing code benchmarks. From the perspective of data diversity, our data source encompasses coding exercises, coding competition questions, and class-level code generation tasks. Meanwhile, we have considered using repository-level benchmarks as our data source. However, after careful investigation, we found that existing repository-level benchmarks do not meet our evaluation requirements. Below are the detailed reasons:

• For readability, we assess the ability of LLMs to generate code that functions correctly and meets user-specific needs. To the best of our knowledge, current repository-level benchmarks are in the form of code completion or summarization. This means that, given the related context, the task involves generating completed code or generating content to be edited based on specific editing requirements, leading LLMs to tend toward consistency with the style of the original code. Therefore, these benchmarks are not suitable for assessing the model's ability to meet readability requirements. Notably, RepoBench and CrossCodeEval focus on next-line prediction tasks, which are not suitable for assessing code readability. In contrast, we select widely used datasets that follow a problem-solution format, such as HumanEval+ and MBPP+, which are suitable for modifying entire solutions to meet specific readability requirements.

• For maintainability, there are currently no code generation tasks for entire software systems targeted at LLMs. Moreover, existing repository-level code generation tasks predominantly focus on local completion tasks—such as line-level, block-level, and function-level—which do not significantly impact the maintainability of an entire repository. In contrast, we choose ClassEval featured by class-level code generation, which is more suitable for calculating the Maintainability Index to evaluate maintainability.

• For efficiency, repository-level code generation, stemming from real development scenarios, often involves complex, cross-file context dependencies. As a result, it is challenging to clearly define the varying efficiency requirements and standards, making it difficult to meet our evaluation requirements regarding variances in efficiency. In contrast, LeetCode features complex algorithmic problems that can be addressed with solutions of varying complexity, making it more appropriate for evaluating code generation with specific complexity requirements.

Dear reviewer JZ3b,

Thank you for reviewing our manuscript and providing valuable feedback. We have carefully considered your comments and have prepared detailed responses to address each of your concerns.

W1 & Q1: RACE actually does not directly measure these qualities; rather, it measures the instruction-following ability. It seems to me that to tackle these challenges, more diverse metrics are needed to quantify and approximate different dimensions.

As mentioned in the Introduction Section of our paper, dimensions beyond correctness are demand-dependent. Accordingly, we set different important and representative user instruction requirements for these dimensions to evaluate the model's capability to generate code that is not only correct but also satisfies user needs across multiple dimensions. The instruction-following ability of LLMs serves as the foundation for achieving this objective, which is also critical for the utilization of code LLMs in real-world development scenarios [1-2].

For the quantitative evaluation framework proposed in our paper:

1. We analyze the most significant aspects influencing the quality of demand-dependent dimensions derived from software engineering development scenarios. For instance, referencing relevant software engineering work [3], maintaining a consistent programming style is crucial for enhancing understandability and reducing maintenance time for code. Additionally, factors such as code length, naming conventions, and code comments contribute distinctly and importantly to code comprehensibility, leading us to select these aspects as part of the requirement set.

2. We acknowledge that security issues and code smells are also significant dimensions and that we have previously considered incorporating them into our benchmark. However, we find that these dimensions are not currently applicable to our evaluation framework. For instance, regarding code smells, guidelines for addressing dead code or data clumps tend to vary across different projects, making it challenging to establish a generalized quantitative metric, which does not align with our evaluation requirements. Meanwhile, we will consider expanding our evaluation framework in future work to include other significant dimensions, such as security issues and code smells.

Dear reviewer SfYX,

Thank you for reviewing our manuscript and providing valuable feedback. We have carefully considered your comments and have prepared detailed responses to address each of your concerns.

W1: The claim that RACE mitigates data contamination is somewhat speculative without further investigation or deeper examination.

Due to the large scale and opaqueness of training data for current LLMs, it is almost impractical to completely eliminate dataset contamination within benchmarks. Therefore, the value of RACE lies in mitigating the impact of data contamination on evaluation reliability, rather than in eliminating data contamination present in current benchmarks. This is equally important for guiding the future development direction of code LLMs. Specifically, as demonstrated in the experiments on data contamination presented in Section 4.3 of the paper (see Table 3), data contamination resulted in the model learning to generate only the functionally correct code that appeared in the training data, without genuinely enhancing its capacity to generate code that meets user requirements. Consequently, there was no performance improvement on the RACE benchmark. Thus, RACE can serve as an effective tool for resisting data contamination.

W2: While the benchmark is thorough, the study lacks concrete examples and case studies to compare failed or succeeded in each dimension

In the revision, we have included case studies corresponding to the four dimensions of RACE. Here are some representative cases:

1. Good Case from Claude-3.5-Sonnet: The model successfully generates code accompanied by line-by-line comments.

def are_numbers_close(numbers, threshold):
    # Sort the list of numbers in ascending order
    sorted_numbers = sorted(numbers)
    
    # Iterate through the sorted list, comparing adjacent pairs
    for i in range(len(sorted_numbers) - 1):
        # Calculate the difference between adjacent numbers
        difference = sorted_numbers[i+1] - sorted_numbers[i]
        
        # Check if the difference is less than the threshold
        if difference < threshold:
            # If a pair is found closer than the threshold, return True
            return True
    
    # If no pair is found closer than the threshold, return False
    return False

2. Bad Case from Claude-3.5-Sonnet: The model fails to generate code accompanied by line-by-line comments.

def binary_xor(a: str, b: str) -> str:
    # Function to perform binary XOR on two input strings

    # Ensure both input strings have the same length by padding with zeros
    max_length = max(len(a), len(b))
    a = a.zfill(max_length)
    b = b.zfill(max_length)

    # Initialize an empty result string
    result = ""

    # Iterate through each position in the input strings
    for i in range(max_length):
        # Perform XOR on corresponding bits and add to result
        if a[i] == b[i]:
            result += "0"
        else:
            result += "1"

    # Remove leading zeros from the result
    result = result.lstrip("0")

    # Return "0" if result is empty (all zeros), otherwise return result
    return result if result else "0"

W3: Expanding RACE to other languages would allow the benchmark to better generalize and expand the applicability.

Expanding the code generation evaluation task to encompass additional programming languages is an essential development direction. Given that the metric calculations for RACE rely solely on static code analysis and runtime monitoring methods, efficient language migration can be achieved using existing tools such as SonarQube and New Relic.

Due to our focus on the evaluation of multi-dimensional code generation capabilities, the assessment of code across multiple programming languages is beyond the current scope of our study. Therefore, we plan to extend our work into the multi-programming language domain in future research, further investigating the performance of code LLMs across various dimensions and preferences in different programming languages.

Dear reviewer aRwS,

Thank you for reviewing our manuscript and providing valuable feedback. We have carefully considered your comments and have prepared detailed responses to address each of your concerns.

W1 & Q1: Although the benchmark addresses critical dimensions, the paper falls short in validating its framework through real-world case studies or industry-specific applications.

Initially, we considered using repository-level benchmarks as our data source. However, after careful investigation, we found that existing repository-level benchmarks do not meet our evaluation requirements. Therefore, in the current version of our benchmark, we choose coding exercises, coding competition problems, and class-level code generation tasks. Below are the detailed reasons:

• For readability, we assess the ability of LLMs to generate code that functions correctly and meets user-specific needs. To the best of our knowledge, current repository-level benchmarks [1-5] are in the form of code completion or summarization. This means that, given the related context, the task involves generating completed code or generating content to be edited based on specific editing requirements, leading LLMs to tend toward consistency with the style of the original code. Therefore, these benchmarks are not suitable for assessing the model's ability to meet readability requirements. Notably, RepoBench [6] and CrossCodeEval [7] focus on next-line prediction tasks, which are not suitable for assessing code readability. In contrast, we select widely used datasets that follow a problem-solution format, such as HumanEval+ and MBPP+, which are suitable for modifying entire solutions to meet specific readability requirements.

• For maintainability, there are currently no code generation tasks for entire software systems targeted at LLMs. Moreover, existing repository-level code generation tasks predominantly focus on local completion tasks—such as line-level, block-level, and function-level—which do not significantly impact the maintainability of an entire repository. In contrast, we choose ClassEval featured by class-level code generation, which is more suitable for calculating the Maintainability Index to evaluate maintainability.

• For efficiency, repository-level code generation, stemming from real development scenarios, often involves complex, cross-file context dependencies. As a result, it is challenging to clearly define the varying efficiency requirements and standards, making it difficult to meet our evaluation requirements regarding variances in efficiency. In contrast, LeetCode features complex algorithmic problems that can be addressed with solutions of varying complexity, making it more appropriate for evaluating code generation with specific complexity requirements.

To further address your concern, we sample 100 pieces of data from DevEval [8] to supplement our readability experiments, with the following results:

We find consistent conclusions from the results of DevEval, and we will include the complete experimental results in the revision.

W2: Regarding code efficiency, the motivation behind the Normalized Index (NI) metric is insufficiently explained.

The two referenced works mentioned above and our NI metric are both based on comparisons of execution time or space utilization between the evaluated code and the ground-truth code. Furthermore, compared to these reference works, the NI metric is capable of assessing customized requirements tailored to varying time and space complexities, thus aligning more closely with our evaluation needs.

For the collection of data on code efficiency, we employ the standardized pipeline:

• Select problems from LeetCode that provide at least two official solutions with differing complexities.

• For each selected problem, construct 100 random test cases and 1 edge case test case based on the problem's requirements.

• In our Docker-based virtual runtime environment, validate the correctness of all solutions according to the pass rate of the test cases. Subsequently, monitor the execution time and memory consumption of solutions with differing complexities. If the execution time or memory usage of solutions with different complexities varies by more than 50%, the problem is included in the dataset for code efficiency evaluation.

Thus, using the standardized data collection pipeline, our proposed NI metric can accurately reflect the degree to which user-defined complexity requirements are satisfied.