DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination

Thanks for your valuable comments.

How Our Dynamic Metric Mitigates Data Contamination

For the static metric pass@k, the same fixed problem prompt is fed to the LLM multiple times, leveraging its sampling capability to generate different outputs. However, since this prompt is publicly available and remains unchanged, pass@k becomes unreliable if the prompt is contaminated in the LLM’s training data.

In contrast, our proposed DivPass@K generates multiple randomized problem mutations using our approach before feeding them to the LLM. These dynamically generated prompts are not static, publicly available, or present on the Internet, reducing the risk of data contamination. We will clarify the working mechanism of our dynamic metric in the final version.

More challenging and uncontaminated benchmarks

We also applied our approach to LiveCodeBench, a newly collected competition-level programming benchmark sourced from LeetCode and other platforms. The results, shown in the table below, demonstrate that DyCodeEval can be effectively applied to challenging and uncontaminated benchmarks.
Notably, Qwen2.5-Coder’s accuracy did not drop as significantly as it did from HumanEval to LiveCodeBench. This is because LiveCodeBench was released after Qwen2.5-Coder, reducing the likelihood of data contamination.

Uniform sampling in our assumption

We will revise our theorems and clarify that they rely on the assumption of uniform sampling.

Other evaluation metrics

Besides the correctness metric, we also consider the test case pass rate to evaluate partial correctness. The results are shown in the following table. We observe that, after applying our transformation, the test case pass rate increases for some models. This is because our transformation generates diverse variants of the problem, which may change the models reasoning and make it poential to get a partial correct solution.

Thanks for reviewing our paper and valuable comments

Semantically Equivalent Validation and Human Study

By “semantically equivalent variations,” we mean that the generated problems can be solved by the same code solution as the original. We validate this through:

1. Automated Validation:
   A large language model (LLM) acts as a probabilistic oracle to check:

   • Whether the rewritten problem retains the original meaning.
   • Whether the original solution remains valid for the new problem.

2. Human Verification (Appendix D):
   Two graduate students independently reviewed N = 30 problem pairs per dataset (60 total), assessing whether the core algorithm and complexity were preserved. They initially disagreed on three pairs but reached consensus after discussion, yielding a 95% agreement rate.

We provide the sampled data on our website (see four CSV files in the directory).

Finetuning details

We fine-tune the selected code LLM on randomly sampled portions of the benchmarking dataset, ranging from 25% to 100%, using a standardized instruction tuning objective. The fine-tuning process employs a learning rate of 5e-5, a batch size of 8, and runs for 20,000 steps.
We acknowledge that fine-tuning on a small subset can impact the instruction-following capabilities of the model due to overfitting. However, this phenomenon is precisely the risk posed by data contamination—overfitted models exhibit artificially high performance on contaminated benchmarks, creating a false sense of intelligence while sacrificing generalizability. This issue is empirically demonstrated in Figure 4 (first row), where the red bars highlight the inflated accuracy due to overfitting, while the blue bars indicate the degradation in general capabilities. The presence of such overfitting underscores the need for contamination-free benchmarking, which is the primary motivation of our work.
Regarding pretraining leakage, we note that instruction fine-tuning can override or mitigate the effects of pretraining data exposure. Our study does not aim to analyze pretraining leakage directly but rather to simulate its effects in a controlled manner. To achieve this, we follow established methodologies in the literature, where instruction-tuning-stage leakage is widely used to approximate the impact of training data contamination. This approach allows us to systematically examine how leakage-induced overfitting distorts benchmarking results.

Unnecessary theorems

We strongly believe that these theorems are both necessary and valuable. They provide probabilistic guarantees to benchmark providers, ensuring that an entity with ulterior motives cannot easily overfit a model to achieve artificially high scores on our benchmarks. Thanks to our hierarchical transformation framework, we can control the search space at each transformation layer, effectively mitigating the risk of collisions. This approach allows us to maintain a manageable search space at each layer while achieving a significantly large total search space.

The claim that "prior LLM steps could simply be repeated or resampled (line 252) in case of a collision" does not apply to our scenario. Benchmark providers keep their scenario and context pools private to prevent manipulation rather than expose them for adversarial exploitation. As these pools act as a private key, our framework ensures transparent benchmarking while minimizing contamination risk. While brute-force overfitting is possible, our theorems show it would require an impractically large number of trials, making it infeasible.

To highlight the advantages of our hierarchical transformation for reducing the rist of collision. We conduct a empirical evaluation, the setup and results are shown on our website.

Baselines in Sec4.4 We did describe and cite the baseline methods in Section 4.4. Below, we provide further details on the specific mutations:

• Token Mutation: Randomly replaces a token in the original prompt with another token.
• Char Mutation: Randomly inserts a character at a random position in the original prompt.
• Func Mutation: Changes the function name style in the prompt, e.g., renaming "MyMethod" to "my_method."
• Insert Line: Randomly inserts blank lines in the original prompt.
• CommSyntax: Modifies the syntax of comments in the prompt (e.g., changing # comment comments to """comment""" style).
  These mutations are derived from the robustness-based mutations proposed by Wang et al. (2023). Additionally, PPM (Chen et al., 2024) concatenates the original problem description with a newly defined problem description to test robustness.
  We used publicly available implementations to ensure consistency and reproducibility.

Thanks for reviewing our paper and valuable comments

---

​Empirical values of our theorical bounds

To empirically evaluate the collision rate of our method, we conduct an experiment on HumanEval. First, we run DyCodeEval on HumanEval to generate an initial set of transformed programming problems. We then repeat this process $N$ times $N = 10, 20, 30, 40, 50$ and measure:

1. Repeat rate – the proportion of problems from the initial transformed set that reappear in the subsequent $N$ runs.
2. Collision rate – the proportion of problems within the $N$ runs that are duplicates of any previously generated problem, regardless of whether they match the initial set.

To highlight the advantages of our hierarchical transformation, we compare it against a baseline where we prompt an LLM (using the following prompt) to generate a new programming problem from a given seed. We report both the repeat rate and collision rate for this baseline as well.

Baseline Prompt

Rewrite the following problem description to create a new problem description for new scenario\n\n

Original Problem Description\n
{ori_inst}\n\n

Please ensure to put your rewritten problem description in <new_problem></new_problem> tags.

---

​Is Sonnet sufficient for more complex tasks

To assess whether Sonnet is sufficient for more complex tasks, we apply our approach to LiveCodeBench, a competition-level programming benchmark. The following table shows the number of tokens in these three datasets. For LiveCodeBench, we randomly selected 100 seed and transformed programming problem pairs and evaluated their semantic equivalence. Our analysis found that 92 out of 100 pairs were semantically equivalent, demonstrating the effectiveness of our transformation approach.

---

​Other evidence to validate that Qwen2.5-Coder-7B contains data contamination

Another indication that Qwen2.5-Coder-7B may be overfitted comes from LiveCodeBench, where its evaluation also shows an unusually large accuracy drop on newly collected programming benchmarks, similar to our findings in Figure 6.

---

​Essential References Not Discussed

We will add all mentioned paper to related work.

Thanks for reviewing our paper and valuable comments.

​Concern about the benchmark's discriminative power

We appreciate the reviewer’s feedback and the opportunity to clarify our findings. However, we believe there may be some misunderstandings regarding Figure 5.

First, the lower number of outliers in Figure 5 does not imply that our benchmark lacks discriminative power. The key evidence for distinguishing overfitted models is in Figure 4, not Figure 5. In Figure 4, we fine-tune models with controlled contamination and observe a significant accuracy drop on our benchmark as leakage increases (second row). This confirms that models overfitted to contaminated data perform well on the original benchmark but fail on ours.

In contrast, Figure 5 evaluates potential data contamination in publicly available LLMs, not discriminative power. Without access to these models’ training data, we cannot confirm overfitting but instead analyze their performance differences. Our findings show:

1. A linear relationship between accuracy on our benchmark and the original, indicating comparable problem complexity.
2. Anomalous behavior in certain models, such as Qwen2.5-Coder-7B, which experiences an unusually large accuracy drop, falling outside the 95% confidence region. While we cannot confirm contamination, this suggests potential data leakage, which is why we use the term “may be contaminated.”

Second, the fact that open-source models perform similarly to closed-source ones does not mean our benchmark is too easy to track future advancements. Our focus is on transparent evaluation rather than increased difficulty. A reliable benchmark should measure true generalization while avoiding misleading performance inflation caused by data contamination.

Relationship with EvoCodeBench

EvoCodeBench is constructed by collecting programming problems from GitHub, following a similar approach to LiveCodeBench, as discussed in our Introduction section. However, this method has several limitations: (1) It shifts the burden of manual question design to coding platform authors. (2) Since problems come from public GitHub repositories, existing models may have already seen them, raising concerns about data contamination. (3) EvoCodeBench relies on external contributions, leading to infrequent updates—the latest update, per their website, was nine months ago, which is inadequate given the rapid pace of model development.

While DyCodeEval is fundamentally different, we acknowledge EvoCodeBench as related work and will include it in the related work section. However, its existence does not diminish our contributions, as DyCodeEval is fully automated and scalable.

Figure 6

We have revised Figure 6, on our website.