W1: Simplistic Benchmarks

Thank you for your comment. We respectfully disagree with the assessment that our datasets are simplistic and do not adequately capture the real-world effectiveness. It is worth mentioning that Automated Programming Progress Standard (APPS) [1], is a comprehensive benchmark that includes problems from various coding platforms such as Codeforces and Kattis. It includes three categories, namely Introductory, Interview and Competition, and the Interview-level problems are challenging, matching the difficulty of technical programming interviews. Additionally, HumanEval and MBPP are currently the most widely adopted benchmarks in the code generation field [2,3,4,5].

While the suggested benchmarks are valuable contributions, they serve different purposes or have temporal limitations. The Performance-Improving Edits benchmark [6] focuses on code optimization rather than code generation, which falls outside the scope of our study. As for BigCodeBench [7], while interesting, it is currently under review at ICLR 2025 and has not yet been peer-reviewed or officially published. Furthermore, given its recent release (June 18, 2024), it has limited community validation and adoption (26 citations on arXiv) compared to our selected benchmarks. We believe our chosen benchmarks provide a well-established and appropriate framework for evaluating our method's effectiveness in code generation tasks.

[1] Hendrycks D, Basart S, Kadavath S, et al. Measuring Coding Challenge Competence With APPS[C]//Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

[2] Chen B, Zhang F, Nguyen A, et al. CodeT: Code Generation with Generated Tests[C]//The Eleventh International Conference on Learning Representations.

[3] Luo Z, Xu C, Zhao P, et al. WizardCoder: Empowering Code Large Language Models with Evol-Instruct[C]//The Twelfth International Conference on Learning Representations.

[4] Zhang T, Yu T, Hashimoto T, et al. Coder reviewer reranking for code generation[C]//International Conference on Machine Learning. PMLR, 2023: 41832-41846.

[5] Nguyen A, Karampatziakis N, Chen W. Meet in the middle: A new pre-training paradigm[J]. Advances in Neural Information Processing Systems, 2024, 36.

[6]Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., ... & Yazdanbakhsh, A. (2023). Learning performance-improving code edits. arXiv preprint arXiv:2302.07867.

[7] Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., ... & Von Werra, L. (2024). Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.

---

To be continued.

Thank you for your valuable feedback.

Regarding your concern about the benchmarks, we respectfully disagree with the assessment that our chosen benchmarks are "well-understood and straightforward, requiring little to no further explanation". First, if this is the case, then SEK, which focuses on providing explicit and additional explanations, should perform no better than the baselines. However, our experimental results show SEK consistently achieves superior performance across different LLMs, supporting that the used benchmarks still require further explanation. Second, the pass rates of the Default baseline on the APPS benchmark are comparable to or worse than those on the BigCodeBench benchmark you recommended [1]. These results imply that the difficulty of APPS is at least comparable to that of BigCodeBench: the pass rates on APPS-Introductory align well with those on the full BigCodeBench, while APPS-Interview demonstrates similar difficulty levels to BigCodeBench-Hard. Moreover, all LLMs consistently achieve the lowest performance on APPS-Competition, indicating APPS-Competition is more challenging than BigCodeBench. What's more, APPS has been extensively cited (488 times) and validated by numerous studies [3,4,5] in the field of code generation, while BigCodeBench is still under review at ICLR 2025 and its impact (28 cites) and adoption in the research community are yet to be established. Our approach has shown superior performance on the APPS dataset. Therefore, we believe that our existing benchmark framework is sufficient to substantiate SEK's contributions.

Regarding CoT's negative impact on model performance, we would like to mention that this finding is actually not unusual. Previous work has already demonstrated CoT's inherent unsuitability for generation tasks [2].

Additionally, we would like to point out that we conducted the experiments strictly following your initial review comments:

“The rephrased description would then be fed back into the language model to determine if this simple rephrasing enhances performance as well as SEK does.”

We have also followed your new suggestion (despite being raised less than 24 hours before the discussion deadline) and conducted additional experiments to connect the refined problem descriptions with the original problem descriptions. Specifically, we add the prefix "Revised Problem:" before the refined problem description. In the table below, Original One-Step CoT uses the revised problem descriptions as input, the New One-Step CoT combines both the refined and original problem descriptions as input.

The results and analysis remain consistent with our findings in Section 4.1. Specifically, One-Step CoT and SEK extract different types of knowledge from the problem description. One-Step CoT tends to simply restate the complete problem description, while SEK emphasizes low-frequency keywords that effectively bridge the knowledge gap between the problem description and the code implementation. Consequently, One-Step CoT's approach fails to address the knowledge gap between problem descriptions and implementations, resulting in weaker performance compared to SEK.

---

To be continued.

We hope our response addresses your concerns and demonstrates the validity of our benchmarks and the differences between One-Step CoT and SEK. We appreciate your suggestions and the opportunity to further clarify and improve our work.

[1] https://bigcode-bench.github.io/

[2] Sprague Z, Yin F, Rodriguez J D, et al. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning[J]. arXiv preprint arXiv:2409.12183, 2024.

[3]Zhang S, Chen Z, Shen Y, et al. Planning with Large Language Models for Code Generation[C]//The Eleventh International Conference on Learning Representations.

[4]Olausson T X, Inala J P, Wang C, et al. Is Self-Repair a Silver Bullet for Code Generation?[C]//The Twelfth International Conference on Learning Representations. 2023.

[5]Xu S, Fu W, Gao J, et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study[C]//Forty-first International Conference on Machine Learning.

Q2: Developing more advanced methodologies

Yes, we are actively exploring a recursive keyword generation and explanation system to enhance SEK. This framework would iteratively identify and explain keywords until all terms in the problem description are clear enough to be mapped into their implementation, potentially improving code generation performance.

However, we believe our current approach, i.e., SEK, already makes significant contributions by demonstrating a fundamental insight: the importance of explaining relatively low-frequency terms in problem descriptions. This insight has proven effective with comprehensive experiments and provides a solid foundation for more advanced methodologies.

S1: Considering pre-defined keyword extraction dictionaries or tools alongside LLMs

Thank you for this constructive suggestion. While pre-defined keyword dictionaries can indeed be valuable in certain scenarios, we identified several potential limitations with such an approach. First, given the diversity of models, it is challenging to pre-define an exhaustive dictionary that can cover all possible keywords, and such dictionaries would require frequent updates to remain relevant, potentially limiting their robustness. Second, keywords can vary significantly across domains and contexts. For example, keywords like "gradient descent" in machine learning or "double-entry bookkeeping" in accounting include specialized phrases that vary across fields. Consequently, this would require maintaining different external knowledge bases tailored to each specific task or domain, which is not only resource-intensive but also inflexible. As a result, we respectfully argue that pre-defined keyword dictionaries might lack the flexibility and portability required for broad applicability.

W1: Dependence on LLM's Existing Capabilities, Simplicity, Novelty, and Impact

Thank you for raising these concerns regarding our reliance on LLMs' inherent capabilities and the novelty and impact of our approach.

Dependence on LLM's Existing Capabilities: We respectfully argue that the inherent bias in LLMs is limited when it comes to extracting relatively low-frequency terms. In our paper, the concept of low-frequency primarily refers to the frequency with which particular terms are translated into corresponding code, rather than their general occurrence in natural language corpora. LLMs are capable of identifying and explaining these terms due to their extensive training on both general and specialized corpora. To provide a concrete example, consider the term "even digits." When comparing its occurrence in code-related corpora (with approximately 3k mentions of python code searched by GitHub) to general corpora like web data (with around 63k mentions searched by Google), it becomes clear that although the term is less commonly found in code-related contexts, LLMs are still capable of comprehending and extracting these terms effectively due to their training on diverse data sources. Additionally, recent research [1,2] has also demonstrated LLMs' robust capabilities in keyword extraction.

Simplicity: We would like to thank you for your encouragement on the effectiveness of our approach in the Summary. Regarding the concern that our approach is simple, we respectfully argue that for an effective approach, simplicity is an advantage instead of a disadvantage. Because it means this approach can be easily implemented and applied to real-world scenarios, and thus can be potentially more impactful than complicated approaches. Examples that support this argument include CoT [3] and Zero-shot-CoT[4], which are also fundamentally simple but are very impactful (attract thousands of citations) and have been widely used in practice.

Novelty: While we acknowledge that our approach is simple, we respectfully argue that this does not necessarily mean our approach is not novel. We would like to emphasize that the novelty of this work is twofold: (1) we demonstrate that explicitly explaining relatively low-frequency terms in problem descriptions benefits LLM-based code generation, which is an insight not explored in previous studies; and (2) our approach strates LLMs' inherent capabilities in a novel way to effectively instantiate this insight and significantly enhances LLMs' code generation performance. We believe SEK demonstrate a fundamental insight: the importance of explaining relatively low-frequency terms in problem descriptions, which has proven effective with comprehensive experiments and provides a solid foundation for more advanced methodologies.

Impact: This simplicity and reliance on LLMs' existing capabilities actually strengthens the method's practical applicability - our approach achieves substantial improvements while remaining lightweight and easily deployable.

[1] Maragheh R Y, Fang C, Irugu C C, et al. LLM-take: theme-aware keyword extraction using large language models[C]//2023 IEEE International Conference on Big Data (BigData). IEEE, 2023: 4318-4324.

[2] Lee W, Chun M, Jeong H, et al. Toward keyword generation through large language models[C]//Companion Proceedings of the 28th International Conference on Intelligent User Interfaces. 2023: 37-40.

[3] Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in neural information processing systems, 2022, 35: 24824-24837.

[4] Kojima T, Gu S S, Reid M, et al. Large language models are zero-shot reasoners[J]. Advances in neural information processing systems, 2022, 35: 22199-22213.

Q1: Integration of RAG

Thank you for suggesting the integration of RAG principles. While RAG is a powerful technology, we deliberately chose not to incorporate it due to its reliance on high-quality knowledge bases and the effectiveness of our proposed approach. First, integrating RAG or similar methods will make the effectiveness of our approach correlate to the quality of the external knowledge base. Constructing a high-quality knowledge base is non-trivial and may require manual efforts. Moreover, a good knowledge for coding tasks in one domain may not benefit coding tasks in another domain.

In contrast, our approach is self-contained. The heuristics used in the ranking stage do not make any assumptions about the coding task and the model used. Therefore, we respectfully argue our approach is more flexible and portable than RAG-based approaches. In addition, our experimental results have demonstrated that SEK can effectively enhance code generation performance across different models and different benchmarks. It would be interesting to further investigate whether RAG can benefit the ranking of keywords in future work.

---

To be continued.

W2: On the choice of beam size=2

We appreciate the thoughtful observation regarding our beam search configuration. Our initial selection of beam size=2 was motivated by our aim to compare performance under equivalent search space explorations, as SEK modifies the LLM's search space once through additional token insertion.

To address your concerns about performance saturation and computational costs, we conducted additional experiments with varying beam sizes (2, 3, 5, and 10) using LLaMA-3.1. We were unable to include Mixtral in these experiments due to memory constraints (Out-Of-Memory issues) at beam sizes $\geq$ 5. Our extended results, presented in Table 7 and Table 8 in the revised paper, show that SEK consistently outperforms beam search across most scenarios, even with larger beam sizes. Interestingly, we observed that beam sizes of 5 and 10 occasionally surpassed SEK's performance on MBPP(+) and APPS-Interview, which may be attributed to more computation cost of beam search (see below for details). For clear reading, we present the results of beam search with different beam sizes.

In practice, especially in LLM application scenarios where computational resources are already heavily constrained, if the GPU memory is large enough to handle $n$ beams, we can also batch $n$ independent user requests and process them simultaneously. However, beam search would require much more computational resources to handle these n requests, as each request needs to maintain multiple beams. So we respectfully argue that the claim "Beam Search is significantly less costly than full generation" may not hold in real-world applications.

To quantify computational resource usage of each approach, we calculated the product of the numbers of generated tokens and maintained paths as the total computational cost. The results are shown in Table 9 and Table 10 in the revised paper. When comparing the scenarios with similar computational costs, SEK consistently outperforms beam search. In the cases where beam search surpasses SEK, beam search typically demands significantly more computational resources. For instance, on MBPP, beam search with sizes 5 and 10 consumed approximately 890 and 1840 computational units respectively, whereas SEK required only 412 units. These results reinforce SEK's efficiency in achieving superior performance. For clear reading, we present the computational costs of different sizes of Beam Search baseline.

---

To be continued.

W3: Results bounds

Thank you for raising these important points about result bounds and evaluation metrics. We would like to clarify that our experimental setup employs greedy decoding, as mentioned in Section 3.4. Under this configuration, the token generation process is deterministic and repeated runs would yield identical results. These explain the absence of standard deviations or confidence intervals in our presented results.

Regarding the suggestion about pass@k (where k >= 1) evaluation, we note that both our method and the selected baselines are not designed to optimize for multiple attempts. Our focus is on the model's ability to generate correct code solutions in a single pass, so we specifically chose pass@1 as our evaluation metric.

W4: Low frequency assumption

We appreciate your careful examination of our low-frequency assumption. In response to your feedback, we have revised our Introduction section to present this relationship more as an observation rather than a strict causative relationship. To provide concrete evidence for the comparative frequency of "even digits" versus "even numbers" in programming contexts, we conducted an analysis using GitHub code-specific searches. We observed that "even numbers" appears approximately 215k times, while "even digits" appears only 20k times.

Regarding your point about keyword selection, we respectfully argue that keywords and low-frequency terms are somewhat similar concepts. According to wikipedia (https://en.wikipedia.org/wiki/Keyword_(linguistics)), a keyword is defined as a word that occurs in a text more often than we would expect by chance alone, based on a comparison between its frequency in a specific text and its expected frequency in a much larger reference corpus. Thus, when we ask the LLM to extract keywords, it is essentially extracting words that are relatively low-frequency in the training set, but appear more frequently in the targeted problem description.

Q1: Details of CoT prompt

Thank you for the advice. We followed the implementation methodology from the original CoT paper [1], where demonstrations are constructed by combining problem descriptions with step-by-step reasoning processes. We share the complete CoT prompt as follows:

Please provide a self-contained Python script that solves the following problem in a markdown code block:

"""
Write a function to find the shared elements from the given two lists.
assert set(similar_elements((3, 4, 5, 6),(5, 7, 4, 10))) == set((4, 5))
"""

Below is a Python script with a self-contained function that solves the problem and passes corresponding tests:
"""python
def similar_elements(list1, list2):
    # Step 1, convert to sets
    set1 = set(list1)
    set2 = set(list2)
    # Step 2, find the intersection
    common_elements = set1.intersection(set2)
    # Step 3, get results
    return tuple(common_elements)
"""

Please provide a self-contained Python script that solves the following problem in a markdown code block:
"""
Write a function to find the kth element in the given array using 1-based indexing.
assert kth_element([12,3,5,7,19], 2) == 3
"""

Below is a Python script with a self-contained function that solves the problem and passes corresponding tests:
"""python

---

To be continued.

if sentence.strip().startswith("I"): -> if sentence.strip().startswith("I "): 

W1: No evaluation of zero-shot CoT

Thanks for this valuable suggestion. We have conducted additional experiments to incorporate Zero-Shot CoT as a baseline for comparison. We follow the method outlined in its original paper [1], which first prompts the LLM to "think step by step" for getting the intermediate reasoning steps and then concatenates the original problem description with the generated intermediate steps as input to get the code solution. We did not implement Zero-Shot CoT with DeepSeekCoder-V2-Instruct as DeepSeek's API for this model has changed and the API we used for evaluation is no longer accessible. Given our limited GPU resources, we were unable to deploy this model (236B) locally.

As shown in Table 1 and Table 6 in the revised paper, Zero-Shot CoT consistently underperforms SEK across most scenarios. This may be attributed to the distinct types of the knowledge extracted by the two methods: while Zero-Shot CoT tends to merely restate the complete problem description, SEK focuses on keywords and their explanations, thereby more effectively addressing knowledge gaps during code generation. The detailed comparison between Zero-Shot CoT and SEK has been added to Section 4.1 in the revised paper. The exceptions where SEK achieves slightly weaker performance than Zero-Shot CoT are all related to the MBPP dataset or GPT-4o-mini. These results are consistent with our original results and were discussed in Section 4.1. For clear reading, we present the results of Zero-Shot CoT baseline below.

[1] Kojima T, Gu S S, Reid M, et al. Large language models are zero-shot reasoners[J]. Advances in neural information processing systems, 2022, 35: 22199-22213.

---

To be continued.

Thank you for your constructive comments. According to your suggestion, we have further restructured the Introduction section to better frame our motivation and hypothesis. We have weakened the claim of rarity and avoided stating it as causative.

Regarding your question about the frequency of "even digits" in training datasets, we conducted an analysis using the Python subset of Stack-V2, which serves as the pretraining data for StarCoder2 [1]. The frequency analysis presented in the table below demonstrates that both "even digit" and "even digits" appear significantly less frequently compared to related terms, empirically supporting our statement about their relative rarity in LLM's training corpora.

To address your concerns regarding the low-frequency nature of the extracted keywords, we have included a new frequency distribution analysis comparing the extracted keywords with other terms in the problem descriptions in Appendix E.5. The results demonstrate the extracted keywords are significantly skewed towards higher TF-IDF values, indicating that they are relatively low-frequency terms.

[1] Lozhkov A, Li R, Allal L B, et al. Starcoder 2 and the stack v2: The next generation[J]. arXiv preprint arXiv:2402.19173, 2024.