Impact-driven Context Filtering For Cross-file Code Completion

Many thanks for your thoughtful feedback. Following is our response regarding your concerns about our work:

Q1: CODEFILTER may require more computational resources than baseline methods.

A1: CODEFILTER adds some overhead during generation because it sequentially evaluates each chunk and emits special tokens. However, by predicting context sufficiency, CODEFILTER avoids many unnecessary retrieval operations and thereby reduces retrieval cost. These savings largely offset the extra generation effort. In our evaluation setup, using a single retriever worker and vLLM for inference acceleration, CODEFILTER actually reduces per-instance processing time by approximately 1.5 seconds compared to a full-retrieve baseline, so the whole framework will not introduce significant additional computational cost.

Q2: The framework relies on multiple thresholds (e.g., T_p, T_n mentioned in Line 143, and inference thresholds mentioned in Line 255), raising concerns about its stability and sensitivity to these hyperparameters.

A2: Regarding inference thresholds (Line 255), we performed a detailed ablation study in Appendix B, investigating their impact on completion performance and efficiency across various datasets and completion settings. While the results confirm that the thresholds indeed influence both effectiveness and efficiency, we successfully identified a unified setting balancing both aspects across multiple scenarios, demonstrating good generalizability. Given the importance of threshold sensitivity, we acknowledge this as a valuable future direction. We intend to further explore diverse factors, such as code complexity, programming language differences, and develop a dynamic, adaptive method to set thresholds optimally according to specific completion scenarios.

The thresholds (T_p, T_n) mentioned at line 143 were determined empirically to achieve high recall when labeling positive and negative chunks, ensuring effective data collection for training. To identify these values, we sampled 10 % of the collected training data and searched for threshold settings based on completion performance under different prompt strategies: retaining only chunks classified as positive and removing chunks classified as negative. We required that both strategies outperform the full-retrieve baseline and that the presence or absence of neutral chunks have minimal impact on completion performance. Using these criteria on this subset, we selected optimal thresholds and then validated them on the RepoCoder and CrossCoderLongEval datasets, with consistent results (Table (a) in the paper delivers the results of different prompt strategies on the RepoEval benchmark according to our threshold setting). These experiments suggest our threshold settings generalize effectively across diverse Python datasets.

Q3: Including example cases of neutral and negative code chunks would make the analysis more concrete and insightful.

A3: We included a representative example illustrating positive, neutral, and negative chunks along with baseline and CODEFILTER results in Appendix G. We agree more detailed interpretation of each chunk would be insightful, and will expand this analysis in our revised manuscript. We hope this addresses your concern.

Q4: Writing issues: Figure 1 is not cited in the main text. Typo: Line 210: “the process” -> “The process.”

A4: Thank you for pointing out these issues. We will revise the corresponding lines in our new manuscript for better clarity.

Thank you for reviewing our paper, we greatly appreciate your feedback and suggestions. Here are our response regarding your concerns:

Q1: The methodological design, which appears to be a modest adaptation of previously established techniques, such as Self-RAG and RepoFormer

A1: These methods are all about selective and adaptive retrieval, but differ in many ways:

Self-RAG adds a separate critic LM that injects reflection tokens to decide, passage by passage, whether the main model should retrieve an entire document—a pattern suited to open-domain QA, where each paragraph usually carries an independent fact. In contrast, CODEFILTER lets the generator tag every retrieved code chunk with lightweight polarity tokens within the same decoding call, so no extra LM invocation is needed. Because repository-level code completion depends on multiple chunks that must work together, contextual sufficiency must be judged collectively. Self-RAG’s per-passage gating cannot capture this, which make self-RAG can't be directly applied to code completion tasks. Whereas CODEFILTER processes the whole sequence in one run and assesses sufficiency holistically.

RepoFormer makes a single, global retrieve-or-skip decision: it predicts one flag indicating whether any retrieval is helpful and, if “yes,” appends all retrieved chunks to the prompt, treating the retrieval result as an indivisible block. CODEFILTER sharpens this pipeline by adding two lightweight special tokens generated in the same decoding run—one to signal that the existing context is already sufficient and another, emitted for each chunk, to mark it keep or drop—so the model can both avoid unnecessary retrieval and discard irrelevant chunks individually. Conceptually, RepoFormer optimises whether to retrieve, whereas CODEFILTER also optimises what to keep, achieving chunk-level adaptive retrieval that produces much shorter prompts(>=50% prompt length reduction) and stronger effectiveness(3% higher ES) by filtering out chunks that could hurt the completion

Q2: The resulting performance gains are marginal, particularly when evaluated with stronger base models like CodeLlama 13B

A2: Our model improves performance by filtering retrieved noisy chunks that degrade completion. Because such instances with negative retrieved chunks account for only ≈ 10–20 % of the general test set, the overall gain on the full benchmarks is meaningful and nontrivial. When we isolate this difficult subset (identified in section 6.2), Improvements are much larger: relative gains over the baseline exceed 50 % across all metrics, and some even surpass 100 %. We have also added CodeLlama-13B results on this targeted subset in the table below, demonstrating similarly significant performance gains compared to the full-retrieve baseline. These numbers show that CODEFILTER is crafted for, and delivers sizable gains on, the hard cases where unfiltered retrieval degrades performance.

Beyond accuracy, the method also cuts prompt length by > 50 %, lowering cost and making completions more clearly attributable to the retained context across all instances.

Thank you for your reviewing and your suggestions. Following are our responses regarding your concern:

Q1: Provide a clear explanation of the key differences between CodeFilter and the RepoFormer framework. Explaining how CodeFilter ensures the "filtering of irrelevant chunks".

A1:

Thank you for your question. We will clarify the differences between our method and other baselines in the revised manuscript.

Specifically, RepoFormer makes a single, global retrieve-or-skip decision: it predicts one flag indicating whether any retrieval is helpful and, if “yes,” appends all retrieved chunks to the prompt. RepoFormer treats the retrieval result as an indivisible block. CODEFILTER refines this pipeline by introducing two lightweight special tokens generated within the same decoding run: one signals that the existing context is already sufficient for completion, and the other—emitted for each chunk—marks whether to keep or drop it. As a result, the model can both avoid unnecessary retrieval and discard irrelevant chunks individually.

Conceptually, RepoFormer optimizes whether to retrieve, whereas CODEFILTER also optimizes what to keep, achieving chunk-level adaptive retrieval that produces much shorter prompts (≥ 50 % prompt-length reduction) and stronger effectiveness (∼ 3 % higher ES) by filtering out chunks that could hurt completion.

For "how CodeFILTER filters code chunks". Statistically, we collected a wide range of general Python repositories and annotated the retrieved code chunks. During training, the model learns which patterns of retrieved chunks consistently lead to a correct completion versus those that do not. Semantically, it builds an implicit sketch of the function’s intent, identifying the names, calls, and types still needed, and then tags chunks whose cues (identifier overlap, call-chain continuity, module proximity, type consistency) align with that intent as positive while marking others as irrelevant or negative. After enough examples, CodeFilter internalizes the mapping {inferred intent → usefulness cues → keep/drop}, so at inference it can filter irrelevant chunks without seeing the ground-truth code before generating the final completion.

Q2: The reliance on Jaccard similarity for initial retrieval might limit the diversity and relevance of retrieved chunks, potentially overlooking semantically relevant but syntactically dissimilar code.

A2: Previous repository-level code completion research (e.g., RepoCoder, CrossCodeEval) compared sparse retrievers like Jaccard similarity with dense retrievers and found no substantial difference in final performance, indicating that dense methods are not necessarily superior in this task. For our main experiments, we therefore adopted the simpler and more efficient Jaccard similarity as the retrieval metric.

Moreover, we also include additional experiments in Appendix C using a dense retriever (UniXcoder). The results align with prior findings: the dense retriever achieves comparable performance, and CODEFILTER provides similar improvements relative to baselines.

One possible reason dense retrievers do not significantly outperform sparse ones is that, when using preceding lines as the retrieval query, dense retrievers select chunks semantically similar to the query itself, but these chunks may not necessarily reflect the underlying intent required for the target completion. As a result, such semantically similar chunks do not always provide the most relevant or helpful context.

Thank you sincerely for your valuable suggestions. Here are our responses regarding your concerns about our work:

Q1: Time costs: provide some data on the total time taken for a single completion task, comparing CODEFILTER with baselines

A1: We thank the reviewer for prompting this analysis.

We share your emphasis on latency. CODEFILTER makes a deliberate trade-off: the iterative loop adds a handful of special-token generation steps, but adaptive retrieval eliminates many costly retrieval calls. Under our current implementation and hyperparameters, we observe:

Because retrieval and generation are typically deployed as independent services, the overall time overhead depends on the configuration of these two modules. In our default evaluation setup (A100-80 GB GPU, vLLM, one retrieval worker), CODEFILTER saves 1.5 – 1.6 s per API-level completion. With 8 retrieval workers the saving shrinks to ~0.2 s, while an extreme setting of 30 workers and a non-accelerated generator incurs a modest 0.8 – 0.9 s overhead.

These results illustrate that hardware (GPU/CPU type), worker-count, and acceleration frameworks all influence latency, yet in typical deployments CODEFILTER reduces or only minimally affects end-to-end time. We will add this discussion and the full latency table to our new manuscript.

Q2: Analysis of the overall "query cost"

A2: We primarily measure query cost in terms of token usage—i.e., the number of input and output tokens incurred when calling an LLM API for code generation. Moreover, there could be some strategies to avoid excessive per‐chunk API calls. As the result, CODEFILTER is cost‐efficient in both deployment modes:

1. Integrated API. When the generation model is accessible, decoding with polarity tokens can be deployed in the API. Therefore, filtering occurs within the same generation call and no per‐chunk requests are required. Using the OpenAI pricing ratio (≈ 1 unit per input token: 4 units per output token), the total paid units per query are ≈ 55 % of the full‐retrieve baseline.

2. External filter (Section 6.4). With an unmodified API (e.g., GPT-4), filtering runs locally on a lightweight model such as StarCoder-7 B; its compute cost is negligible compared to API billing. The subsequent generation call uses a much shorter prompt, still reducing overall cost by > 50 %.

Thus, even under pay‐per‐call billing, CODEFILTER remains more cost‐effective.

Q3: Trade-off between the time spent on filtering and simply feeding more raw context into larger-window models to achieve similar or better results within the same time budget.

A3: As discussed in A1, the filter step adds a generation overhead but removes many retrieval calls, so end-to-end latency is essentially unchanged. Therefore, the efficiency–effectiveness trade-off is negligible. Moreover, though the current model supports very long input, context is also not “free”: attention cost still grows with input length, and API billing remains token-based. Crucially, extra context does not always help. Section 3.2 shows that ≥ 80 % of retrieved lines are irrelevant, and Section 6.2 reports that unfiltered context hurts completion quality on 10–20 % of instances. Feeding all chunks into a large-context model therefore risks both higher cost and accuracy regressions, whereas CODEFILTER keeps prompts concise and reliably improves performance.

Q4: Typo of legend and bars in Figure 3

A4: We really appreciate you pointing out this typo; we will revise it in our new manuscript.