REPOFILTER: Adaptive Retrieval Context Trimming for Repository-Level Code Completion

Q1: Lack of Baselines

A1: Our work primarily focuses on the pre-processing (adaptive retrieval) and post-processing (context filtering) of retrieved contexts, rather than the retrieval process itself. To the best of our knowledge, the only related work in this area is RepoFormer. As highlighted by the papers you mentioned, most existing research focuses on developing more accurate methods for retrieving cross-file contexts. These studies focus on different, independent modules of the framework. Notably, our method can be integrated with retriever-based approaches to jointly improve completion performance. For example, in our revised manuscript, we demonstrate how our approach generalizes to other retrievers, such as RepoCoder, to provide additional improvements.

Q2: New Code LLMs

A2:

We have added StarCoder2-7B, DeepSeekv2, and Qwen-Coder 7B to Section 6.2. Our experiments now cover a variety of model families, including StarCoder2, StarCoder, CodeLlama, QwenCoder, and DeepSeek Coder. We hope this addresses your concern.

Q3: Time Overhead

A3: Our method introduces additional time overhead during the generation phase due to the need to generate more special tokens. However, it reduces retrieval time through adaptive retrieval. Specifically, under our current implementation and hyperparameter settings, the generation time for RepoEval-Line completion increases by 61% compared to full-retrieve, 44% for API-level completion, and 23% for function-level completion. Conversely, the retrieval time is reduced by 22%, 41%, and 36%, respectively, compared to always-retrieve.

The overall per-instance time overhead for the entire system should be calculated as the additional time in the generation phase minus the time saved in the retrieval phase. However, since these two modules are typically implemented as entirely independent components, there is no standardized way to measure the total system time overhead, as it heavily depends on implementation details. For example, in API completion:

• Using a single process for retrieval and vLLM acceleration on an A100 GPU for generation, our method saves 1.5–1.6 seconds per instance compared to the baseline under the same configuration.
• With 8 processes for retrieval, the saving time is reduced to only 0.2 seconds per instance.
• Using 30 processes for retrieval and non-vLLM generation results in an overhead of 0.8–0.9 seconds per instance.

The overall time overhead is thus highly dependent on the implementation details of the two independent modules. Various factors, such as GPU/CPU type, number of processes, implementation specifics, and acceleration frameworks, significantly influence the time for each module. Consequently, there is no standard criterion for evaluating the time overhead of the entire system.

Q4: summarizing contributions in bullet points at the end of the Introduction

A4: Thank you very much for your suggestion. We have made the suggested changes in the revised manuscript, and we hope this makes our contribution clearer.

Thank you for your reviewing and your suggestions. Following are our responses regarding your concern:

Q1: Limited Exploration of Combinatorial Effects

A1: Our current approach independently labels each chunk's impact on completion generation without explicitly accounting for combinatorial effects between multiple chunks. This choice was driven by the already substantial computational cost of our labeling process. Considering combinatorial effects, such as evaluating pairs or groups of chunks, would significantly increase complexity, likely to an exponential degree.

Initially, we attempted to account for such effects using a leave-one-out likelihood metric, which measured the likelihood difference when a single chunk was removed from the top-10 retrieved contexts. While this inherently considered the interplay between chunks, we found it less reliable for accurately labeling chunk polarity. Factors such as inter-chunk dependencies and rank order introduced confounding influences that affected the model's generation.

Consequently, we opted for a simpler and more computationally efficient approach that evaluates chunk polarity independently. The results in Table A of the manuscript validate this method, demonstrating its effectiveness despite not explicitly accounting for combinatorial effects.

Q2: Potential Hyperparameter Tuning Bias

A2: We did not tune these hyperparameters on the test set; instead, we tuned them on the validation set of our constructed database. The test set was only used for ablation experiments to analyze the impact of the threshold setting on model performance. Moreover, to demonstrate the generalizability of this parameter setting, we have included additional ablation results for other tasks, including RepoEval-Line, RepoEval-Function, CCLongEval-Chunk, and CCLongEval-Function, in appendix E of our revised manuscript.

Q3: Unstable Evaluation for Chunk and Function-Level Completion

A3: Execution-based metrics indeed provide a more accurate evaluation compared to reference-based metrics. However, collecting such data is significantly more challenging, as benchmarks with test cases often contain only limited data, especially for repo-level code generation or completion tasks. In contrast, reference-based metrics allow us to evaluate a wider range of repositories, including those with unconventional code patterns. Moreover, metrics like Exact String Match (ES) and Exact Match (EM) can still reflect the overall accuracy of the model's generation, even if there are larger fluctuations in certain specific settings or tasks.

Therefore, covering both evaluation metrics is currently a good choice to ensure accurate evaluation while accommodating a wide range of repositories.

Q4: Masking non-contextual information

A4: Our training objective is to enable the model to effectively perform context filtering and code completion based on the given context. However, in-file and cross-file contexts are repository-dependent and not part of the learning objective. Incorporating these contexts into the loss function could cause the model to memorize irrelevant local information, which does not align with the supervised objectives. To address this, we mask these parts. We have revised this section in the manuscript to improve clarity.

Thank you for reviewing our paper, we greatly appreciate your feedback and suggestions. Here are our response regarding your concerns:

Q1: Are the three baseline techniques RAG fine-tuned for this task?

A1: Among the three baselines, RepoFormer was fine-tuned using the same loss function. This fine-tuning was necessary due to the introduction of new special tokens and modifications to the standard prompt format, requiring the model to adapt for effective generation under the updated setup.

For the remaining baselines, we followed the same configuration as in the original RepoFormer paper, where no-retrieve and full-retrieve baselines were not further fine-tuned. Given the similarity between the code completion task and the pretraining task, task-specific fine-tuning was considered unnecessary. Moreover, our fine-tuning dataset, derived from Stack, may overlap with the training data of existing code LLMs.

To further validate our approach, we fine-tuned StarCoder-3B using all retrieved cross-file chunks for one epoch while keeping all the configuration same as in our main experiment. The results showed no significant improvement over the full-retrieve baseline, which still underperformed compared to our framework.

Q2: Comparing with baseline of retriever trained on perplexity-gain loss.

A2: Thank you for highlighting this method. Our work primarily focuses on pre-processing (adaptive retrieval) and post-processing (context filtering) of retrieved contexts, rather than the retrieval process itself. While numerous studies, including the paper you mentioned, aim to improve retrievers for RAG-based repository-level code completion tasks, these efforts address a different and independent module of the framework (i.e., the retriever). Therefore, direct comparisons between our approach and retriever-focused methods may not be entirely meaningful, even though the loss function you referenced shares a similar motivation with ours.

Importantly, our method is complementary to retriever-based approaches and can be integrated with them to further enhance completion performance. For example, in our revised manuscript, we demonstrate how our approach generalizes to other retrievers, such as RepoCoder, by filtering retrieved contexts to provide additional improvements, as detailed in Appendix C.

Thank you for your detailed responses and clarifications. I appreciate the effort you put into addressing my concerns. Below are my thoughts based on your replies:

1. Regarding RAG fine-tuning, I am surprised by the result showing that fine-tuning StarCoder on all retrieved chunks yielded no significant improvement. Since StarCoder was not originally RAG fine-tuned, the loss function you used should intuitively encourage the model to be robust against irrelevant chunks. My past experience with RAG fine-tuning StarCoder has demonstrated noticeable gains in similar setups, so I wonder if the lack of improvement could be due to specifics of the fine-tuning process.
2. While I understand that your method primarily focuses on filtering rather than retrieval, from a broader RAG perspective, filtering can also be interpreted as improving retrieval quality since it directly modifies the final retrieval result. In this sense, the distinction between filtering and retrieval improvement feels less meaningful. Nevertheless, I agree that your approach could complement retriever-focused methods and appreciate the clarification on its compatibility with other retrievers like RepoCoder.
3. Thank you for explaining the limited scope of your GPT-4 experiments. Could you provide an estimate of the total cost for running your full set of experiments with GPT-4?
4. Yes, you correctly understood my question regarding constrained decoding. I agree that adding an end-of-chunk token can help guide the model to output special tokens for polarity classification, and I think it will also potentially reduce the reliance on constrained decoding.

Thank you for your detailed follow-up and clarifications—you've addressed most of my concerns, and I now have a clearer understanding of your approach and its scope. Based on this, I am raising my score from 5 to 6.

Many thanks for your thoughtful feedback. Following are our response regarding your concers on our work:

Q1: Limited Evaluation on Diverse Repositories:

A1: The majority of existing repository-level benchmarks and related studies primarily focus on Python, with a notable lack of comprehensive datasets for other programming languages [1,2,3,4,5]. Consequently, our experiments and analysis are centered on Python repositories, aligning with current standards in the field. While we acknowledge that extending the evaluation to other programming languages could further demonstrate the generalizability of our approach, we believe the current scope is sufficient for this paper, given the robustness of our results across hundreds of diverse Python repositories. We have addressed these limitations and our intention to explore other languages in future work in the revised manuscript.

Q2: Detailed Explanation of the Adaptive Mechanism

A2: We have revised the sections in our manuscript that introduce our inference pipeline, aiming to enhance clarity and address your concerns. Additionally, the newly added generation case F in the appendix provides further illustration of our inference process to aid understanding.

Q3: Provide Qualitative Examples

A3: Thank you for your suggestion. We have selected a sample along with its corresponding labels, generated outputs, and explanations, which we have included at the appendix F in our revised manuscript. We hope this example provides an intuitive illustration of how our framework operates and how these samples can potentially positively or negatively influence the model's generation process.

Q4: Broader Contextual Relevance Discussion

A4: Following your suggestion, we have added a discussion section in the appendix, where we address the limitations of our work, its potential impact, and future directions. We hope this addition provides a clearer perspective on the scope and implications of our study.

Q5: Inadequate baseline

A5: Our work primarily focuses on the pre-processing (adaptive retrieval) and post-processing (context filtering) of retrieved cross-file contexts of code repositories, rather than the retrieval process itself. To the best of our knowledge, the only related work in this area is RepoFormer, as most existing research explores more accurate methods for retrieving cross-file contexts. These studies focus on different, independent modules of the framework, making direct comparisons less meaningful. Notably, our method can be integrated with retriever-based approaches to jointly improve completion performance. For example, in our revised manuscript, we demonstrate how our approach generalizes to other retrievers, such as RepoCoder, to provide additional improvements. Of course, if we have missed any closely related work that could serve as a baseline, we would greatly appreciate your pointing it out. We will include such comparisons in future revisions of our paper if possible.

Q6: Dependency on Accurate Likelihood Estimation

A6: Thank you for raising this concern. We would like to clarify that our metric is based on the difference in the model's likelihood for the ground truth between prompts with and without a specific cross-file chunk, rather than relying on the absolute value of the likelihood. Therefore, the issue you mentioned with unconventional or complex code patterns is less likely to arise. While it is true that in repositories with unconventional or complex patterns, the model’s likelihood for the ground truth might be generally lower, the presence of a cross-file chunk providing the necessary context for completion is still expected to increase the likelihood. To further mitigate potential variability introduced by complex patterns, we base our evaluation on the percentage increase in likelihood rather than the absolute value. This approach effectively normalizes for differences across varying patterns and coding styles, ensuring robust polarity labeling. While there is no direct metric to measure the accuracy of our labeling process, we have indirectly validated the accuracy of the likelihood-based metric through the model’s generation performance, as shown in Table A. The performance observed supports the effectiveness of our labeling approach in accurately identifying the polarity of cross-file chunks.

To better illustrate our approach, we used the cyclomatic complexity metric to measure the complexity of the code in our completed files. Specifically, this metric evaluates the number of linearly independent paths through the source code. We selected the file with the highest complexity score, exceeding 12, which indicates the need for possible refactoring. Below, we provide part of the preceding code from this sample, the chunk identified as positive by our metric, and the corresponding ground truth:

Preceding Code:

def wrap_iv_filter_server(worker):
    """
    This function is to perform feature selection with iv_filter \
    to data for server.
    Args:
        worker: `federatedscope.core.workers.Worker` to be wrapped

    Returns:
        Wrap vfl server with iv_filter.
    """
    def trigger_for_feat_engr(self,
                              trigger_train_func,
                              kwargs_for_trigger_train_func={}):
        logger.info('Start to execute woe_filter, which requires HE.')
        self.trigger_train_func = trigger_train_func
        self.kwargs_for_trigger_train_func = \
            kwargs_for_trigger_train_func
        self.msg_buffer['feat_dim'] = {}

        # Broadcast client address and feat_engr_public_key
        self.broadcast_client_address()
        self.comm_manager.send(

Positive Chunk:

#         # Broadcast client address and feat_engr_public_key
#         self.broadcast_client_address()
#         self.feat_engr_public_key, self.feat_engr_private_key = \
#             secure_builder(worker._cfg).generate_keypair()
#         logger.info('Sending feat_engr_public_keys to clients.')
#         self.comm_manager.send(
#             Message(msg_type='feat_engr_public_keys',
#                     sender=self.ID,
#                     receiver=list(self.comm_manager.get_neighbors().keys()),
#                     content=content))

Ground Truth:

Message(msg_type='binning',
                    sender=self.ID,
                    receiver=list(self.comm_manager.get_neighbors().keys()),
                    state=self.state,
                    content=self._cfg.feat_engr.selec_woe_binning))

From this example, it is evident that the chunk identified as positive provides a clear example of constructing and sending a Message object using the comm_manager. This serves as a reference for the method call in the model's completion target. Even for patterns deemed complex, there is a direct relationship between the positive chunk and the ground truth.

Q7: Error case and analysis

A7: Referencing the sample mentioned above, during the generation process, the model identified the positive chunk as a neutral chunk and therefore did not append it to the final prompt. As a result, the lack of a relevant reference for instantiating the Message object led the model to generate the following output:

message=msg,
destination=client_id,
msg_type='client_address')

This does not match the ground truth. We believe the error primarily stems from the model's inability to discern the intent of the content to be completed in certain scenarios, which in turn prevents it from accurately identifying useful context.

Thank you sincerely for your valuable suggestions. Here are our responses regarding your concerns of our work:

Q1: Missing details from the paper that need further clarification

A1: Thank you very much for pointing out these missing details. We have revised the corresponding section of the paper to make it clearer. Below, we provide a detailed explanation addressing your concerns about these details.

Threshold Setting: We determined the threshold setting based on the model's performance when provided with only positive samples and when negative samples were removed. This threshold setting directly influences the labeling accuracy. For instance, if the threshold for positive is set too low, more samples will be classified as positive, even if they are irrelevant to the completion task. Conversely, if the threshold is set too high, genuinely positive samples might be incorrectly labeled as irrelevant neutral samples. Through adjustments conducted on the validation set of our constructed dataset, we obtained results aligning with our expectations. Specifically, when the model is provided with only positive samples, its completion performance does not degrade. Furthermore, when negative samples are removed from the prompt, the model's performance improves. This confirms the accuracy of our labeling strategy. Additionally, as shown in Table A, this threshold setting is also applicable to the RepoEval dataset, further validating our hypothesis.

Generating <neg> token: Generating<neg> token: The model sequentially appends cross-file chunks to the current prompt and evaluates their polarity. If the model classifies a chunk as negative or neutral by generating a special token<neg> or <neu>, the prompt will roll back to its state before the chunk was added. As a result, the final prompt includes only positive chunks with <pos> tokens. We have also rewrite the inference section for better clarity

Discrepancy in algorithm and description: We apologize for this inconsistency. In our revised manuscript, we have standardized the expression, clarifying that the model generates a special token based on a threshold rather than simply generating the token with the highest probability.

Q2: Data leakage concern

A2: The repositories included in RepoEval were collected after the publication of the Stack dataset, and there is no overlap between the repositories in CrossCodeEval and the Stack dataset. Since our dataset is an extension of the Stack dataset, there is no data leakage between the training and test sets.

Q3: Problem scope clarification: How does RepoFilter differ or do better than training a new retriever with the RepoFilter training data?

A3: From the perspective of identifying the polarity of cross-file chunks, training a retriever often involves using surrounding code as the query, such as the preceding 10–20 lines of code or a specific number of tokens. However, determining the polarity of a chunk is not based on semantic or lexical similarity to the cross-file chunk, but rather on understanding the intention of the code. This is a more challenging task, as the limited context provided by a retriever's query is often insufficient for the model to fully grasp the nuances of this task. In contrast, our framework is designed to evaluate the polarity of chunks based on the entire prompt context. By leveraging the complete context, the model can make more informed and accurate decisions. This capability is critical for achieving better accuracy in polarity identification. Moreover, beyond identifying polarity, our framework also implements adaptive retrieval. The model can decide whether retrieval is necessary or whether the current context is already sufficient. This dynamic decision-making process cannot be achieved with a traditional retriever, which would always retrieve without assessing the sufficiency of the existing context. Considering these two factors—better context utilization for polarity identification and the ability to perform adaptive retrieval—we are motivated to train our framework rather than simply training a new retriever.