CodeXEmbed: A Generalist Embedding Model Family for Multilingual and Multi-task Code Retrieval

Thank you very much for the thoughtful review and the strong-accept recommendation. We are pleased that you found CodeXEmebed’s unified framework, multi-stage LoRA strategy, and SOTA retrieval results valuable.

We will incorporate your points in the camera-ready version by:

• Highlighting the two-stage separate-LoRA gains in section 3.4.
• Releasing checkpoints and scripts to facilitate reproducibility of the BEIR and CoIR results.

We appreciate your positive assessment and confidence in our work.

Reviewer Comment:
While the paper proposes a multi-stage LoRA-based training strategy, the core methodology—contrastive learning with query-document pairs—is standard in retrieval literature. The approach largely builds upon existing methods (e.g., LoRA, GradCache) without introducing fundamentally new architectures or training paradigms.

---

Reply:

Thank you for the insightful observation. Our goal is indeed to reuse strong primitives (LLM, LoRA, GradCache, contrastive loss) but assemble them in a novel, staged curriculum that has not, to our knowledge, been applied to embedding models before. Concretely:

• Code-centric focus. Prior open embeddings are overwhelmingly text-only; high-quality code retrieval remains under-served. To our knowledge, CodeXEmbed is the first single encoder that supports 12 programming languages and natural-language queries with state-of-the-art code results by unifying diverse code tasks into a retrieval-format training frame.

• LLM to Bi-Encoder. Starting from a single autoregressive LLM, we duplicate its weights to form a two-tower bi-encoder: one tower encodes the query, the other encodes code/document passages. We remove the causal mask, add bidirectional attention, and fine-tune each tower with lightweight LoRA adapters. This conversion preserves the LLM’s rich pre-training while enabling constant-time embedding and ANN search—something prior work has not demonstrated at this scale (2B, 7B) for code embedding.

• Unified model with code and text ability. No prior work demonstrates how to train a single model that excels at both code and text retrieval. Table 4 shows that naïve single-stage training—or other two-stage baselines—inevitably degrades one domain. By contrast, our two-stage schedule employs separate LoRA adapters and a continuous optimizer: Stage 1 trains on text only to establish strong natural-language semantics, and Stage 2 introduces code while freezing the text adapters, thereby preserving those semantics. This design attains state-of-the-art performance on CoIR and competitive scores on BEIR, confirming the effectiveness of our unified approach.

• Demonstrable, non-trivial gains. Table 4 shows that this “Separate-LoRA two-stage” recipe boosts NDCG@10 by +2.85 on CoIR and +2.09 on BEIR over a single-stage baseline with identical FLOPs; alternative two-stage variants either hurt CoIR or under-perform. These gains are achieved without full-model fine-tuning.

Because we could not locate prior work that combines LLM with multi-stage LoRA and GradCache in a staged, adapter-merging schedule for retrieval embeddings, we believe the contribution is sufficient. If you are aware of any prior work we may have overlooked, please share the reference. We would be grateful to include it and will acknowledge it in the final version.

Reviewer Comment:
The retrieval-augmented generation experiments use GPT-3.5 for code synthesis, which may obscure the retriever’s standalone contribution. It's unclear how much of the performance gain is attributable to the improved retriever versus the strong decoder model.

---

Reply:

We isolate the retriever’s impact by holding the generator constant:

• Fixed generator setup. In Section 3.5 we always call the same gpt-3.5-turbo with an identical prompt, temperature, and max-token budget. The only variable is the set of top-k (k=5) contexts returned by the retriever.
• Additional clarification added. In the revision we state explicitly that the generator parameters and decoding hyper-parameters are frozen.

We hope these clarifications make it clear that the reported RAG gains stem from the retriever rather than latent generator variability.

Reviewer Comment:
The paper lists hyper-parameters such as batch size 1024, rank 8 LoRA and learning-rate 5×10⁻⁵ in Table 10 yet gives no information on training hardware, total update steps or FLOPs, leaving reproducibility and energy cost quite difficult to verify.

---

Reply:

Thank you for pointing out the missing training-hardware details.
We will add the below to Appendix D.

*Assumes sustained 1.8 PFLOPS for the 8 H100s and 700 W board power per GPU (5.6 kW total).

Reviewer Comment:
No statistical variance, confidence intervals or significance tests accompany single-run metrics on the ten CoIR datasets, leaving it unclear whether reported gains over baselines exceed random fluctuation

---

Reply:

We agree that statistical evidence is important. In our setting the only nondeterminism comes from the random seed used for (i) LoRA weight initialisation and (ii) shuffling of training batches, because each run starts from a fixed pretrained checkpoint (GTE-large-en-v1.5 → 400M, Gemma-2-2B-it → 2B, Mistral-7B-it-v0.3 → 7B).

To quantify variance we trained the 2B model with four different seeds:

Mean = 67.51, σ = 0.28.

Our reported score (67.41) is well within one standard deviation.

We will include the evaluation script plus a --seed flag in the released training code so that readers can reproduce or extend the significance tests.

Reviewer Comment:
In Section 3.5.1 the authors use gpt-3.5-turbo as the code generation model but do not specify its concrete version (e.g. gpt-3.5-turbo-0125 or gpt-3.5-turbo-1106), making their experiments difficult to reproduce. We would also like to note that the model is generally considered outdated as even within OpenAI's offering there are models that seem to be both cheaper and report higher performance (e.g. GPT-4o mini).

---

Reply:

For consistency with the baselines reported in Code-RAG-Bench [1], we evaluate our model using the same generator, gpt-3.5-turbo-0125. We will add the full model name (including the -0125 suffix) to the camera-ready for reproducibility.

To address your concern about newer models, we reran the CodeXEmbed retrieved output on SWE-Bench-Lite with gpt-4o-2024-08-06. Using the None (no retrieval), OpenAI, Rerank (best baseline), and Gold (oracle) scores reported in [1] as reference, CodeXEmbed delivers the same relative improvements, showing that the retriever’s benefit is decoder-agnostic.

Using gpt-4o-2024-08-06 raises overall performance compared to gpt-3.5-turbo-0125, and the 7B variant of CodeXEmbed surpasses the best baseline OpenAI, Rerank, approaching the Gold upper bound.

[1] CodeRAG-Bench: Can Retrieval Augment Code Generation?

Reviewer Comment:
Analysis in Table 6 indicates that for Java, a model trained solely on Java (84.25 NDCG@10) surpasses the model trained on all 12 languages (81.36 NDCG@10 for Java) [1, Section 3.7]. This finding somewhat contradicts the broader narrative of multilingual training benefits for every included language and suggests specialized models could be more effective.

---

Reply:

Thank you for highlighting this point. We agree that a Java-only model achieves a slightly higher NDCG@10 on Java (+2.9). However, our goal is one encoder that serves all languages simultaneously, because:

• Overall benefit: Averaged across all 12 languages, a model trained with all languages is better than just a single language.
• Low-resource gains: For lower-resource languages like Ruby and SQL, training with all 12 languages yields more gains compared to a single-language (Java) model.
• Simple specialization option: One encoder with 12 languages can lead to simpler deployment and lower memory cost.

In short, while a niche model can edge out the multilingual one on a single language, the unified encoder offers stronger cross-language performance, simpler deployment, and remains easily adaptable when maximal per-language accuracy is required.

Reviewer Comment:
This research shows imbalanced language coverage in training data with Python (27.1%), Go (25.2%), JavaScript (17.0%) and PHP (17.2%) dominating, while languages like Rust, Kotlin, and C# have minimal representation, potentially affecting cross-language performance consistency

---

Reply:

We acknowledge that the raw crawl is skewed toward high-usage languages (Python 27 %, Go 25 %, JS 17 %, PHP 17 %). We deliberately accepted this imbalance, but we mitigate its effect and even leverage it in several ways:

• Real-world demand. Most industrial code-search targets Python and JavaScript, so strong performance there is essential.
• Structural similarity enables transfer. Core programming constructs—loops, conditionals, function/class declarations, and common APIs—are shared across languages. High-resource Python/JS training helps the model learn these language-agnostic patterns, which low-resource languages (Rust, Kotlin, C#) can then inherit with minimal additional data.
• Batch re-balancing for low-resource languages. We up-weight low-resource language datasets and down-weight high-resource ones during sampling, so each batch has a more balanced language mix.

Reviewer Comment:
It is unclear what dataset was the experiment in Section 3.7 done on -- could you please clarify?

---

Reply:

Section 3.7 reports results on language-specific slices of the CoIR test set (CoIR-X). For each language we simply aggregate every CoIR test subset tagged with that language:

No additional re-splitting was performed; we evaluate exactly the official CoIR test queries for each subset. We will add this table to Section 3.7 for clarity.

Reviewer Comment:
The paper is positioned in a confusing way. It seems like the developed embedding models were primarily targeted for code-related tasks, but then the authors addressed text-domain tasks as well. What was the actual focus of the work? Code domain or text domain or both?

---

Reply:

Thank you for pointing this out. Our intent is:

1. Primary goal — strong code retrieval.
   High-quality, openly available code embeddings are still scarce; delivering a family of state-of-the-art code retrieval models is our main contribution.

2. Unified model, not two separate encoders.
   We deliberately keep a single encoder that understands both natural language and code so that developers and researchers can:

   • Maintain one shared embedding space, reducing memory use, deployment complexity, and serving cost.
   • Avoid having to identify whether a task is code retrieval or text retrieval and then call different models for each.

3. Two-stage curriculum preserves both domains.
   No prior work demonstrates how to train a single model that excels at both code and text retrieval. Table 4 shows that naïve single-stage training—or other two-stage baselines—inevitably degrades one domain. By contrast, our two-stage schedule employs separate LoRA adapters and a continuous optimizer: Stage 1 trains on text only to establish strong natural-language semantics, and Stage 2 introduces code while freezing the text adapters, thereby preserving those semantics. This design attains state-of-the-art performance on CoIR and competitive scores on BEIR, confirming the effectiveness of our unified approach.

The result is SOTA on CoIR (code) and competitive on BEIR (text), demonstrating the benefit of a unified, multilingual model. We will clarify this positioning in the introduction to make the dual-domain motivation explicit.

Reviewer Comment:
Limited novelty as all the pieces of this work already exist in the literature. However, we know, “more data, more training, large model”; this recipe works. What other knowledge does this paper share? I am not sure.

---

Reply:

We appreciate the concern. Our contribution goes beyond simply “more data + bigger model” in four ways:

1. Code-centric focus.
   Prior open embeddings are overwhelmingly text-only; high-quality code retrieval remains under-served. CodeXEmbed is, to our knowledge, the first single encoder that handles 12 programming languages and natural-language queries with SOTA code results.

2. LLM to Bi-Encoder.
   Starting from a single autoregressive LLM, we duplicate its weights to form a two-tower bi-encoder: one tower encodes the query, the other encodes code/document passages. We remove the causal mask, add bidirectional attention, and fine-tune each tower with lightweight LoRA adapters. This conversion preserves the LLM’s rich pre-training while enabling constant-time embedding and ANN search—something prior work has not demonstrated at this scale (2B, 7B) for code embedding.

3. Carefully designed training—not just scale.

   • Unified multi-task training data. We curate a unified dataset that merges five retrieval task types (text2code, code2text, code2code, text2text, and hybrid) into one training stream.
   • Staged LoRA curriculum. Our “train → merge → re-train” loop (Fig. 1) improves NDCG@10 by +2–3 over a single-stage baseline (Table 4), showing that architecture-agnostic scheduling—not size—drives much of the gain.

4. Efficiency and open models.
   We provide three sizes of models, offering a tradeoff between performance and cost. The open model weights could benefit the community on downstream tasks like code issue localization, benchmarking, and reproducible research.

Reviewer Comment:
This paper is yet another paper with lot of results (indeed they are strong). Just a couple of lines (Line no. 34-37) to discuss the limitations of the prior works, it is a shame.

---

Reply:

Thank you for the comment. To address it fully, could you please specify which prior work you feel CodeXEmbed most closely with?

Below are the spots in CodeXEmbed where we explicitly discuss the limitations of prior work:

1. Introduction
   In the very first paragraph, we note that “standard IR methods, while effective in text-based retrieval, often fall short in specialized domains such as code retrieval” and point out that existing BERT-based code models (e.g., CodeBERT Feng et al. 2020; CodeGPT Lu et al. 2021; UniXcoder Guo et al. 2022) rely on relatively small encoders, leading to subpar retrieval performance in practice. This passage calls out three key gaps in prior approaches:

   • Text-only focus: Most open embeddings target natural language only.
   • Limited scale: BERT-style code embeddings remain too small to capture complex code semantics.
   • No unified text+code encoder: There is no off-the-shelf single model that handles code ↔ code and text ↔ code retrieval.

2. Related Work (Section 4.1)
   We dedicate Section 4.1 to “Retrieval Models for Text and Code,” where we further elaborate on why existing text retrieval systems (e.g., E5-Mistral Wang et al. 2023; NV-Embedding Lee et al. 2024) fail to adapt to code’s syntax and control-flow nuances. We also explain that closed-source code embeddings (Voyage-Code AI 2024; OpenAI Embeddings Neelakantan et al. 2022) are inaccessible and/or expensive, and small BERT-derived code encoders lack broad language coverage (§4.1).

3. Extended Baseline Discussion (Section 3.1, Table 1)
   When listing our baselines, we explicitly cite UniXcoder, CodeBERT, and other smaller code-domain models and show that they underperform—especially on multi-language CoIR subtasks—compared to CodeXEmbed’s larger bi-encoder variants. This empirically underscores the limitation of “BERT-only” code embeddings (Table 1).

Together, these passages demonstrate that we have not only mentioned but also empirically and qualitatively analyzed the shortcomings of prior text-only or small BERT-based code embedding work, as well as the lack of any unified text+code encoder. If there are specific limitations you believe we have not addressed, please let us know which aspects are missing so we can incorporate them.

Reviewer Comment:
The improvements demonstrated in Table 1 is too good. Have the authors ensured decontamination of the evaluation benchmarks?

---

Reply:

We take benchmark integrity very seriously and designed our data pipeline to prevent any leakage between training and evaluation splits. Below we detail the safeguards we applied for each dataset family:

• CoIR: Our general-training models never see CoIR data. Section 3.1 states they are trained “exclusively on our proposed general training data, without using any CoIR in-domain data.” In the optional in-domain training, we fine-tune only on CoIR train splits; dev/test queries remain held-out.
• Other code data: For tasks that overlap with CoIR (e.g., code-agent dialogs), we explicitly remove any sample that appears in CoIR before training (§A.1, lines 560–563).
• BEIR (text): Stage 1 uses the same BEIR-corpus training protocol adopted by NV-Embed and SFR-V2—documents only, no dev/test queries.
• Reproduction: Model weights and the exact training/eval scripts will be released, so anyone can rerun the splits and verify the numbers.

We hope these clarifications address the decontamination question; please let us know if any additional details would be helpful.