CODE REPRESENTATION LEARNING AT SCALE

Thank you for acknowledging the novelty and impact of our work, especially our efforts on diving deep into the key ingredients for code representation learning. We are also grateful for your constructive suggestions on additional benchmarks and baselines, and we have carefully addressed them in our general response (2/2) above. To complement the discussion, we also summarize our initial motivations for experiment design in our general response (1/2).

Naming Convention It was a hard decision for us. On the one hand, we did want to follow the BERT convention, while we also noticed the recent trend to name the 1B model as the Large model on the MTEB Leaderboard. However, we are seriously considering your question and trying to find a sweet spot between our current names and CodeSage-Base, CodeSage-Large, and CodeSage-XLarge.

Latex Typesetting and Paper Title We will improve the formatting as you suggested. We also want to talk about our motivations for the current paper title.

1. Our work is among the first (in addition to StarCoder, to the best of our knowledge) to train the encoder-based embedding model at a large scale of pretraining data and increased model size. We limit our model size to 1B to balance performance and inference cost in many practical settings, e.g., building a retrieval system over billions of examples.
2. More can be different. As reported in Figure 3b of our paper, the dropout-based contrastive learning does not work well with large-scale pretraining data, though it works for UnixCoder and SimCSE, where less than 3M examples are used for training.
3. We indeed hope our work can motivate more researchers to keep pushing the frontiers of code embedding models by advancing the pretraining strategy on a large scale of data, as what has been doing for text embeddings (MTEB).

We are looking forward to hearing your thoughts and would appreciate it if you have better suggestions.

The Baseline Models Are Smaller One of our goals is to study if larger models would be effective for code representation learning since such models are missing in the literature. Therefore, we urge not to consider that as a weakness of this work. On the other hand, larger decoders are often not primarily trained to serve retrieval or code understanding works. Those models often yield poor performance in our preliminary evaluations, and we exclude them from comparison as we think it’s unfair to compare with them. However, we appreciate your question, which makes us realize that including these baselines would have made our paper more complete. We summarized the evaluation of CodeGen2.5 (7B), Starcoder (15.5B), and CodeT5+ (16B) in Tables (1a)&(1b) in our general response (2/2) above.

Evaluation on Clone Detection We did evaluate BigCloneBenchmark at the beginning of this project. However, we found the task very easy but inefficient to evaluate due to the large data size. Hence, we excluded it and focused on more challenging ones. Presumably, given that CodeSage shows strong performances over previous SOTA models on our chosen tasks across many languages, it will perform on par if not able to largely outperform the baselines. Instead, we dived deep and performed analysis to show the reasons behind the effectiveness of CodeSage (e.g., the effectiveness of MLM/DOBF and CL, performance with scaling), which can drive future research works.

But we are grateful for the suggestion, which motivates us to improve our paper's completeness and overall quality. We summarized our evaluation results on both POJ-104 and BigCloneBenchmark in Table 3 in our general response (2/2) above, where CodeSage-small performs slightly better than UniXCoder.

We sincerely thank you for the constructive feedback. We have taken your comments into serious consideration. We provided the evaluation results in our general response (2/2) accordingly, which we will also use for the next update of our paper to address your questions comprehensively. Let us know if you have any other comments.

Thank you for the feedback. To address your main question, we briefly summarize the fundamental contributions of our work below, which differentiate CodeSage from the existing work.

1. We are the first to train different-sized code embedding models using a much larger amount of publicly available data and study their effectiveness as an off-the-shelf encoder model in downstream tasks.
2. We innovate an effective two-stage pretraining strategy in the presence of large-scale data, which outperforms the previous SOTA model by significant margins on a wide range of tasks. In particular,
   • we propose an effective MLM strategy for code, which is different from the existing literature (including CodeBERT and many others) that mainly follows 80-10-10 corruption convention proposed for text in BERT. We dive deep into why 80-10-10 is suboptimal for code through quantitive and qualitative experiments in Figure 2.
   • we identify the value of the DOBF objective that is not utilized in prior works for training code embedding models and cross the technical hurdles to make it work for training encoders (see Appendix A.5 for details). We further boost the performance by leveraging our proposed MLM with DOBF to complement each other (see Table 3 in our paper).
   • we proposed an effective contrastive learning strategy with effective hard positive construction tricks that boost the performance over the baseline (see Figure 3a).
   • we deep dive into the key ingredients of code representation learning through an exhaustive ablation study, which we believe will help future works advance the performance further.

Differences Between CodeSage and The Combination of CodeBERT and DOBF

Our work is not a combination of CodeBERT and DOBF. CodeBERT adopted the original MLM objective proposed for text in the BERT paper. In contrast, we identified the caveat of such naive adoption for code and proposed an effective MLM variant, effectively boosting the performance (see Figure 2). We are the first to identify the value of the DOBF objective that is not utilized in prior works for training code embedding models and cross the technical challenges to make it work for training encoders (see Appendix A.5). Moreover, we also innovate an effective contrastive learning strategy for stage-II pretraining of our two-stage pretraining scheme. We run exhaustive ablations to show why such two-stage pretraining is necessary and share our findings on the key ingredients for code representation learning.

Why Smaller Models Sometimes Outperform A Larger Model?

Large models should generally outperform smaller ones on most tasks; we observe such a trend for CodeSage, especially in the zero-shot settings. However, there are no guarantees that larger models are universally better than smaller ones. We can also clearly see this on the MTEB leaderboard for text; for example, bge-large-en-v1.5 outperforms bge-base-en-v1.5 (https://huggingface.co/BAAI/bge-large-en-v1.5) on average, but the opposite is also reported on some classification tasks and retrieval tasks, with similar findings for e5-large vs. e5-base (https://huggingface.co/intfloat/e5-large) and many other models.

We have taken your comments into serious consideration and hope our response can help address your question about the contributions of our work. We would be happy to take your other questions, if there are any.

Thank you for acknowledging our work's contribution and efforts in enhancing the zero-shot performance and diving deep into the problem through exhaustive analyses. We also appreciate your questions on additional experiments, which indeed motivate us to make our work more complete. Please see our general response (2/2) above for the evaluation results. We also summarized some key motivations of our current experiment design in (1/2) to complement the discussion.

Missing Comparison with Decoder-only Models As mentioned in our general response (1/2) #1, we tried to avoid unfair comparisons against the larger decoder-only models as they are not primarily trained for discriminative tasks and often yield suboptimal performance. As we can see in Tables (1a)&(1b) in our general response (2/2), CodeGen2.5 (7B), StarCoder (15.5B), and CodeT5+ (16B)-encoder all perform very suboptimal in the zero-shot setting, especially on NL-to-Code search tasks which often require properly designed training objective (e.g., contrastive learning) on the bimodal-data (text-code) to boost the performance. However, we appreciate your question, which makes us realize we should include these results in our paper for completeness.

Why Not Code-CPT-001 We chose Text-Ada002 models per OpenAI’s suggestion on their website and claim that Ada002 is better or at least comparable to CPT-001 on both text and code tasks but at a much cheaper cost. We have validated their claim in tables (1a) and (1b) in our general response (2/2) above.

Some Classification Benchmarks Ignored Instead of going through all benchmarks in CodeXGLUE, we preferred to dive deep and perform analysis to show why CodeSage learns effective code representations. We initially evaluated clone detection, which we found is easy for different models but has high evaluation cost due to the large data size. We excluded it and focused on other challenging classification tasks. However, we have seriously considered your comments and summarized evaluations on POJ-104 and BigCloneBenchmark (BCB) in Table 2 in our general response (2/2). We can see that CodeSage maintains its strong performance.

Missing Fine-tuned Results on CSN, AdvTest, and CoSQA Our primary goal was to build an off-the-shelf embedding model that does not require finetuning in many practical use cases; semantic search is one such case. Strong zero-shot performance would allow wider usage of the embedding models, as annotations for code are rare and very costly to collect. Our evaluation strategy also aligns with the recent trend in the text domain, where strong models have been actively developed to advance the MTEB Benchmarks. We hope our work can motivate a similar trend for code.

However, we are grateful for your question, which motivates us to better connect with the previous work. We summarized the finetuning results in Table 2 in our general response (2/2), where CodeSage-base (356M) consistently outperforms CodeT5+(770M). We want to point out that both CodeT5+ 220M and 770M optimize an additional “matching loss” during finetuning (cross-encoder type finetuning of the decoder with pairs selected by the encoder). We hypothesize that if we finetune a copy of UnixCoder or CodeSage on such matching tasks and use them with the original encoders, we can also get a further performance boost for both models.

Unclear Writing Thank you for pointing out the limitations of the abstract and introduction. We will work on them to clarify our writing in the camera-ready version.

We appreciate the opportunity to improve our paper based on your reviews. We will include these evaluation results reported in our general response above in the next update of our paper to address your questions comprehensively. Let us know if you have any other comments.

We sincerely thank the reviewer for acknowledging the novelty and impact of our work, especially your great attention to the details we mentioned in the paper.

We also appreciate your constructive suggestions and thoughtful questions. Please see our response below.

Removing One Objective At A Time We partially cover it through different experiments.

1. Figure 4a shows that contrastive learning (stage-II pretraining) helps boost performance on top of stage-I pretraining with DOBF & MLM only.
2. Figure 5 shows contrastive learning from scratch yields suboptimal performance and cannot benefit from large model sizes. Stage-I pretraining is essential to provide a good starting point for contrastive learning to be effective.
3. Table 3 compares DOBF vs. MLM and effective ways to combine them for the stage-I pretraining.
4. We initially also tried optimizing all three objectives from scratch. However, we observed suboptimal performance that can be explained by #2 above (which indicates that including contrastive learning from scratch is not effective).

We would like to explore further if we can get computation support for the camera-ready version.

Responses to Specific Questions

Q1: We plan to release the model checkpoints and the evaluation code.

Q2: Your insights align with our hypothesis that random token replacement could hurt MLM training for multilingual text. However, this requires empirical validation; to the best of our knowledge, the existing multilingual MLM encoders are trained with the 80-10-10 practice. Another hypothesis is that the 80-10-10 corruption convention is proposed by the BERT paper, which pre-trained on a comparatively smaller dataset (BooksCorpus + Wikipedia En) over 40 epochs. In such a setting, corruption can help avoid overfitting and cause no significant performance drop. However, nowadays, scaling up the pretraining data allows us to reduce the training epochs, and the effectiveness of the 80-10-10 convention can change consequently.

Q3: We used 128 A100 GPUs for pretraining CodeSage-Small&Base, which takes 3-5 days to finish the two-stage pretraining. We used 256 A100 GPUs for pretraining CodeSage-Large, which takes roughly six days. We will add those details in Appendix A.2.

Q4: We can train even larger models. In this work, we limit the model size to 1B by considering the cost and latency for many practical settings, e.g., providing embeddings for building a retrieval system over billions of examples. However, if we can get enough support on resources, we are also interested in seeing how the performance can be further improved by scaling up both the data and model size, as we can first optimize for performance and then inference cost.

Q5: CodeBERT and GraphCodeBERT are trained on 6 languages, UniXCoder (we used UnixCoder-Nine in this paper) and CodeT5+ embedding models are trained on 9 languages, Starencoder is trained on 86 languages. It is unknown how many programming languages are used to train text-embedding-ada-002.

Benchmarking with Large Decoder Models && OpenAI Ada-002 vs Code-CPT-001

1. Decoders, even with large model sizes, often perform poorly on discriminative tasks in the zero-shot setting, as they are not primarily trained for those tasks. In particular, NL2Code search often requires properly designed objectives on the bimodal text-code data, e.g., contrastive learning or conditional generation.
2. OpenAI-Code-CPT-001 underperforms Text-Ada-002 on code tasks, which aligns with their claim.

• Table (1a) Zero-Shot Code-to-Code Search

• Table (1b) Zero-Shot NL-to-Code Search

Finetuning Results on NL2Code Search

CodeSage maintains the performance gain over the baseline models, especially that CodeSage-base (356M) persistently outperforms CodeT5+(770M). We want to point out that both CodeT5+ 220M and 770M optimize an additional “matching loss” during finetuning (cross-encoder type finetuning of the decoder with pairs selected by the encoder). We hypothesize that if we finetune a copy of UnixCoder or CodeSage on such matching tasks and use them with the original encoders, both models can get a further performance boost. (Note: Finetuning CodeSage-Large is undergoing; we will add the results once it is complete.)

• Table 2 Supervised Finetuning on NL2Code Search

Additional Benchmarking Results on Code Clone Tasks

We report the finetuning results by using the evaluation code from UnixCoder. We are working on the evaluations for other baseline models and will add them once they are ready.

• Table 3 Supervised Finetuning on POJ-104 and BCB