Multilingual Code Retrieval Without Paired Data: New Datasets and Benchmarks

We thank the reviewer for the very detailed review and for providing suggestions to improve our manuscript. We took those into great consideration and carefully responded to comments and questions in what follows.

1. Experiment about the effectiveness of data is not clearly setup.

Response: Assuming that the reviewer’s concerns are related to experiments with results reported in tables 2 and 7 (kindly let us know if not, and we will address it), we clarify that those correspond to estimates of the semantic alignment between text and code for each language combination. We did that by estimating the expected BERTScore between text and code pairs, and did that independently for each language combination. We did that using an encoder that was trained on multiple natural and programming languages.

We also reported what we called the GPTScore which uses a similarity obtained from prompting a language model (GPT-3.5-turbo in this case) with a text-code pair. More details about this approach including the exact prompt we used are placed in the appendix due to space constraints.

We observed both metrics to be predictive of retrieval performance, e.g., they suggest high semantic similarity for python and low for javascript, and those are the languages for which models consistently perform the best and the worst, respectively.

2. The purpose of adding TATOEBA and CodeXGlue is not clearly articulated. They also make the claim of learning through anchor languages weaker because another factor is added to the training mix.

Response: We used a relatively small number of source-source and target-target instances in order to make our training mix more diverse, and then help our models cope with shifts at evaluation (cf. [1] for a discussion on how diverse training data sources enable generalization to new, shifting distributions). CodeXGlue for instance gives us some code-code pairs. Even though the underlying languages differ from those we evaluate on, it’s a more similar task to those we expect at testing than the ones defined by other datasets.

3. It's not sure if the pre-training data of various models already have multi-lingual exposure. The near-0 auMMR number could also be task alignment issues.

Response: We completely agree with the reviewer. While base models were trained on multiple languages, they were never trained to yield chunk/sentence level embeddings that semantically align. Fine-tuning is thus required to align with test tasks of interest.

4. Clarity

Response: We thank the reviewer for kindly remarking points of improvements in terms of clarity and presentation of our contributions. We addressed each related comment in what follows.

5. What's the purpose of the AST parsing phase?

Response: We used AST parsing to be able to collect natural language from code snippets (docstrings and comments). We also followed CodeSearchNet and split files into functions/methods.

6. Table 2: "high scores are observed across all language combinations". It's better to have a baseline to help understand a relative assessment of "high".

Response: Thank you for pointing this out. We modified that sentence to make precise and clear what we mean by high similarity in that case.

We clarify that all of those scores are bounded in $$ 0,1 $$, so that sentence suggests that the scores are much closer to the upper limit than to the lower one. We further remark that a random embedding or random data would score very close to 0, and would not yield a very informative baseline. In other words, our goal in that sentence is to highlight that the scores suggest a reasonable signal-to-noise ratio in the data. This is further confirmed in our experiments in Section 4.3.1 and in table 9 in the appendix: the data can be predicted to high accuracy so the semantic similarity must be high.

7. Section 3.1 last paragraph: "Note that any data realization x observed..." It's not clear why we need to discuss it here. Better to clarify the purpose.

Response: In that paragraph, we are highlighting that when we use $x$ to refer to a data instance, we are referring to a sequence of tokens. This paragraph is intended to introduce some notation to make the manuscript accessible to readers maybe not so familiar with sequence modeling settings.

8. Section 4.1 second paragraph: ".., models able to generate train ..." Reviewer couldn't understand what does this mean with moderate effort.

Response: ”Models able to generate” refer to models that can be used to generate text. In more detail, our evaluation covers encoder-only, encoder-decoder, and decoder-only architectures. As such, encoder-decoder and decoder-only both can be used for text generation, in addition to yielding chunk/sentence level embeddings.

9. Section 4.1 right after Equation 3: "... and define translation auxiliary tasks." What does this sentences mean?

Response: This sentence explains that the models that are able to generate text, i.e., encoder-decoder and decoder-only models, are trained with an additional “generative” objective compared to contrastive-only encoders.

To make the presentation clear, we will rephrase it and add an explicit reference to the generative objective discussed in Section 4.1 - Eq. (3).

10. Section 4.3.2 last sentence: "Finally, the decoder-only model doesn't reach..." it's better to call out the model name (is it CodeGen?) to save the readers some back-n-forth work.

Response: Yes, we agree with the reviewer and replaced it in the text.

11. Would you open source the data and evaluation code?

Response: Yes! The datasets are already (anonymously) available:

• https://huggingface.co/datasets/blindsubmissions/M2CRB
• https://huggingface.co/datasets/blindsubmissions/GH_text2code

Links are also in the manuscript in the captions of tables 1 and 3 for training and testing data, respectively.

Code for data processing, fine-tuning, and evaluation will be released upon publication.

References

[1] Albuquerque I, Monteiro J, Darvishi M, Falk TH, Mitliagkas I. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00804. 2019 Nov 3.

We thank the reviewer for their comments and for the detailed assessment of the strengths and weaknesses they identified. We carefully address the latter below.

1.There are no baselines presented to understand how well the proposed technique actually works. While baselines might be difficult to get for non-English code search (Table 4), I would assume for Python-Java code search (Table 5) and English code search (Table 6) available model embeddings should work well. For instance, CodeBERT…

Response: We kindly highlight that we did evaluate other models not trained by us such as CodeBERT and CodeT5. We observed however that those models have no predictive power in the settings we consider and perform as well as a random embedding. Note that CodeBERT MRR results openly available typically correspond to the performance after fine-tuning in a particular language combination. We went beyond that and evaluated a single fine-tuned model, trained in the multilingual scheme we proposed.

We further refer the reviewer to table 9 for results with state-of-the-art embeddings (OpenAI’s and Sentence Transformers’s) from models not trained by us, both open and closed source. Note that those results were kept in the appendix because we have no control over the training of the evaluated models, and in particular have no control over the data used for training. Still, those results attest for the predictability of the M$^2$CRB data and offer a performance upper bound that follow-up methods should strive to achieve.

2. This work requires paired NL data (such as between Spanish and English), and it incorporates this paired data in the loss function. However, another way to utilize this data could be to learn a translation model from English to Spanish using the paired data and use this translation model to translate English queries in the CodeSearchNet data to create a much larger (Spanish, Code snippet) dataset, albeit with some noise. This has the advantage of creating larger synthetic training data that can be directly used to train the code search model, instead of the objective proposed in the paper. Do the authors have some assessment on why the proposed technique is a better approach than this one?

Response: Thank you for commenting on the use of translation to generate training data. While we agree with the reviewer and, starting from an English-to-code dataset, one can indeed create synthetic variations of it by translating docstrings to different languages, it must be noted that translation step would add significant cost to pre-processing, and its quality could not be good enough considering we would be translating docstrings rather than natural language, and that we could be targeting low-resource languages.

Cost and quality concerns aside, the proposal is indeed valid and orthogonal/complementary to what we show rather than a competing approach: if one does what the reviewer suggested and, for instance, translates docstrings in CodeSearchNet to Spanish and French, we showed that training a model in such a dataset would enable mapping directly between any combination of natural or programming languages. Either approach used to build a set of paired datasets would be equally good.

In other words, combining existing training datasets or synthesizing via translation are both valid and complementary approaches to build a dataset with the properties our training requires. We highlight once more: our goal is to show that generalization is possible to indirectly observed language combinations.

We added a sentence in Section 3.2 remarking that other approaches are possible to create the style of training data we require.

We thank the reviewer for the insightful review. We addressed all comments and questions in detail in the following.

1. While the M$^2$CRB dataset covers multiple language pairs, it can be expanded to include more programming languages for better representation.

Response: We agree with the reviewer. Kindly refer to the Update plan section of our dataset card in the following link where this is listed: https://huggingface.co/datasets/blindsubmissions/M2CRB

2. The study focuses on the code search/retrieval setting, but it would be helpful to investigate the applicability of the introduced data and training approaches in generative settings as well.

Response: We completely agree with the reviewer and also mentioned this in the text. Especially for the case of encoder-decoder architectures that require paired training data, the approach we proposed could enable multi-linguality efficiently for languages not necessarily combined during training.

However, we decided to focus on the retrieval setting given its high practical importance, especially now that retrieval augmented generation (RAG) approaches become the prevalent language modeling framework.

3. The evaluation focuses on models within the 60M-360M parameter range, and exploring larger-scale models could provide insights into the effect of model size on generalization capabilities in this domain.

Response: We completely agree with the reviewer and highlight that the decision on the model scale we focused on was in part influenced by the amount of compute we had available. We also note that recent literature (e.g., [1]) has shown that encoders do not observe scaling gains as pronounced as decoders. For instance, the extremely popular sentence-transformers/all-MiniLM-L12-v2 encoder has only 33M parameters. Another recently released encoder, bigcode/starencoder, has 125M parameters. We thus believe the range we chose to be very well aligned with common practice in this setting.

4. Can the training approach proposed in this paper be adapted for generative models, and if so, how would it affect their performance on text-to-code generation tasks?

Response: As discussed above and in the manuscript, our proposed training strategy would be useful as-is to enable multi-linguality in encoder-decoder architectures. Even more than that, our results show that this can be done while also carrying out contrastive training on top of chunk-level embeddings obtained from the encoder. So to answer your question, yes, conditional generative models can benefit from our proposal, and no, we do not anticipate it affecting the performance of the base model it builds upon.

5. Are there any potential biases in the dataset, such as the influence of specific programming language styles or the quality of non-English docstrings, that may affect the model's generalization capability?

Response: Thank you for this insightful question. We measured language-specific text-code alignment (via BERTScore and our proposed GPTScore) and did observe variation across languages. Indeed, the textual component of the data we make available consists of naturally occurring comments on GitHub and, as such, relatively high variance is expected given how large a community of developers GitHub represents, and subcommunities will have different practices. On the other hand, natural data is reflective of what a real-world codebase would look like, and thus defines a useful "in-the-wild" testbed.

6. How do the models perform when fine-tuned on different programming languages (e.g., C++, Rust, etc.) and less common natural languages? Would the performance be consistent with the results presented in the paper?

Response: Our results suggest strong generalization ability to new language combinations for various model classes and training settings. We do not anticipate any issue specific to the languages highlighted by the reviewer, provided that they are included in the training mix (not necessarily paired as we’ve shown).

7. How would the results change when using larger-scale models, such as GPT-3 or the recent Megatron-LM? Would the generalization capabilities improve with increased model capacity?

Response: We kindly refer the reviewer to item 3 above with related discussion. While this would need to be tested in our specific setting, recent literature suggests gains on encoders are not as pronounced with scaling as they are for decoders. This is also confirmed by the fact that our best performing models were not the biggest ones. We also highlight that our decision in terms of which models to fine-tune was in part motivated by the amount of compute we had available for this project.

References

[1] Wang Y, Le H, Gotmare AD, Bui ND, Li J, Hoi SC. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922. 2023 May 13.

We thank the reviewer for their assessment and for seeking to ensure our contribution is relevant, which we believe will help improve our work. Please see below for detailed responses and comments.

1. training and evaluations are not motivated well.

Response: We agree with your other comment, i.e., “we know that multilingual LMs learn to map text or code in a shared embedding space that enables them to perform on unseen language pairs”. However, the embedding spaces learned by LMs do not immediately lead to embeddings which can be used to query code for retrieval applications. One of the motivations of this work is to show that we can adapt the representations learned by LMs and make them much more effective for cross-lingual search.

We further clarify that evaluations follow the standard information retrieval setting where, given a query, one seeks $k$ relevant entries from a database. That’s our target application as discussed in the text. We discuss motivations further in item 5 below. As per training, we used contrastive learning approaches to yield a semantic embedding that is language agnostic so as to enable multi-linguality.

While we believe the summary above is written down somewhat clearly in the text, we would be happy to modify any part of the content that the reviewer believes is confusing or misleading.

2. The main questions addressed in the paper are somewhat known.

Response: The main question we seek to address is as follows: Can a contrastively trained embedding enable retrieval for language combinations only indirectly seen during training? As we argued in the text, if yes, then training of this kind of model can be carried out without paired data from all languages in the set of languages of interest. To our knowledge, ours is the first work showing evidence of that being the case.

3. I couldn't perceive the value and need of the dataset. If we need multilingual docstring,...

Response: That’s actually the main motivating point of the line of work we carry out in this paper: translation steps add overhead and should be removed. This translation overhead (be it machine or human translation) mainly affects non-English-speaking communities of developers or developers required to write documentation in local language. We thus make available both data and efficient training strategies that help bypass undesirable intermediate translation steps.

We also highlight that existing machine translators are trained in natural language, and don’t necessarily work well on docstrings and code comments which embed jargon, variable names, and other code specific information. Machine translation might also be unreliable in low-resource languages.

4. …isn't it possible to use NMT models to translate the docstring available in English?

Response: Please refer to Table 10 in the appendix where we compare direct search with models that use an external translator of docstrings. We see that multilingual models perform better than translate+search using an English-only model. We also see that multilingual models can even further benefit from translating docstrings into English, so even if translation is an option, multilingual training is beneficial.

5. good reason to study code search tasks?

Response: The main motivation we considered for doing code search as opposed to generation is to address situations where users wish to obtain pieces of code from a known and tested codebase. There are many situations where the user desires retrieval results and in the case of retrieval augmented generation (RAG) models, our approach can serve as the underlying representation used to perform the retrieval step as well as a generator. One can also consider the situation of proprietary codebases that are not in the training set of large code LMs. In those cases, generative models wouldn’t be able to generate code that makes sense in the context of an API or library it never had access to and knows nothing about.

We additionally highlight that code search is an extremely active research field with several papers coming out every week, especially so given the recent trend in combining retrievers with decoders to perform retrieval augmented generation (RAG).

Finally, as noted by the reviewer, we also highlighted in the paper that retrieval applications are typically less expensive in terms of compute resources requirements, and might be more accessible as a research topic to groups with compute constraints.