Alignment-Enhancing Parallel Code Generation for Semi-Supervised Code Translation

Weaknesses:

• The novelty of this paper is limited. The synthetic generation and alignment-ascending curriculum learning seems simple and straightforward.

Response: Thank you for acknowledging that our method is simple and straightforward. We believe that when combined with effectiveness in improving the performance, simplicity and straightforwardness can be strengths instead of weaknesses.

• The discussion of the baseline is not enough. I mean, I cannot get what the contribution this paper achieved.

Response: Thank you for your feedback. We discussed some of the baselines in the related work section, hence we curated the detailed introduction to each of them in the experiment section. We are ready to answer any specific questions about the baselines if you can kindly share them.

• During the generation of synthetic code, can you generate the snippet-level alignment?

Response: The snippet-level aligned data is from an existing dataset, CoST (Zhu et al.). We did not generate them during the parallel code generation.

• During selecting the synthetic code, why not run the code and compare the returned results directly？

Response: The synthetic code is function-level code, and to run them requires necessary dependencies and testcases. It means that we need customized cross-lingual testcase generation for every single code in every language, which is challenging and expensive to acquire. For example, to test 260110 codes generated by BT in table 7 (shown below), at least 260110 different test cases need to be generated in 4 different languages. On the other hand, compilation filtering only takes the compiler of the target language, and neither compilation nor static analysis require customization for each code. Therefore, our methods is significantly cheaper and can be extended to more languages with little overhead.

Table 7: Datasets used for training. Volume means number of parallel codes.

Table 6: Model size and number of parameters.

Table 8: Comparison with new baselines (variations of curriculums). Data Volume means number of parallel codes. The evaluation metric is Computation Accuracy.

Comparison with code LLMs:

In Table 6 (shown above), we presented the size and number of parameters of each model. As shown in the table, all the models are in ther hundred million parameters range. On the other hand, StarCoder has 15.5 billion parameters, and CodeLLama models range from 7 billion to 34 billion parameters. As you have kindly pointed out, comparing our models with code LLMs would be like "comparing apple to pear" due to the significant difference in model size and computation scales.

Low-resource language:

We agree that C is not necessarily a low-resource language in the software engineering community. In this paper, we refer to it as a low-resource language because of its significantly smaller training set size (~150 training samples) compared to other languages. We aim to evaluate how our method performs with a weak generator resulted from limited annotated data, and this setting can be extended to more languages with limited training samples. As shown in Table 4 (in the paper), our method improves computational accuracy of C by up to 43%, demonstrating strong performance under the low-resource setting.

Effect of rearranging the curriculum:

In Table 8 (shown above) , we presented some new experimental results (with PLBART). Here is an overview of each experiment:

• Function: trained on ECoST-function (annotated data).
• Snippet+Function: trained on ECoST-function and ECoST-snippet (annotated data with fine-grained alignment).
• AND+Snippet+Function: trained with AND (high quality synthetic data filtered by both static analysis and compilation).
• BT+Snippet+Function: trained with BT (noisy synthetic data with no filtering).
• AND+COMP+STAT+BT+Snippet+Function: SPACoder with reversed curriculum order on synthetic data.
• BT+STAT+COMP+AND+Snippet+Function: SPACoder

We can see that when using the high quality synthetic data, AND, the performance is not as good as SPACoder, which uses larger amounts of noisier synthetic data. Similarly, when using the unfiltered noisy data, BT, the performance is also subpar. This shows that both the quality and the quantity of the synthetic data hold significant importance to the performance. When we reverse the order of the curriculum to AND+COMP+STAT+BT+Snippet+Function, the performance drops significantly compared to SPACoder, which indicates the importance of the order of the curriculum to the performance. It is worth noting that the reversed curriculum has very similar performance with BT+Snippet+Function, likely due to the larger volume of the BT dataset overpowering the effect of the previous datasets.

Table 6: Model size and number of parameters.

Table 7: Datasets used for training. Volume means number of parallel codes.

Table 8: Comparison with new baselines (variations of curriculums). Data Volume means number of parallel codes. The evaluation metric is Computation Accuracy.

Model size and compute: In Table 6 (shown above), we presented the size and number of parameters of each model.

Comparison between testcase filtering and compilation/static analysis filtering:

1. Difference in cost. Testcase filtering requires customized cross-lingual testcase generation for every single code in every language, which is challenging and expensive to acquire. For example, to test 260110 codes generated by BT in table 7, at least 260110 different test cases need to be generated in 4 different languages. On the other hand, compilation filtering only takes the compiler of the target language, and neither compilation nor static analysis require customization for each code. Therefore, our methods is significantly cheaper and can be extended to more languages with little overhead.
2. Difference in hypothesis. Testcase filtering is essentially an alternative way to acquire annotated parallel code data, because target code that passes expected testcases is functionally parallel to the source code. Therefore, testcase filtering follows the straightforward hypothesis of more annotated data leads to better code translation performance. On the other hand, our hypothesis is that large amounts of noisy parallel data arranged in a specific curriculum benefits code translation more than small amounts of annotated data. Our experimental results (with more presented in Table 8) support this hypothesis.

Effect of rearranging the curriculum:

In Table 8 (shown above) , we presented some new experimental results (with PLBART). Here is an overview of each experiment:

• Function: trained on ECoST-function (annotated data).
• Snippet+Function: trained on ECoST-function and ECoST-snippet (annotated data with fine-grained alignment).
• AND+Snippet+Function: trained with AND (high quality synthetic data filtered by both static analysis and compilation).
• BT+Snippet+Function: trained with BT (noisy synthetic data with no filtering).
• AND+COMP+STAT+BT+Snippet+Function: SPACoder with reversed curriculum order on synthetic data.
• BT+STAT+COMP+AND+Snippet+Function: SPACoder

We can see that when using the high quality synthetic data, AND, the performance is not as good as SPACoder, which uses larger amounts of noisier synthetic data. Similarly, when using the unfiltered noisy data, BT, the performance is also subpar. This shows that both the quality and the quantity of the synthetic data hold significant importance to the performance. When we reverse the order of the curriculum to AND+COMP+STAT+BT+Snippet+Function, the performance drops significantly compared to SPACoder, which indicates the importance of the order of the curriculum to the performance. It is worth noting that the reversed curriculum has very similar performance with BT+Snippet+Function, likely due to the larger volume of the BT dataset overpowering the effect of the previous datasets.

Dear Reviewer 9b25, in light of the updates made in the rebuttal, we would be grateful if you could comment on whether our response addressed your concerns. Thank you for your time and consideration.

Questions:

• Unclear/Imprecise terminology: function-level code translation.

Response: Your understanding is correct. The parallel code generation is at function level, and the computational accuracy evaluation is done by wrapping the functions into executable programs, which contains the necessary dependencies and test cases.

• The word “snippet” is vague.

Response: The word “snippet” in this paper means small chunks of code that are separated by code comments in a program. It is not necessarily a complete function. We will add the definition of both early in the draft.

• What is “SPACoder-function” (Table 2 and Section 4.1)? Is it BT, STAT, COMP or AND? Or is it all of them, but in a curriculum learning setup?

Response: SPACoder-function is a dataset that contains all of BT, STAT, COMP and AND in a curriculum learning setup. Compared to the final dataset we used to train the SPACoder models, the only difference is that it does not include the annotated data ECoST-*.

• I assume “SPACoder-PLBart” refers to a model where the generator as well as the base model are PLBART?

Response: Your understanding is correct. SPACoder-PLBART means a PLBART model trained on PLBART generated data, same as SPACoder-CodeT5. This setup is to demonstrate that our proposed method can be generalized to different models, without the assistance of more powerful models.

Concepts that need more elaboration:

• When doing static analysis on function signatures, how do you match types between different languages? Like int[] in Java and vector< int > in C.

Response: We build the cross-lingual type matching rules by extracting function information from large code corpora like CodeNet through AST parsing. After AST parsing, the information about a function will be stored in a dictionary, from which we can access each function’s signature. All the types will first be converted to a small set of base types, and then checked for more fine-grained alignment. For example, ArrayList in Java and <vector> in C++ will both be mapped to the base type list. The Integer type in Java will be converted to base type int, which can be mapped to the int type in C++. In this way, ArrayList<Integer> and vector< int > will be labeled as a cross-lingual pair of matching types.

• If you are operating at the function level, how do you apply a compilation filter?

Response: The compilation is done by creating a temporary file with the given code and calling the corresponding compiler from the command line. For Java, the function will be wrapped in a class with an empty main() function, and run with the javac command. For Python, the file will be run with the “python2/python3” command, which is the same as executing the code. After each compilation, the temporary file is deleted and the stdout/stderr will be saved for filtering and evaluation later. This whole process is parallelized, with memory limit and timeout for each process.

• Let us say that the generator takes code from Language A and converts it to Language B. Then while fine-tuning, do you train on [B, A] samples, or [A, B] samples or both?

Response: We use both. We have conducted experiments to compare the performance of using A-B, B-A and both as training data, and found that using both consistently gives better performance. Therefore we have been using the combination of both A-B and B-A for all the experiments.

• Could you please clarify this terminology back-translation?

Response: Your understanding is correct. Since we use data generated in both directions, it is indeed different from commonly known back-translation. What we refer to is “parallel data generation with trained translator models”, and for simplicity, we call it back-translation since it’s conceptually similar. We will add the clarification in the draft.

• Typically, curriculum learning starts with easy examples and moves to difficult examples.

Response: Thank you for your feedback. You are right that the alignment-ascending curriculum in this paper appears slightly different from the typical easy-to-difficult curriculum. However, just like you kindly already pointed out, in terms of programming languages, high-quality data can actually be more challenging than low-quality data. Because low-quality parallel data might share more similarities between the source and target language due to the “shallow translation” problem, where the model translates token by token or directly copies from the source language. This makes the low-quality data potentially easier to learn than the high-quality data. We agree that it is important to clarify the subtlety and we will add it in the draft.

Typos / Bad phrasing:

Response: Thank you for your feedback. We will rephrase it to “we denote the synthetic parallel data from cross-lingual static analysis and compilation filtering as STAT and COMP, respectively”.