CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming

Metric issue and Formal definition of compilation

We understand the metrics used may not be comprehensive enough, as noted in [1]. To address this, we manually analyzed and executed the translated code. Please reference to global response. Moreover, as it was mentioned in Out of the BLEU [1] that ChrF is a better fit for evaluating the generated code, we calculated the results based on this metric. The results on ChrF show that CodeRosetta achieves a good score in comparison to other models. Note that we omitted BabelTower from the baseline as neither its model nor its results are available.

Compilation Accuracy is the ratio of compilable generated code to the total number of reference codes in the test set.

Functional correctness

Please refer to the global response.

Details on the model's language translation capability.

The bilingualism of our model stems from several pre-training and training objectives that we have utilized and developed. For example, the Mask Language Modeling helps the model to learn cross-lingual representation. For instance, when a if keyword, which is a shared keyword between C++ and Fortran, is masked, forces the model to learn the context of an if statement; that is how such a statement looks like in each language. On the other hand, AST entity recognition teaches the model about language-specific entities and cross-language entities to further enable the model to map similar programs in different languages to the same embedding space (encoder model). Then, Denoising Auto Encoding and Back Translation enable the model to leverage the embeddings learned in previous steps to train an encoder-decoder model in order to generate code in the target language.

Fine Tuning with synthetic data

Finetuning is an optional step. One of the biggest challenges in training a code translation model is the lack of paired data. Therefore, a common practice is to train a model in an unsupervised way. However, even if paired data is available, the amount of data could be insufficient to train a model from scratch. Here, we aimed to evaluate how a model will perform if a small set of paired data exists for fine-tuning. Since the model already has knowledge of code translation, this fine-tuning step can further improve the result of the model and make its performance comparable to a much larger model. CodeRosetta is one of the few unsupervised trained models that support fine-tuning if the data exists.

Differences in ASTs of different languages

That is indeed true. The ASTs differ slightly depending on which language we are dealing with. However, some entities are common between ASTs, such as identifiers. These common entities enable the model to learn how they are utilized in the source code. For example, how identifiers are being used in C++ and CUDA programs. Some other entities are not common among the languages, such as pointers. The absence of such an entity in a programming language forces the model to not predict the type of an entity as a pointer. This, on the other hand, helps the model to realize that pointers are available in a programming language but are not available in another language.

Dear Reviewer BqwA,

Thank you for your quick response, your thoughtful feedback, and the opportunity to address your questions.

Metrics and Compilation Definition

ChrF Metric: Yes, ChrF refers to the Character n-gram F-score. We apologize for not providing the full name earlier. We used the HuggingFace Evaluate Metric library with the default $\beta$ value set to 2. We used the ChrF metric following the recommendations from the “Out of the BLEU: How Should We Assess Quality of the Code Generation Models?” paper, which states that: ChrF and ROUGE-L are the best-performing metrics for the assessment of code generation models among the metrics we consider. We have also included the results using the ROUGE-L metric:

Compilation Accuracy: We acknowledge the limitation of using compilation accuracy as a metric in code translation, as it is possible for a model to generate code that compiles successfully but diverges from the intended functionality. To partially address this, we conducted a manual evaluation of 30 generated code samples to compare their behavior against reference implementations. While we fully understand the limitations of such evaluations, we found that the functional correctness of generated code was preserved in the majority of samples (~93%). We also did run those generated codes against the reference and found them to be compilable and functionally correct producing the same outputs with the same inputs as the reference. In addition, the use of the compilation accuracy metric enabled us a direct comparison with BabelTower (ICML 2022), especially given that their codebase is not open sourced.

Following this discussion, we will dedicate a section in our paper that covers the limitations of existing metrics in code translation and our efforts to partially address these limitations through manual inspection. We will also include the samples in the Appendix.

Taking all of these additional results into account, we believe we have made our best effort to address this concern, though we acknowledge it may not be perfect. That said, we would be more than happy to include any additional metrics you suggest that may better characterize the performance of our work.

Fine-Tuning

Paper-Tables-1 & 2 report the results from the fine-tuned model. Paper-Table-3-Page-9 reports the results from the pre-trained model.

Discussion on Fine-tuning for Code Translation w/o Verification in Supervised Manner. In code translation, paired data is scarce. However, our model benefits from a foundational understanding of code translation acquired through unsupervised and self-supervised pre-training (243K training examples). We show that finetuning, even on limited synthetic data—without verifying the generated samples and their one-to-one mapping to input code in a supervised manner—generated by larger models (merely 5K paired samples---less than 2% of total data points), can further boost the model’s performance. While synthetic data may introduce some errors (as large models can make mistakes in their translation), the combination of foundational understanding during pre-training and fine-tuning with such a small number of synthetic data points can lead to additional improvements.

Model’s Translation Capability

Thanks for this great question. As you mentioned, there are entities, libraries, keywords, and syntaxes from the source language (e.g., C++) that may not be valid in the target language (e.g., CUDA). std::unique_ptr and other C++ Standard Template Libraries (STL) belong to this category and must be avoided in the translation. The pre-training process in CodeRosetta (esp. Weighted Token Dropping) equips the model to gain a semantic understanding of source and target languages. It reduces the occurrences of invalid tokens in the translation. Nonetheless, there may be cases in which the model fails to correctly translate from common entities in the source code to valid entities in the target code.

Looking into our dataset, there was no occurrence of std::unique_ptr in the test set. However, as the training set has std::unique_ptr, we intended to test the capability of CodeRosetta by developing a C++ code that uses std::unique_ptr, we made sure this sample does not exist in the training set:

Reference C++ Code

void initializeArray(std::unique_ptr<int[]>& array, int size) {
    for (int i = 0; i < size; ++i) {
        array[i] = i * 10;
    }
}

CodeRosetta CUDA Generation

__global__ void initializeArray(int *array, int size) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < size) {
        array[i] = i * 10;
    }
}

This sample is about array initialization. As can be seen, the model is able to provide the CUDA code. It removes the std::unique_ptr from the CUDA kernel as it is not supported in the device code. In CUDA, std::unique_ptr is called in the host kernel, which is the main function. Our model focuses on the generation of device function and hence is successful in this case.

Moreover, from the test set, we checked where our model could potentially fail. Here is an example of a translation that CodeRosetta struggles with:

Reference C++ Code:

void getRho(const int numOfNucl, const double* psi, const double* occNo, double* rho, const char debug) {
    *rho = 0;
    for (int i = 0; i < numOfNucl; ++i) {
        *rho += occNo[i] * psi[i] * psi[i];
    }
    if (debug == 1) {
        printf("DEBUG ▁ print ▁ of ▁ RHO:\n▁ RHO ▁ = ▁ %f\nThis ▁ is ▁ the ▁ last ▁ line (RHO).\n\n", *rho);
    }
}

CodeRosetta CUDA Generation:

__global__ void getRho(const int numOfNucl, const double *psi, const double *occNo, double *rho, const char debug) {
    *rho = 0;
    for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < numOfNucl; i += blockDim.x * gridDim.x) {
        *rho += occNo[i] * psi[i] * psi[i];
    }
    if (debug == 1) {
        printf("DEBUG print of RHO:\nRHO = %f\nThis is the last line (RHO).\n\n", *rho);
    }
}

In this example, CodeRosetta, along with all larger models like GPT-4, Gemini Ultra, and Gemini Pro, failed to translate the code correctly. We verified the generated code by executing it. The line *rho = 0; initializes the variable rho to zero. However, if this kernel is executed by multiple threads across different blocks simultaneously (which is very likely), all threads will write to the same memory location. This results in a race condition where multiple threads are simultaneously updating the value of *rho without any synchronization mechanism, such as atomic operations or reduction techniques. As a result, each thread's write could overwrite others, leading to incorrect and unpredictable outcomes. To rectify this issue, rho should be set to 0 in the host code, and atomicAdd should be included in the computation.

We hope these clarifications address your feedback and questions effectively and you find our rebuttal and additional discussion satisfactory.

We would greatly appreciate it if these additional results and clarifications lead you to reevaluate our work.

At the end, if there are any other lingering questions or requests, please do not hesitate to ask. Our goal is to work with experts like you to improve the quality of our paper and make our research more valuable for the community.

Thank you,

The Authors

Evaluation metrics and runtime correctness

Evaluation metrics were selected for comparison as they have been utilized in the baselines. We understand that these metrics can have limitations. We tried to address this point by manually analyzing the translated code and executing it. Please refer to the global response for further details on this. Thank you.

MLM and BT in ablation

We have performed an ablation study on the parts that are contributions of our work. The intention was to show the contribution of each proposed training objective. Technically, there is no issue with performing an ablation study on MLM and BT. We make sure to add them to the paper per your request. However, since this requires retraining the model two more times, it would not be possible to finish it within the rebuttal time constraint.

Fine-tuning Performance

We have added fine-tuning as an additional optional step to our pipeline. For translation works, one of the biggest obstacles is the lack of paired data corpora. For example, it is very challenging to find a C++ program and its equivalent CUDA code. However, even if such data exists, it would not be sufficient enough to train a model. Therefore, developing an unsupervised translation model is inevitable. One of the benefits of having such a model is that we can train it using self-supervised and unsupervised training techniques as proposed in the paper and then fine-tune it on a small set of paired data. This is a key ingredient as the model already has knowledge of translation, and with a small set of paired data, it can gain a significant boost in its performance.

Contamination/Deduplication for synthetic data:

During Back Translation (BT), no new model or training data is involved. In this training task, only CodeRosetta model and the train set are involved. CodeRosetta is asked to translate from A to B. However, since the model has not translated any code in the past, B would be of poor quality with noises. Then B is used as input to the model to reconstruct A. So, the model leverages weak supervision from its own knowledge, and the test set is not involved at all. For synthetic, we take random samples only from the train set and ask a larger model to translate them. To investigate this further, we applied CodeBERTScore to measure the functional similarity of each sample in the test set against the synthetic dataset. And randomly analyzed some samples that have high CodeBERTScore.

For example, this is one of the samples from test set

__global__ void l1_kernel(int n, float *pred, float *truth, float *delta) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        float diff = pred[i] - truth[i];
        delta[i] = (diff > 0) ? 1 : -1;
    }
}

The most similar CUDA code in the trainset with CodeBERTScore of 0.92 is this:

__global__ void softmax_x_ent_cpu(int n, float *pred, float *truth, float *delta, float *error) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        float t = truth[i];
        float p = pred[i];
        error[i] = (t) ? -log(p) : 0;
        delta[i] = t - p;
    }
}

As we can see, the only common thing about these kernels is the fact that both of them are loss functions. However, the first kernel is L1 loss, and the second kernel is cross-entropy loss.

Here, we are showing the amount of data that falls in each range of similarity score. The ranges with zero data are omitted.

A score below 0.8 shows moderate to small similarity. As can be seen, the majority of the files don’t have high similarity with the test data, and even with those ones that had high similarity, we checked manually and realized the reason for the high similarity is that they belong to the same domain with slight similarity.

Difference with relevant works, CodeT5[1] and PLBART[2]

Thank you for providing us with these references. Indeed, Denoising Auto Encoding (DAE) is a popular technique when it comes to training Encoder-Decoder models, as both CodeT5 and PLBART use it, too. Some of the noising strategies, such as masking and token dropping, are common. One of the key differences in the noising strategies employed by CodeRosetta is their language-specific characteristics. For example, instead of random token dropping, we employ weighted random dropping that gives a higher probability to language-specific reserved keywords, forcing the model to develop a deeper understanding of the target language’s semantics and its structure. Another noising strategy is token insertion, which encourages the model to distinguish between valid and invalid tokens for the target language. We make sure to update the paper and distinguish CodeRosetta’s noising strategies vs. common strategies in the literature, specifically in the related works section.

Details on filtering CPP files from StackV2

When training CodeRosetta, We aim to present the same amount of data for the languages involved in the translation. For the translation of Fortran to C++, we used data from the StackV2 data corpus, which contains far more C++ than Fortran. It is possible to randomly subsample C++ source code files to have an equal number of samples for C++ and Fortran. However, instead of random subsampling, we tried to extract C++ samples that have ‘good quality’. We employ a technique similar to “Textbooks are all you need”[1]. Using a larger model to assess the quality of some portion of C++ data in StackV2 (due to the budget) and then training a classifier model to classify all C++ data in StackV2, then randomly selecting from the C++ files that have been predicted to have good educational value. We make sure to rephrase this part of the paper and provide more clear explanations with references. Thank you.

[1] S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi et al.,“Textbooks are all you need,” arXiv preprint arXiv:2306.11644, 2023`

Clarification on Table 4

The table presents Fortran to C++ translation results (apologies for the mistake in the caption of the title; we make sure to rectify its title) on the dataset provided by Bin et al. [9]. In this dataset, the test set contains 33 paired samples, meaning for each Fortran code there exists an equivalent C++ code. We aim to analyze the performance of CodeRosetta for the task of translating from Fortran code to C++ code. The experimental results in this table indicate that for the Fortran to C++ translation, CodeRosetta is effective.

Comparison with StarCoder or DeepSeekCoder

Thank you for suggesting a comparison with StarCoder and DeepSeekCoder. We have provided the results in the following table using the prompt format provided in Figure 6 of the paper, with adjustments depending on which languages we are targeting. Results indicate that StarCoder is relatively better than DeepSeekCoder, and it has a CodeBLEU score close to Gemini-Ultra. We make sure to update the paper and reflect the result of these open Code LLMs.

Fine-tuned StarCoder(15B)

StarCoder has been fine-tuned for the task of C++ to Fortran Translation by Bin et al. [9]. They have fine-tuned other models as well, among them, fine-tuned StarCoder was able to translate C++ to Fortran with higher accuracy. We compared CodeRosetta against the fine-tuned StarCoder since this model was the one that had the best results among others.

Low scores of ChatGPT and Gemini

Baseline papers have used BLEU and CodeBLEU to evaluate the translation results. We have used the same metrics. However, LLMs generally do not perform one to one translation. They typically provide descriptions and instructions about code and how to execute it. This could impact the metrics. Moreover, to reveal why other LLMs' results are lower than CodeRosetta, we manually examined the generated code. Please refer to the global response for further explanation on this.

Thank you for your rebuttal. My question was about Figure 4, not Table 4. My concerns have been largely addressed, and I have raised my evaluation.

Deduplication analysis

The C++ to CUDA dataset was obtained from BabelTower [26]. This dataset has already gone through a rigorous deduplication and cleaning process. Moreover, there is no paired trained data available in the training set. This means the model can not see a C++ code and its CUDA equivalent during training as such data does not exist. The model has to rely on the self-supervised training objectives to learn to embed source code in different languages into the same embedding space. Only for the test set, we have paired data, which we have used to evaluate our model. We further evaluated the similarity between the test and the trainset from BabelTower [26] using CodeBERTScore. The following table shows the ranges for CodeBERT score and the amount of data that falls in each range. For example, 48.61% of training data have CodeBERTScore between 0.7 and 0.8 when compared against test data. Ranges with zero data are omitted. A score below 0.8 shows low or moderate similarity. As can be seen, the majority of samples in the training set have CodeBERTScore lower than 0.8

We randomly examined the training samples that had high similarity scores. For example, this is a sample from the test set:

    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < N) {
        array[idx] *= scale;
    }
    return;
}

And the most similar sample from the trainset is this one with CodeBERTScore of 0.94:

__global__ void set_val(float *data, float val, int size) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx >= size) return;
    data[idx] = val;
}

We can see that despite some similarities between the two code snippets, their functionalities are different. One is scaling the values of an array; the other is populating an array.

Functional Correctness

Please refer to the global response.

More details on datasets

Details of the datasets have been added to the appendix. Please let us know if any other information needs to be added. We will make sure to move the datasets' table to the main paper if space permits.

AST for monolingual models

AST entity recognition (AER) pre-training task enables the model to identify various entities in the source code. Examples of these entities are identifiers, functions, and primitive types. By allowing the model to identify such entities, the model can recognize where in the structure of the source code each entity should be. For instance, it recognizes that a type identifier precedes a function entity in C++. This would essentially improve monolingual encoders. However, we do not believe it can fully replace self-supervised training objectives such as MLM. Instead, it can be considered a fine-tuning step on top of MLM to improve the model's embedding capabilities further.

Back translation cheat avoidance

Back Translation (BT) technique has been used in unsupervised translation-related works both for natural language and code translation. We couple this training objective with the denoising auto-encoding (DAE) training objective in a way that the model is not solely trained on one of these objectives. Instead, during training, the model iterates between DAE and BT for each batch of data. This, in hindsight, prevents the model from solely depending on BT and cheating. To further investigate this, we looked into intermediate outputs in back translation (Language B, as you mentioned). For example, this is one of the C++ input codes to the model (language A):

static void makexgraph(graph *g, xword *h, int n) {
    setword gi;
    int i, j;
    xword hi;

    for (i = 0; i < n; ++i) {
        hi = 0;
        gi = g[i];
        while (gi) {
            j = FIRSTBITNZ(gi);
            gi ^= bit[j];
            hi |= XBIT(j);
        }
        h[i] = hi;
    }
}

And this is the intermediate result (language B):

__global__ void makexgraph(graph *g, xword *h, int n) {
    setword gi;
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    xword hi;

    for (; i < n; i += blockDim.x * gridDim.x) {
        hi = 0;
        gi = g[i];

        while (gi) {
            j = FIRSTBITNZ(gi);
            gi ^= bit[j];
            hi |= XBIT(j);
        }

        h[i] = hi;
    }
}

We can see that the model has tried to translate the code to CUDA code. However, the translated code contains issues. For instance, variable j is not defined. In the context of back translation, this noisy translated CUDA code will be used as an input to the model, and the model is responsible for reconstructing the original input C++ code. Since we iterate over languages with back translation, sometimes the model has to generate noisy CUDA code and sometimes C++ code. This step also enables the model to be able to translate from noisy input data.

Details on tokenizer

BPE is one of the most potent and popular types of tokenizers. For CodeRosetta, we loaded a pre-trained BPE tokenizer from uniXcoder[1] and then trained it further on our trainsets. Training tokenizers from scratch is notoriously time-consuming; this is why we load a pre-trained tokenizer that has already seen some C++ data.

[1]Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022). Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850.