ADELT: Transpilation Between Deep Learning Frameworks

Thank you for your encouraging comments! We address your concerns below:

W1. Why not add more context information into the placeholders

We appreciate your insightful recommendations. Rather than using generic placeholders like [PLACEHOLDER_1], [PLACEHOLDER_2], we agree that using more specific tags such as [FUNC_1], [ARG_1] could provide valuable context. However, given that our current approach has achieved 100% accuracy for code skeleton transpilation in our benchmark evaluations, this modification might be not necessary for this given task. That said, your suggestion will be useful for more challenging tasks, and will be considered in future work. Thank you again for your valuable input.

Q1. Is it possible to measure the precision of each process (the code skeleton and the API keywords mapping) and how do they effect the final performance?

We appreciate your insightful question regarding the precision of our transpilation process's sub-parts and their subsequent effect on ADELT's overall performance.

For skeletal code transpilation, Section 3.4 describes its accuracy of 100% on the benchmark, because skeletal code transpilation is designed to be simple and straightforward.

For dictionary lookup, the performance is reported in Table 3 (ablation studies), where we show metrics such as Precision@1, Precision@5, and MRR. The dictionary of ADELT has Precision@1 of 87.1% and 90.0% for both transpilation directions respectively.

Q1. For table 9, could the authors provide GPT4's 95% intervals, specifically for F1?

We apologize for the oversight in our original submission. The results with 95% confidence intervals for GPT-4, have now been added to Table 10 of the revised manuscript. We also discussed those results in our response to weakness 1. Thank you for bringing this to our attention.

PS1. 2 decimal digits (in Table 1, etc) make it hard to read the table to quickly compare values. 1 decimal digit will be easier/faster to understand.

Thank you for your valuable suggestion. We appreciate your attention to clarity and have updated our tables accordingly, presenting all numbers rounded to 1 decimal digit.

PS2. Datasets in A.1 are reported in GBs, but often a significant fraction of ipynb notebook files is binary data in output cells

We appreciate your insight. In response, we have revised Appendix A.1 to reflect the size of clean Python code in all datasets, measured in gigabytes, thereby providing a more accurate sizing metric.

W1. Very small evaluation dataset size of 50 samples. 95% confidence interval of upto +- 8.8 points.

Thank you for your insightful feedback regarding the evaluation dataset's size. Recognizing the importance of a robust and reliable assessment of ADELT's performance, we have expanded the evaluation dataset to 100 examples. The additional 50 examples were curated by identifying PyTorch modules sampled from GitHub repositories with over 1,000 stars. Human experts subsequently translated these into the target frameworks, and you can find the selection process detailed in Appendix A.4 of the revised manuscript.

Upon re-evaluating the PyTorch-Keras transpilation with the augmented benchmark, ADELT maintains a substantial performance lead, achieving an improved exact match score increase of 17.4 percentage points over the baseline models. Updated results have been incorporated into the relevant sections of our manuscript. We believe that this improvement in our evaluation substantiates the reliability of our results.

The size of the revised dataset is comparable to the standard in domain-specific benchmarks: the general-purpose code generation dataset OpenAI HumanEval contains 164 examples, and most program synthesis benchmarks comprise less than 100 examples, like Casper (37 examples) [1], C2TACO (71 examples) [2], TF-Coder (70 examples) [3]. While we were able to only double the size of our evaluation dataset due to the limited time over the rebuttal period, we will expand the dataset further post-rebuttal.

Regarding statistical significance, 95% deviation of 8.88 points was observed from F1 score of GPT-3 (35.62 ± 8.88), which is statistically significantly inferior to ADELT originally reported (82.01 ± 3.08).

We have introduced a full results table with error bars for GPT-4 and other models on our new 100-example benchmark (also shown in Table 10 of the revised manuscript):

These results illustrate that ADELT (Base) surpasses GPT-4 in all benchmarks with a statistically significant difference. We hope that this larger dataset and comprehensive re-evaluation address your concerns and bring further rigor to our work.

W2. The evaluation dataset construction process itself introduces biases. By selecting only pairs with high BLEU score

Thank you for your astute observations concerning potential biases caused by our initial data collection method. You are correct that our initial reliance on BLEU score heuristics may have inadvertently skewed our dataset towards simpler examples.

However, it's important to clarify that the BLEU score was used as an initial guide to locate repositories potentially containing parallel examples of code across deep learning frameworks, rather than as a strict selection criterion.

In response to your suggestion, we have doubled our evaluation dataset with more challenging, realistic examples sourced from PyTorch repositories, which were then manually transpiled by human experts, as we discussed in our response to weakness 1.

W3. The pre-training dataset perhaps will not be feasible to release due to copyright concerns.

Thank you for your comment regarding the dataset used in our study. We acknowledge your concern about potential copyright issues related to parts of the data such as StackOverflow. To address this, while we have added these data in the supplementary material, we will release the script used to gather and process such data upon publication of the paper.

[1] Leveraging Parallel Data Processing Frameworks with Verified Lifting, https://arxiv.org/abs/1611.07623

[2] C2TACO: Lifting Tensor Code to TACO, https://dl.acm.org/doi/abs/10.1145/3624007.3624053

[3] TF-Coder: Program Synthesis for Tensor Manipulations, https://arxiv.org/abs/2003.09040

W1. Figure 1 is too simple. It appears that step 4 is executed in parallel with the steps 3, 5

Thank you for your insightful comments regarding the Method section of our paper. We apologize for any confusion caused by our initial presentation in Figure 1. We have revised the figure in the updated manuscript to better depict our approach. The updated Figure 1 now provides a more detailed illustration, clarifying how keyword substitutions from step 4 integrate with the other steps.

Yes, steps 3 and 4 are executed concurrently, each building upon the output of step 2, while step 5 synthesizes the results from both steps 3 and 4.

W2. The size of the evaluation dataset is somewhat small, with only 50 samples

Thank you for your insightful feedback regarding the evaluation dataset's size. Recognizing the importance of a robust and reliable assessment of ADELT's performance, we have expanded the evaluation dataset to 100 examples. The additional 50 examples were curated by identifying PyTorch modules sampled from GitHub repositories with over 1,000 stars. Human experts subsequently translated these into the target frameworks, and you can find the selection process detailed in Appendix A.4 of the revised manuscript.

Upon re-evaluating the PyTorch-Keras transpilation with the augmented benchmark, ADELT maintains a substantial performance lead, achieving an improved exact match score increase of 17.4 points over the baseline models. Updated results have been incorporated into the relevant sections of our manuscript. We believe that this improvement in our evaluation substantiates the reliability of our results.

The size of the revised dataset is comparable to the standard in domain-specific transpilation benchmarks: the general-purpose code generation dataset OpenAI HumanEval contains 164 examples, and most program synthesis benchmarks comprise of less than 100 examples, like Casper (37 examples) [1], C2TACO (71 examples) [2], TF-Coder (70 examples) [3]. While we were able to only double the size of our evaluation dataset due to the limited time over the rebuttal period, we will expand the dataset further post-rebuttal.

W3. Some details of the comparisons were not clearly explained, such as what prompts were used for GPT-4

Thank you for your insightful comments. We apologize for missing the GPT-4 prompt details. To address this, we have included a comprehensive explanation of the prompts used for GPT-4 in Appendix A.6 and detailed the post-processing protocol in Table 8 of the revised manuscript. As we explain in the appendix, our experiments leverage Markdown-formatted prompts in GPT-4's chat mode, and then GPT-4 sometimes generates a mixture of natural language and code. After generation, we programmatically extract code from the first code block from GPT-4's responses as the finalized transpilation. Our prompts for GPT-4 are designed to mimic the straightforward and intuitive approach of a human user, as done in prior approaches in evaluating such models [4].

W4. The paper only provides the construction process for the PyTorch-Keras benchmark in Appendix A.4, but not cover the construction process for the PyTorch-MXNet benchmark

Thank you for pointing out this oversight. We have now detailed the construction method for the PyTorch-MXNet benchmark in the revised Appendix A.4. Briefly, human experts manually transpiled the PyTorch snippets from our PyTorch-Keras dataset into MXNet, ensuring consistency across frameworks. Your feedback is greatly appreciated for the improvement of our manuscript.

W5. The execution efficiency also needs some comparisons, such as runtime

We appreciate your concern on ADELT's computational efficiency. Efficiency is actually one of the strengths of ADELT.

ADELT is composed of two main procedures: large language model (LLM) prompting for code transpilation and dictionary lookup for API keyword mapping. Using Codex for transpiling the code skeleton is as efficient as using Codex to do transpilation end-to-end. The API dictionary lookup process is highly efficient and only takes nanoseconds since it does not involve neural network inference for each example.

To illustrate, ADELT outperforms GPT-4 in transpilation results, and its skeletal code transpilation component uses Codex, which is of a similar scale as GPT-3. Because GPT-3/Codex is significantly faster, ADELT is faster than GPT-4.

[1] Leveraging Parallel Data Processing Frameworks with Verified Lifting, https://arxiv.org/abs/1611.07623

[2] C2TACO: Lifting Tensor Code to TACO, https://dl.acm.org/doi/abs/10.1145/3624007.3624053

[3] TF-Coder: Program Synthesis for Tensor Manipulations, https://arxiv.org/abs/2003.09040

[4] GPT-4 Technical Report, https://arxiv.org/abs/2303.08774

Q1. In section 2.4, what do you mean by “In dictionary generation, we do not allow callable names to be translated to callable names”?

We appreciate your keen observation. The mentioned statement was indeed a mistake. The sentence should be, "In dictionary generation, we do not allow callable names to be translated to parameter names". This mistake has been rectified in the revised manuscript. Thanks for pointing it out!

W1. Lack of Negative Results

Although our paper centered on ADELT's strengths, we have indeed discussed its limitations in detail.

In Table 2, we provided a detailed case study illustrating instances where ADELT demonstrated less optimal performance. Notably, a case where the parameter output_dim was inaccurately mapped to embeddings_initializer due to an error in generating vocabulary mapping.

Also, ADELT is unable to transpile layers.Dense(out_dim, activation='relu') into corresponding PyTorch API calls, nn.Linear and nn.ReLU. This is because the training corpus does not include enough examples that reflect such use cases in Keras. In Appendix A.9, we describe the potential for ADELT's improved modeling of such mappings in a scenario where we add synthetic data into the corpus.

We value your perspective and will expand our discussion of both the pros and cons in our paper.

W2. Generalization of the Approach

We appreciate your insightful suggestion on generalizing the application of ADELT. While our current study focuses on transpilation between specific deep learning frameworks, ADELT's approach can be extended to various API ecosystems within the same programming language. We believe that ADELT’s approach to transpilation can also be applied to transpile between APIs from different domains, such as visualization (Matplotlib vs. Seaborn) or database access APIs (PyMongo vs SQLite). Your feedback has certainly contributed to outlining the future directions of our work.

Q1. ​​How does ADELT handle situations where different versions of deep learning frameworks require different expressions?

Thank you for your insightful question. ADELT addresses variability across framework versions by treating each version-specific expression as a unique entry. Both torch.autograd.Variable and torch.tensor in PyTorch are mapped to tf.Tensor in Keras. The reverse process, subsequently, interprets tf.Tensor from Keras as the more recent torch.tensor expression in PyTorch. This approach maintains consistency across different framework versions and ensures accurate transpilation.

Q2. In the METHOD section, the authors mention that they "convert each API call into its canonical form". How exactly is this achieved?

We convert each API call into its canonical form using Python's built-in inspect module, specifically inspect.bind (https://docs.python.org/3/library/inspect.html). This function interprets Python function calls internally and creates a mapping from positional and keyword arguments to parameters. We have further detailed this procedure in Section 2.1 of our paper.

Q3. What e_i and e_j mean in the last paragraph of Section 2.3.

Thank you for your question. In Section 2.3, $e_i$ and $e_j$ correspond to the unique embeddings of each API keyword. To elaborate, every API keyword, like nn.Conv2d in PyTorch, is assigned its unique embedding, like nn.Linear or nn.Linear.in_features. In Equation (2), we incorporate these embedding matrices as weight matrices for the output classification layer, a strategy that is commonly adopted in Transformer LMs where a shared parameter tensor is used between the word embedding and the output layer [1].

Q4. Have the authors tested the accuracy of each component, such as "Canonicalizion" "Extract API calls" "Code to Skeleton" and "Dictionary Lookup" and so on?

Yes, ADELT indeed follows a pipeline structure that includes “Canonicalization”, “API call extraction”, “Code to Skeleton”, “Skeletal Code Transpilation” and “Dictionary Lookup” among other steps.

For “Canonicalization”, “API call extraction”, “Code to Skeleton”, our approach uses Python's internal library and does not involve machine learning, thereby not affecting transpilation accuracy.

For “Skeletal Code Transpilation”, Section 3.4 describes its accuracy of 100% on the benchmarks. In fact, one of the key novelties of ADELT is the decoupling of the transpilation process into transpiling the skeletal code followed by dictionary lookup. Skeletal code transpilation is designed to be simple using any off-the-shelf LLM, while the dictionary used in the second step requires more sophisticated techniques to learn via adversarial learning.

For “Dictionary Lookup”, the performance is reported in Table 3 (ablation studies), where we show metrics such as Precision@1, Precision@5, and MRR. The dictionary of ADELT has Precision@1 of 87.1% and 90.0% for both transpilation directions respectively.

We hope that this clarifies the efficacy of ADELT's individual components.

[1] Rethinking embedding coupling in pre-trained language models, https://arxiv.org/abs/2010.12821