Function-to-Style Guidance of LLMs for Code Translation

We thank the reviewer for the insightful and valuable comments.

Q1: Further study is needed to analyze the case when LLM judges provide low-quality output.

A1: We observe that LLM Judge is prone to errors when evaluating source code that is either lengthy or contains numerous built-in APIs, e.g., "map", "lambda", etc., in Python.

To evaluate its performance in assessing translation quality, we introduce an LLM Judge benchmark. This benchmark comprises source code and human-annotated translation from our F2STrans benchmark, along with translations generated by Qwen-32B and GPT-3.5. We measure the performance of a Qwen-7B-based LLM Judge by its success rate in identifying the human-annotated translation as the best translation for a given source code. The table below summarizes results for source code with varying API counts and token lengths.

While performance deteriorates with increased complexity, our analysis indicates that LLM Judge still reliably identifies high-quality translations in most scenarios.

---

Q2: Why were style and functional correctness chosen as two aspects for improvement?

A2: We focused on style and functional correctness for two main reasons:

1. Fundamental requirements: Functional correctness ensures the translated code maintains the same behavior as the source code—an essential requirement for any translation system. Stylistic correctness enhances readability and maintainability, making the translated code easier to understand and modify.
2. Translation-specific factors: While efficiency, security, and other aspects are certainly important in code development, these factors are largely determined by the source code itself rather than the translation process.

We are actively exploring ways to optimize code translation quality from a broader perspective in our ongoing research.

---

Q3: The research on code preferences (CodeUltraFeedback) should be discussed.

A3: We agree that the CodeUltraFeedback is highly relevant to our work. It inspires us to explore incorporating user-defined code styles and preferences into our code translation system. Additionally, it illuminates important future challenges beyond our current scope, particularly how to extend our approach to address other crucial non-functional factors like efficiency, security, and maintainability in the translated code. Our paper will incorporate a more detailed discussion of CodeUltraFeedback and its implications for our research.

---

Q4: Clarification on I/O optimization in motivation example.

A4: In the C++ code, the lines

ios::sync_with_stdio(false);
cin.tie(0);

disable synchronization between C++ I/O streams and untie cin from cout, reducing overhead and accelerating input/output operations. In Python, a comparable optimization would involve using faster input methods, such as sys.stdin.readline() or reading all input at once via sys.stdin.read().

---

Q5: The paper lacks a detailed description of the process for manually annotating the ground-truth dataset.

A5: Our annotation team consisted of 30 professional software developers with industry experience, 15 graduate students, and 5 doctoral students in computer science. During code translation, each source code was translated by at least 5 different annotators, ensuring multiple candidate translations. Doctoral students then selected the best translation or refined problematic ones. For test case annotation, we maintained a 95% branch coverage rate, prioritizing edge cases and challenging scenarios. The entire annotation process took approximately 5 months.

---

Q6: Lack of comparison with non-LLM approaches, like CodeRosetta.

A6: We appreciate the suggestion to compare our work with non-LLM approaches for a more comprehensive evaluation. While CodeRosetta focuses on translations between C++ and CUDA/Fortran (which differs from our scope), we will include a more detailed discussion of it in our paper. Below, we compare our approach with two prominent non-LLM code translation systems, TransCode-ST and TransCode-IR.

---

Q7: Did authors try other well-known code embedding methods, like UniXcoder?

A7: The training results of Qwen-1.5B using various embedding models are as follows:

We greatly appreciate the reviewer's insightful and constructive feedback, and we have carefully addressed each point in our response to resolve your concerns. If our response has satisfactorily addressed your questions, we kindly request your consideration of raising the score (currently Rating: 3: Weak Accept). Should any further issues remain, please feel free to share your additional comments, and we will continue actively responding to your comments and improving our submission.

Q1: Comparison between DPO and list-wise loss function.

A1: The list-wise loss function is designed to improve the generation of positive translations by learning from one positive example and multiple negative examples. This approach differs significantly from DPO, which optimizes models based on pairwise comparisons between a single positive and a single negative sample.

To evaluate the effectiveness of these two methods, we conduct experiments on the CodeNet and F2STrans benchmarks, with the results presented in Figure 8 of Appendix E in our paper. The average performance across three training runs is summarized in the following table, from which we can observe that our list-wise loss function is more effective. For more details on the experimental setup and additional analyses, please refer to Figure 8 of Appendix E in our paper.

---

Q2: In addition to the CodeNet and F2STrans benchmarks, the paper should evaluate performance on a broader range of benchmarks, such as Multiple-E.

A2: We fully recognize the importance of comprehensive testing benchmarks. Therefore, Table 5 of Appendix D in our paper demonstrates the model's performance on the xCodeEval and CodeScope benchmarks. The average translation results for various source languages ​​are as follows:

In response to your suggestion, we have conducted additional tests on the Multiple-E and CodeTransOcean benchmarks. For Multiple-E, since it does not include C code, we focus on evaluating its performance across other languages. For CodeTransOcean, we adopt the CodeBLEU metric, consistent with the standard practice described in the CodeTransOcean paper. For other benchmarks, we continue to use the Computational Accuracy metric as outlined in our paper.

Thank you for recognizing the contributions of our work and providing valuable feedback. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score (currently Rating: 2: Weak reject). If not, please let us know your concerns, and we will continue actively responding to your comments and improving our submission.

Before addressing your specific points, we would like to clarify that the "CodeEval" mentioned in the review likely refers to the "CodeNet" benchmark.

Q1: In addition to the CodeNet and F2STrans benchmarks, the paper should evaluate performance on a broader range of benchmarks, such as CodeTransOcean.

A1: Besides the CodeNet and F2STrans benchmarks, we also report results on xCodeEval and CodeScope benchmarks in Table 5 of Appendix D. Here, we show more results on CodeTransOcean and Multiple-E benchmarks. For CodeTransOcean, we adopt the CodeBLEU metric, consistent with the standard practice in the CodeTransOcean paper.

---

Q2: The paper lacks evaluation of code quality.

A2: In Table 4 of Section 3.5, we assess the stylistic quality of translations based on the CCSim metric. The results are as follows:

---

Q3: The paper uses Qwen7B and Qwen32B for LLM Judge and Style Aware LLM Translation, respectively. It does not analyze the impact of inherent biases in these LLMs on the model training.

A3: We build training datasets based on different LLMs (Qwen-7B/32B and DeepSeek-6.7B/33B) and show the training results of Qwen1.5B based on F2STrans benchmark:

We find that the inherent biases of LLMs cause uneven performance across languages. By adopting a Hybrid Mining strategy—using Qwen LLMs for C, C++, and Java, and DeepSeek LLMs for Go and Python—we achieved consistent performance improvements. This demonstrates that assigning tasks according to each model's strengths can alleviate the impact of LLMs’ inherent biases and improve the quality of training data.

---

Q4: Some non-trivial insights into the overall performance decline on the F2STrans benchmark are required.

A4: We have identified two main reasons for the overall performance decline:

• Number of Test Cases. F2STrans includes 50 human-annotated test cases for each code snippet (see Table 1), whereas CodeNet provides only one. The scarcity of test cases in CodeNet may lead to inflated evaluation results, introducing evaluation errors. To quantify this impact, we conducted 10 code translation evaluations, each using a random test case from F2STrans samples. The results, shown below, reveal that relying on a single test case introduces at least a 2.4% evaluation error.

• Code Length. As shown below, source code in F2STrans usually contains more tokens than in CodeNet, which makes evaluations more challenging.

---

Q5: The Python translation performance of various models shows a significant decline in the F2STrans benchmark, yet the paper lacks an analysis.

A5: We find that Python code in our F2STrans benchmark is more complex, characterized by longer average length and more frequent use of built-in APIs, such as "map" and "lambda". As a result, models are more prone to generating code with compilation errors when translating these complex examples.
For instance, over half of the errors produced by Qwen32B are due to compilation failures, as follows:

---

Q6: Presenting concrete numbers for the mean average success rate delta, both with and without Python, can help verify the benchmark advantages of F2STrans.

A6: The table below shows the mean average success rate delta for each language between the CodeNet and F2STrans benchmarks. Each model's performance drops by at least 11.7 on F2STrans compared to CodeNet, and by at least 6.0 even without Python. This confirms that our F2STrans benchmark is highly challenging.

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If our response meets your expectations, we would greatly appreciate it if you could consider raising your score (currently Rating: 3: Weak Accept). If further concerns remain, please let us know, and we are committed to addressing them and refining our submission accordingly.

Q1: Heavy dependence on the availability of larger LLMs to judge translations and generate correctly styled translations.

A1: Using larger LLMs to build training data can indeed enhance model performance. However, we would like to clarify two key points:

(1) While it is important to acknowledge the limitations of this approach, we should not overlook its significant benefits.

For instance, our training framework enables Qwen-1.5B to outperform GPT-4 across a wide range of code translation benchmarks. The specific results are as follows:

(2) Even when we use LLMs of the same scale as the trained model to construct training data, our two-stage training method remains effective.

Here, we trained Qwen-3B exclusively on data constructed entirely by itself. The results are presented in the table below. We find that, without relying on larger LLMs, our framework still enables Qwen-3B to surpass GPT-4 across these code translation benchmarks.

---

Q2: Could you please explain how the negative-style translations are generated using $M_{fun}$ in stage 2 along with prompting techniques used. How did you validate that there is sufficient variability/diversity in the generated negative style samples and that they are different from one another.

A2: We set $M_{fun}$'s temperature to 1.5 and perform multiple sampling generations with the prompt described in line 825 of our paper. A translation is retained only if its CCsim score with any prior translation is below 0.9; otherwise, it is discarded. CCSim, as detailed in lines 606 to 633 of the paper, measures stylistic similarity between code snippets, and a threshold of 0.9 ensures that the retained samples are sufficiently distinct from one another. If fewer than the desired number of translations are retained after 20 sampling attempts (a rare scenario), we incrementally increase the temperature to encourage greater diversity in the $M_{fun}$'s outputs. The table below illustrates the average similarity between our constructed negative-style translations, measured using two metrics: CCSim and CodeBLEU.

We further evaluated the performance of the trained Qwen-1.5B model on various benchmarks using different CCsim filtering thresholds. The results shown in the following table indicate that omitting negative translation filtering—disregarding the diversity of negative translations—led to a noticeable decline in performance. Moreover, setting the threshold too low (e.g., 0.8) also impaired performance. This may be because an excessively low threshold introduces some highly diverse but low-quality negative translations.